Updated xSV% - Save Percentage Accounting for Shot Quality

Goalies are voodoo. That should probably be added as the 11th Law of Hockey Analytics. Goaltending analysis is currently one of the most lacking subjects within hockey analytics. Great strides have been made however, with 5v5 SV%, AdjustedSV% and High Danger SV%. A few months back I revealed a statistic I referred to as xSV%. You can click there to read the article but, more or less, I calculated a goaltender's Expected SV% based on the quality of shooter for each shot faced based on a rolling average of the shooter's individual shooting percentage. xSV% is a goalie's actual SV% minus their ExpSV%. A higher (positive) xSV% is good while a lower (negative) xSV% is bad. I have thought more about that specific methodology since posting that article and have eventually decided that with some substantial changes I could greatly improve upon xSV% . Based on factors in my ExpG model combined with regressed shooting percentage for each shooter (same mindset but different process as the original xSV%) I basically started from scratch to develop this latest rendition of xSV%.

Methodology

The basis of a goalie's Expected SV% comes from the same model I used in my latest ExpG model. Here is a quick breakdown of the different variables and a brief explanation of why they are included in the model:

• Adjusted Distance
• The farther a shot is taken from the lower likelihood it has of resulting in a goal 
• Type of Shot
• Snap/Slap/Backhand/Wraparound/etc...
• Different types of shots have different probabilities of resulting in goals
• Rebound - Yes/No?
• A rebound is defined here as a shot taking place less than 4 seconds after a previous shot
• Rebounds are more likely to result in a goal than non-rebounds
• Score Situation
• Up a goal/down a goal/tied/etc…
• It has been proven that Sh% rises when teams are trailing and vice versa
• This adjustment, while only slight, helps to account for a variety of other aspects that we are currently unable to quantify yet have an impact goal scoring 
• Rush Shot - Yes/No?
• Shots coming off the rush are more likely to result in a goal than non-rush shots

Now that we have the structure of our ExpSV% we need to add shooter talent into the mix since the model currently assumes league average shooting talent for each shot, which we know is not the case in reality. Generally, a shot from Sidney Crosby is more likely to result in a goal than a shot from George Parros. So I wanted to make a multiplier for each player in each season to get a best estimation of their personal effect on each shot's probability of resulting in a goal.

Using the Kuder-Richardson Formula 21 (KR-21) I was able to find that 5-on-5 Sh% stabilizes for forwards at about 375 shots while 5-on-5 Sh% for defenceman begins to really stabilizes around 275 shots. Therefore, for each season I added these shots (375 for forwards, 275 for defenceman) to a players total shots. I also added a certain amount of goals calculated as the added shots (375 or 275) multiplied by league average Sh% (forwards and defence had different league average Sh%) for that season. This would then allow me to calculate regressed Sh% based on these new shots and goals totals. I then divided rSh% by the league average Sh% (forwards and defence had different league averages) to give a Shot Multiplier. Then multiply this Shot Multiplier for each shot they took in that season. In case you didn't quite follow that rough explanation, here is an example of how this process played out for Steven Stamkos' 2011-2012 season, the highest rSh% season since 2007-2008:

Repeatability

The big issue with goalie metrics has always been how well they actually represent true ability. Sample size is a frequent issue with goaltenders and it has been shown countless times that the only way to truly get a good idea of a goaltender's ability is to take very large shot samples. These samples typically take multiple years to accumulate. Looking at year-to-year correlations, so far, it seems as though xSV% is on par with 5v5 SV%. The interesting difference I found, with the vital help of @MannyElk, is shown in the graph below. To paraphrase his earlier work, the experiment was done by drawing random samples of n games for each 40+ GP goalie season since 2007 and each sample was compared with the goalie's xSV% over the whole season. Essentially, the graph below shows that xSV% will show us more signal than noise sooner than 5v5 SV%.

**Disclaimer** Manny and I are not 100% sure of the results here so if anyone has suggestions please reach out. 

Data 

Quick review of the stats below:

• xSV% = Actual SV% - Expected SV%
• dGA (Goals Prevented Above Expectation) = Expected Goals Against - Actual Goals Against
• Similar to GSAA

Below is a wonderful Tableau visualization created by @Null_HHockey, as well as a spreadsheet with all of the relevant information. Once again big thanks to the guys at War-On-Ice for all their help (and data). Everything below will also be stored full time on a separate xSV% page. Enjoy!

How Long Does It Take For A Forward's Shooting To Stabilize?

If a player scores one goal on five shots, does that mean they are suddenly a 20% shooter? What if it is 10 goals on 50 shots? How about 20 goals on 100 shots? This is a classic issue of sample size in trying to separate the signal (talent) from the noise (randomness). That issue being, how big does a sample need to be before it stops being small? The question has been tackled before in other sports, see baseball here and basketball here, and my analysis here will mirror a lot of the methodology laid out in those pieces. 

Relating the problem to a player's shooting talent, how many shots does a player need to take before we can separate the talent from the randomness?

Now if you don't care about the math then please skip to the *** for the answer and analysis. 

Otherwise lets dive in!

The most common method used for testing this problem is typically split-half reliability testing. For example, if we were wondering how stable a player's shooting percentage is after 100 shots we would label each shot from 1-100 and then randomly split these 100 shots into two random 50 shot samples. We would then compare the player's shooting percentage between these two samples. This method is fine but it can be improved upon in our case by using the Kuder-Richardson Formula 21 (KR-21).

This formula will tell us the reliability of a test involving binary outcomes (two results), which is great for this test since when a player takes a shot there are only two possible results, a save or a goal. The KR-21 formula allows us to perform a split half reliability test but instead of only being able to compare one only type of combination it allows us to compare every single possible combination of these outcomes. For example, if we were going to preform a basic split half reliability test for a total sample of 10 shots (each labelled, 1 2 3 4 5 6 7 8 9 10) a simple method would be to compare all the even number shots with the odd number shots. Using the KR-21 formula however goes further and compares every single type of combination (ex. evens vs. odds, 1-5 vs. 6-10, 1 2 3 9 10 vs. 4 5 6 7 9, etc..). The results of this 10 shot KR-21 test will be a much better estimate of how reliable an indicator of a player's true talent level a stat will be over a 5 shot sample (10 divided by 2 = 5). 

Our goal is to reach a reliability of 0.707 at which point the signal (skill/talent) will begin to overtake the noise (randomness/luck) in our sample (0.707 x 0.707 = 50%). Below I have charted shots versus their reliability to show how the reliability of a sample which change as your sample's cutoff point increases. The blue line shows the logarithmic curve of reliability (which had an R-squared fit of 0.99626 with the data points) which I used instead of simply plotting a basic curve graph. I used the log curve because as you might notice in the table below I got a tad lazy and stopped running the numbers as frequently for bigger samples so I used the logarithmic curve which shows the relationship just as well. The red line shows the 0.707 cut off line where talent beings to overtake the randomness. Above the red line = good, below the red line = not good.

***

I found that after about 223 shots the reliability will cross the 0.707 threshold.

Shots	Reliability	Signal (Talent)	Noise (Luck)
25	0.169	2.8%	97.2%
50	0.317	10.0%	90.0%
75	0.410	16.8%	83.2%
100	0.493	24.3%	75.7%
125	0.560	31.4%	68.6%
150	0.604	36.5%	63.5%
175	0.656	43.0%	57.0%
200	0.677	45.9%	54.1%
212.5	0.693	48.0%	52.0%
217.5	0.704	49.5%	50.5%
222.5	0.707	50.0%	50.0%
225	0.712	50.6%	49.4%
250	0.732	53.6%	46.4%
300	0.765	58.5%	41.5%
375	0.805	64.9%	35.1%
500	0.891	79.5%	20.5%

We now know that at 223 shots a player's shooting percentage is about 50% skill and 50% luck, which is still a lot of noise. We have to get about 400 shots before we really see a player's talent begin to shine through. This once again demonstrates how easy it is to be fooled by small sample sizes. While 223 may seem like a reasonable estimate it should be noted that only 40 players last season (2013-2014), or just over 6% of the entire league, record more than 223 shots. Alexander Ovechkin led the league with 386 shots total (along with a 13.6 shooting percentage) and still only gives us a signal strength of about 65%. 

This isn't meant to be predictive necessarily. That is to say, just because John shot 9% over 223 shots doesn't mean that we should expect John to shot 9% over his next 223 shots. If John shoots 17% over his next 50 games did he suddenly become a better shooter? Probably not. However, if John shoots 12% over his next 223 shots, the case can actually be made that this player may have improved his actual shooting talent. 

This all goes to show that it does take quite a bit of time for a player's shooting percentage to stabilize. Many are quick to reach assumptions about a player's actual ability simply based on a single season which we can see here rarely makes sense when the vast majority of the league will have taken so few shots that separating the signal from the noise is incredibly difficult. There is definitely talent at the heart of a player's ability to score goals, it just takes some time for that talent to truly become evident.

Normalized Career Player Stats

Who is the greatest goal-scorer of all-time? What about playmaker? Hockey like all sports has evolved so much over the years that it is extremely hard to compare individuals in different eras. With the help of Rob Vollman's database and Hockey-Reference's Normalized Data I have compared the career's of every player over the past 47 years (since the 1967 expansion) to help shed some more light on these debates.

The Normalized Data is presented just like any player stats except all the stats are scaled to reflect certain changes throughout the league's history. The most common adjustments are to account for different lengths of schedules, amount of players carried on each roster and era adjustment to account for the amount of goals being regularly scored in those games (ex. it was easier to score a goal in 1981 than it is in 2014).

You can filter and sort this table at your own discretion and pleasure, enjoy!

*Players needed at least 300 Games Played by the end of the 2013-2014 season to qualify
**Even if a player's career began before 1967 this chart will only reflect their stats since 1967

Observations

• The data is obviously skewed towards players whose careers have yet to end. It is extremely hard to maintain high levels of play throughout your entire career which is why active players still in or near their primes will see their stats slightly inflated. 
• Bobby Orr was amazing. He absolutely dominated the game from an offensive stand point that we will probably never see again. 
• Sidney Crosby is the greatest player alive and one of the best ever.
• Ovechkin is probably one of the greatest goal scorers to ever lace up the skates. It still amazes me how much garbage is thrown Ovechkin's way by people who have a seriously flawed understanding of the game of hockey or are simply trying grab a headline. Ovechkin is one of, if not the most, lethal goal scorer in the leagues past half-century and we should all just appreciate the opportunity to bear-witness. 
• Kovalchuk's stats will forever be skewed from the fact that he essentially played out his best years in the NHL before bolting to the KHL which essentially ensures that his career rate stats will never suffer as he ages. He did have a great run though, while it lasted. 
• Lemieux and Gretzky come down to the wire here. Lemieux has the better era-adjusted PTS/Game due to his big years occurring in the 90s as opposed to Gretzky who succeeded in the high flying 80s. Gretzky however, played about 500 more games which has to be considered as a positive when considering the two.
• Jagr is ageless. He keeps on clicking at a ridiculous rate despite taking 3 years off to play in Europe only to come back and put up unheard of numbers for a player older than 40.
• Cam Janssen just nudges out Colton Orr for worst PTS/Game of any regular forward in the last half decade. Likewise, Wade Belak takes home the title of least offensive defenceman of the modern era.