Sunday, July 12, 2009
Variations on a Theme: Different Formulations of Quality of Competition
One very nice innovation on Quality of Competition came from Jonathan Willis at Copper 'n Blue. In order to calculate Quality of Competition, you normally need 1) TOI totals for every player; and 2) head-to-head icetime totals for every player combination. This data is only available for the NHL, and it's only available for the last three years. What Willis did was to use goal events as a proxy for icetime. That is, in the absence of TOI, Willis assumed that the total number of goals a player was on the ice for was a reasonable proxy for the amount of time he spent matched up against specific opposing players.
That changes the Quality of Competition from:
QoC = sum[(Opponent Rating)*(Opponent TOI)]/TOI
Where Rating = (GFON/60 - GAON/60) - (GFOFF/60 - GAOFF/60) for each opponent.
To something like:
QoC = sum[(Opponent Rating#2)*(Opponent + and - per GP)]/(Opponent + and - per GP)
Where Rating#2 = (GFON/GP - GAON/GP) - (GFOFF/GP - GAOFF/GP)
Or something very similar - I don't think it matters very much whether you use GP or total events as a proxy for TOI.
At any rate, I generated Quality of Competition four different ways for the Edmonton Oilers. Rankings among regular players are below:
QC0 is Quality of Competition as calculated on my site. QC1 is the same TOI-based calculation, but it includes only the off-ice stats for games the player played in (for simplicity of calculation, QC0 includes all of a team's games.) QC2 assumes total TOI is known, but uses goal events (+ and -) as a proxy for head-to-head icetime. QC3 is the same as QC2, but uses games played as a proxy for TOI.
Some results are similar: everybody loves Horcoff and Moreau; everybody hates Stortini. But there are some substantial differences between these systems too. The TOI based ones favor Sheldon Souray, Dustin Penner and Tom Gilbert, while the proxy systems like Jason Strudwick, Ladislav Smid and Andrew Cogliano.
So my question is - given that I see maybe one Oilers game per season - which system seems to be closest to the right answer? Is the difference tolerable?
That changes the Quality of Competition from:
QoC = sum[(Opponent Rating)*(Opponent TOI)]/TOI
Where Rating = (GFON/60 - GAON/60) - (GFOFF/60 - GAOFF/60) for each opponent.
To something like:
QoC = sum[(Opponent Rating#2)*(Opponent + and - per GP)]/(Opponent + and - per GP)
Where Rating#2 = (GFON/GP - GAON/GP) - (GFOFF/GP - GAOFF/GP)
Or something very similar - I don't think it matters very much whether you use GP or total events as a proxy for TOI.
At any rate, I generated Quality of Competition four different ways for the Edmonton Oilers. Rankings among regular players are below:
QC0 QC1 QC2 QC3
ETHANMOREAU 3 3 5 1
SHELDONSOURAY 2 2 12 12
STEVESTAIOS 9 8 3 5
JASONSTRUDWICK 20 20 8 6
LUBOMIRVISNOVSKY 8 10 17 17
SHAWNHORCOFF 1 1 2 4
ALESHEMSKY 5 7 7 19
ALESKOTALIK 17 17 16 7
DENISGREBESHKOV 11 9 13 14
FERNANDOPISANI 3 6 9 11
MARC-ANTOINEPOULIOT 17 18 15 15
ROBERTNILSSON 11 15 6 10
PATRICKO'SULLIVAN 9 11 11 13
DUSTINPENNER 5 4 10 20
LADISLAVSMID 16 14 4 3
ANDREWCOGLIANO 13 13 1 2
KYLEBRODZIAK 19 19 20 8
ZACHERYSTORTINI 21 21 19 18
TOMGILBERT 5 5 18 16
SAMGAGNER 13 16 14 21
LIAMREDDOX 13 12 21 9
QC0 is Quality of Competition as calculated on my site. QC1 is the same TOI-based calculation, but it includes only the off-ice stats for games the player played in (for simplicity of calculation, QC0 includes all of a team's games.) QC2 assumes total TOI is known, but uses goal events (+ and -) as a proxy for head-to-head icetime. QC3 is the same as QC2, but uses games played as a proxy for TOI.
Some results are similar: everybody loves Horcoff and Moreau; everybody hates Stortini. But there are some substantial differences between these systems too. The TOI based ones favor Sheldon Souray, Dustin Penner and Tom Gilbert, while the proxy systems like Jason Strudwick, Ladislav Smid and Andrew Cogliano.
So my question is - given that I see maybe one Oilers game per season - which system seems to be closest to the right answer? Is the difference tolerable?
Labels: Quality of Competition
// posted by Hawerchuk @ 9:53 PM 
Comments:
Links to this post:
<< Home
I think that the TOI system is going to be necessarily better. With that said, I've always been vaguely troubled by the fact that a player could be perceived as being more difficult competition if he was on a different team. As I understand your system, Crosby on the 05-06 Pens is going to have a higher ranking than he would on the 05-06 Wings.
That seems off to me if we're talking about his absolute level. The way most of us use this stuff though is purely for purposes of understanding how a player is used within his team, which isn't a criticism of the system, only a recognition of the limitations. If that's what we're using it for, the TOI way is probably better. Willis' system - which looks decent stacked up against this, which is good in terms of getting QoC measures for other leagues - is necessarily limited.
That seems off to me if we're talking about his absolute level. The way most of us use this stuff though is purely for purposes of understanding how a player is used within his team, which isn't a criticism of the system, only a recognition of the limitations. If that's what we're using it for, the TOI way is probably better. Willis' system - which looks decent stacked up against this, which is good in terms of getting QoC measures for other leagues - is necessarily limited.
I tend to agree with Tyler for the most part. The time-on-ice method is better at evaluating which players are being trusted with more responsibility.
The proxy methods (QC2 and QC3) are helpful for looking at other leagues but there are some Oiler players that look a bit wacky using them. Smid, Staios and Strudwick, especially, seem like they are ranked too high in QC2 and QC3 while Hemsky and Gilbert are ranked far too low in QC3.
That said, I think Willis is using a different proxy method based around pts/gm instead of GF-GA. He awards "difficult minutes" to the players that score on/are socred on by forwards that have high pts/gm and "easy minutes" to the players that score on/are scored on by forwards that have low pts/gm. I'm not sure how that might change the results.
The proxy methods (QC2 and QC3) are helpful for looking at other leagues but there are some Oiler players that look a bit wacky using them. Smid, Staios and Strudwick, especially, seem like they are ranked too high in QC2 and QC3 while Hemsky and Gilbert are ranked far too low in QC3.
That said, I think Willis is using a different proxy method based around pts/gm instead of GF-GA. He awards "difficult minutes" to the players that score on/are socred on by forwards that have high pts/gm and "easy minutes" to the players that score on/are scored on by forwards that have low pts/gm. I'm not sure how that might change the results.
I'm off-topic here, but still hoping for feedback:
Poring over last season's numbers of my beloved habs, I couldn't help but notice two guys in particular, Bouillon and D'Agostini. Both were, from what I can recall, fairly reliable players defensively. But both Bouillon (.889) and D'Agostini (.895) had low save % while on the ice, compared to the team in general that is (.922). For both players, this added up to 10 extra goals against, thus driving down their RATING, hence driving down the QUALTEAM of those who shared lots of ice with them. Andrei Kostitsyn also had some kind of a raw deal, coming in at .907/-7, while Lapierre was kinda blessed at .936/+7.
I guess when it comes to QUALCOMP, those kind of blips even out, and thus my comment is completely off-topic... In fact, maybe this is by design and I just missed the point, but still: wouldn't QUALCOMP and RATING be more, uh, reliable were it based on Corsi or even shots F/A instead of goals?
Poring over last season's numbers of my beloved habs, I couldn't help but notice two guys in particular, Bouillon and D'Agostini. Both were, from what I can recall, fairly reliable players defensively. But both Bouillon (.889) and D'Agostini (.895) had low save % while on the ice, compared to the team in general that is (.922). For both players, this added up to 10 extra goals against, thus driving down their RATING, hence driving down the QUALTEAM of those who shared lots of ice with them. Andrei Kostitsyn also had some kind of a raw deal, coming in at .907/-7, while Lapierre was kinda blessed at .936/+7.
I guess when it comes to QUALCOMP, those kind of blips even out, and thus my comment is completely off-topic... In fact, maybe this is by design and I just missed the point, but still: wouldn't QUALCOMP and RATING be more, uh, reliable were it based on Corsi or even shots F/A instead of goals?
Scott - I knew I got it wrong...I'll re-run using Pts/GP.
MC and Olivier - definitely a valid point. I'm still trying to figure out the right way to correct for this bias.
MC and Olivier - definitely a valid point. I'm still trying to figure out the right way to correct for this bias.
It's probably most accurate when you add up all the different measures and divide for your average rating.
Probably too much work though.
Probably too much work though.
I'll third Tyler and Scott's comments. The TOI system is almost certainly better by a substantial degree.
My system seems to do better with the forwards than the defensemen, which makes sense intuitively given that there's a greater spread up front (12 vs. 6) and also that MacTavish shuffled around the defensemen who faced the top matchups this year a fair bit.
TOI's the system to use, but I'm fairly pleased with how the goal events proxy stood up to this particular test.
The other difference is grading the competition - are the numbers as they're setup now shoing opposition graded by adjusted +/- or by points/gm?
My system seems to do better with the forwards than the defensemen, which makes sense intuitively given that there's a greater spread up front (12 vs. 6) and also that MacTavish shuffled around the defensemen who faced the top matchups this year a fair bit.
TOI's the system to use, but I'm fairly pleased with how the goal events proxy stood up to this particular test.
The other difference is grading the competition - are the numbers as they're setup now shoing opposition graded by adjusted +/- or by points/gm?
Ok, I re-ran using Pts/GP as the metric for the opposition. The rankings are essentially the same as for TOI-based QoC - except Moreau drops to middle-of-the-pack, and Gagner jumps to the top 6.
Gabe, can you post your findings for the Oilers using Pts/GP.
The fact that it drops Moreau down lower, if that's what it does, makes sense to me. ..
The fact that it drops Moreau down lower, if that's what it does, makes sense to me. ..
Call the PPG system 'QC4'
ALESHEMSKY 1
FERNANDOPISANI 2
SHAWNHORCOFF 3
SHELDONSOURAY 4
TOMGILBERT 5
SAMGAGNER 6
DENISGREBESHKOV 7
DUSTINPENNER 8
PATRICKO'SULLIVAN 9
LUBOMIRVISNOVSKY 10
ROBERTNILSSON 11
ALESKOTALIK 12
STEVESTAIOS 13
JASONSTRUDWICK 14
MARC-ANTOINEPOULIOT 15
ETHANMOREAU 16
ANDREWCOGLIANO 17
KYLEBRODZIAK 18
ZACHERYSTORTINI 19
LIAMREDDOX 20
LADISLAVSMID 21
ALESHEMSKY 1
FERNANDOPISANI 2
SHAWNHORCOFF 3
SHELDONSOURAY 4
TOMGILBERT 5
SAMGAGNER 6
DENISGREBESHKOV 7
DUSTINPENNER 8
PATRICKO'SULLIVAN 9
LUBOMIRVISNOVSKY 10
ROBERTNILSSON 11
ALESKOTALIK 12
STEVESTAIOS 13
JASONSTRUDWICK 14
MARC-ANTOINEPOULIOT 15
ETHANMOREAU 16
ANDREWCOGLIANO 17
KYLEBRODZIAK 18
ZACHERYSTORTINI 19
LIAMREDDOX 20
LADISLAVSMID 21
Gabe, your last list is very interesting. I've always thought Moreau was rated too highly be using traditional plus/minus as the basis for a QualComp rating.
This way (Jonathan's way) of doing it looks to be a very strong method, so it could be quite useful in figuring out QualComp in leagues where numerous stats aren't kept.
This way (Jonathan's way) of doing it looks to be a very strong method, so it could be quite useful in figuring out QualComp in leagues where numerous stats aren't kept.
So, to sum up, I'm not agreeing that the TOI system is a better way of doing it, not from my own eyeballing of the Oilers last year.
Jonthan's way may well be just as good.
Post a Comment
Jonthan's way may well be just as good.
Links to this post:
<< Home
Subscribe to Posts [Atom]
