Post by eric on Apr 12, 2018 20:58:50 GMT
Way back in 4.0 we were investigating an(other) odd apparent change in RPM methodology. Here is the conclusion of that investigation.
So the definition of wins hasn't changed as drastically as in 2016, although it's still hugely more variable than the gold standard Win Shares.
And while the decrease in standard deviation isn't as pronounced, it is still forming a clear downward trend year over year which is not at all what should be happening, again as we can see compared to the gold standard Win Shares.
I also decided to try out doing a brute correlation going from data collected in mid-February and mid-March to data collected at season-end.
A quick (you wish) note on that. RPM has always said it uses a previous season of data to help smooth stuff out. I discovered during the All-Star break that it specifically uses calendar period rather than game period for this. I think this is yet another unwise decision, and here's why: suppose it's the All-Star break and I'm trying to measure the RPM of a guy who played every game for the Blazers the last two years. If my prior is 82 games, at any point in the All-Star break that'll be the 58 games he played this season and the last 24 he played last season. Simple. If my prior is instead 365 days, I'll get a different answer on the first day of the All-Star break (Feb 15th) because the Blazers happened to play a game on Feb 15th last year, so my player will have a different RPM on Feb 15th 2018 (off day) than he does on Feb 16th 2018 (off day), even though neither he nor anyone else in the NBA played a single minute. This is compounded by the fact that not everyone plays every game, so I might be getting only let's say 1 game for a guy if he only played 1 game in a 365 day period.
I bring that up only because "season-end" is as previously established kind of a fuzzy figure for RPM, and that's not even considering that we haven't determined if it uses playoff data or not yet. For the record I took the "season-end" measurements today. Anyway, here are the R^2s:
Higher R^2 is better, so from one perspective RPM is doing great because it has clearly the best correlation from February numbers to April numbers: if you wanted to predict how a guy would look at season end, you would want to use RPM. From another perspective, though, RPM's predictive power doesn't improve much at all when we add in the March data, which in turn suggests that the year end value only barely takes March into account.
I can't overstate how big a cause for concern this is in the world of empirically evaluating independent statistical models of sports. So... not that big a concern overall. But in this world, HUGE!!!
Having more data should make your predictions more accurate.
The only way this doesn't happen is if your predictions aren't using the data.
Your predictions should be using the data!!!
It's literally the only thing they should be using!!!
Now RPM's prediction does get a tiny bit better, so it's probably using the data a tiny bit.
My concerns are not assuaged.
Year WS MP Wins RPM MP Wins
2014 595193 1256 588314 868
2015 595203 1257 591784 870
2016 594863 1256 578523 1170
2017 594404 1253 592964 1180
2018 593860 1252 591954 1130So the definition of wins hasn't changed as drastically as in 2016, although it's still hugely more variable than the gold standard Win Shares.
. . WS . . RPM .
Year Max Min SD Max Min SD
2014 .295 -.036 .052 9.08 -8.44 2.98
2015 .288 -.026 .053 9.34 -6.87 2.94
2016 .318 -.049 .056 9.79 -6.27 2.77
2017 .278 -.023 .055 8.42 -5.69 2.57
2018 .290 -.025 .054 6.94 -6.03 2.45And while the decrease in standard deviation isn't as pronounced, it is still forming a clear downward trend year over year which is not at all what should be happening, again as we can see compared to the gold standard Win Shares.
I also decided to try out doing a brute correlation going from data collected in mid-February and mid-March to data collected at season-end.
A quick (you wish) note on that. RPM has always said it uses a previous season of data to help smooth stuff out. I discovered during the All-Star break that it specifically uses calendar period rather than game period for this. I think this is yet another unwise decision, and here's why: suppose it's the All-Star break and I'm trying to measure the RPM of a guy who played every game for the Blazers the last two years. If my prior is 82 games, at any point in the All-Star break that'll be the 58 games he played this season and the last 24 he played last season. Simple. If my prior is instead 365 days, I'll get a different answer on the first day of the All-Star break (Feb 15th) because the Blazers happened to play a game on Feb 15th last year, so my player will have a different RPM on Feb 15th 2018 (off day) than he does on Feb 16th 2018 (off day), even though neither he nor anyone else in the NBA played a single minute. This is compounded by the fact that not everyone plays every game, so I might be getting only let's say 1 game for a guy if he only played 1 game in a 365 day period.
I bring that up only because "season-end" is as previously established kind of a fuzzy figure for RPM, and that's not even considering that we haven't determined if it uses playoff data or not yet. For the record I took the "season-end" measurements today. Anyway, here are the R^2s:
month WS WP RPM
feb .940 .865 .958
mar .968 .923 .960Higher R^2 is better, so from one perspective RPM is doing great because it has clearly the best correlation from February numbers to April numbers: if you wanted to predict how a guy would look at season end, you would want to use RPM. From another perspective, though, RPM's predictive power doesn't improve much at all when we add in the March data, which in turn suggests that the year end value only barely takes March into account.
I can't overstate how big a cause for concern this is in the world of empirically evaluating independent statistical models of sports. So... not that big a concern overall. But in this world, HUGE!!!
Having more data should make your predictions more accurate.
The only way this doesn't happen is if your predictions aren't using the data.
Your predictions should be using the data!!!
It's literally the only thing they should be using!!!
Now RPM's prediction does get a tiny bit better, so it's probably using the data a tiny bit.
My concerns are not assuaged.