Meta-Analysis: Examining My Own Process vs. (and vis-à-vis) the NFL Draft

My RB evaluation model has changed and expanded substantially since I began doing this work a year ago. I've increased the amount of data points I reference in each evaluation, changed the calculation method for several metrics, revamped my player comparison model, and worked towards developing various score outputs in order to quantify the whole of each player's profile, rather than leaving final evaluations up to my own subjective interpretation of the data.

Each RB in my database now has a Prospect Score, essentially a final number that the model spits out that rates a player relative to his peers on the overall quality of his profile. The Prospect Score doesn't tell you the percent likelihood that a player is drafted at X spot or in Y round, it doesn't tell you the likelihood that a player "succeeds" (however success might be defined) or posts Z amount of RB2 or better seasons. Instead, the Prospect Score is something like a Madden rating: it simply tells you, on a scale of 0-100, how good of a RB prospect a guy is.

Todd Gurley's Prospect Score of 85.1 just edges out Saquon Barkley for the highest mark in my entire database. (Photo by John Bazemore / Associated Press)

Subjective interpretation of the RB model would also take into account things like the Raw Traits Score (which ignores production), the On-Field Proof Score (which ignores everything but rushing efficiency and receiving ability), and player comps (for instance, despite a 75.7 to 75.2 Prospect Score advantage for Leonard Fournette over Joe Mixon back in 2017, I may have preferred Mixon due to to his 83.7 to 60.2 edge in the On-Field Proof Score), but for the purposes of this meta analysis of my own process, I will only be looking at the final, all-encompassing Prospect Scores.

Basically, my aim here is to examine how well my model does at identifying player ability (or inability) when compared to where teams select players in the NFL Draft. Because the model does not account for draft capital (again, it's like a Madden rating -- I'm trying to evaluate players in a vacuum), I am also interested in the usefulness of the model paired with draft capital. If the model is worse than the NFL Draft -- or if it doesn't add much to the insights we gain through the NFL Draft -- then it is likely not of much use.

The sample group used in this analysis is every RB drafted (along with some UDFAs) from 2007 to 2018. There are players in my database who were drafted before 2007, but I don't yet have full draft classes prior to 2007, and I didn't want to introduce to the study the kind of selection bias that would be present if I were to only include the Reggie Bush/Ricky Williams/LaDainian Tomlinson types from earlier classes. Having said that, I do acknowledge that the presence of the UDFA players introduces some survivorship bias to the study; because I started putting my database together just under two years ago, I don't have profiles on every UDFA RB who latched on to a team for three weeks in training camp back in 2008. Most of the UDFA profiles I do have belong to guys from the last few draft classes or guys like Zach Zenner or Thomas Rawls or Raheem Mostert: players who may have experienced some success, but who are mostly back of the depth chart guys that have stuck around for a few years. I'm mostly ok with including them in the study, but I wanted to acknowledge them for the sake of transparency. But I digress: all in all, the study group is made up of 276 players.

In much the same way my model generates Prospect Scores to rate players' college profiles, the other end of it also generates NFL Scores to rate players' profiles as professionals. If I were doing this study completely internally for the purposes of perfecting my own process, I would probably look to examine how Prospect Scores and NFL Scores correlate to one another. Given that most of the machinations of my process are not public (I'm all smoke and mirrors), examining that relationship here for the purposes of observing the usefulness of the model is inherently not very transparent.

I could instead use fantasy points produced as my dependent variable. I'm not much a fan of this as I think (this is surely not controversial) that team environment is a very big factor in how many fantasy points a given player produces. Alvin Kamara produces a lot of points playing with Drew Bress and for the Saints, and much of his production is due to his own talent and skillset, but he surely wouldn't be as prolific if he played for a worse team with a lesser QB, and I don't believe he should be penalized for that in this study.

Alvin Kamara is the truth, but his fantasy production is also fundamentally linked to the offensive environment in which he operates. (Photo by Scott Clause / USA TODAY Network)

Instead, as the dependent variable, I chose to use Dominator Rating (for the uninitiated, Dominator Rating is the percent of a team's offensive production -- yards and TDs -- that a particular player is responsible for). I went this direction because, like fantasy points, Dominator Rating is reflective of a player's production, which (in theory, at least) is reflective of his talent, but unlike fantasy points, Dominator Rating is more situation-agnostic. Joe Mixon can post a 25.9% DR on the Bengals to match Aaron Jones' 26.1% DR on the Packers, and that tells us that Mixon and Jones are responsible for a very similar portion of their respective teams' offensive production, without muddying the evaluation with volume stats that would indicate that Jones is a better player because he had more yards and TDs in the context of a more effective offense.

Dominator Rating can be calculated in slightly different ways, but my preference is for per-game DR. It doesn't make much sense in this context (since we are examining player talent, not player availability) to compare Alvin Kamara's output in 14 games this season to the Saints overall production in the full 16 games. By pro-rating team production to match the games played by the relevant player, you evaluate their contributions only in the context of the amount of games in which they were available to contribute.

I also am using a sort of modified "career long" Dominator Rating. For several reasons, the NFL side of my model evaluates RBs until they have played their age-25 season -- mostly because research I've done has shown that RBs tend to peak at age 25, so including the older and theoretically diminished versions of themselves in their full evaluations doesn't make sense -- and then projects forward after that using an age-based talent curve. Only using until-25 DR also means that I'm only evaluating the impact a player made in the NFL relatively soon after he was available to be taken in a dynasty fantasy rookie draft; if a RB doesn't do anything for years before turning into a stud as a 28-year old, good for him, really, but he didn't give you much return on your investment in fantasy (the inverse is also true: dude dominates from age 21-24, then gets hurt or wears down and isn't good into his late 20s -- you wouldn't say that guy wasn't worth the cost of acquisition just because he isn't Frank Gore). I say all of this to say that the Dominator Rating I am using is a per-game pro-rated DR taking into account each game a player plays until the end of his age-25 season. We will now call this number "Career DR," or just "DR."

Frank Gore might be ageless, but he also posted a Dominator Rating of 32.0% through age 25, the 18th highest mark in my database. (Photo by Elsa / Getty Images)

(Note: I realize that DR is not perfect -- it doesn't always identify the Darren Sproles type guy who plays in a backup role but is super efficient, it doesn't identify the end-of-the-depth-chart guy who is really talented but never gets a real shot. The NFL Scores in my model would do a better job of identifying Guy A here, but neither of those players are particularly valuable in fantasy football, and that's really what we're doing here. I am trying to properly evaluate player talent, but always in the context of fantasy football. Part of that is identifying what the NFL likes, because catching their eye = opportunity = production.)

(Another note: Some of the guys included in the study have not yet played their age-25 season. I don't really care much. Some guys who are younger than 25 will have smashed for one season when the model wouldn't have liked them, some of them will have toiled away as backups waiting to breakout when the model said they would be good, some of them will be doing exactly what the model thought they would be doing; basically, I think it all evens out, and most guys have enough years under their belt to feel pretty confident in what their Career DR says about them anyway.)

Phew. Now that all that nonsense is out of the way, let's get to the actual analysis.

First, I put the sample group through correlation tests in Excel: one examining the relationship between the round in which a player was drafted and his Career DR, and one examining the relationship between a player's Prospect Score and his Career DR. The r-squared values for each independent variable and its relationship with Career DR:

Get rekt, NFL general managers. My Excel spreadsheet is slightly better at projecting the kinds of Dominator Ratings that RBs will post by the time they have finished their age 25 seasons than your draft selections are.

This feels pretty important. The difference between the two methods (drafting players in dynasty based on my model's Prospect Scores vs. drafting players in dynasty based on where they were selected in the NFL Draft) is very small, but given how much production in fantasy football is tied to opportunity, and given how much opportunity is tied to draft capital, producing any kind of metric that comes close to matching the usefulness of the NFL Draft, not to mention beating it, is significant. I say this not to gas myself up (though it's obviously very encouraging, and I feel good about my model because of it), but to acknowledge the credibility this finding lends my process. It works, at least to some degree.

In addition to pitting my model against the NFL Draft, I also was interested in examining the usefulness of the two of them used in conjunction with one another. My process in this undertaking was somewhat convoluted, but I will try to explain it the best I can.

I first binned all the sample players into groups based on the round in which they were selected in the NFL Draft, and then found the average Prospect Score for players drafted in each round. I left UDFAs out of this portion of the study because the UDFAs in my database are probably not reflective of "average" UDFAs (given that I only have profiles for the most recent + most successful UDFAs since 2007). The results of that binning and averaging are as follows:

Results here are pretty intuitive: the NFL and I are on generally the same page when it comes to drafting RBs (at least when it comes to drafting them in the right order). The players that my model has deemed more talented are, on average, the players that NFL general managers have opted to select more highly in the Draft. But that's already been mostly established and is therefore not really what we care about here.

After finding those averages, I then averaged the Career DRs of the players in each group to generate a sort of "success threshold" for players taken in each round. Maybe using the median DR would've made more sense, maybe not; honestly I didn't think about it a whole lot, but I think for the purposes of this mostly quick and dirty analysis, average is fine. The average Career DR for RBs by Draft round is:

The sliding scale here is also pretty intuitive. Players selected earlier in the Draft (who are also generally more talented) are more productive than those drafted later. But, what we're more immediately concerned with is the thresholds for success that these DR numbers create. From here on out, a guy whose Career DR is at least as high as the average Career DR of all players selected in the round that he was selected in will be considered a "success" (or a "hit"), at least in terms of living up to his draft position; those whose Career DRs are lower than their Draft round average will be considered "failures" (or "misses").

I then wanted to determine what percentage of players drafted in each round lived up to their selection -- basically, what percentage of players selected in the first round posted a Career DR of at least 22.9%. However, (as I maybe should have expected) the "success" rate was right around 50% for each round (lower for later rounds, as most players drafted late don't pan out and the average Career DR for 7th round picks, for example, is buoyed even to its relatively low point by the few hits), which is an incredibly uninteresting finding (about half of players in a given population are above average, wow).

I then took a different approach by further binning each group into "good" picks and "bad" picks based on how their Prospect Score compared to the average Prospect Score of all backs taken in the round they were. Basically, I labeled every RB drafted in the first round with a Prospect Score of at least 68.2 a "good" first round pick (positional value aside, of course, I'm only concerned with these RBs in relation to each other), and every RB drafted in the first round with a Prospect Score below 68.2 a "bad" first round pick. This was done for each draft round.

The Career DRs of "good" and "bad" picks were then pitted against the average Career DR of all players taken in their respective rounds in two different ways.

First, I averaged the Career DRs of good picks and bad picks in each round and compared them to each other. My hypothesis was that good picks would post higher Career DRs on average than bad picks (bold, I know). The results:

I was right, but more importantly, I think we have some interesting and actionable findings here. Put succinctly, I believe this shows that letting the NFL Draft tell you which players are the best players is not the most sound process you could be using. If, for example, you treated all 2nd round picks the same, you subject yourself to a player pool in which the average guy posts a Career DR of 15.4%. Not terrible (Latavius Murray's Career DR is 15.3%), but we can do better. By evaluating players on their own merits and not blindly following what the NFL tells us, we can more effectively draft players for our fantasy teams: I know that not all 2nd round picks are created equal, so much so that the good ones are almost twice as good as the bad ones on average. Even good 3rd round picks are almost 50% better than below-average 2nd round picks. Hell, even good 4th and 5th round picks give you 85-95% of the production that a poor 2nd round pick would, and they very likely are not going to cost nearly as much in a dynasty rookie draft.

Put in tangible terms, treating all 2nd round picks the same gives you equal odds of ending up with Ronald Jones or Montee Ball as you do Nick Chubb or LeSean McCoy. Evaluating the players outside the context of where they were taken in the NFL Draft (and importantly, doing so well) means  avoiding the bust zone and focusing in on the Chubb / McCoy area, where the Ameer Abdullah landmines exist but are fewer and farther between. That approach would also see you fade the 2nd round bust-risks in favor of the 3rd, 4th, and 5th round upside plays: the Alvin Kamaras, the David Johnsons, the Lamar Millers, the Jeremy McNicholses (we can't bat 1.000), and the Aaron Joneses, while then pouncing on those lesser 2nd round picks when they fall to a more palatable (read: later) area of your rookie draft that more accurately reflects their talent level.

Montee Ball & LeSean McCoy: Choose Your Fighter. (Photos by Ron Chenoy / USA TODAY Sports and Ron Cortes / Philadelphia Inquirer, respectively)

These findings are all further solidified by the second method through which I examined "good" vs. "bad" picks in each given round.

Here, instead of finding average Career DRs for good and bad picks in each round and comparing them to each other and to the round average, I examined "hit" rates for good and bad picks in each round. As mentioned earlier, a "hit" is a player who meets or exceeds the average Career DR of all players taken in the round that they were; a "miss" is a player who doesn't meet the DR threshold. The results were as follows (I kept average DR for good and bad picks on this table just as a sort of comparison point to hit rate):

Things get muddy here near the end of the Draft, as there's not much to separate 5th or 6th round picks based on hit rate (just as there isn't based on DR), but through the first 4 rounds, differentiating between good and bad picks continues to be very helpful.

Maybe this difference in good pick success and bad pick failure is due to teams taking better players earlier in each individual round (that wouldn't explain why good 3rd-5th round picks are better or almost as good as bad 2nd round picks on average, but maybe the good vs. bad pick -- according to Prospect Score -- dichotomy within rounds is really just players taken earlier in the round vs. players taken later in the round). This possibility struck me at this point in my analysis, so I investigated.

Turns out, the slot at which a player was selected not only isn't better than using my model's Prospect Scores, it's not even more useful than just looking at the rounds players are drafted in:

These differences are not huge, obviously, but it is important (and encouraging) that Prospect Score is outperforming draft capital to any degree.

These findings are consistent even when we narrow our focus to the order in which players are picked within each individual round:

In every round but the 6th (yikes), Prospect Score outperforms draft slot, and in the case of the 2nd through 4th rounds, Prospect Score kinda smashes it. I think we can conclude with some confidence that whether you're looking at Prospect Score vs. draft round, Prospect Score vs. draft pick, or Prospect Score vs. draft pick within draft round, Prospect Score is (on average, of course) the more useful data point in projecting the level of contribution that RB prospects will have within their NFL teams' offenses (at least through age 25).

At this point in my analysis, I had pretty much exhausted the comparison of the utility of my model and its Prospect Score vs. the utility of the NFL Draft. I instead became interested in how the two might be used together. In order to do that, I binned players into groups based on their Prospect Scores, informed by what the NFL tells us an x-round RB looks like. I used this chart from earlier:

Any player with a Prospect Score of at least 68.2, regardless of the round in which they were actually drafted, I labeled a "1st Round Quality" player. Any player with a Prospect Score of less than 68.2 and at least 63.1 was labeled a "2nd Round Quality" player, and so on; any player with a Prospect Score below 50.1 was labeled a "UDFA Quality" player (these aren't necessarily thresholds that I agree with, but I don't really have a great method of making up my own -- I'm not very interested in taking real-life positional value into account here, and any thresholds I did establish would be pretty subjective anyway -- so, the NFL Draft-informed marks are plenty useful for my purposes here).

Then, I made some conclusions based on logical assumptions about dynasty rookie drafts. Given that:

a) draft capital has a large influence on where players are taken in dynasty rookie drafts, and
b) player talent has a large influence on where players are taken in dynasty rookie drafts,

we can assume that:

c) a player drafted in round X or at spot Y in the NFL Draft will have a similar rookie draft ADP (average draft position) as other players drafted in round X or near spot Y in the NFL Draft, and
d) a player with an assumed talent level of Z (measured here by my model's Prospect Score) will have a similar rookie draft ADP as other players with an assumed talent level of approximately Z.

This is not exact science (an in-depth study on how draft capital drives rookie draft ADP would be interesting, and I'm sure it has been done), but I don't know that it needs to be. I'm essentially concluding that it is fair to assume that Leonard Fournette and Saquon Barkley will have similar rookie draft ADPs given the draft capital spent on them in the NFL Draft, or that it is fair to assume that Paul Perkins and Mark Walton will have similar rookie draft ADPs given what their perceived level of talent is.

Leonard Fournette was an early first round pick in the NFL Draft as well as in dynasty rookie drafts. (Photo by Travis Spradling / The Advocate)

I then referred back to the draft round-relative success thresholds I used earlier, applying them now to the round in which a player was drafted in the NFL Draft as well as to the quality of player that his Prospect Score indicates he is. For example, let's examine Kareem Hunt. As a 3rd round draft pick, one of his thresholds for success is the average Career DR of all 3rd round picks, or 13.7%. According to the average Prospect Scores of players by draft round, Hunt's Prospect Score of 64.9 indicates that he is a 2nd Round Quality player, so his other threshold for success is the average Career DR of all 2nd round picks, or 15.4%. In other words, we determine whether or not Kareem Hunt is a hit in two ways:

1. If he lives up to his draft position
2. If he lives up to his Prospect Score

I then established another success threshold, this one fairly subjective. I sorted the players in my database by Career DR and eyeballed 15.0% as a pretty good cut-off spot for "useful fantasy asset" (guys just above are Ahmad Bradshaw, Latavius Murray, and Tarik Cohen, guys just below are Kenneth Dixon, Zac Stacy, and Royce Freeman, so *shrug emoji* good enough for government work), so I decided that would be my flat threshold for player success, regardless of Prospect Score or draft capital.

These three different thresholds obviously mean that a player might be considered a success in one of them but not the other two, or in two of them but not quite meet the other one, or some might meet all of them; this is good, because I am interested in examining player success rates according to different criteria based on each of the factors I am considering. A sixth round NFL draft pick (Elijah McGuire or Theo Riddick, for example) doesn't need to meet the flat success threshold of 15.0% Career DR in order to offer a good return-on-investment for where you took him in the late rounds of a dynasty rookie draft, and another guy who was picked in the first round of the NFL Draft and does meet that 15.0% Career DR flat success threshold (say, Trent Richardson) wasn't necessarily a good pick at the 1.01 or 1.02 of a dynasty rookie draft if he doesn't meet the 22.9% Career DR mark that other first round picks have established as the baseline for success.

After establishing those success thresholds, I simply binned players into groups based on Prospect Score, calculated round-relative, talent-relative, and flat hit rates among those groups, and then further binned those groups based on the round in which players were taken in the NFL Draft, and calculated all three hit rates for each of those groups. It sounds convoluted, and it was a bit of work, but hopefully the charts I made will help. Here are results for 1st Round Quality players, or those with Prospect Scores of at least 68.2:

Essentially what we see here is that there are 34 total 1st Round Quality players in the sample group, with an average Career DR of 21.8% and, among those 34 players, 76.5% of them have lived up to the round in the NFL Draft they were selected in, 47.1% have lived up to their Prospect Scores, and 79.4% of them have been useful fantasy assets. Each row below the full group of 34 is divided up based on the round in which they were selected in the NFL Draft.

One important note here is that the seven 1st Round Quality players who were selected in the 3rd round is an even more impressive group than the numbers indicate they are. The only player of the seven who did not meet the 3rd round success threshold is Ty Montgomery, a player who was drafted as a WR and whose Prospect Score is artificially inflated by the relatively ridiculous receiving production that he posted while playing wideout in college.

Upon switching positions in the pros, Ty Montgomery and his 172 career college receptions were able to trick my RB model into thinking he would be a top-tier all-purpose threat out of the backfield. (Photo by Tim Heltman / USA TODAY Sports)

Another thing to keep an eye on while reading the rest of these charts is the difference between round-relative hit rates and Prospect Score-relative hit rates. Players who were underdrafted (a 2nd Round Quality player taken in the 4th round, for example, like Chase Edmonds) have a harder time hitting their Prospect Score-relative success threshold than players drafted more in-line with their talent level, and conversely, these players will have an easier time living up to their round-relative success threshold, given that they are more talented than most players taken in the round they were. The opposite is also true. An overdrafted player (maybe a 4th Round Quality player taken in the 1st round, like Ryan Mathews) will more easily meet his Prospect Score-relative success threshold than other similarly talented players, given the opportunity that his draft capital affords him, but he will also find it more difficult to meet his round-relative success threshold, considering that he is not as talented as most other players taken as early as he was.

Here is the chart for 2nd Round Quality players:

And for 3rd Round Quality players:

For 4th Round Quality players:

5th Round Quality players:

6th Round Quality:

7th Round Quality:

And finally, for UDFA Quality players (again, the sample of UDFAs in my database is likely not very reflective of the general population of UDFAs):

There are insights to glean in all of these charts, and I won't delineate them all here (I surely haven't even gleaned all of them myself yet), but one thing I was interested in was ranking each group of players by flat hit rate, to see from which populations (divided up by Prospect Score and draft capital) useful fantasy assets are most likely to come. Many of the sample groups don't have enough players in them to really take solid insights from (one 1st Round Quality player has been taken in the 7th round -- Ahmad Bradshaw -- and he succeeded, which doesn't necessarily mean that we should expect all 1st Round Quality players taken in the 7th round to succeed), so I'll limit the examined groups to those with at least 5 players in them. I'll also ignore UDFAs given the limitations of my UDFA group that I've outlined before. Here are the groups ranked by flat hit rate, divided up Prospect Score and draft round:

Some of these numbers are still a bit warped by small sample sizes (I'm not going to prefer UDFA Quality players taken in the 4th round over 4th Round Quality players taken in the 4th round just because one has a 20% hit rate and the other has a 0%), but there are useful nuggets here. You can improve your hit rate even among first round picks just by looking at the quality of their Prospect Scores, as 1st Round Quality players taken there hit 20.3% more often than do 2nd Round Quality players taken in that round, and they hit 41.7% more often than 4th Round Quality players taken in the first. That trend is also true of players taken in the second and third rounds.

After all this, I wanted to return back to the regression analysis from the beginning of the study and see how well the RB model and Prospect Score perform in projecting Career DR when paired with draft capital. I ran tests with Prospect Score combined with draft round, Prospect Score combined with draft pick, as well as Prospect Score combined with both draft round and draft pick, and the results were as follows (along with the tests we ran earlier):

It's useful to know that combining draft capital with Prospect Score gives you a better idea of a player's potential than either of them do on their own. None of the numbers here are especially large, which is a testament to the difficulty of evaluating college prospects as well as to the validity of the "RBs Don't Matter" school of thought. Production for the position is so dependent on opportunity and team environment that the talent of the player receiving the opportunity often doesn't make a massive difference. The best we can do is try to identify players that the NFL is likely to offer opportunity, as well as players who are likely to be more effective and efficient with the opportunities they are given than others. Short shelf-lives, the injury-related attrition that comes from smashing into 300-pound men play after play and game after game for years on end, intangible psychological factors, unforeseeable developments in team management and roster construction, as well as countless other variables, make that a difficult proposition. The fact that the order in which NFL teams select players in their Draft is such a relatively weak predictor of player performance is a testament to that difficulty.

I'd love to do further tests on how Prospect Score can be paired with other elements of my model like On-Field Proof Score, Production Score, and Raw Traits Score to improve hit rates and predictions of player production and fantasy value. For now, we'll have to be happy with beating draft capital. We'll throw our darts and let Prospect Score guide our aim.