An objective way to rank Reid's draft performance

It's easy to identify specific draft picks that have worked out especially well or poorly for the Eagles, but it's harder to compare their performance with other teams. To provide one perspective for such comparisons, this FanPost presents an objective basis for evaluating draft performance.

The evaluations are mainly based on a ratio of return-per-investment, which equals the total career value of all players that a team drafted, divided by the total quality and quantity of draft picks that the team used to get those players. Based on that ratio:

  • The Eagles are ranked 13th in overall draft performance during the Andy Reid era (Table 1). Although that ranking is respectable, Reid's winning percentage is ranked even higher (5th), which suggests that other factors are more responsible for his success on the field. What other factors do you think are most important?
  • Some seemingly disappointing players are actually ranked near the league-wide average, which indicates that subjective evaluations can be excessively harsh (Table 2). Which player rankings are most surprising to you?
  • Four of the Eagles' first six drafts were below average, but four of the next five were above average (Table 3). Although those results are not conclusive, they've got me hoping that the Eagles are getting better at drafting. Do you share my optimism?
  • The Eagles are ranked 10th for drafting offensive players and 16th for drafting defensive players (Table 4). The Eagles have made greater investments in offense than defense, however, so their total career value for offense players is also ranked higher (6th versus 20th). Do you believe that the Eagles draft poorly for defense?
  • The Eagles have been most successful in rounds 3 and 5, where they ranked 6th and 4th respectively (Table 5). Do you like when the Eagles trade down for extra mid-round picks?

Even after considering all draft picks by all teams over eleven years, most of the differences among the rankings are not statistically significant. Therefore, it is still hard to be sure whether those differences are due to skill or luck, so we should be cautious about judging a specific year, or a GM, or even a particular draft strategy. Nonetheless, these rankings offer many potential topics for discussion, and of course I welcome any additional perspectives on the methods, the results, and the Eagles' overall draft performance.


Ranking the total value of all players that each team obtained through the draft

The objective method for grading each draft is based upon the Career Approximate Value (CarAV or CAV) as compiled by Pro-Football-Reference. As the term implies, the CAV approximates the cumulative value of each player's contributions throughout his career (through the 2011 season). The Approximate Value of each player is based on his individual performance (games started, yardage gained, tackles made, etc.) as well as the performance of his team (points scored and allowed).

Conventional wisdom indicates that three seasons are typically needed to evaluate a player's performance, so I skipped the past three drafts and examined the rest of the Andy Reid era from 1999-2009. My first approach was simply to add up the CAV for all the players who the Eagles drafted during those years.

The tallies include the performance of players after they left the Eagles. Therefore, I am mainly evaluating the ability of each team to obtain talent through the draft, and I am not evaluating any subsequent personnel decisions for managing that talent.

Estimating the total quality of all draft picks that each team used

To estimate the "quality" of each draft position in each year, I performed polynomial regressions of the CAV data. The regressions essentially just smoothen the actual data, so the Draft Position Quality (DPQ) of each pick is like the average CAV among players who were selected at a similar point in similar years. In other words, the quality of a draft position (DPQ) represents the value that a team could expect to get from that pick, which is similar to the actual value (CAV) that teams got from nearby picks. I added the DPQ of all picks by a team to produce a combined metric for their total quality and quantity (total DPQ).

As the regression curves show, the DPQ is typically higher in earlier rounds, which indicates that NFL teams can expect to get better players in those rounds (Figure 1). The DPQ is typically lower in recent years, because those players had fewer seasons to accumulate value for their career. The actual CAV data follows similar patterns, except that it has a lot of scatter because many players do much better or worse than expected (Figure 2).

Ranking the relative draft success of each team (return-per-investment ratio)

I calculated the relative draft success of each team as the total value of the players it drafted (CAV), divided by the total quality and quantity of the picks that it used to get those players (total DPQ). The ratio of total CAV:DPQ shows how well each team has done with the picks that it used, which represents a measure of its return-per-investment. If the draft performance of a team is better than average, then the value of its players should be higher than expected from the quality of its draft positions, and the return-per-investment ratio should be above 100%.



Overall team rankings

The total career value (CAV) of players drafted by the Eagles equals 1358 points, which is ranked 11th among the 32 NFL teams (Table 1). Their return-per-investment ratio is ranked 13th. Both rankings are consistent with an earlier FanPost which showed that Andy Reid's draft performance has been similar to the rest of the league.

The NFL awards the best draft picks to teams with the worst won-loss records, so I expected those teams to have a high quality of draft picks (DPQ). Surprisingly, however, teams with a low winning percentage did not have a significantly higher total DPQ, which suggests that the NFL draft policy has limited ability to achieve parity.

For example, the Eagles have the 5th best winning percentage during the Andy Reid era, so I might have expected them to have the 5th worst DPQ (i.e., they would be ranked 28th). Instead, the Eagles were ranked 16th for their total DPQ. They may have overcome poor draft positions by trading veteran players for additional picks and by trading down within drafts. The DPQ curves are less steep than the Draft Value Chart, which has led some studies to conclude that trading down is generally a good draft strategy.

Andy Reid's winning percentage exceeds his rankings for CAV and the return-per-investment ratio, so other factors could be more responsible for his success on the field. The discrepancy could also indicate that the CAV does not perfectly represent each player's contribution to winning. For example, the ranking for Reid's winning percentage matches his CAV ranking for quarterbacks, so Pro-Football-Reference may underestimate the importance of that position.

Some teams with the highest return-per-investment ratio have a good reputation with the draft, so qualitative assessments can be reliable in some cases (IND, BAL, PIT, NYG). Nonetheless, even after considering all draft picks by all teams over eleven years, most of the differences among the rankings are not statistically significant (i.e., a statistician would not be 95% confident that past differences between two teams are due to their drafting acumen rather than luck). For example, the Eagles' ranking may not be significantly different from the best team (IND) or the worst team (TAM) on the list.

Eagles' player rankings

The Eagles' draft picks with the highest career value (CAV) include Donovan McNabb, Brian Westbrook, Trent Cole, Corey Simon, Lito Sheppard, Sheldon Brown, Mike Patterson, Michael Lewis, Raheem Brock, and Bobbie Williams. (See Table 2 for a list of all Eagles' draft picks from 1999-2009).

After adjusting for the quality of their draft positions, the Eagles Greatest Hits include Raheem Brock, Trent Cole, Brent Celek, Moise Fokou, Brian Westbrook, Hotel California, Todd Herremans, Brandon Gibson, Omar Gaither, LeSean McCoy, and DeSean Jackson -- all of whom have more than doubled expectations (i.e., their ratio of CAV:DPQ is greater than 200%).

The players who have come closest to their expected CAV include Lito Sheppard, Sean Considine, Adrien Clarke, Mike McGlynn, Max Jean-Gilles, Brodrick Bunkley, Corey Simon, Victor Harris, Todd Pinkston, and A.J. Feeley (i.e., their ratio of CAV:DPQ is close to 100%). Some of those players have been considered disappointments, which may suggest that subjective evaluations can be excessively harsh.

Eagles' yearly rankings

I can imagine that Andy Reid might have needed many years to master the draft process, but I don't know how important his role has been, nor how much continuity he's had with the rest of the staff. To look for any evidence of progress, I calculated separate rankings for each draft class (Table 3).

Four of their first six drafts were below average, but four of the next five were above average. Although those results are not conclusive, they've got me hoping that the Eagles are getting better at the draft.

The best draft years of the Eagles have been 2002 and 2009; the worst years were 2003 & 2004. Overall I gave them a grade of C+ for their draft performance throughout the Andy Reid era.

Eagles' rankings by position

Recent comments have suggested that the Eagles are not good at drafting defensive players, so I extended these comparisons to look separately at each position (Table 4).

The Eagles ranked 20th for the total career value (CAV) of defensive players that they drafted from 1999-2009, but they ranked only 22nd for the combined quality and quantity of draft picks that they invested in those players (total DPQ). The Eagles ranked 16th for the total return-per-investment ratio. Thus the low value of drafted defensive players arose mainly from a low investment in those positions, whereas they've had an average level of success at evaluating the available players for a given draft pick. The main exception is at linebacker, where the Eagles have low ranks with all three parameters.

In contrast, the Eagles' drafts ranked 6th for total value of offensive players, and 5th for total investment in those positions. The Eagles have been especially good at drafting tight ends, running backs, and quarterbacks (ranked 3rd, 4th, & 8th respectively in their return-per-investment ratio). Despite their recent success with Maclin and Jackson, however, the Eagles have a low return-per-investment ratio at wide receiver (ranked 21st).

Overall, their return-per-investment ratio is only slightly higher for the offense than the defense (ranks 10th and 16th, respectively). From that perspective, it may be a bit harsh to suggest that the Eagles are poor at drafting defensive players, but of course there is room for improvement.

Eagles' rankings by round

A recent FanPost by no1pipelayer proposed that the Eagles typically use the 3rd round to take a reach or a project, especially following their successful selection of Brian Westbrook in 2002. The Eagles have certainly had some disappointments since then, but it's unclear how their performance compares with other teams. To provide one perspective for such comparisons, I used these methods to look separately at each round (Table 5).

As no1pipelayer suggested, the Eagles draft performance has been below average from 2003-2009, with a ranking of 24th in the third round and 20th overall (not shown). Throughout the entire tenure of Andy Reid, however, the Eagles have ranked 6th in the 3rd round, thanks of course to Westbrook (and Derrick Burgess and even Chris Gocong!). Thus the evaluations of the Eagles' draft performance can depend heavily on which time frame is considered.

Throughout the Andy Reid era, the Eagles have been most successful in round 5, mainly thanks to Trent Cole and Brent Celek.



Figure 1. Draft Position Quality (DPQ) for each pick in each draft from 1999-2009. The DPQ represents the expected contribution of a player taken with a particular pick, and it is based on the typical Career Approximate Value (CAV) of other players taken at a similar point in similar years. Another study has converted the CAV (per year) into more meaningful categories for the expected contribution of each draft position.



Figure 2. Sample results from the 2004 draft. The CAV is the data from Pro-Football-Reference for the performance of each player throughout his career. The DPQ is the best fit of the data, as calculated by a polynomial regression versus the draft position of each player. It represents the expected value for each player, which I am calling the "quality" of the draft position when he was selected. The DPQ is similar to the average CAV for all players selected in each round, but it declines smoothly throughout each round. The steal of the draft was Jared Allen (CAV = 84 at pick 126).



Table 1. Draft rankings of NFL teams from 1999-2009. The total "Career Approximate Value" (CAV) represents the total contribution of players selected by each team throughout their career (to-date). The total "Draft Position Quality" (DPQ) reflects the expected contribution of those players, based on the year and draft position when they were selected. The return-per-investment ratio of total CAV:DPQ shows how well each team has done with the draft picks that it used. The "wins" column shows the average number of wins per season from 1999-2011.



Table 2. Actual (CAV) and expected (DPQ) career success to-date for Eagles' draft picks from 1999-2009. The return-per-investment ratio is above 100% for players who have exceeded expectations (based on their draft position), and below 100% for players who have fallen short. To provide a more qualitative perspective, an incredibly thorough FanPost described the career of each player (although unfortunately it is hard to compare the Eagles with many other teams on this basis).





Table 3. Grades for each Eagles draft from 1999-2009. In each year, I gave an "A" to the top 5 teams, a "B" to the next 7 teams, a "C" to the middle 8 teams, a "D" to the next 7 teams, and an "F" to the bottom 5 teams. I gave a +/- to the top/bottom team within each grade classification. All of these rankings are based on the return-per-investment ratio, which equals the total value of players obtained in each year (CAV), divided by the total quality and quantity of draft picks used to get those players (DPQ).



Table 4. Positional rankings for the total value of players who the Eagles obtained through the draft (CAV), the total quality and quantity of draft picks used to get those players (DPQ), and the return-per-investment ratio. The "players" column shows the number of players selected for each position.



Table 5. Eagles rankings-per-round for the total value of players obtained through the draft (CAV), the total quality and quantity of draft picks used to get those players (DPQ), and the return-per-investment ratio. The ratio of total CAV:DPQ is probably the most meaningful ranking due to differences in the number of picks per round.