An Update to "Using Statistical Benchmarks to Find Successful Quarterbacks"

For the past few years, I’ve been looking at ways to evaluate quarterback prospects based on different metrics to look at. You guys might remember a post I made a few years back “Using Statistical Benchmarks to Find Successful Quarterbacks”, which got a moderately big response. I’ve been learning some new data science techniques, and have come up with a few more metrics that I think provide significant value when evaluating quarterbacks. To help do this analysis, I created a relational database of all drafted quarterbacks since 2004, their college gamelogs, and school data since 200, and then used SQL to create a few different metrics based on the data.

Defense Adjusted Production

The first aspect of quarterbacking that I wanted to look at was the effect that quality of defense had on QB performance. My hypothesis was that many quarterbacks played in noisy games versus poor defense, which padded their stats. The first thing I did was look at the base correlation between adjusted yards per attempt (a passing efficiency stat that accounts for touchdowns and interceptions), and see how good at was at projecting QBs into the NFL. The stat I used for NFL performance was average career approximate value, which is a metric created by pro-football-reference.com that attempts to boil player performance down to one number. I used this metric as opposed to others because you can use it to compare different positions, which is important when I bring all the models together to make a big board.

With an r^2 of 0.0747, we can say that about 7.47% of the differences in NFL performance could be explained by their college performance, a weak correlation. As you can see on the scatter plot, many of the lower end college QBs never amounted to much in the NFL, but having a higher AYA doesn’t necessarily guarantee success. You have guys like Cam Newton and Andrew Luck at the top, but the list also includes Tim Tebow, Omar Jacobs, and Stefan LeFors. Now, let’s see what happens when we adjust for defenses:

The r^2 goes from 0.0747 to 0.1217, about a 63% increase. What changed is that many of the middle of the road career AYA guys that ended up being NFL studs were not beating up on bad defenses, and were able to compete at the highest level. Some of these guys included Philip Rivers, Ben Roethlisberger, and Aaron Rodgers. There were also some guys that were exposed by better defenses, such as Jamarcus Russell who went from a career AYA of 8.43 to 6.98 against qualifying defenses.

Here are some interesting defense adjusted AYA from last year’s QB class:
Jared Goff = 7.29
Paxton Lynch = 5.96
Christian Hackenberg = 6.21
Dak Prescott = 7.62
Cardale Jones = 9.17
Cody Kessler = 7.21

Road Warriors

Another interesting trend I found was that production on the road or at a neutral site was better at predicting success than just plain AYA:

The top 3 QBs in this metric are Ben Roethlisberger, Cam Newton, and Philip Rivers. One of the lowest is first round bust Josh Freeman, who had a 14 to 19 TD/INT ratio over his college career on the road. 

Like before, here are some interesting road AYA numbers from 2016 QBs:
Jared Goff = 7.78
Paxton Lynch = 7.51
Christian Hackenberg = 6.85
Dak Prescott = 7.86
Cardale Jones = 9.25
Cody Kessler = 8.23

Proving the Filter is statistically significant

When I made my original post a few years back, I presented a filter using a few era adjusted stats to identify quarterbacks with a greater chance for NFL success. I was inspired by other filters such as /r/jumosq’s force players and the rule of 26-27-60, and feel like I found something that was useful and actionable. While it may or may not have missed on the highly drafted guys last year (Goff and Lynch), it was able to identify two mid round picks that have so far played above their draft slot to this point (Prescott and Kessler). While I think I did a good job of explaining exactly what the filter was, I don’t think I did a good job showing that it was statistically significant. For this reason, I decided to do a couple of statistical tests to show that the filter does a good job of not only explaining the past, but helps when using data that was not part of the initial model.

T-test using training data:

As expected, the filter QBs vastly outplayed the other QBs that were used in the original sample. However, this was to be expected. What is important is what happens when I introduce new data that the original model did not use to make the original analysis. To do this, I collected QB data from the 1994 to 1999 drafts to see if the filter consistently finds successful QBs.

T-test using testing data:

While the t stat is not nearly as big as the t stat using the training data, it is still big enough to be significant at a 0.05 significance level. This means that the filter not only is able to explain past data, but when new data is introduced it is still able to identify better performing QBs. 

If you guys liked this post, I’m going to be doing more analysis on different position groups in the future. I’ll probably move onto WRs (because the data is so useful and easy to use) or safeties (because I still haven’t found good metrics for them and because the 2017 class is so hyped up) as my next post. If you have any questions or comments, don’t be afraid to post!