Sacks: The NFL’s Most Confused Statistic

Welcome to a special edition of the 2018 Crunching The Numbers season preview! For those who are wondering, yes, there will be an actual preview before the season starts. If you’re new to the site, I’m not going to do a deep dive here in this article. Just know that “Crunching The Numbers” is an annual project I share here at BGN that uses statistics to preview Eagles games. Science! Math! Fun!

One of the statistics I look into is sacks, for seemingly obvious reasons. I love sacks. You love sacks. Your churchgoing Aunt Evelyn loves sacks. There are few things more satisfying than watching an opposing quarterback disappear into a sea of swarming defenders, especially if that quarterback is mobile (like Tony Romo or Aaron Rodgers) or inspires fear into the hearts of mortal men (like Peyton Manning or Tom Brady).

So you can imagine my surprise when, as I started to study the correlations between certain statistics and winning, that sacks often had no mathematical connection to how many games your team won.

Wait, what?

But that’s not all. In some years the correlation between sacks and winning was slightly negative (meaning that the fewer sacks you forced, the more games you won), while in others there was an overwhelmingly positive connection between sacks and winning, which is what we would expect. The swings were so severe that I felt it warranted a discussion in a separate article. Behold:

The Data

2013 was the first season I started really tracking data, and it makes for a nice five year analysis from 2013-2017. The term “SACK%” refers to the sack percentage achieved by your defense, while the “OFFSACK%” term refers to the sack percentage allowed by your offense. The correlation terms can vary between -1 and 1, with positive numbers representing a positive correlation and negative numbers representing a negative correlation. There is a correlation threshold that must be achieved to be considered “significant” - for a sample size of 32 (for 32 NFL teams), that threshold is 0.35 for positive correlations and -0.35 for negative correlations. If there is nothing else you take away from this paragraph, remember that 0.35 is the magic number.

Some observations:

• Overall, the sack rates declined between 2013-2016, before spiking again in 2017. This would seem to indicate that the “crisis of quality offensive line play” has subsided some, although further tracking through 2018 and beyond will be necessary to determine whether 2017 was an aberration or a sign of things to come. I’ll touch more on offensive line play in a bit.
• In years that the defensive sack rate was significant, the offensive sack rate was not, and vice versa (for the most part). In other words, if sacking the quarterback had a significant correlation with winning in a particular season, preventing your quarterback from being sacked did not. This seems extremely counter-intuitive; if sacking the quarterback is so important to winning, shouldn’t preventing these sacks be just as important? This warrants its own discussion, which I will (also) get to later in this post.
• The significance of sack rate literally flip-flopped over the past five years. In 2013, 2015, and 2017, sack rate had a strong correlation with success. In 2014 and 2016, sack rate did not have any real correlation with winning, and was in fact slightly negative.
• 2017 was the first year where both sack rate AND offensive sack rate had strong correlations in the ways we’d expect. Both your ability to get after the quarterback and your ability to protect him correlated with winning.

I can understand if all this feels very odd. How can sacks have such a highly variant connection to success? Maybe my data has errors? Well, outside of the fact I double-checked my inputs, let’s look at some raw data. Here are the top 10 sack teams from 2016, the year that had the lowest connection between sacks and winning. They are ordered from highest sack rate to lowest:

For reference, the average sack rate in 2016 was 5.79%, meaning these teams were getting sacks at a rate about 19% above the league average. And yet their combined win percentage was 0.497. It does not get more “unrelated” than that.

But, but, but - there’s the Brandon Graham defense! Sacks don’t tell the whole story! You have to look at pressures, too! That’s what really matters.

Well it turns out that overall pressure is even more inconsistent than sacks. This time, I only have data going back to 2015, using pressure rate as it is defined by Football Outsiders:

The only year where pressure rate has a significantly positive connection to winning is in 2015, and in 2016 it had a significant negative correlation - if you got a lot of pressure on the quarterback that year, your team was probably bad. What’s interesting is that Football Outsiders found in each year that teams with a higher pressure rate had a better defensive DVOA. Obviously, I have no reason to dispute this. Football statistics is their job, and their methods are much more robust than mine. But influencing defensive performance does have some distance from the end goal - winning - which might help explain the discrepancy a bit. Let’s get into that.

The Discussion

For those that pay attention to my “Crunching The Numbers” posts, you may have noticed that I have not included sacks into my formula for quite some time, for this very reason. I’ve been following the inconsistency of sacks for a while now and decided now was the time for a dedicated post on the subject. My original hypothesis behind the sack mystery was that it wasn’t the sack itself that made the difference - it was the result of the sack: the interception, the intentional grounding, the tackle-for-loss, the throwaway, the clinching forced fumble. Essentially, these events were gobbling up the “connection” between the sack and winning, because that was the more significant result of the play.

If this hypothesis were correct, we would see a consistent lack of correlation between sacks and winning. This is not the case, forcing me to come up with a new theory: bad offensive line play.

The overall lack of quality offensive line depth is well-documented to the point where the phrase “dearth of good offensive lineman” is practically a football cliche. On the flip side, we have not seen the same crisis with defensive lineman. They’re valuable enough to warrant first-round picks, of course, but there simply has not been the same desperation for quality pass rushers as there has been for pass blockers.

The result? Over the past few seasons, if there is one thing your team is able to do well, it’s probably rush the passer, even if it’s only a modest rush. Even mediocre defensive lineman have been feasting on untalented and inexperienced offensive lineman, meaning that a 3-13 team like the 2016 Chicago Bears can have a better pass rush than the 11-5 Pittsburgh Steelers. This also provides an explanation for why protecting your quarterback hasn’t always had a strong connection to winning: if your team has been good over the past few seasons, there’s a good chance the weakest link has been the offensive line. This is why quarterbacks that can deliver the ball quickly (Tom Brady, Peyton Manning) or scramble (Russell Wilson, Tony Romo, Aaron Rodgers) have been able to keep their teams competitive.

This is all well and good, but what about the variance? If offensive line play has been fairly poor, why the flip-flops? Well, it turns out the issue is a bit more nuanced than that. It’s not necessarily the play of the starters that’s an issue (unless you’re the Giants), it’s the play of the backups, which is something you need to rely on often in the NFL. Of course, inevitably, you will have a season where offensive lines stay healthy and sacks are harder to come by (think 2016 Cowboys), meaning the team that CAN sack the quarterback will have more success. Then a regression to the mean happens, offensive lines get banged up (hello, 2017 Cowboys), and suddenly your Grandpa Stu is taking down Dak Prescott. Now that everyone’s getting sacks, it’s not as special.

The same holds true for protecting the quarterback. Offensive lines are banged up? If you’re one of the few teams able to protect your guy under center, then you’re probably winning. Offensive lines are healthy? Well, you just need to survive the truly talented pass-rushing teams while your line manhandles mediocre pass rushes for the rest of your games.

I haven’t formally tested this hypothesis yet, and I’m still coming up with how exactly I will tackle it. I will probably start with tracking adjusted games lost, and see if I can sort by position to find out if there is any connection between offensive line injuries and the correlation between sacks and winning.

Closing Thoughts

So, what does this all mean? Should we stop caring about sacks? No, of course not. Sacks are awesome. They’re an exciting play that can change the course of a game. They make the game more fun. And like I said before, 2017 was the first season we saw the “expected” correlations between both sack percentage and offensive sack percentage. Furthermore, if you take the average sack rate of each team over the past three seasons (2015-2017), and check the correlation against the win percentage over that same time span, you get a significant connection between the two (about 0.365). All of this points to a trend that sacks may be regaining their impact.

But at the same token, until a long-term trend is established, don’t sweat things out if the defense is going through a sack “dry spell.” Its high variance means that, during the season, it’s impossible to tell how important those sacks really are. For the time being, a well-timed sack might help you win an individual game, but sacks in general are just too erratic to get really upset about. Be more worried if your secondary is struggling or if you can’t stop the run.

And, of course, cheer wildly as the opposing quarterback gets dragged to the ground as the significance of the sack slowly (and hopefully) makes its comeback.