Which manager had most success with review system in 2015?
After the conclusion of yet another great regular season of baseball, we can now start in on the exciting year in review retrospectives. The stats are in, the playoffs are scheduled, and we can look back on the totality of 2015's regular season with a full sample of what worked and what didn't for players and teams. The first day after the end of the season can be disappointing for fans who don't get to root for their favorite club during October, but it's also our first chance to draw that all-important end line that frames this year along with all the others that have come before it.
The same goes for evaluating managers. We have an idea of the managers who have done a good job; many of them still have games to play. We also have an idea of those who haven’t done a good job -- even up to the point of knowing who has a chance of being fired. The drawback, unfortunately, is that we still don’t have a great way of truly evaluating managers, and so we look to the small amount of data that we do have when we try to gauge their performance. Dave Cameron talked about this in the context of filling out his NL Manager of the Year ballot last year -- here’s a paragraph from that piece relevant to what we’re discussing today:
“Evaluating player performance is tricky enough even with all the amount of information we have about their performance; with managers, we’re basically just guessing. We can speculate about things that we think matter, but we don’t really have much objective data to support these thoughts.”
Dave's right -- we have very little data, and the data that we do have isn't terribly useful for evaluation. That being said, there is one newer area of data with respect to managers that I find interesting, and it lends itself to not only understanding an aspect of performance, but also -- in this year’s case -- serves as a window into the operating style of particular managers.
That data is the result of the fairly new system of manager replay review, and this season of baseball has produced some very interesting results. We already had a post earlier in the season on manager challenges, looking specifically at Kevin Cash, and his rather "unique" style of challenging (not waiting for any sort of video consultation from his coaches/advisors before popping out of the dugout to signal for an official review). That post theorized on a way to rank managers on their challenge ability; this post will go a step further in refining an attempt to do that.
We'll do this a few ways. First, we'll start by looking at successful challenges. This data comes from Baseball Reference, with the site originally ranking managers by challenge success rate, or the percentage of the time managers were correct out of the total number of times they challenged. That actually isn't the best way of looking at this data, however, as an absolute value of how many successes during the season is probably more valuable: since there is no penalty for losing a challenge (other than the loss of a potential opportunity to use the challenge in the future), there shouldn't be any penalty in our ranks for not succeeding with a review.
Think about it another way: challenging unsuccessfully at some point during a game is always better than ending a game without having challenged, as a manager has given their team at least a chance to improve upon their possibility of winning the game (however small that chance might be).
With that said, let's take a look at the number of successes by manager, along with the total number of challenges that they've initiated (represented by the dot). Teams that have had multiple managers in 2015 have been grouped together:
As we can see, Joe Maddon of the Cubs leads the pack, with Lloyd McClendon right behind him. Maddon has also lost a lot of challenges, but again, those don't matter unless a more important challenge-eligible moment in the game arose later on -- which is both rare and difficult to get data for. Total numbers aren't everything, but they mean a lot, especially considering that challenges are underutilized in general: theoretically, it is always better to end a game having challenged than to not have.
On the opposite end of the spectrum we have Brad Ausmus and Matt Williams, who were both stingy with when to challenge and had a low tally of successes. Kevin Cash -- just like when we last checked in -- had the highest overall number of challenges and the lowest success rate, showing that his style of not waiting for any sort of team-led video review before challenging is unique in the major leagues. In a way, Cash should be given credit for understanding that challenges should be used freely, though he should probably employ some sort of dugout video review before challenging in future seasons to increases his chance at success.
Now we're going to add an important element onto these challenge numbers: timing. The concept of clutch is easy to understand, and it impacts every single game we watch: we know intuitively when an important moment in a game has arrived, and players often get a label for being "clutch" for performing in those situations. We keep track of these high leverage moments, and we have data for how important the moments were during every single manager challenge this year. I've averaged them all for each manager to get a sense of who challenged in the most crucial game situations. For background, a rating of 1.00 is a considered a game situation of average importance, with a rating of 2.00 being twice as important to the expected outcome of the game. Take a look:
Here we can see that Ned Yost challenged at the most important moments of games out of all managers: times when the possible change in the expected outcome of games were, on average, at their greatest. A lot of this leaderboard is opportunistic -- a manager must have encountered situations that were important and had a challenge-eligible play occur -- but it still took awareness to know when those moments had arrived. I've also highlighted Joe Maddon to point him out as the leader in our previous graph.
Now we're going to put both of these ideas together, as the true gauge of how well a manager has challenged is not just how often he has succeeded, but how often he has successfully challenged in important situations. To do that, I've simply combined each manager's total challenge successes and clutch rating, then compared them to the league average result for all managers. This gives us the best answer to who the best manager at challenging was during the 2015 season, and frames each manager as a percentage above or below average (100). Here are the results:
We have our answer: Joe Maddon was 43% better than the average manager at successfully challenging during important moments of games. It's difficult to overstate how influential it was for Joe Maddon to be successful at so many challenges; even though he was middle of the pack when it came to how important the moments were when he challenged, the sheer number of overturned calls he had pushed him to the top of the leaderboard.
On the other end, if anyone needed more fuel for the Matt Williams fire, they have it: not only did Williams have among the lowest number of challenges and successes in the majors, he also challenged at relatively inconsequential times. The outcome of the reviews he initiated were not particularly important, and given the rate at which he challenged, he most likely made the poorest use out of the review system.
As said before, a lot of success and failure with the replay system is opportunistic, and one season of challenges is going to be subject to some small sample size problems. However, there are two common traits that successful managers share with respect to replay: they challenge often, and they identify the most important moments to use their challenges. They can't control how often they find themselves in those crucial situations, but they can control how they choose to handle them when they arise. And, while replay is only one part of a huge picture of manager performance, maybe it's not a coincidence that the three best managers above are all headed to the playoffs.
