Is your team’s GM a genius?

Every once in a while, I like to do a quick poll among my baseball friends. Which team is the current “magic” team? The one with the front office that does things so perfectly that no matter what they do, even if it looks strange, it must actually be a stroke of genius. And at this point, we’ve cycled through most of the teams in baseball, from Billy Beane after Moneyball all the way across the Bay to Brian Sabean and the Giants’ three World Series wins in five years. They all had the magic touch. Until they didn’t.

If there’s something constant about the game of baseball, it’s the ability of the game to humble just about anyone.

And we’ve reached the point in the offseason where it’s become clear who the big movers are for 2015. Some have gone big in the free-agent market. Some have been trading players like they were baseball cards (Hey there, Mr. Beane!). Some appear to be sitting there hoping that 2017 comes sooner rather than later. And because pitchers and catchers haven’t even reported yet, there’s still room enough for fans and critics alike to declare all of the moves that have been made successes or failures. Everyone’s got an angle. Everyone thinks that he’s figured out the next Moneyball.

At the end of every season, there are a handful of GMs who look like geniuses. (Hint: It’s usually the GMs of the teams who made the playoffs.) They will be lauded as visionaries for having the foresight to know that Free Agent X was going to have a surprise breakout season and for being savvy enough to draft Random Starter Y a few years ago in the fifth round only to watch him turn into a surprisingly good No. 3 starter. Nevermind that the GM really only signed Free Agent X because he was the last guy left on the market who could correctly point to the position that the team wanted him to play and Random Starter Y was drafted in the fifth round because no one, including the team that drafted him, thought he was that good. That’s why he made it to the fifth round.

Let’s ask a rather uncomfortable question. Are there general managers who are actually able to beat the system on a consistent basis? Guys who are actually geniuses when it comes to finding those hidden gems? Are there organizations that are just better at this than the others?

Warning! Gory Mathematical Details Ahead!

Let’s start with an instructive story. In 2010, the Phillies famously signed first baseman Ryan Howard to a 5-year, $125 million albatross … erm, contract extension, which was notable for the fact that they signed it two years before his then-current contract expired. Howard has disappointed since the new contract took effect. His raw hitting numbers are decent, but for a man who has platoon split issues,is a below-average defender and poor runner, and who doesn’t offer much more than the occasional home run, it’s not good value for the money. But is it the reason that the Phillies find themselves in so much trouble? Matt Swartz argued that while the deal has gone badly for the Phillies, blaming it all on Howard misses the point. Even if Howard were playing up to his contract, he might add five more wins to the Phillies, which last year would have made them a 78-win team.

The real phault in the Phillies phailure has been the phact that their draft and international free-agent signings from a few years ago haven’t really panned out. The result is that the Phillies have gotten the least amount of value from their cost-controlled players in MLB over the past few years. I have previously estimated that cost-controlled players (i.e., players in their first 6 years of service time) produce value at about half the price of those who have reached free agency eligibility. The Phillies basically messed up on their sources of cheap talent and now they’re paying for it.

It’s not a surprise that teams talk a lot about how the draft is a big priority for them. Get it right and you’ve got a nice pool of young talent to power those championship dreams. What might be surprising is that for all of the discussion around the importance of the draft, research has shown that teams, in general, aren’t very good at it, especially once you get past the first round. The correlation between signing bonuses and the value that they eventually produce is actually rather low. In other words, teams seem to be guessing as to who will turn out good and who won’t.

More than that, other research has shown that it doesn’t seem that success in one draft is a particularly good indicator that a team will have success in the next year’s draft. The math is a little complex, but it comes down to the fact that in the same way that a hitter who goes 4 for 5 on a given night is not actually likely to hit .800 for the rest of the season. He just had a particularly good (perhaps we might say lucky?) night. If that same hitter goes 0 for 5 the next night, we don’t assume he’ll forever hit .000. Five at-bats isn’t enough of a sample to base any conclusions on. Well, when you do that same sort of calculation for teams and their drafts, we find that even three years of draft data tell us little about what to expect in the coming years. I can tell you who had a good draft or two in retrospect, but I can’t tell you whether that success was the product of some sort of drafting mojo or just dumb luck.

There’s another way that GMs can show off their genius in acquiring players: finding bargains in the free-agent market. It’s generally accepted that a win (and by “win”, I mean a “win” above replacement level – the WAR that you’ve probably heard all about) costs about $7 million on the free-agent market. Happy is the GM who signs a player for $5 million and gets 3 wins out of him. He just found a nice little bargain.

But are teams particularly gifted at finding these bargains consistently? To find out, I looked at all players from 2005-2014 who had reached free agency eligibility (their seventh year in the league) and looked at their salaries and WAR produced. Generally, in these sorts of studies (the ones that say “a win costs $7 million”), researchers look at what current free agents are paid compared to how many wins they are projected to produce. They don’t count players who are already under contract or players who had just re-signed with their original team. Here, I’m simply looking retrospectively. How much bang for the buck did the team get?

I totaled up the number of WAR produced collectively by these free-agency eligible players and how much salary they earned in that year and divided the two numbers. For example, in 2014, free-agency eligible players made about $2.09 billion dollars combined and produced about 332 wins of value, meaning that the average win, among free-agent eligible players cost about $6.3 million. For 2014, I counted a player as a bargain simply by whether his own dollars-per-win ratio was north or south of this rate, even by a dollar. It’s imperfect, but we’ll go with it for now.

Again, we’ll do the same sort of calculation, called a reliability analysis, on whether the same GMs or teams seem to constantly be finding these bargains. (For the super-initiated, this is Kuder-Richardson Formula 21 reliability analysis, again with a binary “bargain” or “not bargain” outcome.) It turns out that after a sample of 10 player-years among free-agent eligibles, the reliability coefficient on bargain finding is .22, which isn’t that great. By 20 player-years, it climbs to .43, which is frustratingly middling. (Note: those reliability numbers are paneled by team. I re-ran by GM and got the same basic results.) For those who aren’t familiar with this type of analysis, these numbers suggest that there is more skill to finding free-agent bargains than we found previously in finding draft bargains, but that there is still a lot of room for good (and bad) luck to hold sway. (I should point out that there might also be a bias in these numbers that certain teams are able to spend more per win – time to break out the thinly veiled references to the Yankees – and so our “reliability” might more be describing the economics of the game. But even the Yankees would appreciate a decent bargain now and then.)

Genius, Idiot, or Both?

The results here might be a little surprising to both the “Fire the GM” and the “(Insert name of the new “it” GM) is a genius!” crowds. A couple of years of performance in either the draft and in the free-agent market are not a reliable indicator of a general manager’s skill in talent acquisition. He might have had a good couple of years, but we can’t reliably call him a genius because of it. Maybe he just has a really nice four leaf clover. He might have had a few bad years, but we don’t know yet whether he picked up a bad penny or just had too much faith in Brad Penny. But of course, he’ll get fired if he has a few bad years. Such is life.

You can think of it like this: Take a coin out of your pocket and flip it 10 imes, recording the result each time (head/tail). The most likely outcome is that you will get 5 heads and 5 tails. But, you might end up with 6 and 4 or 7 and 3. That’s just random variation. It doesn’t change the fact that the coin itself is a 50/50 chance to come up on either side. A GM might get to make only handful of truly meaningful roster decisions each year, and the evidence suggests that this sort of random variation is a much bigger factor in those results than we might want to admit. Still, does that mean that there’s no talent needed at all to be a MLB GM, just good luck?

There could be a couple of reasons why GM performance in these two vital areas is so unreliable. One is that all 30 GMs are incredibly smart guys, and they have created organizations filled with amazingly bright people who gather and analyze all sorts of useful information. The problem is that everyone else has a similar organization filled with brilliant minds and so everyone is looking at roughly the same information and analysis. When everyone has roughly the same information, it’s hard to beat the market. If everyone knows which guys are risky (and roughly how risky they are), it’s just a matter of picking which risks you are willing to take. It’s why teams spend so much time on almost negligible advantages that they might gain, if they think it will give them some leg up. There will be luck, of course, and you have to hope that you’re one of the ones who catches a bit of good luck, rather than bad.

Another possible explanation is that GMs are all idiots. Or, more charitably, especially with the draft, GMs make the best guesses that they can, but they are trying to evaluate 18-year-olds for what they will do when they are 27. It’s like having to turn in your answers in August for a test that will be given in November. A lot can happen between the ages of 18 and 27. Come to think of it, a lot can happen to a player within a year or two, so the same goes with evaluating free agents. Teams, of course, expend a lot of energy trying to figure out who will be good next year and in five years, but that’s not an easy task. There’s a lot of guesswork, so there are a lot of mistakes and a few unwitting successes.

Actually, those two explanations can be reconciled. Anyone at a bar who’s had a few drinks will tell you he can run the team better than that fool in the GM’s office now. I assure you that’s not true. Talk for about 15 minutes to any GM (pick one), and you’ll realize that (yes, even him). There might be small differences between them, but there are no idiots in the bunch. And the advantages that they do discover are quickly copied.

In high-level competitive chess, there are draws between grandmasters (a different kind of GM!) all the time. These are guys who have vast knowledge of the game, but rarely does one dominate another. Victories, even in single games, are usually because of one tiny mistake. But put that grandmaster up against someone who is a true novice (me, for example) and I’d probably be checkmated before I even knew what happened.

Baseball GMs are much the same. There’s a certain knowledge base that you need to have before you can play with the big boys and essentially hold your own against them. Not beat them, just play them to a draw on the things that you can control. A true novice (me, for example) would be fleeced before he got to the table. But on the other side, a good amount of what a GM is asked to do — predict five years into the future — is hard for anyone to do. And it conspires to look like randomness and we then confuse randomness for foolishness. The truth is that GMs are both incredibly smart and live and die by a good deal of luck.

So as we near the endgame of this offseason, and you are tempted to call your team’s GM an idiot (or perhaps something we can’t print here), I’d recommend that you look a little deeper. He’s probably a lot smarter than you (think), but he’s up against other really smart guys and a great deal of uncertainty that no one can control.

It’s a tough game to play. We are too quick to assume that GMs have figured out some magic strategy that no one else is smart enough to figure out when things are going well. We are too quick to assume that he’s an idiot when he’s not.