Players & Picks

A common analytical error

Last night was a calendar event for those, such as yours truly, who favor football of the American kind over the rest-of-the-world-kind: in traditional over-the-top fashion, the annual NFL draft began.

To summarize for the uninitiated, it is an event during which professional teams pick eligible college players. The order in which selections are made is usually the inverse of how teams finished the previous season; the worse you were, the higher you get to pick, the better player you can choose and the more improved for next season you can be. Once all 32 teams have grabbed their desired player, the process is repeated six more times over a total of four days.

Of course, this being professional sports in America, there is no end to analysis of choices made. From the official NFL website to the farthest reaches of social media and beyond, seasoned scouting veterans, newspaper columnists and average Joe sports fans alike chime in with their takes on what teams did right or wrong.

However, year after year in take after take, they all repeat the same fundamental analytical mistake.

During the draft itself, player selections are given an instant grade. But it is then, without fault, caveated with (or met by) a version of “this is all nonsense, of course, it is pointless to grade a pick until a few years down the line, when we know how it turned out”.

In reality, grading the draft pick the moment it is made is the only way to do it. The explanation lies in something called the halo effect.

Psychologist Edward Thorndike first coined the term in The Constant Error in Psychological Ratings, published in 1920. In short, it refers to a cognitive bias that distorts how one interprets new information based on previous impressions; we bend our judgment to predisposition. If an observer finds a particular aspect of a subject attractive, they will generalize the attraction and become positively dispositioned towards the subject.

The illustration that I most often use is televised talent shows. We have all seen it – the voice of an attractive singer is presumed by judges and audience alike to be equally beautiful, whereas the voice of an unattractive singer is presumed to be equally unappealing (leading to displays of shock and bewilderment as nobody had seemingly foreseen that aesthetically challenged people too could be talented). 

The reason can be found in the human inability, or at least unwillingness, to deal with cognitive dissonance. Most people struggle to independently measure different features and the halo effect allows them to create and maintain a coherent and consistent version of reality.

So what does this have to do with American football and, more to the point of this newsletter, strategy?

Well, for analysts (be it of professional sports or business behavior), there is nothing as attractive as superior performance, nor as unattractive as subpar performance. Consequently, organizations that are doing well are routinely believed to have outstanding strategies, astute management, brilliant marketing and so forth. Those less fortunate are instead thought to suffer from poor strategies, ignorant management and shoddy marketing.

Of course, the underlying reason may be something altogether different, not least luck or lack thereof.

The presumption that a positive outcome is the result of a good decision and a negative outcome is the result of a bad one is potentially dangerous, particularly from a strategic perspective. In practice, the opposite can be just as true; bad decisions habitually lead to positive results and vice versa.

What matters, therefore, is what those who made the decision knew or could have known at the time, not what becomes known later. Or to posit it differently, would a person who lost money betting on a game make the same bet if they had known at the time what they came to know once it had been played?

Whether something is good or bad depends on context and frozen moments in a time continuum; most events that we label as outcomes are never really endpoints. Instead, as Duncan Watts once wrote, they are artificially imposed milestones, and where we choose to impose an “end” leads us to infer very different lessons indeed. It may appear a good idea to imitate a company that is doing exceptionally well, but anything but a year later if the company, say, has managed to lose most of its market share by then. Often, that is also the case; just as how Michael Porter has been criticized for praising successful companies that subsequently failed, Clayton Christensen has been criticized for choosing arbitrary time periods that fit his argument. In the long run, as we know, performance tends to revert towards the mean.

As strategists, we should therefore make the best of what we have and what we know. If things turn out well, we accelerate. If they do not, we dampen and alter the course. And when we analyze decisions made and directions taken, we should attempt to establish what information was obtainable, ontological and otherwise, at the time, not what became known with the benefit of hindsight.

Similarly, NFL analysts should look at what the professional teams could have known about the college player they drafted and how they fit with the overall team composition at the time, not how their eventual career turned out after a myriad of unforeseen events inevitably impacted it.

Strategic analysis is about attempting to find objective truth, not creating subjective narratives. In order to succeed, we must account for the fact that we are evolutionary primed for the latter, not the former, and view the world through a conceptually blended lens of previous experience. While it has its undeniable uses, it can also easily lead us astray.

Remember, just because pride goes before the fall, it does not mean that if there is no fall, there would be no hubris.

Onwards and upwards
JP