Biases Don't Exist, and Humans Are Not Irrational.
Some points of agreement and disagreement with Kahneman
In my last article, I had a fairly click-bait-y subtitle where I claimed that Naturalistic Decision-Making convinced me that biases don't matter.
I chose that subtitle on a whim right before pressing publish. I regret doing that. Not because the subtitle was wrong or misleading, nor because I am opposed to click bait titles in principle (clearly not). But rather because I didn't follow through on explaining what I meant. That was pretty rude of me. I am a firm believer that if you have a click-bait title that you should fully follow through with it. I will do better here and convince you that even Daniel Kahneman would agree with my my title for this article.
This will not be a comprehensive critique of Cognitive Biases. I just don’t have space to talk about the theoretical1 and methodological2 issues. I am not even going to spend much time defending heuristics and intuition. Nor will I talk about the lack of imagination that researchers have on different ways they could study, conceptualize, and address errors in decision-making. Instead, I will focus on the relationship of biases to rational models, and whether we should care.
Biases don't exist
First, let’s agree on what heuristics and biases even are because it is something that is very easy to misunderstand.3
A heuristic is a rule of thumb: an easy to implement guideline for complex situations. Most people are going to be familiar with various prescriptive heuristics; measure twice, cut once; A bird in the hand is worth two in the bush; never pee onto a flat rock. All good advice.
The heuristics psychologists talk about are descriptive rather than prescriptive. That is, they describe how people do think instead of prescribing how they should think. For example, when deciding between two risky options, people tend to avoid the option with the more memorable risk. We call this the Availability Heuristic, and it is typically adaptive: if you can easily recall someone dying, then it is very likely a dangerous thing.
But sometimes this heuristic can lead you astray. The most famous example is the thousands of people who decided to drive instead of fly after the 9/11 attacks. Driving is vastly more dangerous than flying, and some have calculated that more people died after 9/11 from choosing to drive than people who actually died on 9/11 itself. This seems like a pretty major failure to understand probabilities. Even if there was a plane hijacking every week of the year, flying would still be safer.
When heuristics lead to mistakes from rational theories (such as from probability), we call it a bias. So instead of talking about the Availability Heuristic which is the cognitive process used that led to the error. We might instead say the Availability Bias, emphasizing how the heuristic led us astray from Probability Theory.
But here is an important point to take away from what I just described; biases don't exist. Biases aren't things in the world and they are not cognitive processes. Biases are an artifact of what we are measuring against.
Some time ago, Gary Klein, who had a friendly but adversarial relationship with Kahneman, was going to grab dinner with him and asked if I had any questions for the Nobel Prize winning scientist. I wrote up some hard hitting questions to ask, including extensive notes in my email to Gary so that he could fully understand why I thought the questions interesting.
But when the time came, the dinner conversation turned to other topics. It was a meeting of friends, not academics, and Gary didn’t want to disrupt the conversation with my hard-hitting questions.
So instead, after he got home, Gary forwarded my entire email to Kahneman, and Kahneman responded. I was both starstruck and mortified. Here is the opening of Kahneman’s letter.
Hi Jared,
You seem to think that biases are a thing, and that I study that thing and am committed to the existence of biases. If you read a few chapter of ‘Thinking Fast and Slow’, you will find that this description is incorrect. What we really try to study are the psychological rules that describe human thinking.
Ouch. Nothing like one of your idols telling you that you have misunderstood his work!
But here he confirms the first part of my click bait title; biases don’t exist. And I don’t mean that empirically we have discovered they don’t exist, but rather, by definition they do not exist. They are not the type of thing which have existence. They are like a correlation coefficient in that if you try to use a correlation coefficient to explain something then you have fundamentally misunderstood what a correlation coefficient is. Correlation coefficients and biases don’t exist in a way that they have causal force in the world.
When you decide to drive instead of fly because of a recent high-profile crash, we can use the Availability Heuristic to explain what happened. The heuristic has causal power in the world. But biases have no causal power. It’s like saying an arrow missed the target. The property of missed is not in the arrow, the bow or the archer. You cannot explain why the arrow missed by evoking a process called missing. That is circular.
Similarly, a bias just means you missed the target, and so can't be used to explain anything. But in this case, the target is what you would have done if you had appropriately applied various prescriptive theories such as Decision Theory, Bayes Theorem, or Probability Theory.
To recap:
Bias: When a heuristic leads us to make a different decision than a rational theory.
Rational theory: What some academics decided was the correct way to make a decision (sometimes also called a Normative Model).
Heuristic: Psychological rule we follow which explain the bias.
Jonathan Baron listed 53 biases in his book (4th edition). Wikipedia lists more than 2004. That is a lot of ways that people systematically miss the target!
But remember, for every bias there is an underlying target (a model) from which it deviates, and it is essential to keep that target in mind. Why? Because at some point you need to decide whether you actually care about hitting it.
Personally, I often don’t care about these models which is why I tend to define biases not as “the systematic ways that human reasoning and decision making deviate from rational models”, but instead as “the systematic ways that economic models are wrong about how humans reason and decide.” Same meaning, completely different emphasis.
Who died and made academics the arbiters of rationality?
I am sympathetic to the list of rational models that academia has developed over the decades and centuries. They are certainly good for some situations.
But these “rational” models are not capital-R Rational. They were not delivered to us on Mount Sinai, written on stone by the finger of God. Academia has not solved the problem of rationality, and it does not have the answer to life, the universe, and everything. If you live your life according to the maxim WWHED (What Would Homo Economicus Do), I think you are seriously hampering your ability to make good decisions, and will wind up believing and acting on very bizarre and irrational things.
This is partly a moral argument. I am not a utilitarian, and I have to admit that the fact that Jonathan Baron considers my moral philosophy a bias does rub me the wrong way.
But even more so it is an argument based on the the very premises of Heuristics and Biases. When Kahneman and Tversky set out to disprove the assumptions underlying economic theory, they didn’t intend to say that the rational theories that economists relied on were capital-R Rational and fully general decision-making formulas by which to live your life. The word ‘rational’ was a technical term which they got from Rational Agent Theory. Kahneman and Tversky were excited about biases because they falsified the idea that humans reasoned using rational models, and also provided a methodology for discovering how people actually reason.
This is why Kahneman has said that his work has been misinterpreted as an indictment of humans when that isn’t really what his work showed. He goes on to clarify that the term “rational” is a technical term and that he hates the word “irrationality” and has never used it.5
And there it is! My clickbait title has been fully endorsed by the man himself; biases don't exist, and humans are not irrational.
Now with that context, here is Kahneman’s full letter.
Hi Jared,
You seem to think that biases are a thing, and that I study that thing and am committed to the existence of biases. If you read a few chapter of ‘Thinking Fast and Slow’, you will find that this description is incorrect. What we really try to study are the psychological rules that describe human thinking. We often illustrate these rules of thinking by showing they lead to characteristic mistakes. We do so not because we are interested in mistakes for their own sake, but because correct thinking is the default case, which requires no explanation6. This is not a universal rule, but it is often the case that one can learn a lot about a system by examining circumstances in which it fails. For example, memory is studied by looking at forgetting.
All best,
Danny
Biases are not the object of study, they are a methodology for understanding the object of study (i.e., heuristics). They are a research tool, not a diagnosis. Just as how you might study forgetting in order to understand memory, or visual illusions in order to understand perception.
But as Kahneman himself has noted elsewhere, the medium became the message. Rather than studying biases to understand heuristics, many researchers began to study biases in order to prove human were irrational. And despite himself, Kahneman seemed very interested in this work.
And so perhaps my title was click-bait after all. Despite explicitly claiming otherwise, Kahneman seemed interested in “mistakes for their own sake”, and did seem to think people were irrational.
And that is my principal disagreement with Kahneman and the entire Heuristics and Biases approach. Biases may be a good tool for studying heuristics in a lab, but I do not think they are a good tool for studying human errors in real-life situations. The principle reason for this being that despite all the hubbub around these models, they are not actually rational in the vast majority of real-world settings.
You would think there would be lots of empirical evidence about how the best decision-makers all use rational models. But yet the literature is lacking on this point, and the literature that does exist tends to show the opposite. It is not the best of the best, but the newest of the new who use rational models; novices will sometimes use rational models because they have so little knowledge of a domain. But as novices gain experience and start to understand and appreciate concrete specifics, they put away childish things and start making decisions based on a deep understanding of the context.
Similarly, you would think there would be lots of literature on how biases lead to many real-world errors. And indeed there is. But most of it is post-hoc; identifying errors and then describing the error as a bias as opposed to focusing on the underlying process which caused the mistake. Worse, some will even claim the mistakes were caused by a bias as if biases were the type of thing which have causal power.
I find this type of research extremely dangerous. Yes, heuristics can underlie various types of errors, but they also underlie expertise, adaptive cognition, and the entire learning process. Blaming the very thing which facilitates excellence is perilous and I do not think current research practices have adequately addressed this.
But maybe you are not convinced that rational models are not capital-R Rational, and the moral argument doesn’t bother you. Well, let’s get even more specific then.
No Free Lunches…well, almost none
Even by their own standards, the rational models are not always rational. For every bias, there is always an exception where the thing that has been declared “irrational” is actually optimal.
This is proven by a set of theorems called the No Free Lunch Theorems, which are simultaneously both trivial and controversial. The theorems state that no model can be optimal across all possible situations; for any model that says A is optimal, there is a possible world where B is actually optimal. Because of this, it's logically impossible for a model to be optimal in all possible worlds.
If you take the theorems seriously, the takeaway is this; you cannot always rely on the same model. There are no free lunches when it comes to rationality. You have to put in the work not only to figure out what is optimal, but to figure out which model can get you that optimal result. Why else do you think we have things like Fuzzy Probability Theory and Non-Aristotelian Logics? Because there are always exceptions where the traditional model doesn’t apply. Remember, all models are wrong! Yet somehow we are supposed to believe there is some set number of models which are the standard for rational decision-making? Do not mistake the entirely of reason with the single field of statistical inference!
Some people feel this in their bones and get anxious when rational models are naively applied. But others think to themselves, “Well maybe there are some exceptions, but by and large the traditional7 rational models will work for most situations and we can easily identify the exceptions.”
When I hear these people, I swear I can almost hear the roar of the T-Rex from Jurassic Park.
Consider two examples.
First, Confirmation Bias, one of the most popular and supposedly well replicated of the biases. Everyone knows that Confirmation Bias is irrational, right?
Well, no. When uncertainty is high, confirmation isn’t only not irrational, it can be optimal as it can be more informative than falsification. This isn’t to say Confirmation Bias is always rational. But do Behavioral Scientists really fully understand the types of situations where it is and isn’t? Should we de facto label all instances of Confirmation as bias or irrational? Had you even considered asking when it might be rational and optimal?
Another example is Expected Utility Theory: a theory so popular that some people base their entire moral philosophy on it. For fans of the theory, Jason Collins has a bet for you: “Suppose you have $100 and are offered a gamble involving a series of coin flips. For each flip, heads will increase your wealth by 50%. Tails will decrease it by 40%. Flip 100 times.”
Do the math. Expected Utility says take the bet.
But common sense says that I am likely playing a trick on you, and you should be careful.
Turns out the bet is non-ergodic, which is a term I barely know how to pronounce let alone describe, so read Collin’s full article here. But the takeaway you need to know is this; if you calculate expected utility it looks like a great bet, but you will probably wind up poorer than you started if you accept the terms.
I use non-ergodicity as an example not in spite of its mathematical complexity but because of it. Had you ever heard of the term? Do you think the average Behavioral Scientist going around declaring people biased has heard it, internalized it, and can tell when they are in a non-ergodic situation? If not, are you sure they are sophisticated enough to know when rational models should and should not be used?
A bias is, by definition, a deviation from a rational model. But for every given rational model, there are times when the bias is more optimal than the model. And frankly, I do not trust people to know when those exceptions apply.
Some people come up with work arounds to this criticism, saying that biases are only biases when they are actually irrational. If Expected Utility doesn’t give you the “correct” answer, then it is not a bias to deviate from it. I find such reasoning very circular and dangerous. The bearers of rationality are no longer unbiased equations but researchers. This leads to situations where a behavior previously considered irrational is later declared to be rational because researchers finally understand the logic of it. I find this problematic. Declaring something irrational shouldn’t have such a high correlation with the ignorance of researchers.
So specific you can only find it in a lab
But maybe there are some domains where these exceptions don’t apply? Maybe there are entire areas of problem solving where rational models are, in fact, capital-R Rational?
Well, it can’t be any quick paced domain as rational models are too slow, laborious and complex. Certainly, for the domains we care about at ShadowBox where I work the rational models just don’t cut it.
We can also ignore domains which are too simple, as such models will not be needed anyways. You don’t need Expected Utility Theory to pick out the right cereal at the grocery store. Similarly, we should ignore domains where intuitive expertise is likely to develop and perform as well as the rational models. Especially since using abstract rational models can get in the way of learning in such domains.
Ditto for domains where values are not clear, stable, and well-ordered. If you are still trying to figure out what you want, or your values are too dynamic, then rational models will just fail you entirely.
It also can’t be a domain where there is too much non-linearity or too many unknown unknowns. In domains which we call complex or chaotic, the list of rational models will lead you astray and will get in the way of the sort of probing, sensing, and acting that you need to do in order to make sense of what is happening.
So, we need a domain that is slow and complicated in a very particular way; you can’t develop expertise, relationships are linear, values are stable and well-ordered, and you have all the information but can’t quite compute it.8
That is a very narrow slice of life! In fact, the only domain I can think of that meets all those criteria is lab experiments which are purposefully built to have these criteria so that people are forced to rely on flawed heuristics. And perhaps that is the fundamental issue with the Heuristics and Biases approach! We use lab experiments to evoke biases to discover how humans reason. But real life isn’t like these experiment, and so generalizing the mistakes to other domains is extremely theoretically and ethically fraught.
Unfair? Let me relax the constraints a bit. Surely there are situations such as stock picking or internet debates where rational models are helpful?
I guess? But they are hard to identify. Even in domains where you should be using models, you typically don’t use the default rational models, but instead something much more context specific. But sure, there are Goldilocks Zone, like Tetlock style forecasting and maybe some others, where I would recommend using Bayes Theorem. My point is not that the rational models are useless. In fact, I would recommend everyone spend some time learning probability, Expected Utility, Bayes Theorem, and logic. These are really great tools for thinking clearly!
But as you learn them, recognize that these are not general purpose tools. The situations in which rational models should be used for decision-making are far fewer than many assume. Rational models are not the default right way to reason, but a very exceptional use case in extremely constrained settings. More useful for post-hoc explanations of a decision than for making a new one.
In a now famous debate between the economists Richard Thaler and Kevin Binmore, Binmore argued that intuition becomes more rational over time as people learn from their mistakes, a point I agree with. Thaler pushed back saying that might be all well and good for grocery shopping, but there isn’t enough feedback for major life decisions like marriage or retirement. Thaler claimed victory with a quip: Binmore’s highbrow theories about the rationality of intuition were only good for “buying milk.”
It is a clever come-back. But yet, I wouldn’t recommend choosing a wife or husband based on Expected Utility Theory. Certainly, I wasn’t trying to avoid biases when I decided to marry my wife. I was trying to construct the future I wanted for me and my kids. Such qualitative as opposed to quantitative approaches still strike me as vastly more capital-R Rational than the rational models that are so popular within academia. And not just for small decisions, but rather, even more so for life’s most important decisions.
Conclusion
So is there anything left to salvage of biases?
Maybe. I sometimes use the term “Natural Bias” when talking about the current craze around “all natural” products and the aversion to chemicals, GMOs, and vaccines. Such thinking seems to me systematically flawed and confused. To be opposed to chemicals is just non-sensical, and vaccines and GMOs have saved billions of lives. All hail Norman Borlaug! 500 million, but not a single one more!
But of course, from the perspective of the those skeptical of scientific consensus, I am the biased one.
And that right there is the difficulty of the whole thing. If you don’t have an objective definition of rationality that everyone can agree on, and that can be applied perfectly and without fail by researchers, then you’re not doing research, you are simply imposing your own standards and calling it rationality.9 And even when you are right, this is a dangerous thing. More rhetoric than science.
This is why I cringe when the word cognitive bias is used. Not because the term is useless, but because misuse and misunderstanding are so pervasive, and I’m not sure the concept of biases serves a useful function outside of the lab. As a cognitive scientist, I would much rather talk about the actual processes underlying cognition than trying to explain human behavior by guessing what the mythical Homo Economicus would have done in the same situation.
Whether we actually reason using heuristics, macrocognition, how decision-making actually works, the ever-expanding zoo of biases.
Issues of replication over time and between situations, generalization between people and situations, small and inconsistent effect sizes
Even advocates of biases tend to think the Wikipedia page pretty silly and think the list does the field harm.
Boo on you, Ariely!
I am still shocked Kahneman wrote this. Correct thinking is the default case and requires no explanation? What?!
Traditional as defined by whom? Perhaps part of the reason for the 200+ biases is that there is no finite set of models that are optimal across all possible decisions. I would argue that as currently theorized, there is no upper bound on the number of biases which exist.
Sometimes these are called Small Worlds.
This should not be taken as an argument against the concept of a “nudge” which is a term that is extremely broad and is not inherently about correcting biases.
What are typically called 'biases' are the natural consequences of an abductive process, that acts/decides based on plausible (heuristic) guesses about the potential consequences of action in a complex, uncertain world.
Great piece. Reminds me when Management teams use the word “culture change”. Culture is an emergent property of a complex environment. When you focus on changing culture you get lost in all sorts of mindset myths….