The Failure of Generalizable Models
Why context-specific mental models outperform generalizable models in the real world
Norbert Wiener once said that the best model for a cat is another cat, preferably the same cat. After all, why consult a simulacrum when you have the real thing?
But as was recently pointed out to me, the best model for ALL cats is, in fact, not a cat. Any particular feline is far too overfit to be a stand-in for the whole genus. There are just too many dimensions on which a given cat may differ from all others, and so every cat will be an outlier in one way or another. This is as true for cats as it is for humans.
Because of this, a perfectly normal cat does not exist. It is an abstraction, a fiction.
Do not interpret me as saying this is bad. Sometimes a good fiction helps to make sense of a diverse population of instances. The fictional perfectly normal cat may be less ideal for understanding a particular cat, but it is helpful for understanding the set of all cats.
This trade-off between a highly specific model representing a small number of instances versus a more fictitious model which represents the larger set of all instances is known formally as the Bias-Variance Trade-Off. But I will give it the less complicated name of the General-vs-Specific Trade-Off: highly specific and context sensitive models do not generalize, but generalizable models are not specific and context sensitive.
The question I want to answer here is this: does the General-vs-Specific Trade-Off apply to decision-making?
I am going to claim that the answer is no. The trade-off is largely irrelevant as there is typically no need for generalizable models when making decisions in real life.
How to not make decisions just like an expert
One thing to note is that decisions do not exist in an objective sense. Reality is not a multiple-choice test. If we perceive options, it is because we have modeled the situation as one in which there are options. A decision is a fiction (like our perfectly normal cat) which we have layered on top of a messy, complex, and formless reality to give it structure and normality. Decisions are tractable and we know how to solve them, so modeling a situation as a decision makes our life easier.
The process by which we give structure to a messy, complicated, and formless reality is called sensemaking, and the subsequent structure is sometimes called a frame or mental model.1 But just because we can frame a situation as a choice between options doesn’t mean we typically will. In fact, it is rather rare and exceptional to do so. Since most of our lives happen in familiar domains, most of our “decisions” (if you can call them that) don’t require us to generate multiple options. Instead, our familiarity means there is typically one clear answer.
For example, how many decisions have you made today? Did you decide to adjust how you are sitting? To not commit an act of murder? Whether to get the milk or cereal out first? Or did all those “decisions” feel so natural that to even label them a decision misrepresents your experience?
Deep understanding of a situation results in effective frames which bypass the need to consider multiple options. The example I gave in What is Naturalistic Decision-Making is that how someone got lost implies how to search for them. So rather than considering multiple search methods, Search-and-Rescue tends to focus on how they got lost in the first place; did they wander off trail, miss their turn, or fail to enter the trail in the first place? Increased specificity in how someone got lost results in increased clarity in how to search.2 This means someone’s level of expertise is better measured by how they distinguish between situations than by their ability to weight options.
Since a good framing implies the answer, expertise must be understood not in terms of abstract models which generalize, but instead as a deep understanding of how to frame the particulars. As Dreyfus and Dreyfus argue:
We must be prepared to abandon the traditional view that runs from Plato to Piaget and Chomsky that a beginner starts with specific cases and, as he or she becomes more proficient, abstracts and interiorizes more and more sophisticated rules. It might turn out that skill acquisition moves in just the opposite direction: from abstract rules to particular cases.
Particulars, not abstractions, underlie expertise.3
This way of thinking about human cognition collapses the descriptive and the prescriptive, as well as problem solving and decision-making. And in that collapse, abstract generalizable decision models lose any advantage they might have had. Abstractions cannot be more informative than a frame which makes it obvious what to do.
And the trade-off?
But what about the General-vs-Specific Trade-Off?
It is true that expertise doesn’t generalize across domains. Years of Chess experience won’t make you a better Go player. The context-specific mental models of experts are just too domain specific and narrow for that.
But conversely, Expected Utility won’t make you a better player either. Nor is there any general model (mathematical or otherwise) which will make you a better player at both games. Your best bet is to actually learn Chess and Go.
I think what is going on here is that complex domains can be incredibly sensitive to context.4 And while a particular situation is unlikely to be an outlier on any given dimension, each situation consists of a thousand dimensions and so every situation is an outlier in one way or another. Typical situations, like typical cats, are a fiction. And as a result, abstract models which are tied to what is typical or average, will (typically and on average) fail.
Chess and Go are incredibly complex, nuanced, and context specific, more so than many assume, which makes them less amenable to generalizable abstractions than one might naively expect. But reality is obviously far more complex and fractally nuanced than either.
But there is a second and even more important reason the General-vs-Specific Trade-Off fails to hold water when it comes to human decision-making which is this; humans are not models, we are the modeler. We do not rely on any single mental model, but are dynamically sensemaking, framing, and adjusting our mental models depending on the needs, constraints, and particular context in which we find ourselves. We are not beholden to a single general model or framing, but can craft a wide variety of context-sensitive frames on the fly.
Because of this ability, we bypass the General-vs-Specific Trade-Off entirely. You simply don’t need generalizable models when you can, in the moment it is needed, construct a new frame which makes sense of what to do.5 Finding out how experts do this so consistently is (what I consider) the central question of Naturalistic Decision-Making, and is why I find expertise a more interesting topic than rationality.
How do humans find the right model in the moment?
Mathematical models do not change. Once the math is written, it is set in stone. A variable once present is always present, and an absent variable always absent. Newton’s equations didn’t change themselves when Einstein came along.
But humans are not like that. If cognition is an algorithm, it’s an extraordinarily strange one which self-organizes, adapts, and changes based on context and changing information.6 I suspect our brain works more like a cloud of starlings than like a computer. As I describe elsewhere, “to understand the collective intelligence of that moving mass, you have to understand how the birds interact with each other, with the wind, the predator birds they are avoiding, the ground, trees, etc. All the various interacting features of the system (living or not) contribute information to the entire system.” This ability requires an open ontology where anything can become relevant.
But of course, humans are even more open to novel variables than starlings. Consider how we can conceive of things which no one else has ever conceived. Our ancestors had no need to understand quarks or quasars. Yet humans can add these to our ontology and make decisions based on their existence.
Humans not only add to our ontology, but also ignore. We create what Gary Klein calls Just-In-Time (JIT) models.7 These fragmentary (non-comprehensive) models are constructed in the moment that they are needed. For example, a cardiologist working on heart structure will only consider the aspects of the heart relevant to the problem at hand. It’s not that the cardiologist has multiple mental models, but rather, their rich mental model of a heart takes multiple forms depending on which information is relevant.
This ability is why expert mental models, despite being incredibly particular, do have a type of generalizability. While skills don’t transfer between domains, they can transfer across problems within the same domain quite well because experts can adjust their mental model based on the needs of the situation.
It is this ability to dynamically frame a problem based on what is relevant in the moment which most amazes me about expertise. How do we realize that something is relevant without already knowing it is relevant? How do we identify the right frame without already knowing it?
I think the circularity required for relevance realization, insights, and expertise should shock you. It should set off red flags which say, “CIRCULAR LOGIC!” in all caps. It is so counterintuitive that relevance realization is possible that I can understand why people think abstract models which generalize should be superior, or even try to do away with models and representations all together (as ecological psychologists do). After all, how can you be sure that your context-specific model that you constructed on the spot is going to solve the novel problem you just came across? How do you know you are considering all the relevant variables?
Yet despite the circularity, realizing relevance is a central cognitive process, maybe even THE central cognitive process. I consider it the most interesting problem not just in decision-making, but all of psychology. Relevance Realization is what makes us modelers and not just models.
Of course, relevance realization sometimes fails. Novices don’t know what they don’t know, don’t notice what is missing, and can’t interpret warning signs because their JIT models are insufficient for guiding relevance realization. And in tough situations experts will struggle with this too. Overcoming the circularity is difficult.
So, what is the solution? How do you realize relevance? Or at least, how do you train yourself to be able to realize relevance?
The full circle
Well, you don’t fall back on abstractions! Instead, you get more specific and concrete. Do a deep dive into the context. Explore corners of the domain you hadn’t looked at closely before, dig into details with more granularity than you had henceforth been willing to do. Find contrasts which will throw into relief the relevant aspects you had previously glossed over.
This has become immensely clear in some re-working we are doing of Gary Klein’s framework on insights which he wrote about in his book Seeing What Others Don’t. For years Gary has been collecting stories of insights from interviews, books, and his personal life. Insights from firefighting, financial markets, science, or even fantasy baseball. The list currently stands at 138 stories, some of successful insights, and some of failures to have an insight.
While coding these insights, it occurred to me that not a single one of the insights involved going more abstract. In every case, the insight came from getting more familiar with intimate details of the problem than anyone else. For example, every person that predicted the 2007 financial crisis did so because they were deep in spreadsheets.
I think of insights like science. Specific hypotheses are falsifiable because they tell you what an anomaly or deviation from the theory is (they tell you what is relevant), whereas vague and abstract theories cannot guide your attention to any deviation. There is nothing to grab onto which can change your mental model and send you off in a new direction. It is only by getting specific than you can have a revolution in your understanding.
Similarly, you need concrete specific mental models to figure out what is wrong with them. This is why in every single case of an insight which Gary has collected over the years, deep familiarity with the topic (call it expertise) is the necessary pre-condition for the insight. A fact which has made me much more reluctant to use LLMs—I do not want to outsource the deep familiarity that is the pre-condition to having insights.8
But what about therapists who have less familiarity with a patient’s life than the patient? Is this not a situation where less familiarity is better?
I think that is the wrong way to frame the problem. Despite intimate familiarity with their own life, patients do not meet the conditions for expertise. They are like a firefighter who only ever fights the same fire, in the same house, using the same tools, day after day. They may be incredibly familiar with that fire, but they are not an expert at fighting fires. Their experience is impoverished and so they suffer a sort of relevance-blindness.
By analogy, patients who commit the same problems over and over again also have a type of relevance-blindness, whereas the therapist has seen enough to know how things could be different. The therapist has a richer mental model of cause and effect than the patient.
So, I will amend my original statement; you need to get deeply familiar with a domain to make expert decisions and have insights, but deep familiarity requires cross-contextual understanding. You cannot understand a phenomenon if you have only ever seen it in one context and have only ever interacted with it in one way. Reality is far too context sensitive for a single context to be sufficient for deep understanding and insight. Similarly, you will never learn to develop effective JIT mental models if you do not practice realizing relevance in changing circumstances.
Of course, introducing variety can be risky, especially in high stake domains like firefighting. But this is why mental simulation is so essential to expertise. Despite mental simulations being based on our own mental models, they still brings things to the foreground9 we hadn’t before considered, and so help with relevance realization. And, as I have argued elsewhere, stories are open ended models capable of going in unseen directions (quite unlike mathematical models). There is a reason mental simulation is an essential component of Recognition-Primed Decision-Making, and why scenario-based training can be so effective.
“Meow means woof in cat”
Academics are so intensely interested in trying to find the most normal and typical thing that they have three different terms for it; mean, medium, and mode. And when they do experiments, how do they determine whether their results are statistically significant? When something deviates too far from what would be considered normal.
I don’t hold this against the academic. The fiction of normal has its utility because it generalizes quite well. I quite like abstractions and generalities and depend on them when doing research.
But decision-making is not like research and so the General-vs-Specific Trade-Off is largely irrelevant. Reality is complex and context sensitive, and when making decisions it is far better to have dynamic mental models which adjust to the context than abstract models which generalize. Expertise, insights, and good decision-making come from not increasingly better abstractions, but from increasing familiarity with all the subtle details and nuances of a situation.10
When it comes to decision-making, the best questions are not about how we might be biased from rational models, but about how we are able to expertly form effective mental models in the moment they are needed.
Frames and mental models are processes, not things. But that is hard to talk about so I will treat them as nouns. And even though I typically think of all mental models as a type of frame, but not all frames as mental models, there is no universal agreement on how to use these terms.
One nuance here is that Search-And-Rescue sometimes uses Bayesian models which are abstract and generalizable. So yes, there are some use cases (such as predicting ocean currents) where abstract general models are better.
The fact that LLMs beat out GOFAI (Good Old Fashion AI) is perhaps another piece of evidence that bears this out.
Some complex systems are incredibly sensitive to initial conditions (i.e., butterfly effect), while other complex systems appear insensitive because they are extremely adaptive. However, adaptation necessitates changing as the context changes, and so also reflects extreme context sensitivity.
I am sure some ecological psychologists will disagree with how I am speaking here, but I do think framing and mental models are essential even if some behavior is non-representational. If someone wants to write a rebuttal taking an ecological view, I’d be interested in reading it.
The argument that LLMs cannot do true relevance realization (see here and here) is the most plausible argument against AGI I know of, and is also a fundamental question in the “representation wars.” Have organisms (but not LLMs) overcome the Open-Endedness Problem by avoiding language like internal notation? Could there be an unintelligible remainder which could never be either perceived or thought of, even if only in principle? Can analogy bootstrap organisms or LLMs into an open ontology? Are these questions even tractable, or is it the case that “whereof one cannot speak, thereof one must remain silent”?
An LLM helped me identify some typos and come up with a title. But everything here was written by me. I used the em-dash just to mess with you.
Check out Elspeth Kirkman’s book Decisionscape for more on Psychological Distance which she explains through the metaphor of art. The book is very aligned with my view.
Cedric Chen has a great series on mental models which is worth checking out.
Very good article. I agree with you a lot. I just think, depending on the type of decision and context, I'd put more emphasis on generating more alternatives to solve problems, using the same methodology you indicated. As you know, I work with law, where there is a lot of complexity and multiple possible solutions, in a complex mix between things that can be solved intuitively, and others that need to be studied because they are new and unknown territory. Visualizing the simulation of different alternatives and their respective pre-mortems may educate one's intuition for the decision-making, as well as may be indicators of information that needs to be studied, researched and tested before moving forward.
Great article. Do SAT's used in intell analysis fall into the abstract models? And how would dynamic models look like?