Psychology Experiments Are Gardens, Not Digsites

(Well, most of the time)

I. My overactive cringe factor

There is a particular type of LinkedIn post that makes me cringe. Well, actually there are many LinkedIn posts that make me cringe, but there is one in particular I would like to focus on.¹

These posts always have the following form: "Look at what this company is doing. They clearly don’t understand Behavioral Science. They’re doing X when they should be doing the exact opposite!"

The classic example, which I have seen multiple times, is a post about how Wikipedia asks for donations by emphasizing how FEW people donate. Behavioral Scientists see this request and conclude it is wrong because Social Proof clearly states that you are supposed to emphasize how MANY donate.

Wikipedia had so many people writing to them about this “mistake” that they actually released a statement saying something to the effect of, “Shut up! We tested it. This works better.”

Maybe my nerves have over-updated on this one example, but every time I see a post similar to these criticisms I wince a bit. Maybe some of them are justified, but it seems more likely the company ran a test and found the opposite of the famed effect than that the person criticizing the company has additional insight into the context. Just as in the Wikipedia example.

However, it’s not the overconfidence that bothers me so much as the entire theory of psychology underlying these posts.

And maybe I am weird in the way I think about it—I suspect this post could be quite controversial—but I don't think psychological effects are properties of the human condition that we have discovered. Rather, I think they are something we prompt through the context of the experiment. More like flowers cultivated in a garden than fossils at a digsite. That’s why we call it choice architecture and not choice archaeology, and behavioral design instead of behavioral excavation.

So here is my argument: stop thinking of psychological effects as truths we have dug-up and discovered about human nature, and instead think of them like flowers which grow in certain contexts and not others.

Once you see psychology in this way, it becomes hard to take these confident LinkedIn takes at face value, and the current ailments in psychology (the twin crises of replication and generalization) become much easier to understand. Because if this view is right, then the question isn’t whether Wikipedia violated some universal human principle, but about whether Wikipedia readers see “social proof” in the same way as the subjects in our lab. (Hint: They do not)

---

II. Gardens not digsites

Just because you use a trowel in both gardens and digsites does not mean they are the same.

At a digsite, such as some ancient Mayan ruins or a Cretaceous era fossil bed, Archeologists and Paleontologists dig to find what was already there without regard to the current conditions of dirt, sun, and moisture. In fact, the current conditions are a nuisance to be removed as quick as possible so that the ancient and immutable truths can stand on their own without being disrupted. Such ancient truths cannot be changed by context—they can only be obscured by it.

By contrast, a gardener isn’t uncovering hidden truths about a seed. The plant isn’t ever the mystery; the conditions necessary for growing the plant are the mystery. In some conditions the plant will grow tall and fast, in another short and slow, and in yet a third, it will wither and die. The question is always about how to shape the context of dirt, sun, and moisture to get the desired response.

So, by metaphor, the question is this: when we observe a psychological effect in an experiment (e.g., Loss Aversion), are we unearthing some fundamental property buried in the human mind? Or are we crafting the context in ways that get us consistent reactions?

It's too late. I have already depicted myself as the chad gardener and you as digging up zombie theories which won't die.

Let’s return to Social Proof as a well replicated example. The idea is that if you emphasize how many people do a certain behavior, the person to whom you are emphasizing this fact will be more likely to do that same behavior. This effect is so basic that every elementary-school-age-kid knows it; “everyone else is doing it.”

But the thing is, it's not universal. Sometimes people resist the crowd. Sometimes the fact that something is popular is exactly what makes it suspect. Remember, hipsters exist. And just about everyone has a little hipster in them.

If Social Proof were like a fossil buried in the ground—a fixed and immutable structure—we’d expect it to show up more reliably. But it doesn’t. It varies wildly by context, culture, subculture, framing, motivation, salience, the perceived intentionality of the message, etc. Social proof is less like a tibia and more like a tulip. It is not something we dig up and discover, but is instead something that grows in certain conditions but not in others.

Loss Aversion, Default Bias, Fresh Start Effect, Sunk Cost Fallacy…I don’t believe any of these findings are inherent to the human condition. They are what happens when someone perceives the context in a certain way. These effects can disappear or even reverse given certain conditions. Just consider the opposing nature of Primacy Bias and Recency Bias, or of Status Quo Bias and Novelty Bias. Or compare Loss Aversion to the risks we take when we gamble, or those taken by adrenaline junkies. All these effects that psychologists find and label are not laws of human nature that we have discovered. They are not the default human condition independent of any context. They’re just what crops up when we manage to craft the context in a very specific way such that people see the pay-off differently.

Of course, to argue against myself, it is true that some variation is consistent with the digsite approach to psychology. No one expects psychological laws (if such a thing were to exist) to be immutable. But how much variability does that approach allow before we should reconsider whether they are laws at all? How far can such a view be stretched before it breaks?²

When I look at experimental procedures, I see tightly controlled conditions that we designed. A garden we planted with cues and instructions. More like a stage on Broadway than a half excavated amphitheater in Greece. And it makes sense to me that we see consistency in such a situation. But just not the kind of consistency you get from a physics experiment. Instead, it’s the kind of consistency you get when you give an actor some dialogue to read and put them on that Broadway stage. I promise you they will perform! If you set up the conditions, you can control how people perceive the situation in which they find themselves and get consistent reactions. But that doesn’t make The Script Reading Effect a fundamental property of human nature.

With the replication and generalization crises as bad as they are, perhaps it is time to reconsider the idea that psychological effects are deep human truths, and instead recognize that they are artefacts of how someone sees the context. i.e., their frame. These frames can be cultivated by the experimental design, and the shared history of the participants, but they are not fundamental properties of the brain. A study doesn’t reveal “what people are like.” It only reveals what people do under the conditions we designed. As Avel Guénin–Carlut puts it: “Psychological effects should be understood as brought about by the experimental context rather than revealed properties of human cognition.”

---

III. Psychological Effects Are Frame-Dependent, Not Fundamental

Take this real-life example: a friend recently told me his child lost a helium balloon and was still feeling down even after being given a new one. The child, confused, asked, “Why am I still sad even though I have a new balloon?”

My friend passed the question on to me, adding, “Don’t just say Loss Aversion.”

I would never dream of it. I think it is silly to assume that Loss Aversion explains much if anything at all, and even sillier to assume it applies to a particular child. The child could have anthropomorphized the balloon and felt like they lost a friend. Or maybe they felt responsible and ashamed. Or maybe the child really just thinks losses are worse than gains! Each is a plausible answer that could apply to some children some of the time. But none of these frames are universal. Each depends on how the child sees the situation, not on some universal principle of human cognition.

Every American saw this film in school. None of us remember what it was about.

For example, if a parent says, “This is Bozo the Balloon, take good care of him,” or if the parent says “Don’t lose it or I’ll beat you with a belt,” the child will be upset when he loses the balloon, but for very different reasons. The presentation completely changes the perception and therefore the experience. The emotional landscape the child inhabits is shaped by so many factors that it would be a fool's errand to conclude that it is just a single effect (loss aversion) that dominates the reaction of all or even most children.

Some might argue that these two examples (the anthropomorphization and threat by the parent) are somehow not neutral cases, and that the child has been primed to see the balloon in a certain way. They might argue the context is getting in the way of understanding the child’s true reaction to a lost balloon. With the typical digsite mentality, they think that such context needs to be ‘dug away’ and controlled for as it can only obscure the deep human truths which is a child’s natural reaction to losing a balloon.

But this is my entire point. There is no default relationship with the balloon independent of the context. There’s no neutral way to relate to anything. Every relationship is a frame, and every frame is shaped by the child’s context. Whether the child treats the balloon like a friend, a responsibility, or a meaningless object will determine their emotional response, and as a result, their emotional response doesn’t reveal something universal about cognition, but rather their emotional response reveals something about the frame they’ve adopted.

There is no escaping context and so all behavior must be understood in light of that context. When studies provoke consistent behavior, it’s not because they’ve uncovered a universal law of human cognition. It’s because they’ve built a reliable garden that is able to grow certain responses with consistency. The experiment is shaping how they think, rather than revealing how they think.

---

IV. Scaling it back

Now that I have (hopefully) made my point clear, let me try and scale it back.

This doesn’t mean there are no human universals, and nothing ever which we can “dig up” about the human mind. But a capacity, such as the ability to anthropomorphize, while being something we can discover, is not the same as a behavior which is something we evoke in an experiment. What matters in most experiments isn’t whether someone can see something a certain way, but whether we can get them to see it a certain way. And that depends on our ability to design effective interventions which cultivate certain frames.

Consider an infamous failed replication. In the original study, researchers found people were more likely to buy French wine when French music played in the background. However, a follow-up study found no effect.

Would a replication renew your faith in the finding? Would a second failed replication consolidate your disbelief?

For me, it is neither. Obviously French music can influence someone if it successfully activates a patriotic, nostalgic, or cultural frame. But also just as obvious is the fact that it won’t work on everyone—they might not recognize the music, might not care, or might even resist the perceived manipulation. I don’t even need to run the experiment to know that both results could happen given the right experimental design and the right population. Either result is trivial if it is treated as a “digsite” study about human cognition.

This guy would totally buy French wine if you played the French Eurovision song.

Once you see experiments as tests of our ability to consistently cultivate certain frames, that is as gardens instead of digsites, replication and non-replication often become rather trivial. The experiments we run are tests of whether the garden was strong enough to provoke a certain frame consistently. Turns out many of them aren’t. Boohoo. So sorry for psychology’s loss, whatever it was.

Which brings us back to my original Wikipedia example; is it surprising that sometimes social proof (or any other psychological effect) works and sometimes not?

Not at all. Social Proof isn’t a psychological law we discovered, but a tool for evoking a certain type of frame. In some contexts it is reasonable to do what others are doing, and those are situations where we should expect Social Proof to work. But in other situations, such as when Wikipedia readers are grateful for an underfunded resource, then an emphasis on how few people donate can be a more effective strategy for getting them to loosen their purse strings.

I think this is why so many behavioral change interventions can feel more like common sense than a cognitive trick. More “the subjects finally did the thing because we made the thing easier to do”; and less, “we exploited a brain module which misfires under specific conditions.”

In fact, to all you Behavioral Scientists out there, I would say that if you haven’t found a way to understand the effectiveness of an intervention in a way which feels like common sense, then you probably haven’t understood your own intervention. Behavior change isn’t voodoo or subliminal messaging. Behavior change techniques should be intelligible even to someone who doesn’t have a degree because behavior change isn’t about manipulating brain modules, but about shaping how people see the context.³

Am I taking this view too far again?

Consider the following possible claims:

1. People are always loss averse.

2. People, by default, are loss averse except under extraordinary circumstances.

3. People tend to be loss averse.

4. People will be loss averse if you manage to successfully get them to frame the situation in a way that makes losses seem worse than the gains are good.

By my reckoning, claims 1 and 2 are false, and if they were true we likely wouldn't need an experiment to confirm them. Claim 3 is so vague that it is unfalsifiable unless we can somehow find a representative sample of all possible choices. And finally, claim 4 is true. Claim 4 may even sound trivially true at first glance, but I do not think it is trivial what Kahneman and Tversky found in (for example) the Asian Disease problem. Treating mathematical identical choices differently is indeed an interesting finding.

However, somehow the classic finding is almost always interpreted as being about claims 1-3. This results in people on LinkedIn saying, “McDonald’s emphasized that you can get an extra toy if you buy two Happy Meals. But they should have framed it as losing two toys if you only buy one!” Completely ignoring the the rest of the context which would make a Loss Averse framing less effective in that situation.

So, in a sense, yes. We dug and discovered Loss Aversion. But having discovered that it can be the case that people can treat mathematically equivalent results differently depending on the frame, the question has now shifted not to whether Loss Aversion replicates, or even how common it is. But it instead shifts to be about whether we can design experiments that reliably reproduce the results if we so desire.

Choice Architecture, not Choice Archeology.

Behavioral Design, not Behavioral Excavation.

Gardens, not digsites.

---

V. Let go of the bones

If I have overstated my case, it is intentional. This is at least my 7th iteration of this essay, and the only way I could make it work, it seems, was to scream into the microphone. As I have said, I do think there are things we can discover and dig up about the human mind, and I am perhaps a little nervous in taking this argument too far and saying there is no module-like entities we can discover.

But I do believe that the strongest impact on how people behave is how they perceive their situation, not the various latent constructs of tendencies and effects which we find in psychology textbooks.

Not recognizing this can be dangerous as we begin to reify experimental averages as if they has some material reality. We call the average in an experiment X-bar (x̄). But x̄ isn’t real. It is not a module in the brain. It doesn’t generalize, and even replication can be pretty shaky. A psychology based on x̄ will be perpetually in crisis. Don’t do x̄-Psychology. Don’t reify x̄ into some property of the human brain. x̄ is a result grown in a particular context, not a cognitive process you can discover.

Sometimes the job of psychologists and behavioral scientists is to excavate the human mind and find fundamental principles. But other times we’re just gardeners trying to find the right set-up to promote a certain kind of growth and reaction. Maybe part of the problem with the current psychological paradigm is that we are very bad at telling when we are doing which.

1. If this is you, don’t pay attention to my cringe. Some people will certainly disagree with my point, and my cringe reaction is overactive (I still can’t bring myself to watch The Office).

2. Paraphrasing Lisa Feldman Barrett in her paper “Context Reconsidered”.

3. This isn’t the first time I have written about this. For more, check out my article Behavioral Science as a lens to solve problems with Ellis Morlock