De Finetti’s Game: How to Quantify Belief

What do people really mean when they say they’re “sure” of something? Everyday language is terrible at describing actual levels of confidence – it lumps together different degrees of belief into vague groups which don’t always match from person to person. When one friend tells you she’s “pretty sure” we should turn left and another says he’s “fairly certain” we should turn right, it would be useful to know how confident they each are.

Sometimes it’s enough to hear your landlord say she’s pretty sure you’ll get towed from that parking space – you’d move your car. But when you’re basing an important decision on another person’s advice, it would be better describe confidence on an objective, numeric scale. It’s not necessarily easy to quantify a feeling, but there’s a method that can help.

Bruno de Finetti, a 20th-century Italian mathematician, came up with a creative idea called de Finetti’s Game to help connect the feeling of confidence to a percent (hat tip Keith Devlin in The Unfinished Game). It works like this:


Suppose you’re half a mile into a road trip when your friend tells you that he’s “pretty sure” he locked the door. Do you go back? When you ask him for a specific number, he replies breezily that he’s 95% sure. Use that number as a starting point and begin the thought experiment.

In the experiment, you show your friend a bag with 95 red and 5 blue marbles. You then offer him a choice: he can either pick a marble at random and, if it’s red, win $1 million. Or he can go back and verify that the door is locked and, if it is, get $1 million.

If your friend would choose to draw a marble from the bag, he preferred the 95% chance to win. His real confidence of locking the door must be somewhere below that. So you play another round – this time with 80 red and 20 blue marbles. If he would rather check the door this time, his confidence is higher than 80% and perhaps you try a 87/13 split next round.

And so on. You keep offering different deals in order to hone in on the level where he feels equally comfortable selecting a random marble and checking the door. That’s his real level of confidence.


The thought experiment should guide people through the tricky process of connecting their feeling of confidence to a corresponding percent. The answer will still be somewhat fuzzy – after all, we’re still relying on a feeling that one option is better than another.

It’s important to remember that the game doesn’t tell us how likely we are to BE right. It only tells us about our confidence – which can be misplaced. From cognitive dissonance to confirmation bias there are countless psychological influences messing up the calibration between our confidence level and our chance of being right. But the more we pay attention to the impact of those biases, the more we can do to compensate. It’s a good practice (though pretty rare) to stop and think, “Have I really been as accurate as I would expect, given how confident I feel?”

I love the idea of measuring people’s confidence (and not just because I can rephrase it as measuring their doubt). I just love being able to quantify things! We can quantify exactly how much a new piece of evidence is likely to affect jurors, how much a person’s suit affects their persuasive impact, or how much confidence affects our openness to new ideas.

We could even use de Finetti’s Game to watch the inner workings of our minds doing Bayesian updating. Maybe I’ll try it out on myself to see how confident I feel that the Ravens will win the Superbowl this year before and after the Week 1 game against the rival Pittsburgh Steelers. I expect that my feeling of confidence won’t shift quite in accordance with what the Bayesian analysis tells me a fully rational person would believe. It’ll be fun to see just how irrational I am!

Loaded words

It’s striking how much more unreasonable you can make someone sound, simply by quoting them in a certain tone of voice. I’ve noticed this when I’m listening to someone describe a fight or other incident in which he felt that someone was being rude to him — he’ll relate a comment that the person made (e.g, “And then she was like, ‘Sure, whatever…'”) And when he quotes the person, he uses a sarcastic or cutting tone of voice, so of course I think, “Wow, that person was being obnoxious!”

But then I wonder: Can I really be confident that he’s accurately representing the tone of her comment? It’s pretty easy for someone to, intentionally or not, distort the tone of a comment they’re recounting, while still accurately quoting the person’s official words. Especially if they’re already annoyed at that person. So to be fair, I try to replay the comment in my head in a more neutral tone to see if that makes it seem less obnoxious. (“Sure, whatever” could be said in a detached, I-don’t-have-a-strong-opinion-about-this way, just as easily as it could be said in a sarcastic or dismissive way.)

And there’s an analogy to this phenomenon in print. Journalists and bloggers can cast a totally different light on someone’s quote just through the word choice they use when they refer to the quote. Compare, for example, “Hillary objected that…” to “Hillary complained that…” The former makes her sound controlled and rational; the latter makes her sound whiny. The writer can exert an impressive amount of influence over your reaction to the quote, without ever being accused of misrepresenting what Hillary said.

One insidious example of a loaded word that I’ve found to be quite common is “admitted.” It’s loaded because it implies that whatever content follows reflects poorly on the speaker, and that he would have preferred to conceal it if he could. Take these recent examples:

“In his blog, Volokh admitted that his argument ‘was a joke’” (Wikipedia)
“Bill O’Reilly Admits That His TV Persona Is Just An Act” (Inquisitr)
“Paul Krugman admitted that the zero bound was not actually a bound” (Free Exchange)
Kennedy admitted that ‘the end result’ under his standard ‘may be the same as that suggested by the dissent…’” (Basman Rose Law)

I investigated the original quotes from the people whom these sentences allude to, and as far as I can tell, they gave no sense that they thought what they were saying was shameful or unflattering. The word “admitted” could just as easily have been replaced by something else. For example:

“In his blog, Volokh clarified that his argument ‘was a joke.’”
“Bill O’Reilly Says That His TV Persona Is Just An Act”
“Paul Krugman explained that the zero bound was not actually a bound”
“Kennedy acknowledged that ‘the end result’ under his standard ‘may be the same as that suggested by the dissent…’”

Suddenly all the speakers sound stronger and more confident, right? This isn’t to say that the word “admitted” is never appropriate. Sometimes it clearly fits, such as when someone is agreeing that some particular point does, in fact, weaken the argument she is trying to advance. But in most cases, reading that someone “admitted” some point can subtly make you think more poorly of him without reason. That’s why I advise being on the lookout for this and other loaded words, and when you notice them, try mentally replacing them with something more neutral so that you can focus on the point itself without your judgment being inadvertently skewed.

Bayesian truth serum

Here’s a sneaky trick for extracting the truth from someone even when she’s trying to conceal it from you: Rather than asking her how she thinks or behaves, ask her how she thinks other people think or behave.

MIT professor of psychology and cognitive science Drazen Pelec calls this trick “Bayesian truth serum,” according to Tyler Cowen in Discover Your Inner Economist. The logic behind it is simple: our impressions of “typical” attitudes and behavior are colored by our own attitudes and behavior. And that’s reasonable. You should count yourself as one data point in your sample of “how people think and behave.”

Your own data is likely to influence your sample more strongly than other data points, however, for two reasons. First, because it’s much more salient to you, compared to your data about other people, so you’re more likely to overweight it in your estimation. And second, through a ripple effect — people tend to cluster with other people who think and act similarly to themselves, so however your sample differs from the general population, that’s an indicator of how you yourself differ from the general population.

So, to use Cowen’s example, if you ask a man how many sexual partners he’s had, he might have a strong incentive to lie, either downplaying or exaggerating his history depending on who you are and what he wants you to think of him. But his estimate of a “typical” number will still be influenced by his own, and by that of his friends and acquaintances (who, because of the selection effect, are probably more similar to him than the general population is). “When we talk about other people,” Cowen writes, “we are often talking about ourselves, whether we know it or not.”

Asking for reassurance: a Bayesian interpretation

Bayesianism gives us a prescription for how we should update our beliefs about the world as we encounter new evidence. Roughly speaking, when you encounter new evidence (E), you should increase your confidence in a hypothesis H only if that evidence would’ve been more likely to occur in a world where H was true than in a world in which H was false — that is, if P(E|H) > P(E|not-H).

I think this is indisputably correct. What I’ve been less sure about is whether Bayesianism tends to lead to conclusions that we wouldn’t have arrived at anyway just through common sense. I mean, isn’t this how we react to evidence intuitively? Does knowing about Bayes’ rule actually improve our reasoning in everyday life?

As of yesterday, I can say: yes, it does.

I was complaining to a friend about people who ask questions like, “Do you think I’m pretty?” or “Do you really like me?” My argument was that I understood the impulse to seek reassurance if you’re feeling insecure, but I didn’t think it was useful to actually ask such a question, since the person’s just going to tell you “yes” no matter what, and you’re not going to get any new information from it. (And you’re going to make yourself look bad by asking.)

My friend made the valid point that even if everyone always responds “Yes,” some people are better at lying than others, so if the person’s reply sounds unconvincing, that’s a telltale sign that that they don’t genuinely like you/ think you’re pretty. “Okay, that’s true,” I replied. “But if they reply ‘yes’ and it sounds convincing, then you haven’t learned any new information, because you have no way of knowing whether he’s telling the truth or whether he’s just a good liar.”

But then I thought about Bayes’ rule and realized I was wrong — even a convincing-sounding “yes” gives you some new information. In this case, H = “He thinks I’m pretty” and E = “He gave a convincing-sounding ‘yes’ to my question.” And I think it’s safe to assume that it’s easier to sound convincing if you believe what you’re saying than if you don’t, which means that P(E | H) > P(E | not-H). So a proper Bayesian reasoner encountering E should increase her credence in H.

(Of course, there’s always the risk, as with Heisenberg’s Uncertainty Principle, that the process of measuring something will actually change it. So if you ask “Do you like me?” enough, the true answer might shift from “yes” to “no”…)

Does it make sense to play the lottery?

A very smart friend of mine told me yesterday that he buys a lottery ticket every week. I’ve encountered other smart people who do the same, and it always confused me. If you’re aware (as these people all are) that the odds are stacked against you, then why do you play?

I raised this question on Facebook recently and got some insightful replies. One economist pointed out that money is different from utility (i.e., happiness, or well-being), and that it’s perfectly legitimate to get disproportionately more utility out of a jackpot win than you lose from buying a ticket.

So for example, let’s say a ticket costs $1 and gives you a 1 in 10 million chance of winning $8 million. Then the expected monetary value of the ticket equals $8 million/ 10 million – $1 = negative $0.20. But what if you get 15 million units of utility (utils) from $8 million, and you only sacrifice one util from losing $1? In that case, the expected utility value of the ticket equals 15 million utils / 10 million – 1 util = 0.5 utils. Which means buying the ticket is a smart move, because it’ll increase your utility.

I’m sympathetic to that argument in theory — it’s true that we shouldn’t assume that there’s a one-to-one relationship between utility and money, and that someone could hypothetically have a utility curve that makes it a good deal for them to play the lottery. But in practice, the relationship between money and utility tends to be disproportionate in the opposite direction from the example above, in that the more dollars you have, the less utility you get out of each additional dollar. So the $8 million you would gain if you won the lottery will probably give you less than 8 million times as much utility as the $1 that you’re considering spending on the ticket. Which would make the ticket a bad purchase even in terms of expected utility, not just in terms of expected money.

Setting that theoretical argument aside, the most common actual response I get from smart people who play the lottery is that they’re buying a fantasy aid. Purchasing the ticket allows them to daydream about what it would be like to win $8 million, and the experience of daydreaming itself gives them enough utility to make up for the expected monetary loss. “Why can’t you just daydream without paying the $1?” I always ask. “Because it’s not as satisfying if I know that I have no chance of winning,” they reply.  Essentially, they don’t care how small their chance of winning is, they just need to know that they have some non-zero chance at the jackpot in order to be able to daydream about it.

I used to accept that argument. But in talking with my friend yesterday, it occurred to me that it’s not true that your chances of winning a fortune are zero without a lottery ticket. For example, you could suddenly discover that you have a long-lost wealthy aunt who died and bequeathed her mansion to you. Or you could find a diamond ring in the gutter. Or, for that matter, you could find a winning lottery ticket in the gutter. The probability of any of these events happening isn’t very high, of course, but it is non-zero.

So you simply can’t say that the lottery ticket is worth the money because it increases your chances of becoming rich from zero to non-zero. All it’s really doing is increasing your chances of becoming rich from extremely tiny to very tiny. And if all you need to enable your Scrooge McDuck daydreams is the knowledge that they have a non-zero chance of coming true, then you can keep those daydreams and your ticket money too.

How to argue on the internet

At least a dozen people have sent this XKCD cartoon to me over the years.

It’s plenty hard enough to get someone to listen to your arguments in a debate, given how attached people are naturally to their own ideas and ways of thinking. But it becomes even harder when you trigger someone’s emotional side, by making them feel like you’re attacking them and putting them automatically into “defend myself” mode (or worse, “lash out” mode), rather than “listen reasonably” mode.

Unfortunately, online debates are full of emotional tripwires, partly because tone isn’t always easy to detect in the written word, and even comments intended neutrally can come off as snide or snippy… and also because not having to say something to someone’s face seems to bring out the immature child inside grown adults.

But on the plus side, debating online at least has the benefit that you can take the time to think about your wording before you comment or email someone. Below, I walk you through my process of revising my wording to reduce the risk of making someone angry and defensive, and increase my chances that they’ll genuinely consider what I have to say.

DRAFT 1 (My first impulse is to say): “You idiot, you’re ignoring…”

Duh. Get rid of the insult.

DRAFT 2: “You’re ignoring…”

I should make it clear I’m attacking an idea, not a person.

DRAFT 3: “Your argument is ignoring…”

This can still be depersonalized. By using the word “your,” I’m encouraging the person to identify the argument with himself, which can still trigger a defensive reaction when I attack the argument. That’s the exact opposite of what I want to do.

DRAFT 4: “That argument is ignoring…”

Almost perfect. The only remaining room for improvement is the word “ignoring,” which implies an intentional disregard, and sounds like an accusation. Better to use something neutral instead:

DRAFT 5: “That argument isn’t taking into account…”

Done.  Of course, chances are I still won’t persuade them, but at least I’ve given myself the best chance possible… and done my part to help keep the Internet civilized. Or at least a tiny bit less savage! 

Thinking in greyscale

Have you ever converted an image from greyscale into black and white? Basically, your graphics program rounds all of the lighter shades of grey down to “white,” and all of the darker shades of grey up to “black.” The result is a visual mess – same rough shape as the original, but unrecognizable.

Something similar happens to our mental picture of the world whenever we talk about how we “believe” or “don’t believe” an idea. Belief isn’t binary. Or at least, it shouldn’t be. In reality, while we can be more confident in the truth of some claims than others, we can’t be absolutely certain of anything. So it’s more accurate to talk about how much we believe a claim, rather than whether or not we believe it. For example, I’m at least 99% sure that the moon landing was real. My confidence that mice have the capacity to suffer is high, but not quite as high. Maybe 85%. Ask me about a less-developed animal, like a shrimp, and my confidence would fall to near-uncertainty, around 60%.

Obviously there’s no rigorous, precise way to assign a number to how confident you are about something. But it’s still valuable to get in the habit, at least, of qualifying your statements of belief with words like “probably,” or “somewhat,” or “very.” It just helps keep you thinking in greyscale, and reminds you that different amounts of evidence should yield different degrees of belief. Why lose all that resolution unnecessarily by switching to black and white?

More importantly, the reason you shouldn’t ever have 0% or 100% confidence in any empirical claim is because that implies that there is no conceivable evidence that could ever make you change your mind. You can prove this formally with Bayes’ theorem, which is a simple rule of probability that also serves as a way of describing how an ideal reasoner would update his belief in some hypothesis “H” after encountering some evidence “E.” Bayes’ theorem can be written like this:

… in other words, it’s a rule for how to take your prior probability of a hypothesis, P[H], and update it based on new evidence [E] to get the probability of H given that evidence: P[H | E].

So what happens if you think there’s zero chance of some hypothesis H being true? Well, just plug in zero for “P[H],” all the way on the right, and you’ll realize that the entire equation becomes zero (because zero times anything is zero). So you don’t have to know any of the other terms to conclude that P[H | E] = 0. That means that if you start out with zero belief in a hypothesis, you’ll always have zero belief in that hypothesis no matter what evidence comes your way.

And what if you start out convinced, beyond a shadow of a doubt, that some hypothesis is true? That’s akin to saying that P[H] = 1. That also implies you must put zero probability on all the other possible hypotheses. So plug in 1 for P[H] and 0 for P[not H] in the equation above. With just a bit of arithmetic you’ll find that P[H | E] = 1. Which means that no matter what evidence you come across, if your belief in a hypothesis is 100% before seeing some evidence (that is, P[H] = 1) then your belief in that hypothesis will still be 100% after seeing that evidence (that is, P[H | E] = 1).

As much as I’m in favor of thinking in greyscale, however, I will admit that it can be really difficult to figure out how to feel when you haven’t committed yourself wholeheartedly to one way of viewing the world. For example, if you hear that someone has been accused of rape, your estimation of the likelihood of his guilt should be somewhere between 0 and 100%, depending on the circumstances. But we want, instinctively, to know how we should feel about the suspect. And the two possible states of the world (he’s guilty/he’s innocent) have such radically different emotional attitudes associated with them (“That monster!”/”That poor man!”). So how do you translate your estimated probability of his guilt into an emotional reaction? How should you feel about him if you’re, say, 80% confident he’s guilty and 20% confident he’s innocent? Somehow, finding a weighted average of outrage and empathy doesn’t seem like the right response — and even if it were, I have no idea what that would feel like.

%d bloggers like this: