In this episode, former World Series of Poker champion and author, Annie Duke, explains how poker is a pertinent model system for decision making in the real world, a system which blends imperfect information with some unknown percentage of both luck and skill. We go through the decision-making matrix, and how we spend most of our energy focusing on just one of the four quadrants at the expense of the learning opportunities that come from the other 75% of situations. Annie also shares how this evaluation of only the bad outcomes (and our tendency to judge others more harshly than ourselves in the face of a non-status quo decision), leads individuals, leaders, and teams to avoid bad outcomes at all costs. This avoidance is at the cost of the types of decisions which lead to progress and innovation both personally, and societally, across many realms from poker to sports to business to medicine. We also dive deep into a framework for learning, and the levels of thought required to rise to the top of a given domain. Finally, we talk about something that resonated deeply with me in terms of how I think about extending healthspan, which is the concept of “backcasting”.

Subscribe on: APPLE PODCASTS | RSS | GOOGLE | OVERCAST | STITCHER

We discuss:

• Annie’s background, favorite sports teams, and Peter’s affinity for Belichick [7:30];
• Chess vs. poker: Which is a better metaphor for decision making in life (and medicine)? [12:30];
• Thinking probabilistically: Why we aren’t wired that way, and how you can improve it for better decision making [18:15];
• Variable reinforcement: The psychological draw of poker that keeps people playing [25:15];
• The role of luck and skill in poker (and other sports), and the difference between looking at the short run vs. long run [38:00];
• A brief explanation of Texas hold ‘em [47:00];
• The added complexity of reading the behavior of others players in poker [53:15];
• Why Annie likes to “quit fast”, and why poker is still popular despite the power of loss aversion [58:30];
• Limit vs. no limit poker, and how the game has changed with growing popularity [1:01:00];
• The advent of analytics to poker, and why Annie would get crushed against today’s professionals [1:10:30];
• The decision matrix, and the ‘resulting’ heuristic: The simplifier we use to judge the quality of decisions —The Pete Carroll Superbowl play call example [1:16:30];
• The personal and societal consequences of avoiding bad outcomes [1:27:00];
• Poker as a model system for life [1:37:15];
• How many leaders are making (and encouraging) status-quo decisions, and how Bill Belichick’s decision making changed after winning two Super Bowls [1:41:00];
• What did we learn about decision making from the Y2K nothingburger? And how about the D-Day invasion? [1:46:30];
• The first step to becoming a good decision maker [1:48:45];
• The difference between elite poker players and the ones who make much slower progress [1:55:30];
• Framework for learning a skill, the four levels of thought, and why we hate digging into our victories to see what happened [1:58:15];
• The capacity for self-deception, and when it is MOST important to apply four-level thinking [2:06:15];
• Soft landings: The challenge of high-level thinking where there is subtle feedback and wider skill gaps [2:16:45];
• The benefits of ‘backcasting’ (and doing pre-mortems) [2:19:30];
• Parting advice from Annie for those feeling overwhelmed (and two book recommendations) [2:28:30]; and
• More.

§

Show Notes

Annie’s background, favorite sports teams, and Peter’s affinity for Belichick [7:30]

• Annie is lives in Philadelphia, but grew up in New England

• Her favorite sports teams are Eagles Peter, a Bill Belichick fan, is still upset that the Patriots lost to the Eagles in Super Bowl LII Annie doesn’t feel bad for Peter Red Sox Players mentioned: Fred Lynn Carl Yastrzemski Carlton Fisk

• Eagles Peter, a Bill Belichick fan, is still upset that the Patriots lost to the Eagles in Super Bowl LII Annie doesn’t feel bad for Peter

• Red Sox Players mentioned: Fred Lynn Carl Yastrzemski Carlton Fisk

• Peter, a Bill Belichick fan, is still upset that the Patriots lost to the Eagles in Super Bowl LII

• Annie doesn’t feel bad for Peter

• Players mentioned: Fred Lynn Carl Yastrzemski Carlton Fisk

• Fred Lynn

• Carl Yastrzemski

• Carlton Fisk

• Celtics Players mentioned: Larry Bird Kevin McHale

• Celtics Players mentioned: Larry Bird Kevin McHale

• Players mentioned: Larry Bird Kevin McHale

• Larry Bird

• Kevin McHale

Bill Belichick

• Peter says he’s actually more of a Bill Belichick fan than a Patriots fan
• Would love to have Belichick on the podcast

• Annie says she is going to introduce Peter to Mike Lombardi as a potential podcast guest He worked with Bill Belichick and Bill Walsh He wrote Gridiron Genius

• He worked with Bill Belichick and Bill Walsh

• He wrote Gridiron Genius

“Beautiful” article about Bill Belichick : No More Questions

Chess vs. poker: Which is a better metaphor for decision making in life (and medicine)? [12:30]

Figure 1. Cover of Thinking in Bets. Image credit: amazon.com

Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts by Annie Duke

• Peter reached out to Annie Duke immediately following finishing her book, Thinking in Bets
• Peter related strongly to the idea of making decisions with incomplete information , something he does all the time in medicine

Chess vs. Poker

• In chess, you are dealing with complete knowledge
• In poker, you have incomplete information
• This makes poker a better metaphor for life
• In life, we are constantly making decisions with incomplete information

⇒ Luck

• In chess, Luck plays no factor in the outcomeIn poker
• In poker: Luck is a bit part of it, and you can only play probabilities, and you could still lose even if you made the right decision

⇒ In medicine:

“ Even if you did have complete information, if you apply a treatment in a situation and it happens to work, there’s all sorts of stuff going on, I assume, that’s relatively stochastic that we don’t really understand. Where if we were to do that same thing again, it may not work in the next situation. So we always want to think about that because that has a very big effect on our decision making. ”

Why chess just doesn’t work as an analogy to decision making in life :

“ Let’s say that we’re dealing in a world where we had perfect information. I think it’s important to think about this to understand why chess is really bad in terms of thinking about these kinds of decisions. . .

. . .For example, if I have a coin and I’ve weighed it so that I know that the weight is correct, I’ve examined it and I can see that it’s two sided, then I now have perfect information. I know exactly what I need to know about the coin to understand how often it’s going to flip heads and how often it’s going to land on tails. But what I can’t know is what it will land on the next try . So that’s that issue of luck. . .

. . .So if you flip and it lands heads, it’s reasonable for me to say ‘it could’ve landed tails’. If we did that again, that tails is totally a possibility regardless of what I happened to have seen on this particular flip. . .

. . .And then what we can do is say, okay, so even in conditions of perfect information, we don’t really know which outcome we’re going to get. We might know the frequency, how often that might occur, or how often we could expect it to occur, but we don’t know for sure what will happen this time. . .

. . .And then you can take it further and say, but ‘what if you haven’t examined the coin?’ Based on some set of outcomes, you’re now trying to derive what the coin looks like. Now that becomes even harder. That’s where you get into something that looks very little like chess. ”

Thinking probabilistically: Why we aren’t wired that way, and how you can improve it for better decision making [18:15]

People in all disciplines (business, politics, science, etc.) often see an outcome and say, “Well, that didn’t happen, therefore is couldn’t have happened.”

• In reality… that other thing could have happened And even with the same initial circumstances, it could happen in the next situation

• Annie says even the most probabilistic thinker (someone like Peter), is not thinking that way for many situations throughout a single day

• that other thing could have happened

• And even with the same initial circumstances, it could happen in the next situation

*Book rec from Annie: Superforecasting: The Art and Science of Prediction by Philip Tetlock *

Example of a game that Peter played which at McKinsey & Company to test his probabilistic thinking :

• A moderator asks you up to 20 questions
• Each question has a quantitative answer
• And your job is to try to give an answer as a range with a 95% confidence interval
• Example questions: How many votes did Abraham Lincoln receive? What is the distance between Jupiter and its nearest moon?

• Peter got 12 out of 20, a score of 60% (“ I’m so pissed about how bad I was ”)

• How many votes did Abraham Lincoln receive?

• What is the distance between Jupiter and its nearest moon?

Everyone is somewhere on the probabilistic-thinking spectrum :

• Depending on your personality and the way you were born, we are each somewhere on the distribution
• Far left thinking is not thinking in probabilities, and far right is thinking in probabilities all the time
• Our goal should be to shift over to the right , no matter who you are

“The idea is that wherever we sit on the [probabilistic] distribution, or whatever our personal distribution looks like, we just want to shift it to the right.”

Two changes we can make to improve our decision making :

1 Try to think probabilistically a little bit more

• The world is probabilistic and so the more accurately you can think about what the outcomes of your decisions might be, the better, because they’re not deterministic
• “ When you make a decision, there’s many, many things that could occur with some likelihood of each of those things occurring and understanding that particular fact is incredibly important to good decision making .”
• The more that you can shift yourself a little bit more on the side of probabilistic thinking, the better off you are, no matter where you started.

2 Catch yourself when you are thinking deterministically

• Understand that we are just wired this way (to think deterministically instead of probabilistically)
• So it’s really hard at all times to think probabilistically
• The idea is that when we do make an error to catch it more frequently, and more quickly

“When we make an error and we have too much confidence, or we’re thinking about something as deterministic when we should be thinking about it as probabilistic, for most things we don’t catch it at all. So if we can just catch it a little bit more often and if we can catch it a little bit more quickly, we’re a lot better off. These small shifts, if you can get a couple percentage points better, you’re way better off in your decision making.”

Variable reinforcement: The psychological draw of poker that keeps people playing [25:15]

When did Annie start getting interested in poker?

• Annie got interested in poker when she was in high school watching her older brother, Howard Lederer , become a professional poker player

• Howard learned the game by playing at places like the Mayfair Club in NYC, a place where a bunch of pros were born Peter makes the analogy to the pockets of specialized talent that comes out of unexpected places that Daniel Coyle writes about in Talent Code (i.e., soccer players that come out of Brazil)

• Peter makes the analogy to the pockets of specialized talent that comes out of unexpected places that Daniel Coyle writes about in Talent Code (i.e., soccer players that come out of Brazil)

“ The thing about poker that’s part of this stuff about like the hidden information and luck is that that kind of uncertainty leaves open the array of explanations for why you might not be doing well. ”

• In chess, by contrast, it becomes very clear after a few games why you are losing I.e., I’m losing because you’re better than I am. “I have nothing else to protect my ego.”
• Take any other sport or activity (like football or tennis), and there is some variable element or excuse that can be made “ The refs are screwing m e” “ The wind blew my ball ”
• But there’s an equanimity to luck… If you’re playing tennis and there’s wind, your opponent is also playing with that wind. If you’re in a game, the refs generally are making bad calls for both teams rather than trying to screw over a particular team But the problem is that we only notice is the wind blew OUR ball

• Poker is basically a fertile ground for that type of thinking The implications of this on poker is that players can convince themselves they are losing for reasons other than that they aren’t as skilled as those around them

• I.e., I’m losing because you’re better than I am.

• “I have nothing else to protect my ego.”

• “ The refs are screwing m e”

• “ The wind blew my ball ”

• If you’re playing tennis and there’s wind, your opponent is also playing with that wind.

• If you’re in a game, the refs generally are making bad calls for both teams rather than trying to screw over a particular team

• But the problem is that we only notice is the wind blew OUR ball

• The implications of this on poker is that players can convince themselves they are losing for reasons other than that they aren’t as skilled as those around them

Variable reinforcement of winning

• Annie was studying cognitive science in graduate school at U Penn
• She learned about the difference between a variable reinforcement schedule versus fixed reinforcement schedule and began to think about this in terms of poker

⇒ Annie describes it as follows :

First, let’s differentiate between a ratio or interval schedule…

Example : Put a rat in a skinner box with a level that when pushed may give them a pellet (i.e., positive reinforcement)

Ratio schedule

• A ratio schedule means the reward is determined by number (in this case number of times the rat presses the lever)
• E.g., Every fifth press the rat gets a pellet

Interval schedule

• And an interval schedule is by time
• E.g., every five minutes the rat gets a pellet

In the case that it’s a FIXED schedule…

• That means the rat gets a pellet every five presses (fixed ratio)
• Or every five minutes (fixed interval)

What happens on the fixed ratio schedule…

• The rat has this very even, controlled pressing
• I.e., press, press, press, press, food, press, press, press, press.

In the case of the fixed interval …

• The rat twiddles its thumbs until it’s almost five minutes and then it goes crazy
• So you get this burst of activity, it’s almost like they don’t want to miss a second

Now let’s change it to a VARIABLE schedule:

• With a variable schedule, you can think in averages
• So on average, every twenty lever presses you’ll get food ( variable ratio )
• Or on average, every five minutes you’ll get food ( variable interval )

The rats behavior is different than fixed in the following ways…

⇒ In the case of variable ratio …

• The rat is quickly pressing the lever over and over, because they don’t know is it the next press or not

⇒ In the case of variable interval …

• There is never a big “burst” activity after 5 minutes because rat isn’t conditioned to the five minutes thing, they will just keep on waiting and waiting for it to come

The KEY POINT is observing the behavior of the rat when you try to extinguish the behavior by withdrawing the reward

• When it’s fixed schedule… It stops doing the thing (i.e., pushing the lever) super fast I.e., “I’ve been pressing a hundred presses of this stupid lever, you haven’t to give me food, I give up. I have figured this out there’s no food coming my way.”

• When it’s a variable schedule… They basically never stop And it’s particularly bad when it’s a variable ratio schedule

• It stops doing the thing (i.e., pushing the lever) super fast

• I.e., “I’ve been pressing a hundred presses of this stupid lever, you haven’t to give me food, I give up. I have figured this out there’s no food coming my way.”

• They basically never stop

• And it’s particularly bad when it’s a variable ratio schedule

Slot machines : The perfect example of a variable ratio schedule

• With slot machines, you’ll get a reward which will come to you on average after a certain number of pulls of the lever I.e., You could get a reward five plays in a row, you could go eighty plays without really getting much in return, and you don’t know because it’s on average
• But people keep playing because their mind is telling them… “ I’m due, I’m due. It must be the next press.”

• Annie says, “ I always imagine the little rats in the cages saying, ‘I’m due.’”

• I.e., You could get a reward five plays in a row, you could go eighty plays without really getting much in return, and you don’t know because it’s on average

Poker : a more complex type of variable ratio schedule

• What makes it more complex is that it involves luck as well as skill
• Skill can help someone win in the long run but… In a particular moment you could win a lot of hands in a row In a particular moment you could not win so many hands in a row
• And this makes it VERY difficult to determine whether you are playing well or not

• And this explains why people keep on playing even when they are losing in the “long run”

• In a particular moment you could win a lot of hands in a row

• In a particular moment you could not win so many hands in a row

The role of luck and skill in poker (and other sports), and the difference between looking at the short run vs. long run [38:00]

• A slot machine is pure luck, the underlying probability is knowable
• In poker, you layer skill and luck on top of each other
• And in the long run, with enough iterations your results will be solely determined by skill

Annie’s definition of skill: Skill would be the umbrella of all the things that are within your control in the game that have an effect on the outcome.

• People mistake games that have a strong influence of luck in the SHORT RUN as not being skill game Example: In poker I could have the best hand, I could play it perfectly and I can lose to you because of the turn of a card. So, therefore isn’t it just luck?

• But Annie is making the assertion that in the LONG RUN, your results in poker are going to be s olely determined by skill

• Example: In poker I could have the best hand, I could play it perfectly and I can lose to you because of the turn of a card. So, therefore isn’t it just luck?

How can you tell if something is a game of skill?

• Try to lose on purpose
• In games that involve skill, you can’t win on purpose , but you can lose on purpose
• Example : Much easier to lose on purpose when playing someone in Tic Tac Toe
• You can only “win on purpose” when the skill gap is huge, e.g., a child vs. an NBA player in one-on-one

Luck plays a larger and larger role as the skill gap closes

• This is where the confusion comes in
• When you narrow the skill gap , luck starts to play a much bigger role
• Luck starts to show its influence and you realize that a situation could have easily turned out differently

⇒ Peter says look at Super Bowls:

• A high percentage of those games come down to the last possession
• It’s because those 2 teams are the best of the best and one “lucky” bounce can decide the game
• By contrast, if the Red Sox played a little league team, there is basically no influence of luck that is going to go the way of the little league team

“If we think about a game like poker, and a lot of these decision making processes that we do in life, the skill gap is actually quite narrow.”

A brief explanation of Texas hold ‘em [47:00]

Here’s a helpful YouTube video: How To Play Poker | Texas Holdem Poker For Beginners

Texas hold ‘em

• The goal is to make a five card hand Those five card hands range from “none of the five cards match up” in which case all you care about is the top card in the hand In most poker games aces are high (including Texas Hold Em) Those five card hands range from Very bad, meaning none of the cards match up to each other in any way To really, really good, meaning something like a… Straight (e.g., five, six, seven, eight, nine) Flush (i.e., all five of the cards are of the same suit) Full house (i.e., you have three of one rank and two of another rank, e.g., king, king, king, ten, ten)

• The goal is to make a five card hand

• Those five card hands range from “none of the five cards match up” in which case all you care about is the top card in the hand
• In most poker games aces are high (including Texas Hold Em)

• Those five card hands range from Very bad, meaning none of the cards match up to each other in any way To really, really good, meaning something like a… Straight (e.g., five, six, seven, eight, nine) Flush (i.e., all five of the cards are of the same suit) Full house (i.e., you have three of one rank and two of another rank, e.g., king, king, king, ten, ten)

• Very bad, meaning none of the cards match up to each other in any way

• To really, really good, meaning something like a… Straight (e.g., five, six, seven, eight, nine) Flush (i.e., all five of the cards are of the same suit) Full house (i.e., you have three of one rank and two of another rank, e.g., king, king, king, ten, ten)

• Straight (e.g., five, six, seven, eight, nine)

• Flush (i.e., all five of the cards are of the same suit)

• Full house (i.e., you have three of one rank and two of another rank, e.g., king, king, king, ten, ten)

• What is the highest ranking hand?

• A royal straight flush Which is ace, king, queen, jack, ten (royal straight) and all of the same suit (flush) Probability of a royal straight flush = one in 649,740 , or nearly 0.000154% of the time

• A royal straight flush Which is ace, king, queen, jack, ten (royal straight) and all of the same suit (flush) Probability of a royal straight flush = one in 649,740 , or nearly 0.000154% of the time

• A royal straight flush

• Which is ace, king, queen, jack, ten (royal straight) and all of the same suit (flush) Probability of a royal straight flush = one in 649,740 , or nearly 0.000154% of the time

• Probability of a royal straight flush = one in 649,740 , or nearly 0.000154% of the time

Texas Holdem: More about the game

• You are trying to make your best 5 card hand out of 7 cards
• You have 2 private cards and then there are 5 community cards
• So we can think about the variants of poker
• Note: there are other poker games where more (or less) cards are private
• So there amount of information about the possible hands that you and other players could have is more (or less) knowable
• Examples: In the film, The Cincinnati Kid starring Steve McQueen , in that particular game people are dealt their own five cards, and so you don’t get to use any of the five cards that I have. So, our cards are unique from each other, and you don’t get to see any of my cards. A variant of a stud games where all of your cards are “private” (meaning no community cards), but other players get to see some of your cards

• The more cards that are exposed, the more ability is given to the players to figure out what possible hand the other players could have

• In the film, The Cincinnati Kid starring Steve McQueen , in that particular game people are dealt their own five cards, and so you don’t get to use any of the five cards that I have. So, our cards are unique from each other, and you don’t get to see any of my cards.

• A variant of a stud games where all of your cards are “private” (meaning no community cards), but other players get to see some of your cards

In Texas hold ‘em…

• It’s a stud game with community cards
• Your two private cards are dealt to you face down (and you can look at them immediately but no other players will ever see them)
• Five cards eventually will end up in the middle, so that’s seven total, but those five cards that are sitting in the middle everybody gets to use
• What that means is that, these five cards in the middle define for me, what your possible holdings are.
• So that now changes for me, what I think that the strength of my hand is

⇒ Example:

• If you have two private cards, the most of a suit that I could have in my hand would be two, so I could have two hearts
• In order for me to ever make a flush, which is five of the same suit, (e.g., five hearts) I need to have three more of those out on the board
• If I look on the board, and you look on the board, and we both see that there aren’t three of a suit that’s on the board, it eliminates the possibility of a flush from the other person’s hand
• That becomes important, because if I have a hand that is vulnerable to a flush, if I have three of a kind, or two pair or something like that, that now increases how I think about the strength of my own hand

Where the skill of poker comes into play :

• The skill in poker comes from: Understanding what those community cards mean in terms of what you have AND what the other player have E.g., What the probability is that the other players cards match up in some way, to those five cards in the middle, and how it compares to the hand I have been dealt

• Understanding what those community cards mean in terms of what you have AND what the other player have

• E.g., What the probability is that the other players cards match up in some way, to those five cards in the middle, and how it compares to the hand I have been dealt

The added complexity of reading the behavior of others players in poker [53:15]

How many players at the table typically for Texas hold ‘em?

• 8-10 players

Why players add more complexity to your decision making

• Now you are not only determining your own hand
• You are figuring out the probability of other players hands
• And additionally, you are now having to read the behavior of the other players to figure out how you should play
• In very rare cases, you can know you have the best hand possible based on the community cards showing

⇒ Most of the time you are somewhere in between the worst possible hand that a person could hold, and the best possible hand that a person can hold …

• So you have to figure out where you’re sitting on that spectrum (and it’s probabilistic in nature, whether I’m likely to be beating you or not)
• But then I also have to adjust those probabilities for what you’re doing and how you’re acting

Individual differences in play and behavior:

• Two people could act in the exact same way, “but depending on my model of who they are, it could mean very different things.”
• So, I could bet strongly with a hand and if you call, that could mean that you have a particular range of hands that’s X.
• And if another person calls in the same situation, their range of hands that they might call with could be narrower or wider (e.g., they could be willing to call with either a hand that’s much worse than the bottom end of the range of hands that you or another person might be willing to call with)

Newcomers to the game

Does a newcomer to a game have an advantage or a disadvantage?

• All things being equal, against the game, he’s at a disadvantage
• Because he does not know how to range six people
• “ But me personally, I’m going to be better with somebody I’ve already played with before ”

Newcomer success

• There are many cases in poker where someone has come in and they’re doing something that’s just really unusual that is not something that would be expected (so the others are “mis ranging” the strange newcomer)
• They do really well for a certain period of time, and then the market basically figures out what they’re doing (they are usually being over-aggressive with their hands)
• Most newcomer success can be attributed to a short term burst of luck, but in some cases they’re actually doing something really unexpected, and people are actually mis-ranging them

“In poker, the first thing that you’re trying to do is figure out, in an absolute sense, what I have. But I don’t know where that sits relative to what you have. And then, once I figure that out, and that’s probabilistic in nature, I now have to figure out for you, what’s the best line of play for me.”

Why Annie likes to “quit fast”, and why poker is still popular despite the power of loss aversion [58:30]

The flow of betting in Texas hold ‘em

• You start off getting dealt two cards and there’s a round of betting
• You can choose at that point whether to bet to start in the game, or to fold
• Annie says people don’t fold enough at the start

In the book, Freakonomics , there is a section on the benefits of quitting

“Quit fast”

• Annie is a big fan of the action of “quitting fast”
• “We spend a lot of time encouraging people to stick to it. For everything that you stick to, there’s a huge opportunity cost involved” says Annie

• You shouldn’t be sticking to something that has a negative expectancy to it And not just negative money/financially, but also… Negative expected happiness Negative expected health, etc.

• And not just negative money/financially, but also…

• Negative expected happiness
• Negative expected health, etc.

“My theory is do a lot of things that are basically a free roll, where there’s very little downside to it. Go try out piano, right? It’s a little bit of your time, you can’t die doing it, just go see if you like it. But then quit fast, just quit fast and move onto something else. And then if you find out, ‘Oh I really love this, I really want to spend my time doing it.’ Now stick to it.”

In poker…

• Peter has always wondered how so many people stick to poker given the power of loss aversion
• But in poker, you actually aren’t “losing” that often because you can fold at any time and cut your losses… so it’s more like a “non-win”
• While you do lose real money and real hands, the player can avoid big losses and continue to play with a long term view

Limit vs. no limit poker, and how the game has changed with growing popularity [1:01:00]

Limit vs. no limit betting

• In limit, there is a cap on how much you can bet You bet in increments, and you can only raise in increments

• In no limit, There is no upper limit that you can bet You can go “all in” This is the super exciting poker you see on ESPN

• You bet in increments, and you can only raise in increments

• There is no upper limit that you can bet

• You can go “all in”
• This is the super exciting poker you see on ESPN

*The movie Rounders is no-limit poker ⇒ Here’s the oreo scene*

Great limit players are winning the series 100% of the time

• An excellent limit player is going to have won about 56% of the hands played
• That means the person that they’re playing against is going to win 44% of the time (which is an incredible amount of reinforcement the losing player is getting
• But with enough iterations (e.g., an 8 hour session at the table), the better player is almost assuredly going to win the series
• NFL analogy : if the Patriots beat the Eagles 56% of the time, that means that if they play enough games, the Patriots will win the series 100% of the time
• So for an excellent limit poker player, at the end of the year they will have won money

In no limit poker…

• You’re going to see huge volatility
• You have enormously asymmetric outcomes at times

How the game of poker has changed with growing popularity

Before poker became popularized on television with these big high stakes, no limit games, l imit poker was actually the more common version played at casinos

⇒ Why?

• Before it was super popular, there was a lot fewer players
• So for the better players, making sure the “not so great” player were getting that frequent enforcement was REALLY important
• By contrast, if you have an unlimited supply of new players coming into the game, you have no incentive to NOT just take all your money right away even if it means they will never want to play again
• Now that the game is so popular, it is moving more in that direction

Not a zero sum game

• But when Annie was playing, you needed to make sure the newcomers were enjoying themselves
• And so playing these limit games was the way to go in order to edge out a player over a longer period of time
• There is value other than money
• Just because a person is losing it doesn’t mean that it is an awful experience
• Just like buying a ticket to a sporting event or restaurant, you are paying for an experience
• they’d get to play with really good players, and they’d be learning the game, which is exciting. And they’d feel like, oh I’m playing at table one, at the Bellagio or Aria or whatever. And they’re winning enough to feel good, they’re probably learning and this is entertainment for them
• So it’s no longer a zero sum game when you look at what’s the scope of what people value
• Annie mentioned Chip Reese as an example of someone who people always wanted to play with even though he would always win

“If we just think about it as money, it seems like a losing proposition. But of course, that’s not all we should be really thinking about. So I think that that gets lost a little bit as the game grows and becomes really, really, really big. And so if you don’t have a particularly good time, and you’re sad, and you leave, there’s somebody else just to take your place.”

The advent of analytics to poker, and why Annie would get crushed against today’s professionals [1:10:30]

• Annie became pro in 1994
• She admits that during her career, there wasn’t nearly the mathematical data and analytics that is available for players today to study different hands and situations and optimize their strategy
• Analytics are been widely adopted in poker (like the Moneyball in baseball)
• Annie says she would get “crushed” if she played today simply because hasn’t kept up with that part of the game

⇒ Example:

“[Players] can sit there and they can just set the parameters. If I am in a particular situation where I think that I have to bluff, and if I give you this range of hands, how often do you fold? If I give you this range of hands, how often do you fold? And they can figure out exactly, at what point if I bet a particular amount, am I winning enough to that bet that I can now try to bluff in that situation… maybe I would have never thought in a million years that was the situation that I would be willing to, if I were to bluff there, because I just didn’t have the data.”

Annie loved the learning part of the game

• Annie says poker always remained fun for her simply because there was always room to improve
• Like an onion, each time you peel back a layer you see how much more you don’t know about the game
• Quote Peter uses to describe this situation: “the further you get from shore, the deeper the water”

“I’m immersed. It’s like I’m living and breathing and talking poker all the time, with peers, with mentors, so on so forth. So it’s not really like a job in that sense. Certainly not during that first eight or 10 years that I’m playing, I’m learning. And every second is learning.”

The decision matrix, and the ‘resulting’ heuristic: The simplifier we use to judge the quality of decisions —The Pete Carroll Superbowl play call example [1:16:30]

Two types of poker players

1. Players that think that when they win, it’s because of how well they’ve done things. And when they lose, it’s because of bad luck.
2. Players who realize that sometimes I’m winning when I make the wrong bet. And sometimes I’m losing when I make the right bet.

Decision making matrix

4 quadrants

1. Right decision, good outcome
2. Wrong decision, good outcome
3. Right decision, bad outcome
4. Wrong decision, bad outcome

Figure 2. The decision matrix.

Which one of these quadrants is the worst one to be in?

• The common thought is that the hardest one to accept is the good decision, bad outcome

• But Annie says it’s the reverse… the worst situation is when you make a bad decision and you get a good outcome Why? … Because we don’t end up digging into our decision since the outcome was good we just leave it alone (more to come on this point later)

• Why? … Because we don’t end up digging into our decision since the outcome was good we just leave it alone (more to come on this point later)

The “resulting” heuristic

• When we’re evaluating other people’s decisions, particularly people that we don’t compare to, particularly in a situation where the decision is complicated, we have the simplifier, the shortcut, called resulting .
• In other words, we say, “if I know what the quality of the outcome is, I can work backwards from that to the quality of the decision .”
• But understanding the quality of the decision is very complex and multifactorial

Example situation : Pete Carroll made a controversial call in Super Bowl XLIX where instead of making the expected play call of handing off the ball to Marshawn Lynch at the goalline, he elected to make the unexpected play call of a pass, and the pass was intercepted and the Seahawks lost the Super Bowl to the Patriots.

• People were quite critical of this play call
• But to evaluate the decision, you have to know a lot of things What were the possible outcomes? How often would those outcomes occur? What are the consequences of each of those outcomes? How much do I prefer each of those outcomes, so I can understand what the probability of getting a preferred outcome is? If I have an outcome that I don’t prefer, like an interception, how often does that happen?
• Then you have to perform that exact thought process with all the other possibilities/the other decisions that you could have made.
• With the goal of figuring out: How do I get the highest expected value? The highest chance of actually winning the Super Bowl?
• There’s other factors such as the remaining time on the clock/time management How many timeouts are left? How do I get three plays off, instead of two? If you run the ball you may only get 2 plays If you pass you can get 3 but what is the interception rate?

• And all these things are almost impossibl e to know off the top of your head

• What were the possible outcomes?

• How often would those outcomes occur?
• What are the consequences of each of those outcomes?
• How much do I prefer each of those outcomes, so I can understand what the probability of getting a preferred outcome is?

• If I have an outcome that I don’t prefer, like an interception, how often does that happen?

• How many timeouts are left?

• How do I get three plays off, instead of two?
• If you run the ball you may only get 2 plays
• If you pass you can get 3 but what is the interception rate?

⇒ So what we tend to do in those situations is we say…

• That’s all really hard and I don’t know what any of that is, and it’s all very opaque to me
• So if I understand what the quality of the outcome is (was it good or bad), then I know what the quality of the decision is
• And this is what’s happening in that Pete Carroll thing, that’s called resulting

When is using “resulting” useful and when is it not?

• It works great for chess : If I lose to you in chess, I played worse than you
• And works very poorly in poker : If I lose to you in poker, who knows?
• And certainly very poorly in football : We don’t really know, depending on how the Super Bowl play worked out, whether it was a good or bad decision

When people tend to use the resulting heuristic most often

• People don’t do this resulting thing in situations where they feel like the quality of the decision is known

⇒ Example: Your friend tells you they got in a car accident

• Do you know if it’s my fault or not ? … You have no clue.

• But you won’t use the fact that your friend got in a car accident to work backwards in the same way we did with Pete Carroll Why? ⇒ Because you understand that there are certain things that really are knowable about driving I.e., Did you break the rules of the road? Were you drinking?

• Why? ⇒ Because you understand that there are certain things that really are knowable about driving I.e., Did you break the rules of the road? Were you drinking?

• I.e., Did you break the rules of the road? Were you drinking?

Consequences of feeling like you know the quality of the decision :

• When you think you know the decision was good, you actually tolerate a bad outcome much better

• In business, for example: The CEO executes on a decision that was made in a committee where the whole team agreed that this was the right way to go And then it doesn’t work out… people aren’t like, “idiot” But if the CEO does something that is not something that everybody always does, then all of a sudden, if it doesn’t work out, are really under the gun This is an issue of transparency

• The CEO executes on a decision that was made in a committee where the whole team agreed that this was the right way to go

• And then it doesn’t work out… people aren’t like, “idiot”
• But if the CEO does something that is not something that everybody always does, then all of a sudden, if it doesn’t work out, are really under the gun
• This is an issue of transparency

The transparency problem

⇒ Example of the transparency problem: You’re driving to the airport with your spouse

• Scenario A : You go the normal way that you’ve always gone, and there’s a big accident on the road which causes traffic and you miss your flight. Are you getting blamed for that? Is that your fault? Is your spouse mad at you? The answer is no.
• Scenario B : You tell your spouse you know a quicker route to the airport, but then it turns out there is an accident on the alternate route and you miss your flight. Now is your spouse upset with you? Absolutely.

What is the difference?

• It’s that scenario A is the ‘expected’ decision (meaning to take the normal route) and scenario B is the ‘unexpected’ decision
• In the Pete Carroll situation, if he had handed the ball the Lynch and he had fumbled, nobody would be blaming Pete Carroll since he made the “expected” decision

Expected versus not expected.

• It comes up in personal life all the time For example, people staying put in a job they don’t like… If someone’s really unhappy in their job, they won’t be aggressive enough about going and finding another position And the reason is that the unhappiness in their job has become expected and they’re not really themselves for that But if they go and find a new position, and they’re not happy there, they’re going to feel like it’s their fault

• This happens in business strategy This is the way that teams end up operating So this becomes a really big problem

• For example, people staying put in a job they don’t like… If someone’s really unhappy in their job, they won’t be aggressive enough about going and finding another position And the reason is that the unhappiness in their job has become expected and they’re not really themselves for that But if they go and find a new position, and they’re not happy there, they’re going to feel like it’s their fault

• If someone’s really unhappy in their job, they won’t be aggressive enough about going and finding another position

• And the reason is that the unhappiness in their job has become expected and they’re not really themselves for that

• But if they go and find a new position, and they’re not happy there, they’re going to feel like it’s their fault

• This is the way that teams end up operating

• So this becomes a really big problem

Transparent and opaque decisions

What decisions do you think are the most transparent to you?

• “Your own.”

• What happens with our own decisions, is that we allow uncertainty as the explanation for a bad outcome more often than we would for somebody else’s decisions Why? 1) Because we feel like we know what the quality of our decision is 2) We don’t really want to question the quality of our decision

• Why? 1) Because we feel like we know what the quality of our decision is 2) We don’t really want to question the quality of our decision

• 1) Because we feel like we know what the quality of our decision is

• 2) We don’t really want to question the quality of our decision

When are decisions the most “opaque”?

• When it’s someone else making the decision
• Especially when that decision is innovative or unexpected
• In the case of a bad outcome we don’t allow uncertainty to explain the bad outcome

The personal and societal consequences of avoiding bad outcomes [1:27:00]

There are two “discordant” quadrants:

1. Wrong decision, good outcome
2. Right decision, bad outcome

Figure 3. Discordant quadrants of the decision matrix (in yellow).

Thought experiment : In which situation are you more likely to dig into it more?

• Most people are more eager to dig into the “ right decision, bad outcome ” rather than the other situation where the outcome was good… people tell themselves it was a good outcome so they assume it was the right decision
• Annie says she thinks we learn a lot from all four quadrants
• However, she says that we don’t pay enough attention to the “ bad decision, good outcome ” quadrant so we are missing a big chunk of the lessons we could learn to improve
• This is actually a really big problem

⇒ Morbidity and mortality conference :

• In med school, Peter said there is a conference where everyone comes together to discuss every single situation in which a patient died
• Annie says this is a great example of a situation where you are only evaluating the bad outcome… what about the “good outcomes” that happened despite bad decision?
• You’re only “getting into the room” where there’s a bad outcome

See Zdogg episode of The Drive for more detail about the morbidity and mortality conference

Why is this such a big problem ?

• You have two types of decisions : 1) Status quo/expected decision 2) Non-status quo/unexpected decision
• W hat happens if you make the expected decision, a consensus decision, and you have a good outcome ? Nobody is hailing you a genius but they say “good job”
• W hat happens if you make a status quo decision that has a bad outcome ? Nobody is excoriating you, they are saying “tough luck”
• Now, what happens if you make an unexpected, an opaque, a non status quo, a non consensus decision that has a good outcome ? They’re calling you the biggest genius of all time

• But what happens if the unexpected decision ends up in a bad outcome ? Everybody is calling you an idiot

• 1) Status quo/expected decision

• 2) Non-status quo/unexpected decision

• Nobody is hailing you a genius but they say “good job”

• Nobody is excoriating you, they are saying “tough luck”

• They’re calling you the biggest genius of all time

• Everybody is calling you an idiot

Figure 4. Expected/unexpected decision matrix and the dreaded “idiot quadrant”

What does that do to somebody’s decision making?

• First, they want to do everything they can to avoid the bad outcome (i.e., loss aversion) We never want to be the “idiot”

• Secondly, we may want to avoid a bad outcome all together That leads people to non-action Or very, very low volatility decisions that won’t move the needle

• We never want to be the “idiot”

• That leads people to non-action

• Or very, very low volatility decisions that won’t move the needle

Avoiding “idiot” status quashes innovation

“We want people to be thinking, ‘I just want to make the best decisions.’ And I don’t really want them to be afraid of the bad outcomes. I particularly don’t want people to be afraid of this ‘idiot quadrant’, because we would like people to be innovative and … to push against what the status quo is, because that’s how we move forward as a society, as a business, as an individual.”

⇒ Medicine example : It doesn’t matter if 80% of ear infections are viral, if a patient comes into my office, I’m giving them antibiotics so that I won’t get yelled at, if this happens to be the time, and I want to be able to cover my butt

⇒ Business example : Imagine that we’re investing in real estate as a real estate investing group.

• We have a model of the market. We have limited resources that we can deploy. We end up investing in a particular project, and it ends up doing 15% worse than we expected.
• The result is we’re having a long meeting in a room trying to figure out WHY the project did 15% worse than we expected.
• In the opposite scenario, the investment does 15% better than expected, is the same conversation happening? … No.
• The problem is that it matters just as much as if you overestimated the market
• Because your goal is to efficiently allocate your resources within the market, and if you’re wrong in either direction, underestimating or overestimating, that’s a problem for efficient allocation of resources
• Not only that, but when it’s 15% better than you expected, it’s a signal that there may be risks that you didn’t identify .
• So for example, you could have the mean/average correctly identified, but you may not have the variant accurately pegged.
• Peter chimes in… people are very fixated on return on capital , but they generally aren’t as appreciative of risk adjusted return on capital
• The point here is that we want to dig in either way because something unexpected happened

Poker as a model system for life [1:37:15]

The importance of asking, “ Should I have done worse? ”

• When you have a situation when you underperform, people using ask, “could we have done better?”, but they rarely ask, “could we have done worse?”
• This line of questioning is reinforcing for people to only care when things go badly ⇒ thererore, when I want to AVOID things going badly at all costs

⇒ Poker is a great example

• “Where should I have done worse?” is an important question to ask, regardless of whether I won or lost a hand
• For example , let’s say that I estimate that your hand is in a range that is very weak, and I feel that I’m very strong in comparison to you
• So because you’re weak, I would be slow playing (i.e., not betting big) because I’m trying to draw as much money out from you as possible
• Now, at the end of the hand, you hit a “lucky” card and I ended up losing the hand
• But when we turn the hands over, it turns out that you had a much stronger set of cards than I thought
• In this case, I should have lost more because you hit a lucky card, HOWEVER, I should have been extracting much more money from you because you would have tolerated given the strength of your hand
• I need to explore that after the fact because now what I know is that if I knew what your cards were I would have lost a lot more money than I did
• In short, I actually played the hand very poorly given what your hand actually was… in other words, I actually underplayed my hand

Thinking in Bets isn’t a book about poker: It’s a book that uses poker as a model system

⇒Think about biology: Why do we do so much biology in C. elegans (worms), or mice, or yeast?

• Do we care that much about mice and yeast and worms? No.
• They are model systems that allow us to do things in cleaner ways
• This is the beauty of poker
• It’s not often in life we will get that much clarity after the fact, but we can still bring that level of critical thought to our decisions

“If you remember nothing else from this podcast, this is the part I’d want someone to remember: I think of poker as the C. elegans of decision making in life .” —Peter Attia

Should you have won more (or less)?

• If you win a hand in poker, you should ask yourself: Should I have won more money or less money had I played the hand perfectly using perfect information?

⇒ For example:

• I could misread your hand and I could play a hand really strongly and then I can be the one who hits the lucky card.
• But actually, if I had perfect information, I would not have ever put that much money at risk.
• So we want to explore whether the outcome is good or bad.

“We want to have equal enmity toward the direction that we’re exploring.”

Thinking orthogonally

• We need to also think orthogonally
• Such as, maybe you lost for reasons that had nothing to do with your decision process
• Or maybe you won for reasons that you totally didn’t predict

⇒ Example: Financial markets/investing models

• Often times people will invest in say a company because they think that a particular product of theirs is going to do really well, but then that product fails… But the company happened to acquire another company and stock rose Or they happen to have a totally different product do really well that made up for it

• If you happen to have won, don’t take credit for it if it wasn’t in your model

• But the company happened to acquire another company and stock rose

• Or they happen to have a totally different product do really well that made up for it

“So we want to be thinking up, down, orthogonal in all these directions.”

How many leaders are making (and encouraging) status-quo decisions, and how Bill Belichick’s decision making changed after winning two Super Bowls [1:41:00]

• They are more likely to make status quo decisions
• Simultaneously they are trying to cover their butt in the case of a bad outcome They want to be able to say, but it’s not my fault, it was bad luck “ my decision process was good and everybody had consensus and I handed the ball off to Marshawn Lynch. What could I do? That was what I was supposed to do. ”
• So we’re driving people into this place where they’re very likely to be trying to build consensus, not necessarily of the good kind… the false kind

• They may be using data not to find the truth , but to tell a data story that supports them in case they happen to get into the room.

• They want to be able to say, but it’s not my fault, it was bad luck

• “ my decision process was good and everybody had consensus and I handed the ball off to Marshawn Lynch. What could I do? That was what I was supposed to do. ”

“What really ends up coming out of it is that people tend to move slowly. They tend to default toward omissions versus commissions. In other words, not deciding versus deciding, and they tend not to like to innovate too much.”

Do the best people in their craft transcend this aversion to being the “idiot”?

• Peter has a hard time believing that Pete Carroll was making a decision based on wanting to make sure the decision was defensible at the press conference
• Annie agrees, but that’s most likely because he was an established, trusted person at the very top of his field (even among head coaches)

Why the NFL coaches/teams have been slow to adopt analytics:

• The slowness to adopt analytics in the NFL “has been remarkable”, says Annie
• The people who are judging your decisions are owners, the general managers, and the fans
• And they are mainly judging you for the unusual decisions

How Bill Belichick’s decision making evolved

• Annie saw a great talk by Toby Moskowitz in which he talked about Belichick
• When Belichick was with the Browns, his decision making was “dismal compared to the analytics”
• And it’s not actually until he wins two Super Bowls that you start to see his decision making starting to align with the analytics
• Why? Because the fans/GMs/owners are willing to tolerate the stuff that they don’t understand

So what are we encouraging with things like the morbidity and mortality conference (to use Peter’s previous med school example)?

• You’re encouraging the behavior which is focused on trying to stay out of the lower right quadrant (idiot quadrant)
• In other words, “I don’t want anything bad to happen to me, but if it does happen to me, I’m going to make sure there’s so much CYA there that it’s going to be okay that nobody’s going to sit there and say I made a crazy decision.”

⇒ And the reason we are encouraging that is because:

• We’re not exploring both directions (good outcomes and bad outcomes)
• But particularly because we’re not getting in the room when something unexpectedly good happens

“When something unexpectedly good happens, there’s so much to learn from that.”

What did we learn about decision making from the Y2K nothingburger? And how about the D-Day invasion? [1:46:30]

Case study: The Y2K problem

Turned out to be a giant “nothingburger”… but why?

⇒ Possible reasons:

1. You wasted a bunch of time and human productivity on an irrelevant problem (bad decision, good outcome)
2. Maybe nothing happened because of the work you did (Good decision, good outcome)
3. “It could be, it doesn’t really matter because when you sort of look at what the range of possibilities were given the unknown information, the downside was so bad that you were actually supposed to protect against that regardless. And where are you going to waste a whole bunch of human productivity doing it? Well, it’s not really a waste because it just acts as a hedge against, just in case it’s the worst case scenario. And in particularly with something like that, we would want to protect ourselves against that. But unless you dig into the win you don’t find that stuff out, and I think that where the problem is .”

Case study: Eisenhower ’s decision to proceed with the D-Day invasion

“A great model for how do we think about our own outcomes in our own lives .”

• In life, when it’s a bad outcome , we are excited to go in and discover that I didn’t actually do anything wrong
• When it’s a good outcome , it’s much more painful to figure out what we did wrong and potentially find out that we simply just got lucky
• With D-Day, when we land on the beaches of Normandy and it’s a pivot point in the war, and we win, people aren’t looking as hard to try to figure out what we did wrong, to really explore the counterfactuals

The first step to becoming a good decision maker [1:48:45]

Annie says that one of the first steps to being a really good decision maker is exploring the good outcomes in order to:

• Figure out how much luck played a role; and
• Figure out if you could have done better (or worse)

“We have this asymmetry in how willing we are to explore these outcomes. And it’s what you just said, we’re all living our lives having a morbidity conference. Now, the morbidity conference that we’re having in our head is specifically to go find the luck in the whole thing. But that’s the only conference we’re having. We’re never having the, ‘My life is great conference. Let me go figure out if that’s because of my own decision making or because of luck or if there was a better way, or I could have actually opened up a whole other set of outcomes’… nobody is doing that.”

The following are the consequences of only digging into the bad outcomes :

1 We are losing half of our opportunities to learn (or more)

2 We are reinforcing the risk aversion

• We’re saying, hey, what we really care about is downside outcome. So you better avoid those.

3 We are completely quashing the likelihood of anybody ever doing anything unexpected

• Because if you do something unexpected, the chances that somebody’s going to allow for luck, or that you yourself are going to allow for luck, as the explanation is just going to “go poof”

Steps to digging into good (and bad) outcomes :

• First, we observe the outcome

• Then, before I make any conclusions about it , let me think about what are the other possible things that could have occurred (i.e., the counterfactuals): What were the other possibilities? What were my preferences for those possibilities? What was the probability of each of those possibilities occurring?

• What were the other possibilities?

• What were my preferences for those possibilities?
• What was the probability of each of those possibilities occurring?

⇒ Example: Peter’s archery hobby

• Peter points out how he spends almost zero time evaluating the situation when he hits a bullseye
• He spends nearly 100% of his effort thinking about the 10% of the time when he “misses”
• Annie’s advice: Start forecasting your results I.e., At a particular range given what I think that my skill level is… What percentage of the shots do I think I’m going to get a bullseye? On a particular day, whether you are well below… or well above… your expected percentage, you should dig into the reason why
• Peter realizes that his skill improvement is being hampered by the fact that he isn’t digging into the good days

• In other words, by not fully understanding why he did better on a particular day, he isn’t able to implement it and make it part of his permanent approach

• I.e., At a particular range given what I think that my skill level is… What percentage of the shots do I think I’m going to get a bullseye?

• On a particular day, whether you are well below… or well above… your expected percentage, you should dig into the reason why

The difference between elite poker players and the ones who make much slower progress [1:55:30]

Annie uses Erik Seidel as the example :

• When you hear him talk, you never know whether they won or lost the hand

• The reason ? ⇒ He’s constantly exploring all the counterfactuals: Did I have the person’s hand right? What if I had bet a little bit more? What if I had bet a little bit less? What if I hadn’t played the hand at all? What would have happened if I played this hand that I didn’t play?

• Did I have the person’s hand right?

• What if I had bet a little bit more?
• What if I had bet a little bit less?
• What if I hadn’t played the hand at all?
• What would have happened if I played this hand that I didn’t play?

⇒ What you hear from players who aren’t making that kind of progress ?

• 1) With the hands they won, it’s just sort of like “I was great”… and they don’t really spend a lot of time on it
• 2) With the hands they lost, it’s a lot of exploration of the luck element (what bad luck caused me to lose)
• The problem is that, in poker, the luck element in a given instance is so prominent that it’s so easy to focus on the bad card that came
• So rather than focusing on what they could have done better (such as betting more to make the person fold), they dwell on that unlucky card that flipped over on the river

Framework for learning a skill, the four levels of thought, and why we hate digging into our victories to see what happened [1:58:15]

Peter’s framework for learning a new skill:

As we learn, we advance through 4 stages…

1 Unconsciously incompetent ⇒ 2 Consciously incompetent ⇒ 3 Consciously competent ⇒ 4 Unconsciously competent

Figure 5. The four stages of learning.

⇒ Example : Peter learning to swim at age 31

• Level 1: Peter didn’t know how to swim, and he couldn’t tell you why he couldn’t swim (unconsciously incompetent)
• Level 2: The first step of learning how to swim is understanding why you are really bad at this (consciously incompetent) E.g., learning about Archimedes’ principle and about buoyancy NOTE: You’re no better as a swimmer in level 1 vs. level 2, you still can’t swim

• Level 3: You don’t actually start swimming until you reach level 3: consciously competent But at this point you tend to vacillate a lot between level 2 and level 3 (you go from consciously incompetent to consciously competent and back to being consciously incompetent) In swimming, Peter found physical fatigue to be the thing that will push you from level 3 back to level 2

• E.g., learning about Archimedes’ principle and about buoyancy

• NOTE: You’re no better as a swimmer in level 1 vs. level 2, you still can’t swim

• But at this point you tend to vacillate a lot between level 2 and level 3 (you go from consciously incompetent to consciously competent and back to being consciously incompetent)

• In swimming, Peter found physical fatigue to be the thing that will push you from level 3 back to level 2

Figure 6. Vacillation between level 2 and level 3 is common among people trying to advance their skill in a given domain.

• Level 4: Unconsciously competent, this is reserved for experts like Michael Phelps in the swimming example “ You transcend to this fourth box. You are now unconsciously competent. Everything becomes sort of autonomic at that level. ”

• “ You transcend to this fourth box. You are now unconsciously competent. Everything becomes sort of autonomic at that level. ”

The four levels of thought (using poker as the example):

Level 1 : I mostly think about the discordance on the loss side (good decision, bad outcome i.e., bad luck!) and on the concordance on the win side (good decision, good outcome) and I just kind of do that naturally and I don’t spend a lot of time thinking about either of them.

Level 2 : Still mostly thinking about the concordance on the win side (meaning you’re not examining those very much) … but now you start to examine the concordant losses (bad decision, bad outcome) so you examine the losses that are actually due to your own undoing)

Level 3 : You start looking in all four quadrants

• Looking at wins that are due to skill that are due to my good decision making
• Looking at wins that are despite my bad decision making
• Looking at losses that are due to my bad decision making
• And looking at losses that are despite my good decision making
• Starting to explore all those boxes equally

Level 4 : When it’s concordant on the win side (good decision, good outcome) but you don’t accept it

• In other words that you say, I made good decisions and I won, but it does not mean there wasn’t a better way.
• Digging to make sure I really came up with the best possible solution that I could have made given what I knew
• That I won’t be satisfied with just finding out that there was concordance…I want to go beyond that on the win side.
• Peter Attia: “ I promise you I’ve never done what you just said. I’ve never once had a level four thought in my life. ”

Why do we HATE to dig into our wins ?

• Our beliefs are the threads that weave the fabric of our identity
• By doing a level 4 thought, you risk tearing a hole in that fabric
• We want to do is feel like that fabric is strong, kike our beliefs are good and we’re competent

• We need to feel competent:

• This idea of competence and being able to bank credit for the way that things have turned out in our life is so core to what forms or identity When you actually dig around in there, you’re risking now turning something that was a win into a loss

• This idea of competence and being able to bank credit for the way that things have turned out in our life is so core to what forms or identity

• When you actually dig around in there, you’re risking now turning something that was a win into a loss

“When you get to level four [thought], what you say is, ‘okay, I had a good outcome. I looked and I saw that my decision making was pretty good, but now I’m willing to turn that win into a loss in order to get a long run win there.’”

The capacity for self-deception, and when it is MOST important to apply four-level thinking [2:06:15]

• Peter says, in poker and a business like investing, it seems like “ you can’t really hide from your decisions ”
• Since your decisions lead to a traceable outcome, your decisions are wed to the outcome and you are exposed
• Annie rebuts: “ I agree in theory that there’s nowhere to hide. But the interesting thing is how long people CAN hide from it . ”

Self-deception

• Let’s say a poker player at the end of the year has lost money overall
• The player may then ask, Why did I lose?
• That person has an incredible ability to chalk it up to bad luck

⇒ This is ESPECIALLY true if you started out doing really well

• Like if a poker player comes out of the gate in the first 6 months and loses everything in sight, they’re very unlikely to stick around because they’ll say, “Oh, I must be really bad at this.”
• On the contrary, if you come out of the gate and you do really well , your ability to fool yourself afterwards for a very long time “ becomes almost infinite. ”

• Business example: You see this in hedge fund managers, those that do great from the start was more willing to chalk their failures in the subsequent years to luck (and their investors are more tolerant as well) Annie refers to this phenomenon as path dependence

• Annie refers to this phenomenon as path dependence

“When you come out of the gate strong, you can fool yourself for a really, really, really, really long time. That’s the thing that I find so fascinating. . . Trying to figure out the capacity for self-deception is a bottomless well.”

When it’s MOST important to apply this kind of four-level thinking

What conditions are most amenable to either i) self-deception and/or abject failure or ii) critical thinking/four level thinking that leads to success?

1. 1 Luck plays a factor : In an environment where there’s an influence of luck The opposite example is chess, if you lose to someone in chess, it’s nearly impossible to blame bad luck Also, few people are fooling themselves that bad luck is the reason why they are not an NFL linebacker
2. Quick feedback loop : Now, the tighter the feedback loop, the more feedback you’re getting, the more that somebody needs to rise to the occasion

3. Skin in the game : People have their own money or fortunes on the line

4. The opposite example is chess, if you lose to someone in chess, it’s nearly impossible to blame bad luck

5. Also, few people are fooling themselves that bad luck is the reason why they are not an NFL linebacker

The takeaway : Those are three conditions tend to push people towards either i) abject failure or ii) getting batter via moving up the ladder of the 4 levels of thinking. The individuals that are going to thrive more in those systems are definitely the ones who are much more willing to dig in and dig around.

What about when the feedback loop is long and less direct?

• In absence of the world telling you quickly, this type of critical thinking becomes more difficult and actually becomes much more important
• When you spread the feedback loops out, you lose the direct “decision leads to outcome” feedback

• The longer the time horizon. . .the more leeway the world is giving you to fool yourself Why? ⇒ Because it allows for uncertainty to come in

• Why? ⇒ Because it allows for uncertainty to come in

“Uncertainty gives you the leeway to allow bias to come in. That’s the difference between chess and poker. In chess, because you’ve taken a lot of uncertainty away, you don’t have the leeway to fool yourself.”

How to challenge your own thinking in a long feedback horizon situation :

Given the ease with which to fool yourself, it becomes that much more important to be thinking counterfactually and digging in to every scenario

It becomes MORE important to:

• Have people challenging your ideas
• Think about whether your decision making is status quo to protect yourself from blame if it doesn’t go well? (i.e., am I innovating enough?)
• Am I looking at that discordant box that has to do with good outcome, or am I only willing to look at the discord when it’s a bad outcome?

⇒ In a leadership role:

• How am I propagating, or am I not propagating, these problems across the team.?
• Am I telling the team through my actions that they should be terrified of bad results ?
• Is the longer feedback back loop that’s giving me leeway actually allowing for my beliefs to lead me around on a leash?
• Am I infecting my team with that same problem?

“You should be digging into [your decisions] as if you’re getting the answer tomorrow, as if you’re playing chess and you’re going to find out right away. You should be doing everything you can to make sure that you don’t find out that you hung your rook.”

Soft landings: The challenge of high-level thinking where there is subtle feedback and wider skill gaps [2:16:45]

Two conditions which make it harder, therefore more important, to focus on four-level thinking:

1. When the feedback is subtle
2. When the skill gap is wide

1 Subtle feedback vs. obvious feedback : When the feedback is more subtle and harder to discern, it makes it more important to apply this type of thinking

⇒ Example: Swimming vs. riding a bike

-Swimming is harder than riding a bike, says Peter

• On a bike: First, the feedback loop is very quick ⇒ Anytime you’re out of balance, your body naturally feels it Secondly, the consequences are quite high (painful if you fall on pavement)

• When learning to swim: Swimming is about balance just like a bike, but the medium (the water) that is so much more dense than air that the feedback loop is much slower Secondly, the consequences are much more gentle (it doesn’t hurt when you’re out of balance)

• First, the feedback loop is very quick ⇒ Anytime you’re out of balance, your body naturally feels it

• Secondly, the consequences are quite high (painful if you fall on pavement)

• Swimming is about balance just like a bike, but the medium (the water) that is so much more dense than air that the feedback loop is much slower

• Secondly, the consequences are much more gentle (it doesn’t hurt when you’re out of balance)

“So in that sense swimming is gentler, meaning you have more excuses, it’s a longer time horizon. But to improve, it’s that much more important that you do the deep examination. So swimming then becomes the example of where it’s more important when you bring that level of thinking if you want to make any progress.”

2 Wider skill gap: “ The wider the skill gap between you and the people that you’re competing against, there’s so much more cushion for a soft landing. ”

⇒ Thought experiment: Imagine a zero sum game where we are trading against each other

• And in this particular game, if I were to make perfect decisions against you, every time that we put a dollar on the table I would get a quarter
• But because of the decisions that I’m making, every time we put a dollar on the table, I’m only getting a nickel
• If we think about those quadrants, I’m in the “good decision, good outcome” situation
• But by not exploring, I don’t see the 20 cents that’s sitting there

“ That’s the problem with the soft landing. That’s the problem with having that cushion, it doesn’t let you know that there’s something better that you could be doing. ”

The benefits of ‘backcasting’ (and doing pre-mortems) [2:19:30]

In Annie’s book, Thinking in Bets, the section about “backcasting” really resonated with Peter

• Backcasting, in contrast to forecasting, is when you imagine it’s the day after you’ve reached you goal, and we look back and ask: How did I get here? What decisions did I make?

• How did I get here?

• What decisions did I make?

Peter’s example of backcasting: The Centenarian Decathlon

• Prior to reading Annie’s book, Peter has referred to this same concept as “reverse engineering”
• Peter thinks about backcasting in terms of the physical, structural demise of bodies
• When asked why they exercise, 90% of the answers are one of the following: I like the way it makes me look I like the way it makes me feel I like the freedom and flexibility it gives me with what I eat All of those are fine reasons to exercise, but I don’t think they’re good enough The best reason to exercise comes from the concept of backcasting: What do you want your body to function like the day before you take your last breath ?
• Centenarian Decathlon : Think about what an Olympic games would look like of hundred year olds, and ask yourself, what do you want to be able to do in that Olympic games?
• Peter has about 18 things on his list including: A 30 pound goblet squat (simulating picking up a great grandchild) Another one is to be able to stand up off the floor using only a single point of my own (i.e., one arm and both my legs and get up off the floor without help)

• So backcasting for the Centenarian Decathlon look like: If I can do X at 100, what was I able to do at 90? If I could do that thing at 90, what much have been true at 60? Until you get to the point where you say, “now that I’m 46 I actually have to be able to do all these other things.”

• I like the way it makes me look

• I like the way it makes me feel
• I like the freedom and flexibility it gives me with what I eat
• All of those are fine reasons to exercise, but I don’t think they’re good enough

• The best reason to exercise comes from the concept of backcasting: What do you want your body to function like the day before you take your last breath ?

• A 30 pound goblet squat (simulating picking up a great grandchild)

• Another one is to be able to stand up off the floor using only a single point of my own (i.e., one arm and both my legs and get up off the floor without help)

• If I can do X at 100, what was I able to do at 90?

• If I could do that thing at 90, what much have been true at 60?
• Until you get to the point where you say, “now that I’m 46 I actually have to be able to do all these other things.”

“That’s why that concept for me just resonated so much. Because I really don’t think medicine spends enough time backcasting.” —Peter

For more details about Peter’s idea of the Centenarian Decathlon see AMA #5

Pre-mortems

• A pre-mortem is imagining that you have failed your goal and asking, Why did I fail?
• With backcasting, you focusing about the things within your control (skill-related things)
• Particularly with pre-mortems, you allow luck into the equation
• Example, if I get to 100 and cannot do a 30 lb goblet squat, why did I fail? Well, I could have fallen and broken my hip due to some bad luck situation But then you start thinking, how can I mitigate the odds of that “bad luck” happening? You don’t see that bad luck can occur if you’re not working backwards
• You can now ask yourself some questions because you see the luck:
• If it’s on the good luck side… can I increase the chances that that thing happens to me?

• On the bad luck side… This is really important to explore How can I decrease the chance that I trip and break my hip? Maybe there is something you can do, maybe there is nothing you can do In the case of there being nothing you can do, ask whether there are any hedges available to me, Can I hedge against this bad luck thing happening to me? Sometimes the answer to that is also “no” Regardless, what you can say is, “I’ve got my hedges set up, I’m doing what I can to decrease the chances that it occurs, but obviously it still could.”

• Well, I could have fallen and broken my hip due to some bad luck situation

• But then you start thinking, how can I mitigate the odds of that “bad luck” happening?

• You don’t see that bad luck can occur if you’re not working backwards

• can I increase the chances that that thing happens to me?

• This is really important to explore

• How can I decrease the chance that I trip and break my hip?

• Maybe there is something you can do, maybe there is nothing you can do In the case of there being nothing you can do, ask whether there are any hedges available to me, Can I hedge against this bad luck thing happening to me? Sometimes the answer to that is also “no” Regardless, what you can say is, “I’ve got my hedges set up, I’m doing what I can to decrease the chances that it occurs, but obviously it still could.”

• In the case of there being nothing you can do, ask whether there are any hedges available to me, Can I hedge against this bad luck thing happening to me? Sometimes the answer to that is also “no”

• Regardless, what you can say is, “I’ve got my hedges set up, I’m doing what I can to decrease the chances that it occurs, but obviously it still could.”

• Sometimes the answer to that is also “no”

⇒ Planning your response to bad luck before it happens

• Your next step would be to think about, before that bad luck intervenes , what am I going to do in response?
• By planning in advance, you are making decisions right now in a calmer state of mind rather than reactive to what the world might deliver at the time

• This is one of the big differences of working forward and working backwards: When we think forward (forecast), we’re developing a strategic plan for how we get there and we are really focused on what are the things that we’re going to do that are going to get us there But we DON’T ask, “ what are the ways that luck might intervene?” But when you work backwards, you naturally start to see the luck, then you can start to explore all these questions

• When we think forward (forecast), we’re developing a strategic plan for how we get there and we are really focused on what are the things that we’re going to do that are going to get us there

• But we DON’T ask, “ what are the ways that luck might intervene?”
• But when you work backwards, you naturally start to see the luck, then you can start to explore all these questions

In the end, backcasting (and pre-mortems) makes your strategic plan so much better.

Note: Annie is putting this framework in her workout that is nearly complete.

Parting advice from Annie for those feeling overwhelmed (and two book recommendations) [2:28:30]

Annie has two book recommendations :

• Superforecasting: The Art and Science of Prediction (mentioned earlier when discussing probabilistic thinking at 18:15)

• The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing by Michael Mauboussin It’s all about the influence of luck and skill in our lives and how you can parcel those apart “ If people are really interested in the conversation that we had today, then I guarantee they would be really interested in The Success Equation. ”

• It’s all about the influence of luck and skill in our lives and how you can parcel those apart

• “ If people are really interested in the conversation that we had today, then I guarantee they would be really interested in The Success Equation. ”

Parting thoughts from Annie for those feeling overwhelmed …

“It’s really, really good to have a framework to be thinking about decisions, to really recognize the probabilistic nature of the world. . .and an understanding of what the structure of a really good decision is.

That being said . . . for the majority of decisions that you make in your life, you can actually decide quite fast. And what’s interesting is that the way to really understand when can I decide fast is to understand this broader framework … this broader framework that would allow you to take the bigger decisions in your life and actually start to think about them in this way where you are really forecasting them is actually the way to understand when can I go really fast.

And what it will allow you to do is stop taking 15 minutes to decide what to order in a restaurant, because you have to understand this framework in order to get when you can go fast and when you can go slow.

Because when I do talk to people about this kind of framework, they do feel a little bit overwhelmed and they’re like, ‘Oh my gosh, how am I ever going to make a decision again? Like I’m going to need a super computer in order to do it.’ There’s very few decisions that you actually need to take a ton of time on.

On the ones that are bigger decisions, embracing the probabilistic nature actually gets you out of the analysis paralysis. Because what you realize is once you’ve identified your options and you’ve got an option that you see is better, relative to the other ones, you stop focusing on achieving absolute certainty, and start focusing on achieving ‘relative to the other things I can do, this one looks better to me.’

Because I recognize that there’s all sorts of stuff I don’t know, and there’s all sorts of different ways the world can turn out, and a s long as I’m taking the things that occur in my life and trying to learn from them in a real way, I’m not going to be so terrified as sort of the downside. That really frees you up. ”

Selected Links / Related Material

Annie’s book that prompted Peter to invite her on the podcast : Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts by Annie Duke | (amazon.com) [4:15, 12:40, 1:38:45, 2:19:30]

Annie wins $2 million in a winner take all invitation only World Series Poker Tournament Of Champions : 2004 Tournament of Champions 8 | andtom01 (youtube.com) [4:45]

2010 she won the prestigious NBC National Heads Up Poker Championship : Annie Duke Wins the 2010 NBC National Heads-Up Poker Championship | Ryan Lucchesi (cardplayer.com) [4:45]

Peter is still bitter about the Eagles beating the Patriots in Super Bowl LII : Super Bowl LII | (wikipedia.org) [8:00]

Carlton Fisk famous walk-off home run in the 1975 World Series : Must C Classic: Fisk waives walk-off home run fair to win Game 6 of 1975 World Series | MLB (youtube.com) [9:00]

“Beautiful” article about Bill Belichick : No More Questions | David Fleming (espn.com) [10:00]

Book recommended by Annie : Gridiron Genius by Michael Lombardi | (amazon.com) [10:30]

Consulting firm Peter worked at where they made him take the probability test : McKinsey & Company | (wikipedia.org) [20:00]

Annie recommends two books by Phil Tetlock : [20:30]

• Expert Political Judgment: How Good Is It? How Can We Know? by Phil Tetlock | (amazon.com) [20:30]
• Superforecasting: The Art and Science of Prediction by Phil Tetlock | (amazon.com) [20:30, 2:28:30]

Where Annie watched her brother play poker when she was in high school : Mayfair Club | (wikipedia.org) [27:00]

Daniel Coyle writes in a book about these centers of excellence, small places that breed an outsized amount of talent : The Talent Code: Greatness Isn’t Born. It’s Grown. Here’s How. by Daniel Coyle | (amazon.com) [29:45]

Movie about a poker player starring Steve McQueen : The Cincinnati Kid | (wikpedia.org) [49:30]

The benefits of quitting were written about by Steve Dubner and Steven Levitt in this book : Freakonomics: A Rogue Economist Explores the Hidden Side of Everything (P.S.) | (amazon.com) [59:00]

Famous movie about no-limit Texas hold ‘em : Rounders | (wikipedia.org) [1:01:45]

Oreo scene from Rounders : Rounders (11/12) Movie CLIP – Spotting KGB’s Tell (1998) HD | Movieclips (youtube.com) [1:01:45]

Peter mentions “Moneyball” to compare how analytics have been introduced to poker : Moneyball: The Art of Winning an Unfair Game by Michael Lewis | (amazon.com) [1:11:45]

Super Bowl that ended with the Patriots beating the Seahawks via an interception after a controversial call by Pete Carroll to pass instead of hand the ball to Marshawn Lynch : Super Bowl XLIX | (wikipedia.org) [1:18:15]

Video of the controversial play call to end Super Bowl XLIX : Butler picks off Wilson to seal Patriots Super Bowl XLIX victory | NFL (youtube.com) [1:18:15]

Episode of The Drive where Peter and Zubin Damania reminisce about the Morbidity and mortality conference from med school : #37 – Zubin Damania, M.D.: Revolutionizing healthcare one hilariously inspiring video at a time | Peter Attia (peterattiamd.com) [1:28:45]

Year 2000 scare known as Y2K that turned out to be nothing : Year 2000 problem | (wikipedia.com) [1:45:45]

The decision to invade Normandy by Eisenhower as an example of a good outcome but was it a good decision? : Normandy landings | (wikipedia.com) [1:45:45]

Previous episode of The Drive where Peter talked about ‘backcasting’ : #50 – AMA #5: calcium scores, Centenarian Decathlon, exercise, muscle glycogen, keto, and more | Peter Attia (peterattiamd.com) [2:21:30]

Book that Annie highly recommends if you enjoyed this interview : The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing by Michael Mauboussin | (amazon.com) [2:28:45]

People Mentioned

• Larry Bird (player on the Celtics that made Annie root for them) [8:30]
• Kevin McHale (player on the Celtics that made Annie root for them) [8:30]
• Fred Lynn (former center fielder for the Boston Red Sox) [8:45]
• Carl Yastrzemski (former left fielder for the Boston Red Sox) [8:45]
• Carlton Fisk (former catcher for the Boston Red Sox, famous home run in G6 of World Series) [8:45]
• Bill Belichick (Peter is a big fan of this NFL head coach of the Patriots) [9:15, 12:00, 1:43:15]
• Tom Brady (Patriots quarterback, the GOAT?) [9:15, 12:15]
• Mike Lombardi (American football executive and author of “ Gridiron Genius ”) [10:30]
• Bill Walsh (iconic NFL coach) [10:45]
• Phillip E. Tetlock (author of books that Annie recommends, Super Forecasting and well as Expert Political Judgement ) [20:30]
• Abraham Lincoln (16th President of the United States) [22:00]
• Al Gore (created the internet?, 45th vice president of the United States) [25:45]
• Howard Lederer (Annie’s brother and professional poker player) [28:40]
• Dan Harrington (won the Main Event at the 1995 World Series of Poker) [29:15]
• Erik Seidel (pro poker player that Annie uses as the example of how to evaluate your decisions) [29:15, 1:55:30, 2:12:00]
• Steve Zolotow (pro poker player) [29:15]
• Jason Lester (professional poker player)[29:30]
• Daniel Coyle (writer that talks about pockets of specialized talent) [29:45]
• Rob Gronkowski (New England Patriots tight end) [44:30]
• Steve McQueen (actor from The Cincinnati Kid ) [49:30]
• Steve Dubner (co-author of Freakonomics) [59:00]
• Steven Levitt (co-author of Freakonomics) [59:00]
• John Malkovich (American actor in the movie Rounders) [1:02:00]
• Chip Reese (famous poker player Annie thinks might be the greatest poker mind ever) [1:09:15]
• Howard Lederer [1:13:45]
• Bob Kaplan (Peter’s head of research) [1:15:00]
• Pete Carroll (NFL coach, made a controversial call in the Super Bowl) [1:17:45, 1:42:15]
• Marshawn Lynch (Seahawks running back who people think should have been handed the ball in the Super Bowl) [1:18:15]
• Russell Wilson (Seahawks quarterback) [1:18:15]
• Benjamin Morrison 538 [01:18:15] ???
• Malcolm Butler (Made the game winning interception for the Super Bowl 2015) [1:19:15]
• Zubin Damania (previous guest on The Drive, spoke about the morbidity conference in med school) [1:28:45]
• Sam Hinkie (76ers President of Operations, “Trust the process”) [1:30:45]
• Toby Moskowitz (financial economist Annie saw a great talk from) [1:43:15]
• Donald Rumsfeld (former politician) [1:45:30]
• Dwight D. Eisenhower (president during the D-Day invasion) [1:46:45]
• Daniel Bernoulli (mathematician and physician) [1:58:15]
• Michael Phelps (most decorated Olympian Swimmer) [2:00:00]
• Michael Mauboussin (author of The Success Equation) [2:28:45]
• Travis Denson (show notes guy for The Drive) [2:30:30]

Annie Duke is an author, and experienced corporate speaker and consultant on the behavior of decision making. In 2018, Annie’s first book for general audiences, “Thinking in Bets: Making Smarter Decisions When You Don’t Have All the Facts” was released by Portfolio, an imprint of Penguin Random House. It quickly became a national bestseller.

As a former professional poker player, she has won more than $4 million in tournament poker. During her career, Annie won a World Series of Poker bracelet, and is the only women to have won the World Series of Poker Tournament of Champions, and the NBC National Poker Heads-Up Championship. Annie is a mom of four who has written five books.

In 2014, Annie co-founded The Alliance for Decision Education to build a national movement that empowers teachers, school administrators and policymakers to bring Decision Education to every Middle and High School student. She also serves on the National Board of After School All Stars and The Franklin Institute, and has won a televised championship in rock-paper-scissors. [annieduke.com]

Twitter: @annieduke