Knowledge and Uncertainty

It is said that we live in an “information age.” Indeed, information is a powerful force that shapes our world and the range of human experience — from the highs, to the lows, to the downright paradoxical:

  • “The key to Alice’s success is she always has the right information.“
  • “It was a complete failure, I had bad information.“
  • “Somehow, I feel like I have less information after talking with you.”\\[1.2em]

But what does it mean to have information? Is it something that we can have more or less of, like money? And how do we go about obtaining it?

Guess Who

                           

The answers to these questions lie at the heart of three closely related strands of thought that this course will uncover:

  • information theory: how to quantify and transmit information
  • Bayesian inference: how to update our beliefs in the face of new evidence
  • causal Bayesian networks: how to infer causes from correlations.\\[1.2em]

Together, these ideas can help us interpret the flood of information in our lives.

Guess Who

                           

We'll start this journey with Alice, a budding information theorist like yourself. She's trying to figure out how to ask efficient questions. In other words, how to obtain the most information possible with any given question.

She's playing her friend Bob in the game Guess Who, a board game where the goal is to guess the face your opponent has chosen out of a set of faces by asking as few questions as possible.

When we find her, she's fallen into a daydream. The characters have taken on a life of their own and she's square in the middle of a detective thriller. Bob has become a security guard who witnessed the crime but will only respond to questions from Alice.

With no further ado, enter stage left.

Guess Who

                           

A terrible crime has been committed.

The main attraction, a rare and precious diamond, was stolen just before the museum opened its doors. Patrons were horrified and the detectives were called immediately.

Here are their main suspects:

Alice needs to find the criminal by asking as few "yes"/"no" questions as possible. How to achieve this might not be clear, but what about the opposite?

Suppose that Alice's strategy is to go through the suspects one by one, asking

  • "Was it Audrey?"
  • "Was it Bill?"
  • etc.

How many such questions would she have to ask, on average, to identify the criminal?

Guess Who

                           

Some potential "yes"/"no" questions Alice could ask are ones like "Does the criminal have stripes?" or "Does the criminal wear sunglasses?" In fact, Alice can ask any question about the criminal where the answer is yes or no.

Which of these questions would be the most useful to ask?

Guess Who

                           

We already know that none of the suspects are grey and that all the suspects are 2D2\text{D} shapes. That’s why the shape and color questions are not worth asking.

We don’t know the answer to the question "Does the criminal have a yellow hat?", but it's very likely that the answer is "no." So, that question isn't very valuable either.

By contrast, asking whether the criminal wore sunglasses would eliminate a full half of the answers — much better than we do with the other questions.

It seems a question's usefulness is somehow related to how much uncertainty we have about the answer.

Guess Who

                           

For the most useful of questions, what is the probability that the answer is yes?

Guess Who

                           

We've just discovered an important principle we'll come back to throughout this course:

The most informative questions are the ones whose answer we are the most uncertain about.

These questions, where both answers are equally likely, are so important that we'll give them a special name: maximally informative questions.

A useful way to think about maximally informative questions is like flipping a fair coin. Before the coin is flipped, we have no insight into what the outcome will be — if we tried to guess, we'd be wrong half the time and right the other half of the time.

It is impossible to know less than this about the coin's outcome.

This is what is meant by maximal uncertainty. If we ask what happened, we'd go from zero knowledge to complete knowledge.

By contrast, if we had a coin that comes up heads 90%90\% of the time, we're already pretty sure it will land heads. If we ask what happened, we learn a little bit, but it doesn't do much to change our understanding.

Guess Who

                           

Which of the following questions are maximally informative?

Select all that apply.

Guess Who

                           

Select one or more

Let's suppose we asked the question "Do they wear a hat?" and the answer was "yes".

We've used up 11 of our questions, and there are 88 suspects left:

How many more maximally informative questions would you need to ask to find the criminal?

Guess Who

                           

We were able to find the criminal with just 44 maximally informative questions!

Maximally informative questions seem to be the best way to get to the truth, after all 🎉

... or are they?

Guess Who

                           

Alice knows that 88 of the suspects wear hats, which is why she asked this question:

"Does the criminal wear a hat?" (maximally informative)

But she knows a bit more than that — only 77 suspects wear black hats. Could she have done better by asking the more specific question below?

"Does the criminal wear a black hat?" (not maximally informative)

No matter what the answer is to the first question, Alice will have 88 suspects left. On the other hand, if the answer to the second question is "yes," then she'd only have 77 suspects left.

It seems like this might actually be the better question.

What's wrong with this reasoning?

Guess Who

                           

When we evaluate the question "Does the criminal wear a black hat?", we need to consider both possible answers.

  • If the answer is "yes" then there are 77 suspects left.
  • If the answer is "no" then there are 99 suspects left.

You might then think that on average this question eliminates half of the suspects, just like the maximally informative question, but we need to remember something:

The answer "no" is more likely than "yes".

On average, how many suspects will you have left after asking this question?

Guess Who

                           

We've just shown that maximally informative questions cut the number of suspects in half, whereas other questions leave more suspects behind on average. The more suspects left, the longer it will take to find the criminal, and that's why you're better off with only asking maximally informative questions.

This mathematically proves our statement from earlier:

The most informative questions are the ones whose answer we are most uncertain about.

This might seem like an odd insight, but in this course we'll see how we can leverage this fact again and again to understand knowledge and uncertainty.

For now, let's see how it can help to win another game in the next exploration!

Guess Who

                           
×

Problem Loading...

Note Loading...

Set Loading...