Probability - Independent events

In probability, two events are independent if the incidence of one event does not affect the probability of the other event. If the incidence of one event does affect the probability of the other event, then the events are dependent.

There is a red 6-sided fair die and a blue 6-sided fair die. Both dice are rolled at the same time. Let $A$ be the event that the red die's result is even. Let $B$ be the event that the blue die's result is odd. Are the events independent?

---

Consider whether rolling an even number on the red die will affect whether or not the blue dies rolls odd. The outcome of the red die has no impact on the outcome of the blue die. Likewise, the outcome of the blue die does not affect the outcome of the red die.

$P(A)=\dfrac{1}{2}$ regardless of whether $B$ happens or not.

$P(B)=\dfrac{1}{2}$ regardless of whether $A$ happens or not.

Therefore, the events are independent.

Determining the independence of events is important because it informs whether to apply the rule of product to calculate probabilities. Calculating probabilities using the rule of product is fairly straightforward as long as the events you're working with are independent. Calculating probabilities of dependent events can be more challenging and less straightforward. Thus, it is important to think about whether events are independent or not, as it affects the approach to problem solving.

The example in the introduction demonstrated events that were clearly independent. However, it can sometimes be a challenge to identify whether events are independent or not. Consider the following example:

There are 3 green marbles and 5 blue marbles in a bag. Two marbles are drawn from the bag at random. Let $G$ be the event that the first marble drawn is green. Let $B$ be the event that the second marble drawn is blue. Are the events independent?

---

Case 1: $G$ happens

When the first marble drawn is green, there are $7$ marbles left in the bag, and $5$ of them are blue. In this case, $P(B)=\dfrac{5}{7}$.

Case 2: $G$ does not happen

When the first marble drawn is blue, there are $7$ marbles left in the bag, and $4$ of them are blue. In this case, $P(B)=\dfrac{4}{7}$.

The incidence of $G$ affects the probability of $B$. Therefore, these events are not independent. In other words, they are dependent.

In the previous example, the first marble drawn affected which marbles were left in the bag. Whenever events happen in sequence, and the incidence of an event affects the sample space of the next event, the events will be dependent.

Events do not have to occur in sequence to be dependent. Consider this example:

There are 12 horses in a race. Nyquist and Exaggerator are two of those horses. Each horse has an equal chance of winning. Let $A$ be the event that Nyquist wins the race, and let $B$ be the event that Exaggerator wins the race. Are the events independent?

---

Case 1: $A$ happens

If Nyquist wins the race, then Exaggerator cannot win the race. In this case, $P(B)=0$.

Case 2: $A$ does not happen

If Nyquist does not win the race, then there are $11$ other horses that could possibly win the race, each with an equal chance of winning. Exaggerator is one of those horses, and so $P(B)=\dfrac{1}{11}$.

The incidence of $A$ affects the probability of $B$. Therefore, the events are dependent.

When trying to determine whether events are dependent or independent, consider how the incidence of one event affects the probability of the other. If the probability is affected, then the events are dependent. If there is no effect on the probability, then the events are independent.

The concept of conditional probability is closely related to the concept of independent events. You might notice that some of the previous examples can be restated using conditional probability. For instance, in the marble example, it can be stated that $P(B\mid G)=\dfrac{5}{7}$ and $P(B\mid G')=\dfrac{4}{7}$.

With an understanding of conditional probability, the definitions of independent and dependent events can be restated:

Two events $A$ and $B$ are independent if:

$P(A\mid B)=P(A\mid B')$ and $P(B\mid A)=P(B\mid A')$

Two events $A$ and $B$ are dependent if :

$P(A\mid B)\ne P(A\mid B')$ or $P(B\mid A)\ne P(B\mid A')$

Note: $A'$ and $B'$ are the complements of $A$ and $B$, respectively.

There are some results in probability that can be surprising due to the nature of dependent events. Sometimes, events are worded such that they do not seem to be related. However, further analysis shows that there is some dependence, and this has an effect on probability.

A couple have two children. What is the probability that both of the children are boys, given that one of the children is a boy?

Assume that when a child is born, it has an equal chance of being a boy or girl.

---

A common mistake on this problem is to assume that with one child being a boy, the other child simply has a $\dfrac{1}{2}$ chance of being a boy. However, this solution ignores how conditional probability is defined, and it also ignores the dependence of the events described in the problem. The actual solution is much more surprising.

Let $S$ be the sample space of this experiment. Let $B$ represent a boy and $G$ represent a girl, with the order of the births mattering. $S=\{BB,BG,GB,GG\}$.

Let $C$ be the event that both children are boys. $C=\{BB\}$.

Let $D$ be the event that one of the children is a boy. $D=\{BB,BG,GB\}$

$C\cap D=\{BB\}$.

The sample space is uniform, so $P(C\mid D)=\dfrac{|C\cap D|}{|D|}=\dfrac{1}{3}$

Thus, the probability that both children are boys given that one of the children is a boy is $\boxed{\dfrac{1}{3}}$.

This solution might seem non-intuitive, but it can be proven with real-world evidence. If you found many families that had exactly two children with at least one child being a boy, then roughly $\dfrac{1}{3}$ of those families would have two boys.

Part of the reason this result is so surprising is that the events $C$ and $D$ are dependent. $P(C\mid D')=0$, and so $P(C\mid D)\ne P(C\mid D')$. The knowledge of one child being a boy has a dramatic effect on the probability that both children are boys.

Let $A$, $B$, and $C$ be events, and let them be pairwise independent. That is to say, each pair of events is independent: $A$ and $B$ are independent, $A$ and $C$ are independent, and $B$ and $C$ are independent. Does this mean that $A$, $B$, and $C$ are mutually independent? Unfortunately, mutual independence of more than two events has stricter requirements:

Given a set of more than two events, the set of events is mutually independent if each event is independent of each intersection of the other events.

If even one independence is not satisfied, then the set of events is mutually dependent.

Two fair 6-sided dice are rolled, one red and one blue. Let $A$ be the event that the red die's result is 3. Let $B$ be the event that the blue die's result is 4. Let $C$ be the event that the sum of the rolls is 7. Are $A$, $B$, and $C$ mutually independent?

---

$P(A\mid B)=\dfrac{1}{6}$ and $P(A\mid B')=\dfrac{1}{6}$. Thus, $A$ and $B$ are independent.

$P(A\mid C)=\dfrac{1}{6}$ and $P(A\mid C')=\dfrac{1}{6}$. Thus, $A$ and $C$ are independent.

$P(B\mid C)=\dfrac{1}{6}$ and $P(B\mid C')=\dfrac{1}{6}$. Thus, $B$ and $C$ are independent.

Theses events are pairwise independent. However, in order for all three events to be mutually independent, each event must be independent with each intersection of the other events.

$P\left(A\mid (B\cap C)\right)=1$ and $P\left(A\mid (B\cap C)'\right)=\dfrac{1}{7}$

These are not equal, and so $A$, $B$, and $C$ are mutually dependent.

In the previous example, one might suspect that something fishy is going on given that event $C$ involves both dice rolls. Given this, we would usually be skeptical of finding an event that is independent of $C$. It turns out that it is coincidence that these pairs of events satisfy the definition for independence.

This definition of mutual independence is relevant for the rule of product, as the rule of product requires independent events. In the previous example, if we (incorrectly) try to obtain $P(A\cap B\cap C)$ by the rule of product, we obtain$\dfrac{1}{216}$. However, the correct probability of the intersection of events is $P(A\cap B\cap C)=\dfrac{1}{36}$.

The following theorem can sometimes be useful as a "sanity check" to ensure that you are applying the principles of independence properly:

A set of events $\{A_1,\dots,A_n\}$ is mutually independent if and only if, for every subset of events, the probability of the intersection of those events is equal to the product of the probabilities of those events.

Two random variables $X$ and $Y$ are called independent if, for any outcomes $a$ and $b$,

$$P(X = a \text{ and } Y = b) = P(X = a) \cdot P(Y = b).$$

More generally, a collection $X_1, \ldots, X_n$ of random variables is called independent if for any outcomes $\{a_i\}_{1\le i \le n}$,

$$P\left( \bigcap_{i=1}^{n} (X_i = a_i) \right) = \prod_{i=1}^{n} P(X_i = a_i).\ _\square$$ Note: $\bigcap$ is the symbol for intersections of a series, and $\prod$ is the symbol for products of a series.

As an application of this definition, one can show that $\mathbb{E}[X \cdot Y] = \mathbb{E} [X] \cdot \mathbb{E} [Y]$ if $X$ and $Y$ are independent random variables. This is quite useful; linearity of expectation implies $\mathbb{E}[X+Y] = \mathbb{E}[X] + \mathbb{E}[Y]$ regardless of whether $X$ and $Y$ are independent or dependent, but generally $\mathbb{E}[X \cdot Y] \neq \mathbb{E}[X] \cdot \mathbb{E}[Y].$

If $X$ and $Y$ are independent random variables, then $\mathbb{E}[X \cdot Y] = \mathbb{E}[X] \cdot \mathbb{E}[Y]$.

---

Suppose $X$ takes on values $a_1, \ldots, a_n$ and $Y$ takes on values $b_1, \ldots, b_m$. By the definition of expectation,

$$\mathbb{E}[X \cdot Y] = \sum_{i,j} P(X = a_i \text{ and } Y = b_j) a_i b_j.$$

Since $X$ and $Y$ are independent, $P(X = a_i \text{ and } Y = b_j) = P(X = a_i) \cdot P(Y = b_j)$ and it follows that

$$\begin{aligned} \mathbb{E}[X \cdot Y] &= \sum_{i,j} P(X = a_i) \cdot P(Y = b_j) a_i b_j \\ &= \left(\sum_{i} P(X = a_i) a_i \right) \left(\sum_{j} P(Y = b_j) b_j \right)\\ &= \mathbb{E}[X] \cdot \mathbb{E}[Y].\ _\square \end{aligned}$$

Probability - Rule of Product

Conditional Probability