Continuous Random Variables - Joint Probability Distribution
In many physical and mathematical settings, two quantities might vary probabilistically in a way such that the distribution of each depends on the other. In this case, it is no longer sufficient to consider probability distributions of single random variables independently. One must use the joint probability distribution of the continuous random variables, which takes into account how the distribution of one variable may change when the value of another variable changes.
Contents
Definition of Joint Probability Distribution
The probability that the ordered pairs of random variables take values in the (open or closed) intervals and respectively, is given by the integral of a function called the joint probability density function
In the discrete case, if and are two random variables, then to each pair of possible outcomes and can be assigned the number , the probability of that pair of outcomes. The sum over all possible pairs of outcomes is then equal to one in the discrete case:
As before, the generalization to the continuous case follows by replacing the sums with integrals and with
This is the normalization condition for joint probability density functions.
Intuitively, the joint probability density function just gives the probability of finding a certain point in two-dimensional space, whereas the usual probability density function gives the probability of finding a certain point in one-dimensional space.
A certain joint probability density function is given by the formula
where and both range over the entire real number line. Find the normalization constant .
Computing the normalization integral using polar coordinates,
Thus the constant is
A normalized joint probability density function on the square is given by
Find the probability that is between and and is greater than .
By the definition of the joint probability density function, this probability is
Marginal Distributions
Suppose that one has the joint probability density function for and , . But perhaps only the variable is relevant to the problem at hand, i.e. one only cares about the probability regardless of the value of . Fortunately, the marginal distributions and can be extracted from the joint probability distributions.
In the discrete case, recall that every ordered pair of outcomes is assigned the probability . Since the discrete case is discrete, these may be thought of as matrix elements by ordering the possible outcomes in some way, where each fixed corresponds to fixed and vice versa. If one is looking for the probability of , one wants to sum all the probabilities in the matrix where this is true:
If fixed corresponds to row , this probability is
That is, the probability that is found by summing the probabilities of every possible outcome where .
Given the last formula above in the discrete case, the generalization to the continuous case is now easy by replacing the sums with integrals. The marginal distributions are found by integrating over the "irrelevant" variable:
In probability, two random variables are independent if the outcome of one does not influence the other. Independence can be stated in terms of joint probability density function using marginal distributions via the statement
That is, two random variables are independent if their joint probability distribution function factors into the marginal distributions.
A certain joint probability density function is given by the formula
where and are both drawn from the interval Find the marginal distribution
The marginal distribution in is given by integrating out
Expectation, Variance, and Covariance
The expected value, variance, and covariance of random variables given a joint probability distribution are computed exactly in analogy to easier cases. The expected value of any function of two random variables and is given by
For instance, the expected value of is
The variance of each variable independently is defined accordingly:
Note that the expected values can be computed using either the joint probability distributions or the marginal distributions, since the two cases will be mathematically equivalent (in one case, the two integrations are performed together; in the other, they are performed one at a time).
A new quantity relevant to joint probability distributions is the covariance of two random variables, which is defined by
There are a few important observations to make about this expression. The first is to note that and similarly for . Secondly, note that the independence of and is equivalent to their covariance vanishing. This is because if and are independent then since the joint probability density function factors. The covariance thus encapsulates how much changing one random variable affects the other.
A certain joint probability distribution is given by the joint PDF
where and are sampled from the interval . Compute .
To compute the covariance, one must compute a number of expectation values according to the above definition. Computing each gives
where, in the last line, the symmetry of and in the joint probability density function allows one to say without doing more computation. Now
Since , .
This should have been expected; since the joint PDF factorizes into marginal distributions, and are independent.