Normal Distribution

The normal distribution is a probability distribution that (roughly) describes many common datasets in the real world. It is the most common type of distribution, and it arises naturally in statistics through random sampling techniques.

The normal distribution, also known as the Gaussian distribution, originally arose in the context of estimating human and mechanical error in the use of scientific equipment, like telescopes. Nowadays, it is more common to show up as a model for the "lifespan" of a product, like a lightbulb, or the outcome of standardized tests, like IQ. Biological measurements, like height or weight, are often estimated with normal distributions (though log-normal may be more accurate), and stock values were originally estimated in the same way (though they tend to have fatter tails).

The graph of a normal distribution with mean of \(0\) and standard deviation of \(1\). Its shape is known as the bell curve.

The plot of a normal distribution is symmetric about the mean and has skinny tails, which are less formal ways of saying its skewness and kurtosis are each \(0\). A given normal distribution is then characterized entirely by the values of its mean and standard deviation. The median and mode of a normal distribution are equal to its mean.

The central limit theorem states that the probability distribution of the average of \(n\) independent, identically distributed random variables with finite variance converges to the normal distribution as \(n\) approaches infinity. No matter what the distribution of the random variables are (even if it is very non-normal), the distribution of sample means is normal! In empirical cases, \(n = 30\) suffices for a sufficiently "normal-like" distribution. (This number was determined by computers running way too many Monte Carlo simulations.)

So, even if a distribution is not normal, the distribution of its sample means (for sample size \(>\) 30) is normal! With this in mind, it becomes more reasonable that normal (and normal-like) distributions appear so much in nature or the real world.

Many common examples of measurements in statistics can, from another perspective, be thought of as the combination of a multitude of independent factors. For instance, a person's height is the result of a complicated jumble of genetic, environmental, and personal factors. The accuracy of measurements from a telescope may be dependent upon the eyesight or steady-handedness of the astronomer, the effectiveness of the mechanisms that constructed the telescope, the presence of microscopic dust on the lens, and atmospheric interference from weather, gravity, and background noise. While these examples do not meet the explicit criteria for the central limit theorem, they hopefully at least show why it might be intuitive that the normal distribution appears so often: it comes from a synthesis of other distributions.

It is worth noting that while many standard measurements in social sciences and elsewhere are often assumed to be normal, they may not be in reality. However, the distribution of averages from simple random samples is indeed normal.

Suppose all \(2000\) students at a school each wrote on a slip of paper their favorite number from \(1\) to \(100\). Then, all the numbers were collected and put in a box. The math teacher then chose \(30\) slips at random, wrote down the average of the numbers written on them, and put the slips back in the box. The teacher then does this many, many more times. No matter what the probability distribution was for the individual students, the probability distribution for the numbers the math teacher writes down is normal! If those numbers were plotted, the teacher would get the typical bell curve.

The takeaway is that many naturally occurring processes are susceptible to statistical techniques that work for normal distributions. In particular, even very non-normal distributions (with finite variance) have a sampling distribution of means that is normal.

However, the assumption of normal distribution fails to hold in many cases. Here are some good rules of thumb for evaluating whether or not it is reasonable to assume normality:

There are statistical tests of the normality of a probability distribution, like the Kolmogorov-Smirnov test, the Shapiro-Wilk test, and Pearson's chi-squared test. These attempt to disprove a null hypothesis that the distribution can be modeled by a normal distribution by finding sufficient evidence that the distribution departs from a normal distribution. The quantile-quantile plot provides a popular and graphical technique for determining whether two datasets have the same distribution. This requires independent sampling from a known normal distribution.

Student scores on history quizzes are likely to be non-normal since their performance is dominated by whether or not they read the material before class. The distribution is likely to be left-skewed.

The 2008 financial crisis was arguably caused by long-term adherence to the assumption that stock prices are normal when, in fact, there is often a herd mentality contributing to swift rises/falls in price. Dependencies among contributing factors lead to distributions with fatter tails than the normal distribution.

In many empirical studies (for instance, surveying a group of people), it is important that respondents do not hear other participant's answers before giving their own. This is because people may change their answer based on what they hear, and the previous person may have affected the next person's response, skewing results even if the people were selected at random.

Overall, statisticians are well-disposed to talk about whether a distribution is normal or not.

The normal distribution has two important properties that make it special as a probability distribution.

The average of \(n\) normal distributions is normal, regardless of \(n\).

There exist other distributions that have this property, and they are called stable distributions. However, the normal distribution is the only stable distribution that is symmetric and has finite variance. Such sums are known as multivariate normal distributions.

Given a simple random sample from a random variable with a normal distribution, the sample mean and sample variance are independent.

This property is unique (among all probability distributions) to the normal distribution. It emphasizes the overall symmetry and "balance" of the bell curve.

Histograms show how samples of a normally distributed random variable approach a bell curve as the sample size increases. The following graphs are of samplings of a random variable with normal distribution of mean \(0\) and standard deviation \(1\).

\(n = 10\)	\(n = 100\)
\(n = 1000\)	\(n = 10000\)
\(n = 100000\)	\(n = 1000000\)

Note how the graphs become more and more symmetric as \(n\) increases. The proportion of numbers in a certain region also begin to have fixed ratios. For instance, \(68\%\) of the numbers in the last graph appear between \(-1\) and \(1\). In fact, all normal distributions have these same ratios, and tables of \(z\)-scores are used to determine the exact proportions.

A new product was released and a survey asked customers to give the product a score between 1 and 100. At first, when the number of subjects \((n)\) was still relatively low, the company couldn't pull much information from the surveys. For example, after four people had taken the survey, one person rated it a 92, one rated it a 72, one rated it a 63, and the last one rated it a 34. However, as more customers took the survey, the company was able to create a histogram showing the results. Once 5,000 surveys had been taken, the company found that the average person rated the product a 67 out of 100, and the rest of the scores were normally distributed in a bell-curve out from there (with a standard deviation of 9). Based on this, the company decided that its product was not meeting customers' desires.

The \(z\)-score of a value from a random variable is the number of standard deviations away from the mean the value is. If \(\sigma\) is the standard deviation of the distribution, \(\mu\) is the mean of the distribution, and \(x\) is the value, then \[z = \frac{x - \mu}{\sigma}.\] This value is used in many tests in statistics, most commonly the \(z\)-test. By calculating the area under the bell curve, a \(z\)-score provides the probability of a random variable with this distribution having a value less than the \(z\)-score.

A visual representation of the \(z\)-score \(-0.68\)

A \(z\)-score table usually takes the following form, where the column determines the hundredths digit of the \(z\)-score and the row determine the tenths and units digit.

Consider a population with a normal distribution that has mean \(3\) and standard deviation \(4\). What is the probability that a value selected at random will be negative? What about positive?

---

A negative number is any number less than \(0\), so the first step is to find the \(z\)-score associated to \(0\). That is \(\frac{0 - 3}{4} = -0.75\). By finding the row with the first two digits \((-0.7)\) of the \(z\)-score and choosing the column with the next digit \((5),\) we find that the value in the table associated to a value of \(-0.75\) is \(0.2266\), so there is a \(\color \red \text{22.66%} \) probability that the value will be negative. There is a \(1 - 0.2266 = 0.7734\) or 77.34% probability of it being positive. \(_\square\)

The normal distribution with mean \(\mu\) and variance \(\sigma^2\) is denoted \(\mathcal{N}\big(\mu, \sigma^2\big)\). Its probability density function is

\[p_{\mu, \sigma^2} (x) = \frac{1}{\sigma \sqrt{2\pi}} e^{-\frac{(x-\mu)^2}{2\sigma^2}}.\]

There is no closed form expression for the cumulative density function.

If \(X_1\) and \(X_2\) are independent normal random variables, with \(X_1 \sim \mathcal{N}\big(\mu_1, \sigma_1^2\big)\) and \(X_2 \sim \mathcal{N}\big(\mu_2, \sigma_2^2\big)\), then \(aX_1 \pm bX_2 \sim \mathcal{N}\big(a\mu_1 \pm b\mu_2, a^2\sigma_1^2 + b^2\sigma_2^2\big)\).

The bell curve is a probability density curve of binary systems. Then the probability at some displacement from the medium is

\[P(n, k) = \left( \begin{matrix} n \\ k \end{matrix} \right) {2}^{-n}= \frac{n!}{\big(\frac{1}{2}n + k\big)!\, \big(\frac{1}{2}n - k\big)!\, {2}^{n}}.\]

Using the Stirling's approximation and treating \(k = \frac{\sigma}{2}\), we have

\[P(n, \sigma) \sim {\left(\frac{n}{2\pi} \right)}^{\frac{1}{2}} {\left(\frac{n}{2}\right)}^{n} {\left(\frac{{n}^{2} - {\sigma}^{2}}{4}\right)}^{-\frac{1}{2}(n+1)}{\left(\frac{n + \sigma}{n-\sigma}\right)}^{\frac{-\sigma}{2}}.\]

For \(n\gg \sigma \), \(\frac{n + \sigma}{n-\sigma} \sim 1+\frac{2\sigma}{n}\); hence, for large \(n\)

\[P(n, \sigma) \sim {\left(\frac{n}{2\pi} \right)}^{\frac{1}{2}} {\left(1- \frac{{\sigma}^{2}}{{n}^{2}}\right)}^{-\frac{1}{2}(n+1)}{\left(1+\frac{2\sigma}{n}\right)}^{\frac{-\sigma}{2}}.\]

Taking the logarithm yields

\[\ln\big(P(n,\sigma)\big) \sim \frac{1}{2}\ln \left (\frac{2}{\pi n}\right) - \frac{1}{2}(n+1)\ln \left (1- \frac{{\sigma}^{2}}{{n}^{2}}\right) - \frac{\sigma}{2}\ln \left (1+\frac{2\sigma}{n}\right).\]

For small \(x\), \(\ln(1+x) \approx x\); subsequently,

\[\ln\big(P(n,\sigma)\big) \sim \frac{1}{2}\ln \left (\frac{2}{\pi n}\right) - \frac{1}{2}(n+1) \left (-\frac{{\sigma}^{2}}{{n}^{2}}\right) - \frac{\sigma}{2} \left (\frac{2\sigma}{n}\right)\]

or

\[\ln\big(P(n,\sigma)\big) \sim \frac{1}{2}\ln \left (\frac{2}{\pi n}\right) + \frac{{\sigma}^{2}}{{n}^{2}} - \frac{{\sigma}^{2}}{2n}.\]

Since \(\frac{{\sigma}^{2}}{{n}^{2}}\) vanishes faster than \(\frac{{\sigma}^{2}}{2n}\) for very large \(n\), we arrive at the result

\[P(n, \sigma) = {\left(\frac{2}{\pi n} \right)}^{\frac{1}{2}} {e}^{\frac{-{\sigma}^{2}}{2n}}.\]

• Central Limit Theorem
• Mean
• Standard Deviation