Selection Bias

A selection bias is a bias that comes from a difference between the distribution of data sampled in a study and the distribution of the population selected from. The fundamental problem of statistics in science is to try to infer general properties from a small set of observations. Concerns about selection bias are important for designing any scientifically valid experiment. Randomized controlled trials are the gold standard for attempting to eliminate selection bias, but the question must always be: randomized relative to what? For instance, if a study looking at academic achievement picks randomly from students visiting a website on college applications, it will get very different results than if it had randomly picked from all high school students.

• polling
• survivorship bias
• Berkson's Paradox Berkson's Paradox is a specific type of selection bias resulting when the data is selected from a range where $A$

When

In addition to bias from who the polling agency contacts, there is self-selection bias in who decides to respond. The surveyors cannot force people to participate, so they are more likely to An even stronger bias results from determining public opinion by looking at letters sent or calls made to politicians' offices. The people who call or write are the ones with the largest grievances, so looking at them will give an inaccurate representation of public opinion.

The Müller-Lyer line illusion

Psychologists conduct studies on the way people perceive themselves and the world, in the hopes of figuring out universally how humans work. However, most research psychologists are based out of universities in wealthy cities in the United States. Out of those cities, the subjects are overwhelmingly likely to be students, especially students in psychology. The results from any individual study are likely not to be universal psychology, but WEIRD psychology: Western, Educated, Intelligent, Rich, Democratic psychology.

The Müller-Lyer line illusion, depicted to the right, shows two lines with arrows on either side. Although both lines are the same length, many studies showed that people systematically assume the first line (with arrows pointing inward) is shorter than the second line (with arrows pointing outward). However, this turned out to be remarkably strong in American undergraduates and European samples, but very small in other cultures. Something as seemingly basic as visual perception varied in ways that were not captured by the sample of people studied.

A funnel plot looking at studies on stereotype threat. The triangles represent the ideal plot. Notice that there are many more data points to the left of the triangle than to the right^[1].

Individual studies can only tell researchers a certain amount about whether a particular hypothesis is true. A meta-analysis is a study that combines the results of several other studies to get a sense of what the literature as a whole says. In addition to bias resulting from selection of data for individual studies, there can be an additional level of bias when considering the selection of studies that are available for study, otherwise known as publication bias. There are many reasons that studies that are done either don't get reported or have distorted statistical results. For instance, scientific journals prefer to publish studies that found an effect was true rather than no effect at all. Even if in reality there is no effect, if researchers do small enough studies they are likely to find one. If there are a number of small studies performed, some of which find an effect and some of which don't, the papers that end up getting published will make the effect seem stronger than it actually is.

To help identify publication bias, researchers can use a funnel plot. It plots the results of the studies considered in the meta-analysis, comparing the effect size to the certainty of that result. Ideally, the largest studies with the most certain results should be closest to the true effect size, whereas smaller studies with less certain results should be spread out more. This creates the funnel shape, which is a triangle that is symmetric about the true effect size. If there is publication bias, the plot will instead be skewed to one side or the other. In the example plot, there are more studies on the left-hand side, indicating that studies systematically published more negative effects than they should have.

[[heading|]] It is possible to have results that are systematically skewed from the

Very small changes in the cosmological constant (one of the constants in general relativity) can lead to very large changes in the shape of the universe^[2].

Selection bias plays a large and controversial role in physical theories of the multiverse. In our best physical theories, there are a number of constants, such as the mass of the electron and the cosmological constant, that determine the shape of our universe. However, if we thought of these constant as "knobs" that could be adjusted, there is an extremely small range of values they could take before the universe would be completely incapable of supporting human life as we know it. This is known as the fine-tuning problem. One way physicists try to explain this is by imagining that all possible universes with all possible combinations of constants "exist" in some sense, and then applying selection bias.

Anthropic bias ("anthro-" meaning "human," as in "anthropology") is the bias resulting from the fact that humans can only exist in a universe that is capable of supporting human life^[3]. Almost tautologically, we can only observe universes in which observers exist. If this is so, it would be difficult or impossible to find what the true distribution of other universes that exist is. There is not much that can be said about how likely or unlikely it is for any given universe to be able to support life or consciousness. Unlike the case of WEIRD psychology, we can't just get more funding to do studies in universes with alternate physics.

Anthropic reasoning is also important for reasoning about catastrophic risks. When the Large Hadron Collider was turned on, there was public fear about whether the high-energy collisions it made would create a black hole that would consume the Earth. Physicists had good reason to believe this wouldn't happen, since there are many higher-energy collisions that happen all the time, when cosmic rays hit the Earth.

However, if a black hole did consume the Earth, we would no longer exist, and therefore wouldn't be around to question how likely it would be for a black hole to consume the Earth. This makes reasoning about catastrophic risks extremely difficult.

1. Flore, P., & Wicherts, J. (2015). Does stereotype threat influence performance of girls in stereotyped domains? A meta-analysis. Journal of School Psychology.
2. Davis, O. File:Flatness problem density graph.svg. Retrieved from https://commons.wikimedia.org/wiki/File:Flatness_problem_density_graph.svg
3. Bostrom, N. (2010). Anthropic Bias. Routledge.