Why simpsons paradox

This article gives some examples of where these cases happen, discusses how and why they happen, and suggests ways to automatically detect these situations in your own data. This might be due to the relationship between the variables, or simply due to the way that the data has been partitioned into subgroups.

In this example, when looking at the graduate admissions data overall, it appeared that men were more likely to be admitted than women gender discrimination! This leads us to ask: which view is the correct view?

Do men or women have a higher acceptance rate? Is there a gender bias in admissions at this university? In this case, it seems most reasonable to conclude that looking at the admissions rates by department makes more sense, and the disaggregated view is correct.

David Justice had a higher batting average in both and individually, but Derek Jeter had a higher batting average over the two years combined. Figure 1: Knowledge Studio Decision Tree displaying the imbalanced number of at-bats by each player in and Again, we can ask: which view is the correct view?

Was Derek Jeter or David Justice the better hitter? In this case, it seems most reasonable to conclude that the aggregated view is the correct view, and Derek Jeter was the better hitter over the two years.

As we see in the examples above, both cases are possible: sometimes the aggregated view is correct, and sometimes the disaggregated view is correct. There are benefits to both, however, this can get difficult very quickly, especially when working with big datasets. There are also further challenges to consider, even if we have searched for and found all possible Simpson's Pairs.

These challenges relate to interpretation, for example:. Simpson's Paradox is a tricky issue, but a good analyst or data scientist can handle it with the right tools and knowledge. Intervening on Treatment disrupts the evidential relationship with Gender —for example, by controlling for the proportion of male and female patients in each sample—so that any remaining probabilistic relationship between treatment and recovery can only be explained by having taken the treatment.

Such an experimental design, where Treatment and Gender are made probabilistically independent, suffices to rule out association reversal cf. Section 2. Using the notion of an ideal intervention, one can explicate causation as follows Pearl []; Woodward This does not mean, however, that one can only get causal knowledge in cases where one can experimentally intervene. In contrast, the following two expressions are equivalent given the DAG:. A confounding set of variables is one that biases the effect measurement.

For instance, an unmeasured common cause is a confounder because it makes it impossible to differentiate the probabilistic dependence between the variables resulting from the common cause from that resulting from a causal relationship between them. This notion of confounding can diverge from a common colloquial understanding of confounders as alternative explanations of an observed outcome other than the treatment.

A useful sufficient condition for identifiability is the back-door criterion Pearl , [ 79]. First we need to introduce some graphical terminology. This reflects the fact that independent causes of a common effect will typically be dependent conditional on a common effect. This is what we already saw in Section 2. Yet such a derivation is only licensed by causal assumptions about the relationships between the variables.

In our original example, the treatment increased the probability of recovery in each subpopulation, but not it in the population as a whole. Should one approve the drug or not? The causal approach makes it easy to see why one should. The probabilistic relationship between Treatment and Success in the population is an evidential rather than a causal one.

Learning that someone took the drug provides evidence about their gender, and this information is relevant to predicting whether they will recover.

But this does not tell one about whether the drug is causally efficacious. To learn this, one needs to know how the chances of recovery for individuals in the population would change given an intervention on treatment.

This can be determined by conditioning on gender which enables one both to learn the gender-specific effects of the drug, and to derive the average effect in the whole population using the back-door criterion. This case is shown in Figure 3 and contrasted with our running example where the third variable is a confounding factor.

In order to identify the effect of birth control on thrombosis, it is crucial that one does not condition on pregnancy. If there are no unmeasured common causes of birth control and thrombosis, then a probability-raising relationship between birth control and thrombosis in the population as a whole would reliably indicate that taking birth control pills promotes thrombosis.

It is worth emphasizing that there is no basis for distinguishing the two causal structures in Figure 3 using statistics alone. Any data generated by the model on the left could also have been generated by a model with the causal structure of that on the right. Accordingly the judgment that one should partition the population in one case but not the other cannot be based on the probabilities alone, but requires the additional information supplied by the causal model.

For example, if one assumes that Gender is not an effect of Treatment , it cannot be the case that the drug raises the probability of recovery in both males and females, but has no effect on recovery in the general population.

What should not, however, be controversial is that recent causal modeling techniques enable one to systematically distinguish between causal and probabilistic claims in a much more general and precise way than had previously been possible. Theorists of probabilistic causality were to some extent aware that one did not need to hold fixed all causes of the effect in order to eliminate confounding, but they lacked a general account of which variable sets are sufficient for identifying the effect.

Of course, a positive average effect is compatible with the cause lowering the probability of the effect significantly in many subpopulations. This reflects the fact that the partitioning variable s could interact with the cause of interest. But such possible interactions do not make the effect any less genuine as an average effect for the whole population.

Properly understood, it does not. It is certainly true that a cause can raise the probability of its effect in one population and lower it in another, or that it can have a positive effect in a whole population, but not in some of its subpopulations.

But it is not as if only some of these causal relationships are genuine and that philosophers must therefore find a privileged background context within which the true relationship is revealed. It is simply a fact about causation that different populations can have different sets of interactive background factors, and thus the average effects will genuinely differ across the populations.

As shown in Section 2. Bandyopadhyay et al. Question i is essentially a question about the psychology of reasoning: one must offer an account of why the mathematically innocent association reversals seem paradoxical to many. Such accounts help to identify valid forms of inference that leads individuals to mistakenly rule out association reversals, and thereby provide answers to question ii. Such analyses can differentiate among subtly different forms of reasoning, and open the door to empirical work testing whether humans systematically fail to attend to particular differences.

Section 3. If one interprets the claim that taking the drug raises the probability of recovery as the causal statement that intervening to give the drug will make patients more likely to recover, and plausibly assumes that taking the drug has no influence on gender, then the drug cannot lower the probability of recovery both among males and among females.

But, of course, if one is considering ordinary conditional probabilities without any do-operators, such reversals can occur. Accordingly, the appearance of paradox results from conflating ordinary conditional probabilities with conditional probabilities representing the results of interventions.

This answer presupposes that the aim of partitioning the population is to identify causal relationships. Questions about how to proceed in light of the paradox only make sense given a context and given the kind of inference one wishes to draw.

Pearl presents several reasons supporting his analysis of the paradox. Pearl accounts for this story-relativity by showing that whether one should partition a population is decided not by the probabilities but rather by the causal model generating the probabilities.

These causal models cannot be distinguished by conditional probabilities alone. They note that there can be instances of the paradox that do not seem to invoke any causal notions. Suppose that in either bag the big marbles have a higher red-to-blue ratio than the small marbles.

If there are cases of the paradox that still exhibit surprise despite having nothing to do with causality, then the general explanation of the paradox cannot be causal.

As we know from Section 2 , this need not be the case. Given the widespread literature revealing how seemingly error-prone humans can be when reasoning about probabilities e. Yet Bandyopadhyay et al.

Or, more specifically, they do not propose a valid form of reasoning that reasoners are mistakenly appealing to when falling prey to the paradox. The fact that people expect that the ratios in subpopulations to be preserved in the combined population just shows that people are tricked by the paradox. It does not illuminate the underlying mistake that they are making when they are tricked.

In this sense, Bandyopadhyay et al. They also, by their own admission, do not provide a general answer to iii. They view this as a virtue of their account, since they believe that discussions of iii ought to be divorced from discussions of i and ii. His analysis relies on identifying confirmation with increasing the subjective probability of a proposition.

In particular, Fitelson distinguishes between the suppositional and conjunctive readings of a confirmation statement. In our running example, these statements would be as follows:. While the suppositional and conjunctive reading coincide for some accounts of confirmation e. More importantly, while the suppositional reading allows for association reversals, on the conjunctive reading it cannot be the case both that being a female treatment-receiver and being a male treatment-receiver raises the probability of recovery, but being a treatment receiver simpliciter does not Fitelson — In the conjunctive reading there cannot be association reversals, and because the suppositional and conjunctive reading do not differ for many accounts of confirmation, people mistakenly assume that there cannot be such reversals, even when they are relying on the suppositional reading.

Both Bandyopadhyay et al. Ultimately, it is an empirical question whether the paradox can be accounted for exclusively by errors in probabilistic reasoning, or, as Pearl suggests, due to a conflation of causal and probabilistic reasoning.

Not necessarily. The question of whether the source of the paradox is causal cannot be resolved purely by appeal to the mathematical conditions under which the it arises. Rather, it depends on substantive psychological hypotheses about the role of causal and probabilistic assumptions in human reasoning. The empirical evidence on the paradox shows that reasoners find trivariate reasoning i. A famous example is the analysis of SAT scores—the results of college admission tests—in the United States as a function of the high school grade point average GPA of students.

This phenomenon is, however, very natural. As soon as there is a bit of grade inflation at high schools, each group loses their best students to the next higher group, lowering the SAT average per group.

But this is of course consistent with the overall SAT average remaining equal, or even rising from to , like in our dataset. Since societal developments such as grade inflation affect both the grade distribution and the SAT scores, one should not condition on the GPA of a student when studying SAT scores over time compare the back-door criterion from Section 3.

Each cluster of values corresponds to a single person repeated measurement. A similar example is presented in Figure 4 , adapted from Kievit, Frankenhuis, Waldorp, and Borsboom The figure shows the results of coffee intake on performance at an IQ test.

Suppose that coffee actually decreases performance slightly because it makes drinkers more nervous and less focused. At the same time, coffee intake co-varies with education level construction workers are too busy for drinking coffee all the time! When we measure performance repeatedly for different individuals, we see that their performance is slightly negatively affected by their coffee intake.

However, the unconditional regression model of performance as a function of coffee intake suggests misleadingly that coffee consumption strongly improves performance! The reason for the confounding is the causal impact of the hidden covariate, education level, on both coffee consumption and performance.

One of the aims behind the methodology of randomized controlled trials RCTs is to eliminate the effect of potential confounders on whether a person is treated or not. This was described in Section 2. For example, if we ensure the same proportion of both genders in the treatment and control group, the same prevalence of different age groups, etc. However, the log- odds ratio, a popular measure of effect size in epidemiological research, shows a deviant behavior.

The odds ratio is thus a particularly tricky association measure. Greenland gives the instructive example of an odds ratio that is equal in all subpopulations with row-uniform design, but halved when data are pooled. How should such studies be aggregated? If this is indeed the case, then the overall dataset is row-uniform and AR and for most measures, AMP is avoided, as shown in Section 2.

Another reason for not pooling the data is that study populations are often heterogenous and that calculating the strength of association i. In particular, while at the level of studies patients are usually assigned randomly to the treatment or control group, this cannot be said about the aggregate data Cates Proper meta-analysis therefore proceeds on the basis of weighting the effects rather than pooling the data, either by a fixed effects model or e.

That principle is supposed to guide rational decisions under uncertainty, and has been stated by Savage as follows:. Savage 21— But this inference is mistaken if association reversal occurs: it is perfectly compatible with the above scenario that the overall frequency of recovery is higher for non-treated than for treated patients! Blyth concludes that. Association reversal means that. However, Savage certainly did not intend the sure-thing principle to be a theorem of probability.

Specifically, buying the property raises the chances that the Democratic candidate, whom he dislikes, will win. In that case he would certainly buy the property after the election, regardless of the outcome, but he may refrain from buying it before the election.

In response to this challenge, Jeffrey restricts the sure-thing principle to the case where. That is, buying the property should not change our rational degree of belief in who wins the election. Therefore he proposes a causal sure-thing principle that we have encountered in Section 3. Throughout this entry we have assumed knowledge of the causal facts pertinent to a situation.

Scenarios in which an agent lacks such knowledge raise additional complications for decision theory. A distinct concern is that an agent may not be sure whether her action counts as an intervention e. See the entries on decision theory and causal decision theory for further discussion. Within the philosophy of biology, the units of selection debate Sober [ ch. Since altruistic individuals harm their own chances of survival and reproduction, they are less fit, and it is thus unclear how altruism could evolve as a result of natural selection.

If, however, groups with more altruists are fitter than groups with fewer, and selection can act on groups, this could potentially explain how altruism could still evolve.

Within the units of selection debate, Simpson reversals have played an important role in explaining the possibility of group-level selection.

Consider the following naive argument against the conceptual possibility of group-level selection. In this context, altruistic individuals are, by definition, those with traits that reduce their individual fitness while improving the fitness of other group members. For instance, crows that issue warning cries when a predator approaches benefit the group while increasing the chances of being harmed themselves.

Natural selection explains the evolution of traits on the basis that individuals with the trait are fitter than those without it all else being equal.

Since selfish individuals are by definition fitter than altruistic ones, it follows that groups with more altruistic individuals cannot be fitter. Or so one might argue. By now it should be clear what is wrong with this type of argument—it does not follow from the fact that altruistic individuals are less fit than selfish ones in every population that populations which average over selfish and altruistic individuals cannot be fitter than populations with just selfish individuals.

It could be that being an altruist is correlated with being in a population with more altruists, and that populations with more altruists are fitter. This dispenses with the naive argument. Note, however, that within every single group selfish individuals are fitter, so if the groups change membership only through reproduction as opposed to migration and mutation then over enough generations every group will end up consisting only of selfish individuals.

So whether groups selection can occur depends on additional facts about population structure and dynamics. The group selection hypothesis remains controversial among biologists. Weinberger Walsh presents an example in which a correlation in a population disappears when one splits the population into two parts.

As Otsuka et al. In the prior discussion, dividing the population was not a matter of changing its size, but rather of partitioning its probability distribution based on a variable. Bickel et al. The authors use the paradox to explain why the higher university-wide acceptance rate for men does not show that any department discriminated against women. Specifically, women were more likely to apply to departments with lower acceptance rates. While the probabilistic structure of the Berkeley case is similar to other instances of the paradox, it raises an additional question.

But assuming that gender is a cause here, then the department variable is a mediator , and one should not condition on mediators in evaluating the mediated causal relationship. So what is the justification for conditioning on department? The answer is that in evaluating discrimination, what often matters are path-specific effects , rather than the net effect along all paths Pearl [ 4. To give a different example Pearl , consider whether a hypothetical black job candidate was discriminated against based on her race.

It is possible that as a result of prior racial discrimination, the candidate was denied opportunities to develop job-relevant qualifications, and as a result of lacking these qualifications was denied the job.