LARS P. SYLL

Why your dates are attractive jerks

A helpful intuition for understanding ‘collider bias’ — the spurious correlation induced when one controls for a common effect rather than a common cause — can be found in modern dating behaviour.

In a population at large, it is reasonable to assume that attractiveness and personality are independent. Mean people are not necessarily attractive, nor are nice people. However, once we shift our focus from the general population to the specific subset of people who are currently dating, we are conditioning on a collider. The choice to date someone is caused by both attractiveness and personality.

Within the dating group, a false trade-off emerges. If an individual has a mean personality, the reason they might still be dating is probably their attractiveness. Conversely, if someone is unattractive, they must have a nice personality to be successful daters. Looking only at the dating pool, one would conclude that there is a negative correlation between the two traits, guessing that mean people are attractive and that unattractive people are nice.

This simple analogy should serve as a powerful warning for economists. In recent decades, the discipline has developed a tendency to address the problem of omitted variable bias by adding as many controls as possible to a regression. The logic seems sound — if leaving a variable out biases your estimate, putting it in should fix it. This mechanical approach, however, crucially depends on the type of variable being added. Controlling for a confounder — a variable that causes both the treatment and the outcome — removes bias. But controlling for a collider — a variable that is caused by both the treatment and the outcome — creates it.

When economists add variables that may themselves be outcomes of the processes they are studying, they risk opening up back-door paths and creating correlations that have no basis in reality.

Reducing complex research questions to manageable statistical hypotheses is a difficult art, and the assumption that one can simply treat real-world processes as if they were random statistical experiments is a phantasmagoric fantasy. If our goal is a relevant and realistic science, we must remember that more controls are not always better — sometimes, they are the very source of the illusion.

How (not) to make valid causal inferences

Traditionally, analysts use data on stopped individuals to study bias by computing the difference in violence rates between stopped minority and white civilians, while controlling for observable differences between these two sets of encounters. We term this the “naïve estimator” … However, without further assumptions, this quantity will have no causal interpretation so long as the treatment affects the mediator (i.e., civilian race affects whether officers detain a civilian) … This is because treated encounters (with minority civilians) that result in a stop will not be comparable to those with stopped control (majority) civilians. As a simple example, suppose officers exhibited racial bias as follows: they detain white civilians if they observe them committing a serious crime (such as assault, potentially warranting the use of force) but detain nonwhite civilians regardless of observed behavior. When this is true, comparing stopped white and nonwhite civilians amounts to comparing fundamentally different groups. The analyst will observe force used against a greater proportion of stopped white civilians because of the differential physical threat they pose to officers. Under the traditional approach, the analyst would naïvely conclude that anti-white bias exists, yielding an erroneous portrait of racial discrimination in the use of force.

Dean Knox, Will Lowe, Jonathan Mummolo

This study is a must-read for researchers seeking to identify causal relations from proprietary administrative datasets.

Looking only at data often gives the wrong causal impression — especially when, as is the case here, the data are biased from the outset and lead to sample selection effects due to post-treatment conditioning. Comparing white bank robbers to black civilians who have committed no crime does not provide the apples-to-apples comparison necessary for valid causal inference.

From a statistical perspective, the fundamental error in the traditional analyses is the all-too-frequent and deeply problematic assumption that one can simply treat convenience samples and real-world processes as if they were random statistical samples. They are not — and every statistical hypothesis tested is consequently of dubious value. Relying on unwarranted, phantasmagoric model assumptions is a risky manoeuvre if our goal is relevant and realistic science.

Reducing interesting research questions to ‘manageable’ statistical hypotheses is a difficult art. Data — whether ‘big’ or otherwise — never by themselves yield credible causal inferences.

Is smoking — really — good for babies?

Judea Pearl, in his The Book of Why, discusses the problems that arise if we thoughtlessly try to ‘control’ for too much in our quest to identify causal relationships.

One of his examples concerns the paradox that when we want to find out whether mothers’ smoking increases the risk of infant mortality, but only study newborns with a low birth weight, it can lead us to draw the incorrect conclusion that smoking is beneficial.

To understand the paradox, one must first appreciate the true causal structure. A mother’s smoking during pregnancy (MS) can cause her baby to have a low birth weight (LBW). Separately, a completely independent factor — genetic anomalies in the baby (GA) — can also cause low birth weight. Furthermore, these genetic anomalies (GA) are a direct cause of infant mortality (IM), just as low birth weight (LBW) itself can be. Crucially, smoking and genetic anomalies are independent—smokers are no more likely to have babies with genetic defects.

The trouble begins when a researcher decides to study the effect of smoking on mortality within a specific group: babies with low birth weight. By restricting the study to these babies, the researcher inadvertently conditions on the ‘collider’ LBW, which gives rise to a false correlation between the two independent causes of low birth weight: smoking and genetics.

If a baby has a low birth weight and the mother smoked, its low birth weight is easily attributed to the smoking. Consequently, these babies are less likely to have a severe genetic anomaly. However, if a baby has a low birth weight and the mother did not smoke, its low birth weight cannot be explained by smoking, implying a higher likelihood of an underlying genetic defect. Since genetic defects are a powerful cause of mortality, the non-smoking mothers’ babies with low birth weight will have a higher death rate than the smoking mothers’ babies.

This leads to the bizarre — and false — conclusion that smoking appears to be good for newborns. In reality, smoking is harmful. By mechanically controlling for birth weight (by choosing only to study low birth weight babies), the researcher creates a statistical illusion. This example powerfully underscores the importance of understanding the causal structure behind the data. It demonstrates that naive statistical adjustments, performed without a causal map, can transform a genuine harm into a perceived benefit.

The point yours truly made last week bears repeating: econometric models are frequently the result of researchers mistaking statistical control for causal identification. The problem? You can’t ‘control’ your way out of confounding — it requires a causal solution. Control for everything, and you’ll inevitably hit a collider, opening back-door paths and creating confounding that wasn’t there to begin with.

‘Overcontrolling’ (III)

.