Confirmation bias and the death penalty

So we're continuing on our death penalty theme. Don't worry, soon we'll move onto happier subjects!

Justin Wolfers posted this piece today explaining how the Supreme Court misinterpreted his research on the deterrent effect of capital punishment. In the post (which is a follow-up to an op-ed in the Washington Post), he links to some background information he put together on the empirical evidence in the debate.

The relevance of the deterrent debate is that many people have taken the position that the death penalty is only justifiable as a means of preventing murder. Thus, the question about deterrence cuts to the heart of the death penalty's reason for existence.

Wolfers critiques both Justice John Paul Stevens, writing for the dissent, and Justice Antonin Scalia, writing for the majority. Stevens wrote that, "In the absence of such evidence, deterrence cannot serve as a sufficient penological justification for this uniquely severe and irrevocable punishment". To this, Wolfers appropriately responds, "the absence of evidence should not be confused with the evidence of absence." Just because we cannot find a deterrent effect doesn't necessarily mean it doesn't exist.

The more consequential critique is of Scalia's decision, in which the Justice writes, "a significant body of recent evidence" shows "that capital punishment may well have a deterrent effect, possibly a quite powerful one." Wolfers responds that Justice Scalia may want to update his reading list to include the large body of research contradicting his claim.

Since in my last post on this topic I discussed the statistical reasons why an answer to the deterrence question remains elusive, I thought it was important to follow up with the other factor hindering the debate: ideology. Justices Stevens and Scalia differ on the issue of the death penalty, for reasons that go beyond the issue of deterrence, and these differences affect how they filter new evidence. Maybe Stevens and Scalia we legitimately swayed by virtues of different econometric techniques. But my guess is that this is the result of "confirmation bias". According to the idea of confirmation bias, people have a tendency to seek out information that supports their preexisting world view. In statistical analysis, researchers may dismiss contradictory findings on methodological grounds or only report the versions of their results that worked and not the ones that didn't.

Supreme Court judges aren't the only ones who fall pray to this bias. It is often hard to find an academic who's willing to change his/her position because of contradictory empirical evidence. In his 2006 testimony to congress, economist (and deterrence proponent) Paul Rubin states that:

“Another recent paper by Lawrence Katz, Steven D. Levitt, and Ellen Shustorovich uses state-level panel data covering the period 1950 to 1990 to measure the relationship between prison conditions, capital punishment, and crime rates….In several estimations, both the prison death rate and the execution rate are found to have significant, negative relationships with murder rates…”

However one of the paper's authors, Steven Levitt, disagrees with this assessment of his work. According to Levitt, he and his co-authors ran 20 different models (containing different mixes of variables), and found "significant, negative relationships" in only 3. Generally if the results of a statistical model are so sensitive to the model's specification, the relationship is not that strong or may not exist at all.

Unfortunately there's not much we can do about confirmation bias, except to be open about it. It takes a lot of effort to approach issues with a clear and open mind. But it's important to try; sometimes it's an issue of life and death.

Lies, damn lies and statistics: death penalty edition

My blog partner recently posted a terrific piece on this week's Supreme Court ruling regarding the death penalty and child rape. I thought this was a good chance to talk about how difficult it can be to parse the data on such complex social issues. Warning: this gets a little technical, so please fight the urge to fall asleep.

In discussing the evidence on the deterrent effect of the death penalty on crime, she cites an Amnesty International fact sheet, which provides two graphs showing murder rates in states without the death penalty to be lower than those with the death penalty:

So we should conclude that not only does the death penalty not have a deterrent effect, but that it actually increases murders, right? Well, not really. These graphs show a clear correlation between the death penalty and murder rates, but they provide no information about which causes which. This type of analysis suffers from problems of "endogeneity", where you have unclear or two-way causality. So while the death penalty could be causing differences in murder rates, it is also possible that causality runs the other way; that is, states with already higher murder rates felt the need to implement the death penalty in order to reduce crime. As a result, we need more sophisticated statistical techniques to determine exactly what is happening with the data.

To overcome the endogeneity problem, empirical papers have used an "instrument variables" approach. An instrument variable is an independent variable that is correlated with the independent variable of interest in the model (in this case the use of the death penalty), but is otherwise uncorrelated with the dependent variable (in this case the murder rate). This will control for the endogenous effect of independent variable. Here's a technical explanation of how this works.

Unfortunately, as Justin Wolfers and John Donohue point out, finding a good instrument variable is not so easy. For example, one well known empirical paper on this topic used "percent of Republican voters in the previous election". There's good reason to think that this would be correlated with the explanatory variable (i.e. the death penalty), but there's also good reason to think that this would be otherwise correlated with the murder rate, violating one of the rules for instrument variables. Republicans tend to take tougher stances on crime, favoring harsher sentencing laws and a stronger police presence. As a result, states with more Republican voters may have lower murder rates, but it might not have to do with their use of the death penalty.

Why does this need to be so complicated? Well the only real way to identify the effect of the death penalty is to run a controlled experiment where, for example, you randomly assigned some states to have the death penalty and some not to. That way, the only difference between the groups of states is the death penalty. Of course, we can't do this. So we have to resort to observational studies, despite their difficulty in identifying the direction of causality.

So what does the research say? The evidence is mixed. There's a very interesting discussion between Justin Wolfers, John Donohue and Paul Rubin in The Economists' Voice, an excellent collection of essays on a wide range of public policy issues. The authors debate the results of several papers on the issue of deterrence, all of which contain conflicting results. Wolfers and Donohue, looking at the longest dataset of all the papers involved, find no net lives saved per execution. While Rubin and others disagree with this finding, I think that the preponderance of evidence in this area suggests that the death penalty has no consistent or significant deterrent effect. Even Gary Becker, Nobel Laureate and proponent of the death penalty deterrent theory, notes that, "the evidence is decidedly mixed...the weight of the positive evidence should not be overstated."

When faced with morally charged issues like the death penalty, many people look to empirics to settle the debate. While I feel that that impulse is the right one, there's a danger in overstating or oversimplifying the results of empirical research in the social sciences. Social systems are enormously complex, and sifting through reams of data requires a great deal of time, patience and humility. Statistics has a lot to teach us about politically charged issues like the death penalty; we just shouldn't be in a rush to judge what it is they have to say.