Are liberals more biased in social science?
Matthew Martin 6/22/2015 08:52:00 AM
On twitter, Avik Roy pointed out this post on a study that performed an experiment on how biased people are when reading social science research papers, and found that the bias was much higher in liberals than conservatives.
Here's the experiment. Two groups were each shown two papers. In the first group, they gave subjects a research paper that concluded that affirmative action was bad for black people and a paper that concluded that same-sex relationships were just as healthy as opposite-sex relationships. The second group was shown essentially identical papers, but with the conclusions (and presumably data, etc) reversed so that they showed that affirmative action was good for black people and that same-sex relationships were less healthy than opposite-sex relationships. The result:
"People were asked to indicate how true they considered the article to be, and how biased they considered the author to be.Liberals were far more likely than conservatives to label the papers whose results they agreed with as true and unbiased.
And the resounding answer was: Liberals were far more biased. Liberals viewed the articles reporting "liberal" results (affirmative action and same sex relationships are good) as truer and reflecting less author bias than the articles reporting "conservative" results. Conservatives, in contrast, viewed the truthfulness and bias in the articles as nearly identical, regardless of their results."
Ok, this is a bit weird. I don't think anyone would seriously suggest readers should accept a result as "true" based on one paper, even if the methodolgy looked sound. In science, replication is everything. There are one-off results, there is fake data, there are typos, there is p-hacking. I do think people are inadequately critical of papers whose results they agree with--weak methods should fail to revise our priors, not strengthen them--but this study asked respondents whether they thought the result was "true" not (as far as I can tell from the description) to list out criticisms of the methods. Criticisms of methods should be independent of our priors, but as for assessing "truth" of the result, a rational Bayesian updater would require more evidence to revise a strong prior belief than a weak prior belief because, as they say, "extraordinary claims require extraordinary evidence."
Hence, rational, critical evaluators should be somewhat "biased" towards their prior beliefs. At issue in this experiment was the fact that the "bias" was asymmetric across groups despite the fact that both groups saw evidence of identical strength. Under the assumption of rationality, this means Liberals had stronger prior beliefs. The authors claims that this means liberals are more "biased."
But take a look at the topics chosen. I've never heard a conservative argue that affirmative action was harmful to the black people it is designed to help. That's an extraordinary claim for any part of the political spectrum. And while there are far-right factions that do argue that gay relationships are less healthy than straight ones, this is not generally a strongly held belief among conservatives. On both questions, the authors chose topics about which conservatives have fairly weak priors that don't necessarily even correspond to the belief that the authors label as "conservative." But these are two questions about which most liberals do have fairly strong prior beliefs.
Thus, I'm left to wonder whether liberals really are more biased in general--that is, have stronger beliefs about most things--or whether the authors merely singled out two questions on which liberals have stronger priors.
I don't really think that liberals and conservatives have exactly equal amounts of "bias." But since I have no prior beliefs about which group is more biased, I take equivalence as the default until I come across good enough evidence to revise that prior. This article failed to revise my priors.