I’ve always wondered whether the rigorous application of statistics is underutilized in the social sciences. This is less so a problem in economics, where the subject is, by nature, highly quantitative. But in fields like psychology, sociology, and political science, where a background in mathematics is not common (unlike for biology, chemistry, and physics), researchers can intentionally or, very often, unintentionally (this is a really good Economist article) produce wrong results by abuse or misunderstanding of statistical inference.
As an onlooker whose training is in mathematics, I cannot help but to feel frustrated by the lack of numeracy in our “scientists.” The Economist article does a good job at showing how failure to understand statistical concepts leads to false results being published, even past peer review.
What triggered me to write this post was an assigned reading for a comparative politics class. In it, Adam Przeworski discusses the inherent selection bias in matching countries for experimentation. Noting that democracies have higher economic growth rates than authoritarian regimes, Przeworksi brings in the relevant data that democracies have a significant chance to die off when faced with economic failure whereas authoritarian regimes are not as affected. Hence, observing that democracies have higher growth rates does not signify that democracy leads to economic growth, but rather that economically failing democracies are not observed because they tend to disappear.
“What we are observing here is what the statistical literature calls ‘selection bias.’ Indeed, I am persuaded that all the comparative work we have been doing may suffer potentially from selection bias.” (p. 19, stable JSTOR link)
In context of a comparative politics theory symposium, this makes a lot of sense to state. But the phrasing is really interesting to a math person: selection bias is a given, and is one of the tools we use to analyze anything. My instinctual reaction to the reading was “Duh, obviously there is selection bias.” While I am sure the field of comparative politics is more aware of selection bias than Przeworski makes it appear to be, the fact that Przeworski framed it as such (“what the statistical literature calls ‘selection bias'”), as if to imply that the formal tools of statistical inference are generally beyond the scope of comparative politics theory, is a bit unnerving.
Przeworski, Adam in The Role of Theory in Comparative Politics: A Symposium, World Politics, Vol. 48, No. 1 (Oct., 1995), pp. 1-49.