Hazards of statistical inference, Part … ah who knows anymore?

The New Yorker has a very interesting piece about the hazards of publication bias and skewed data in the hard sciences, particularly medicine and biology.  Recommended.

Apparently there is a phenomenon in which effect sizes for established causal correlations decrease over time.  That is, early studies establishing correlations tend to report much higher effect sizes than later replication studies, and it does not appear this is the result of regression toward the mean.  Probably the result of unintentional skewing of data by researchers and of publication bias.


About Jake Wobig

I teach international relations and comparative politics at Wingate University in Wingate, North Carolina
This entry was posted in Philosophy of Science and Epistemology, Statistics. Bookmark the permalink.

2 Responses to Hazards of statistical inference, Part … ah who knows anymore?

  1. Amanda B says:

    Replication is tough, so if the effect is in the expected direction and is of at least minimal size (.15 or above), I’m not sure that’s a huge problem. But I guess it depends on what we are asking. If the effect is that 10 percent of cancer patients are “cured” and 90% are not, we may be willing to accept that. But if 90% are adversely affected, that’s another story.

    As for political science, most scholars seem to ignore or just not know about effect sizes so they are rarely reported.

  2. Most of our top journals should be very, very ashamed for not requiring reportage of effect sizes, or standardized regression weights at the very least. I’m to the point that I don’t really believe anything in a large-n test unless effect sizes are reported. This disbelief is positively correlated with the prestige of the journal (p < .0000001).

    This is especially a problem with large samples such as the ANES, but is even worse for those studies/researchers that have a lot of money for their work. A large enough n will make almost any random variation reach statistical significance. For the sake of the discipline we need to stop with the asterisk frenzy; it's a ridiculous race to the bottom and acts as a detriment to our ability to explain the world (and, as a result, our ability to explain our utility to the world).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s