"If one cannot find a publishable p-value in one’s data—...—then one is being lazy."

During my classes and through reading countless academic papers, I am continuously bombarded with statistical regressions. Frequently the results are claimed to be “statistically significant” based on a small p-value. The implication is that these results should garner more attention and the hypotheses should be presumed true. Results of this kind typically lead to journal publications and often inform policy making.

Unfortunately, the basic intuition laid out above is technically incorrect and likely detrimental. In a recent manifesto, William Briggs displays the truth behind the meaning of p-values. He argues It is Time to Stop Teaching Frequentism to Non-statisticians, from which I offer a couple select passages:

The hunt for publishable p-values is nearly always fruitful. If one cannot find a publishable p-value in one’s data—with the freedom to pick and choose models and test statistics, to engage in “sub-group” and sequen-tial analysis, and so on—then one is being lazy. P-values can and are used to prove anything and everything. The sole limitation is the imagination of the researcher.

(p.4)

Civilians just can’t remember that it is forbidden in frequentist theory to talk of the probability of a theory’s or a hypothesis’s truth. They insist on translating the certainty they have in the value of some test statistic via the p-value to certainty that their hypotheses are true, despite that this is impossible to do so in frequen-tist theory. The result is that too many people are too certain of too many things.

(p.7-8)

The entire paper is worth reading, but the last sentence highlights my main concern. The desire for certainty in academia and policy making cannot overcome the reality of living in an uncertain world. Attributing excessive certainty to our results may increase our hypothesis’ chances of acceptance, but will not alter the likelihood of its being true in practice. In my view, there is a general need for greater humility, especially in academia and public policy. Classrooms are a great place to start.

(h/t Ryan Murphy @ Increasing Marginal Utility)