“What data will show the truth?” is a fundamental question emerging early in any empirical investigation. From a statistical perspective, experimental design is the appropriate tool to address this question by ensuring control of the error rates of planned data analyses and of the ensuing decisions. From an epistemological standpoint, planned data analyses describe in mathematical and algorithmic terms a pre-specified mapping of observations into decisions. The value of exploratory data analyses is often less clear, resulting in confusion about what (...) characteristics of design and analysis are necessary for decision making and what may be useful to inspire new questions. This point is addressed here by illustrating the Popper-Miller theorem in plain terms and using a graphical support. Popper and Miller proved that probability estimates cannot generate hypotheses on behalf of investigators. Consistently with Popper-Miller, we show that probability estimation can only reduce uncertainty about the truth of a merely possible hypothesis. This fact clearly identifies exploratory analysis as one of the tools supporting a dynamic process of hypothesis generation and refinement which cannot be purely analytic. A clear understanding of these facts will enable stakeholders, mathematical modellers and data analysts to better engage on a level playing field when designing experiments and when interpreting the results of planned and exploratory data analyses. (shrink)

Gillies introduced a propensity interpretation of probability which is linked to experience by a falsifying rule for probability statements. The present paper argues that general statistical tests should qualify as falsification rules. The ‘goodness-of-fit paradox’ is introduced: the confirmation of a probability model by a test refutes the model's validity. An example is given in which an independence test introduces dependence. Several possibilities to interpret the paradox and to deal with it are discussed. It is concluded that the propensity interpretation (...) properly reflects statistical practice, but it is not as objective as some adherents claim. (shrink)