Persistent Experimenters, Stopping Rules, and Statistical Inference

Abstract

This paper considers a key point of contention between classical and Bayesian statistics that is brought to the fore when examining so-called ‘persistent experimenters’—the issue of stopping rules, or more accurately, outcome spaces, and their influence on statistical analysis. First, a working definition of classical and Bayesian statistical tests is given, which makes clear that (1) once an experimental outcome is recorded, other possible outcomes matter only for classical inference, and (2) full outcome spaces are nevertheless relevant to both the classical and Bayesian approaches, when it comes to planning/choosing a test. The latter point is shown to have important repercussions. Here we argue that it undermines what Bayesians may admit to be a compelling argument against their approach—the Bayesian indifference to persistent experimenters and their optional stopping rules. We acknowledge the prima facie appeal of the pro-classical ‘optional stopping intuition’, even for those who ordinarily have Bayesian sympathies. The final section of the paper, however, provides three error theories that may assist a Bayesian in explaining away the apparent anomaly in their reasoning.

Appendices

Appendix: Illustrative Example

The following concern the two hypotheses:

$$ \begin{aligned} &H_{1}: ch(Heads)=0.5\\ &H_{2}: ch(Heads)=0.6 \end{aligned} $$

For the Bayesian tests in Tables 5 and 6, the prior probabilities and decision utilities are such that $\frac{p_{2}\cdot (u_{r2}-u_{a2})}{p_{1}\cdot (u_{a1}-u_{r1})}=1$. (This is the critical likelihood-ratio threshold for rejecting the H ₁ hypothesis.)

The example optional stopping test is as follows: the test is stopped when H ₁—that the coin is fair—is rejected, or after 7 tosses, whichever happens first. Given the assumptions above, this means that a sample continues, $x=(t_{1},t_{2},\ldots )$ where $t_{i}\in \{H,T\}$ until the first t _n such that $\frac{Pr(x|H_{1})}{Pr(x|H_{2})}<1$. If the experiment has not already stopped before the maximum m trials, then it is stopped at this point, regardless of the test result. A summary of the outcomes is given in Table 5.

Table 5 Bayesian optional stopping test (stop when $\frac{Pr(x|H_{1})}{Pr(x|H_{2})}<1$, or n = 7)

We see that H ₁ is more likely than not to be rejected, whether or not it is true: The probability that H ₁ will be rejected, when it is true, is the Type I error, which is 0.727 (sum the ‘reject’ entries in column 2: 0.5 + 0.125 + 2 × 0.03125 + 5 × 0.007813 = 0.727). The probability that H ₁ will be rejected, when it is false, is 1 - Type II error, which is 0.855 (sum the ‘reject’ entries in column 3: 0.6 + 0.144 + 2 × 0.03456 + 5 × 0.008294 = 0.855).

Now compare the above error probabilities with those of the fixed 5-toss test in Table 6. (For instance, it might be the case that the sequence of tosses is x = TTHHH, and we want to entertain the possibility that the test performed was either the optional stopping test or the fixed 5-trial test.)

Table 6 Bayesian fixed 5-trial test

The type I error probability is substantially less for this fixed 5-trial test than for the optional stopping test (sum the ‘reject’ entries in column 2: 0.03125 × 16 = 0.5). The Bayesian draws the same inference for the outcome x = (TTHHH) in both cases, however.

Note that we get a different classical inference for the outcome x = (TTHHH) , depending on whether the outcome space was that associated with the optional stopping test defined above, X _O , or the outcome space associated with tossing the coin 5 times, X _F . For the first case, the relevant p-value, v ₁, of the outcome x = (TTHHH), is 0.688 (from column 2 of Table 5: 0.5 + 0.125 + 2 × 0.03125 = 0.688), while for the second, v ₁ is 0.5 (from column 2 of Table 6: 16 × 0.03125 = 0.5). Thus, if we are speaking of a typical classical test (where rejection hinges on the choice of α), the experimental outcome ‘speaks more’ for the rejection of H ₁ if the outcome space was X _F rather than X _O , because the p-value is smaller for X _F than X _O .

We might also consider the other sort of classical test mentioned above: testClass2 _X (x). Here too, the outcome x = (TTHHH) will lead to different decision results for our two outcome spaces X _O and X _F , for certain values of k. For instance, k = 1 might be deemed appropriate to the context. This would mean rejecting H ₁ just in case the ratio of p-values, $\frac{v1}{v2}$ is less than 1. This is indeed the case when the outcome space is X _F : v ₁ = 0.5 < v ₂ = 0.663 (where the v ₂ value is calculated from terms in column 3 of Table 6: 10 × 0.03456 + 10 × 0.02304 + 5 × 0.01536 + 0.01024 = 0.663). Thus H ₁ would be rejected. If the outcome space is X _O , however, v ₁ = 0.688 > v ₂ = 0.256 (where the v ₂ term is calculated similarly to the above) and so there would not be sufficient evidence to reject H ₁.

Proof: Errors for Optional Stopping Tests

Claim: Optional stopping tests (2 simple exhaustive hypotheses) with maximum m trials, designed to stop as soon as a ‘reject’ result is achieved (assuming that the rejection threshold ratio t remains constant), have greater or equal type I error than any fixed-trial test of up to m trials.

$$\ldots$$

Consider any fixed-n-trial test (n ≤ m) for which H ₁ is rejected if the relevant likelihood ratio <t. The rejection set is denoted here R.

To prove the claim above, it suffices to show that all the sequences in R, and perhaps more, are also in the rejection set, denoted $R^{\prime}$, for the optional-stopping test of maximum m trials that is designed to stop when the relevant likelihood ratio <t. If so, the type I error for the optional stopping test is clearly greater than (when $R\subset R^{\prime}$) or equal (when $R=R^{\prime}$) to the type I error for the fixed-n-trial test.

To show this, we appeal to a super-outcome space that is the set of all sequences of length m. Call this set M. Any sequence of length n ≤ m, which we’ll denote S ⁿ _i , is just a subset of M, i.e. the subset of length-m sequences that begin with the relevant length-n sequence. When n = m these sets S ⁿ _i contain only one length-m sequence, otherwise they contain more. Note too that $\bigcup\nolimits_{i}S_{i}^{n}=M$.

We denote the set of length-n trials S ⁿ _i that result in rejection, $\mathcal{S}_{R^{n}}$. The rejection region for the fixed-n-trial test, R, can be given as $R=\bigcup\nolimits_{S_{i}^{n}\in \mathcal{S}_{R^{n}}}S_{i}^{n}$. This is to say that R contains all length-m sequences that are elements of a set S ⁿ _i that is rejected.

Turn now to the optional stopping test. It will stop if any $S_i^n\in \mathcal{S}_{R^{n}}$ occurs, and the set of those that may occur, denoted $\mathcal{S}_{O^{n}}$, will be rejected. The remaining sequences $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}$ are those that cannot occur, which implies that the optional stopping test would stop on a rejection result before the first n trials for all length-n sequences in $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}$.

Now there are two cases to consider: (1) $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}\neq \varnothing$, and (2) $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}=\varnothing$.

First case: $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}\neq \varnothing$

All S ⁿ _i in $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}$ must themselves be subsets of smaller-length sequence sets that resulted in rejection. Call the set of these smaller-length sequence sets that resulted in rejection $\mathcal{S}_{S}$. Then $\bigcup\nolimits_{S_i^n\in \mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}}S_i^n\subset \bigcup\nolimits_{S_{i}\in \mathcal{S}_{S}}S_{i}$.

It follows that $\bigcup\nolimits_{S_i^n\in \mathcal{S}_{R^{n}}}S_i^n\subset \bigcup\nolimits_{S_{i}\in \mathcal{S}_{O^{n}}\cup \mathcal{S}_{S}}S_{i}$.

The latter term in the above expression is a subset of the full set of length-m sequences in the rejection region for the optional stopping test, which we denote $R^{\prime}$. (There may additionally be sequences of length greater than n that are also rejected (cf. case below).) The former term in the expression is R, as defined earlier.

So in this first case $R^{\prime}\supset R$, and thus the type I error for the optional stopping test is strictly greater than the type I error for the fixed-n-trial test.

Second case: $\mathcal{S}_{R^{n}}\backslash \mathcal{S}_{O^{n}}=\varnothing.$

It is easy to see here that the type I error for the optional stopping test is at least as great as the type I error for the fixed-n-trial test, since $\mathcal{S}_{O^{n}}=\mathcal{S}_{R^{n}}$.

Note, however, that $\bigcup\nolimits_{S_{i}^{n}\in \mathcal{S}_{O^{n}}}S_{i}^{n}\subseteq R^{\prime}$ . The reason the left-hand term (which = R) may be a proper subset of $R^{\prime}$ is that there may be other sequences that were accepted in the fixed-n-trial test that in fact also result in rejection in the optional stopping test. (These rejected trials will be such that the likelihood ratio for the first j trials is <t, where n < j ≤ m.)

So in this second case $R^{\prime}\supseteq R$ , and thus the type I error for the optional stopping test is greater than or equal to the type I error for the fixed-n-trial test.

$\ldots$

Corrollary Note also that the type II error for the optional stopping test with maximum m trials must be less than or equal to the type II error for the fixed-n-trial test where n ≤ m. This is because the type II error is the probability, given hypothesis H ₂, of $M\backslash R^{\prime}$ , for the optional stopping test, or else M\ R, for the fixed-trial test.

Since $R^{\prime}\supseteq R$ , then $M\backslash R^{\prime}\subseteq M\backslash R$ , and from this follows the stated corollary.

$\ldots$