ON EQUITABLE NON-ANONYMOUS REVIEW Abstract. Remco Heesen has recently argued in favor of the editorial practice of triple-anonymous review on the grounds that \an injustice is committed against certain authors" under non-anonymous review. On the other hand, he concedes that the information waste of triple-anonymous review does handicap editors, in particular sacricing a boost in the average quality of accepted papers that would otherwise be conferred by non-anonymous review. In this paper it is observed that by devoting comparatively greater reviewing resources to the papers of unfamiliar authors, editors practicing non-anonymous review can, without loss of information, avoid subjecting authors to the sorts of injustices observed by Heesen. Thus they can reap the eciency gains of non-anonymous review without sacricing fairness. 1. Introduction Remco Heesen (2018) invites us to consider a simplied scholarly1 community having a single journal. Scholars submit papers (one each, chosen at random from their output) to the journal's editor. Some scholars tend to produce better papers than others. Let i be the mean quality of a paper produced by scholar i, and assume that i  N(0; 1). That is, if a scholar is chosen at random, then the mean quality of their papers lies in a standard normal distribution.2 Papers by a xed scholar meanwhile are assumed to lie in a normal distribution of variance 1. So, if qi is a random paper produced by scholar i then qi  N(i; 1). The editor sends each submitted paper out for review, obtaining from the reviewers an aggregate estimate ri of the quality qi of the paper. Suppose that ri  N(qi; 1). (In particular, the score is unbiased; its expectation is the actual quality of the paper.) Assume that there is a threshold q > 0 such that the editor accepts a submitted paper just when her posterior expectation of its quality is above q; for simplicity I'll take q = 2. Now it happens3 that E(qijri) = 23ri. Indeed, this is a consequence of the following theorem. (Take W = qi, so that W = 0 and  2 W = 2, and Z = ri  qi, so that 2Z = 1 and x = W + Z = ri.) 1Heesen speaks of \scientic" communities rather than \scholarly" communities, but the issues he addresses are relevant more generally. 2I am assuming a variance of 1 for each of the random elements that aect the reported value ri; Heesen makes no such assumption, but I don't see any reason not to, as it simplies the presentation without sacricing any of the qualitative features that make the model interesting. 3An odd feature of this model is that the reviewer herself (should she know the relevant variances) would, post-review, put the expectation of the paper's quality at 2 3 ri{even though it was she that estimated the quality of the paper to be ri! So there is a supposition here that reviewers are not making this sort of post-mortem \regression to the mean" correction prior to submitting their scores. (I don't nd this supposition realistic; real-life reviewers are conservative; tending in particular to underestimate the quality of strong papers.) This observation can be glossed over, though, for if reviewers are doing the post-mortem prior to report and the editor knows this then she can reconstruct the original \raw" estimate and proceed as in the text. 1 ON EQUITABLE NON-ANONYMOUS REVIEW 2 Theorem 1. Suppose that W  N(W ; 2W ) and Z  N(0; 2Z) are independent. Then conditional on W + Z = x, W  N( 2Z 2 Z +2 W W + 2 W 2 Z +2 W x; 2 W 2 Z 2 W +2 Z ). So if the editor's only evidence is the value ri, she will accept the paper if and only if ri > 3; since Var(ri) = 3, the probability of acceptance is therefore Z( p 3)  :0416. But the key feature of Heesen's model is that for some of the papers, the editor does have additional evidence{she is acquainted with the authors of these papers, in the sense that she knows, for these authors i, the corresponding values i. (She knows, for these authors, the average quality of their papers.) Again invoking Theorem 1 (with W = qiji, so that W = i and 2W = 1, and Z = ri   qi as before), E(qiji; ri) = i+ri2 . So for the group the editor will accept paper i if and only if i+ri 2 > 2. Since Var(i+ ri) = Var(2i)+Var(qi i)+Var(ri  qi) = 4+1+1 = 6, the probability of acceptance is therefore Z( 4p 6 )  :0512. Heesen concludes: \Thus I have established the surprising result that an editor who cares only about the quality of the papers she publishes may end up publishing more papers by her friends and colleagues than by scientists unknown to her, even if her friends and colleagues are not, as a group, better scientists than average." I contend that this result is not surprising. For conditionalization on any partition whatsoever would increase (strictly, in all but the extreme case of a partition independent of the conditional expectation of the quality of the paper with respect to the algebra generated by the existing review) the variance of the editor's posterior expectation of the paper's quality.4 And while this does not guarantee an increase in the probability of acceptance in general,5 it does guarantee this if the editor's posterior expectation of the paper's quality is itself assumed to have a normal distribution. (As is the case in Heesen's model.) So the phenomenon on display here is more general. What causes the acceptance rate in one of the two pools of papers to be higher is simply that the editor has more information about the papers in that pool. Heesen opines: \On the one hand, the editor is simply making maximal use of the information available to her. It just so happens that she has more information about (scholars) she knows than about others. But that is hardly the editor's fault. Is it incumbent upon her to get to know the work of every (scholar) who submits a paper? This may well be too much to ask." So although he ultimately concludes that the editor is \ethically and epistemically culpable" for the injustice that some scholars suer under this system and implicitly recommends that editors should correct this injustice, his language suggests that the editor's role in the injustice is passive. 4The general claim is that if X is a random variable on probability space ( ; ) with E(X) = 0 and A is a ner -algebra than B then R E(XjA)2 d  R E(XjB)2 d. For RHS = R   R X dy 2 d(y) = R   R R X dyz dy(z) 2 d(y)  R R   R X dyz 2 dy(z) d(y) = LHS. (Here (y) is the decomposition of  over B and (yz) is the decomposition of y over A.) 5Here is a counterexample. Imagine that the editor learns the truth or falsity of the event qi > 1:9, and nothing else. Then she will accept the paper if and only if qi > 1:9. But if she learned the true quality, she would accept the paper if and only if qi > 2. So here learning more yields a lower probability of acceptance. ON EQUITABLE NON-ANONYMOUS REVIEW 3 Against this, I would say that the editor's role is active. To see why, assume for the sake of convenience that in the original model two reviewer scores were solicited and averaged in the computation of ri. (The variance of each of these scores is 2 conditional on qi and they are independent conditional on qi, so their average, ri, has variance 1 conditional on qi, as described above.) Then she could have achieved fairness by sending papers written by familiar authors out for review only once. For in that case Theorem 1 (with W = qiji, so that W = i and 2W = 1, and Z = ri   qi, so that 2Z = 2) would yield a posterior expectation of paper quality equal to 2i+ri 3 = i + 1 3 (ri   i). So the editor would accept the paper if and only if i + 1 3 (ri   i) > 2. Since Var(i + 13(ri   i)) = 43 , the implied acceptance rate is Z( p 3)  :0416, the same as the acceptance rate in the group of unfamiliar authors.6 What makes the editor's role active, then, is that she chose to send the papers by the familiar authors out for review a second time. Yes, it \just so happened" that she started out having more information about papers by scholars that she knows. But it didn't \just so happen" that things ended up that way. That was her doing. Concomitantly I disagree with Heesen's recommendation, which is elucidated in: \But an alternative option is to remove all information about the authors of submitted papers. This can be done by using a triple-anonymous reviewing procedure, in which the editor is prevented from using information about (scholars) she knows in her evaluation." I don't dispute that this levels the playing eld (in theory), but the cost in wasted information (a debt that must ultimately be cashed out in either a decrease in quality of accepted papers or the additional sweat of reviewers) is prohibative. A more ecient way to equalize the prospects of familiar and unfamiliar authors is to expend comparatively fewer reviewing resources on papers by familiar authors. This argument doesn't require that the editor is the oracle that Heesen's model makes her out to be. Its recommendations stand (with weaker eect) in a case where the editor's estimates of the i are (normally distributed and) unbiased. Since few real-life editors approximate even this premise, the argument doesn't advocate strongly for non-anonymous review in practice. I can't resist mentioning however that mathematics uses non-anonymous review and it and it seems to work extremely well, whereas philosophy uses anonymous review and it seems to work extremely badly. Putting aside endish hypotheses as to why, Heesen's work indicates a purely statistical one...even when review is ocially triple anonymous, editors will still know some authors' identities{comparatively more often for \well-connected" authors. The ocial reason we give for shopping a paper at talks prior to submission is that such vetting might improve the papers. Heesen's work suggests a more cynical motive, however...to subvert the \anonymity" of review. This suggests that (even double) anonymous review may act more as cover for connection bias than as solution to it. References Heesen, Remco. 2018. When journal editors play favorites. Philosophical Studies 175:831-858. 6It may also be worth noting that the posterior variances of the qi are equal (they are both 2 3 ) in the two groups, so the expected quality of an accepted paper will be independent of familiarity.