Skip to main content
Log in

Acceptance, Aggregation and Scoring Rules

  • Original Article
  • Published:
Erkenntnis Aims and scope Submit manuscript

Abstract

As the ongoing literature on the paradoxes of the Lottery and the Preface reminds us, the nature of the relation between probability and rational acceptability remains far from settled. This article provides a novel perspective on the matter by exploiting a recently noted structural parallel with the problem of judgment aggregation. After offering a number of general desiderata on the relation between finite probability models and sets of accepted sentences in a Boolean sentential language, it is noted that a number of these constraints will be satisfied if and only if acceptable sentences are true under all valuations in a distinguished non-empty set W. Drawing inspiration from distance-based aggregation procedures, various scoring rule based membership conditions for W are discussed and a possible point of contact with ranking theory is considered. The paper closes with various suggestions for further research.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Jeffrey Sanford Russell, John Hawthorne & Lara Buchak

Notes

  1. Note that the assumption made here, that acceptability is a function of an underlying probability model, whilst commonplace in the literature, is not entirely uncontroversial. It rules out, for instance, the views that the acceptability of a sentence depends also depends on the practical payoffs associated with true/false negatives/positives (see Rudner 1953) or again that it is relative to a specific question, modeled as a partition of the language (Levi 1967). A discussion of these issues is however beyond the scope of the present paper.

  2. In what follows I will be using the expressions ‘full belief in the truth of’ and ‘acceptance of’ interchangeably.

  3. The name for this constraint originates in Chandler (2010), and derives from the fact that were it to be violated, Structurality and Deductive Closure could be marshaled, lottery paradox-style, to yield a violation of Zero-Normalisation. The lottery paradox -proofness involved was dubbed ‘weak’ in the original paper for reasons that do not apply to the current model.

  4. In fact, Douven and Williamson prove a slightly stronger result, making use of a principle that its strictly weaker than Deductive Closure, namely:

    • Aggregativity: For any probability model \(\mathcal{M}=\langle\mathcal{L},\hbox{Pr}\rangle\) and sentences \(\varphi,\psi\in\mathcal{L}\), if \(\hbox{Acc}(\mathcal{M},\varphi)=1\) and \(\hbox{Acc}(\mathcal{M},\psi)=1\), then \(\hbox{Acc}(\mathcal{M},\varphi\wedge\psi)=1\).

  5. The reader familiar with the judgment aggregation literature will notice a considerable amount of simplification going on here. For instance, I am rather severely restricting the class of possible opinion models, which, in the aggregation-theoretic literature, notably involve opinion profiles that are tuples of possibly inconsistent sets of sentences in \(\mathcal{L}\). For reasons that will become clear shortly, these expository niceties can be dispensed with.

  6. See most notably Theorem 2 (a) of Dietrich and List (2008), in which, in contrast to many other previous results, Opinionation plays no role.

  7. Distance-based approaches to aggregation are also more recently discussed in Miller and Osherson (2009) and Pigozzi (2006).

  8. On this option, to ensure that the set of selected valuations isn’t empty, one could specify that, in the event that all valuations are further than t, all valuations are selected.

  9. Strictly-speaking, this is not a distance, as the triangle inequality d(vv *) + d(v *v **) ≥ d(vv **) need not be respected.

  10. In fact, Konieczny et al. use the non-normalised counterpart; the normalisation is introduced here to harmonise with the range of d D .

  11. Clause (i) is crucial here. Without it, in the event that stage (b) allows for W to be the empty set, we would have a violation of Zero-Normalisation, since, obviously, \(\varnothing\subseteq[\![ \varphi ]\!]\), for all \(\varphi\in\mathcal{L}\), including contradictions.

  12. Indeed, where \(c\in (0,1]\) is the degree of confidence under consideration, simply construct a model such that a single most probable valuation v receives a probability of c and the remainder of the probability mass is evenly distributed among a sufficient number of alternative valuations for each of these to receive a probability that is strictly inferior to c. It will follow that \(\varphi_{v}\) is accepted, although \(\hbox{Pr}(\varphi_{v})=c\).

  13. It is worth remarking that some noteworthy scoring rules satisfy ≥Pr-Reversal but violate D-Independence. A case in point is the normalised counterpart D N log of the logarithmic score:

    $$ D_{N \log}(v,\hbox{Pr}):=-\log\left(\frac{\hbox{Pr}(\varphi_{v})} {\max_{v^{\ast}\in V}\hbox{Pr}(\varphi_{v^{\ast}})}\right) $$

    As an anonymous referee has remarked, the use of this rule in conjunction with Sat yields a proposal due to Levi (1996, p. 289), to the effect that the strongest sentence that one accepts is the one that is validated by all and only those valuations whose probability ratio with respect to the most probable valuations is high enough. Like Acc1, this suggestion allows for acceptance of sentences with arbitrarily low probability.

  14. Both suggestions were very briefly mentioned by Chandler (2010), where the connection with distance-based aggregation functions had not been drawn.

  15. This is of course not a suitable occasion to address the controversial issue of the normative status of a requirement of translation-invariance. The issues at play are complex and, as far as I can see, remain unresolved at this point. See Zwart (2001), chapter 5 for a detailed overview of the debate.

  16. In the original draft of this paper, I had suggested that the issue of translation-sensitivity of distance-based methods had been overlooked in the judgment aggregation literature. As an anonymous referee pointed out to me, however, I was wrong. Indeed, the issue is in fact briefly discussed in a recent piece by Cariani et al. (2008). There, they first offer a theorem (Theorem 5, p. 17) to the effect that translation sensitivity is a property, not only of the Hamming distance, but of any distance measure that is not ‘trivial’, in the following sense:

    A measure d of distance between valuations is trivial iff there exists \({r\in \mathbb{R}^{+}}\) such that, for all \(v, w\in V\), d(vw) = r × d D (vw).

    After presenting this result, they then go on to say:

    In short, only trivial distance measures are translation-invariant… Thus any judgment aggregation procedure that depends on a non-trivial distance measure will fail translation-invariance.

    Now the inference from translation-sensitivity of the distance measure to translation-sensitivity of the corresponding aggregation procedure seems basically correct, given certain assumptions about the latter. But these assumptions are not provided and neither is the precise derivation: a little more work is required to establish the result, as Cariani et al. have acknowledged in recent correspondence.

  17. The restriction to finite models is important here. Universal quantification over all probability models—finite or otherwise—would yield a principle that is incompatible with the conjunction of Aggregativity and Zero-Normalisation. Note, furthermore, that Consensus Preservation entails Responsiveness.

  18. Note that the notion of a ‘refinement’ is not to be understood dynamically here, as an diachronic alteration of a specific agent’s state of mind, resulting from, say, an expansion of his or her conceptual repertoire or a shift in his or her awareness. Rather, the term is simply intended to pick out a certain atemporal relation between probability models.

  19. When t > 0, since for t = 0, Independence is satisfied.

  20. As an anonymous referee has pointed out to me, there may be grounds to hold that conservative refinements are perhaps not as doxastically neutral as I have suggested and hence that a requirement of preservation of acceptability under such refinements may not be in order. Indeed, the standard Bayesian suggestion of modeling a lack of opinionation with respect to a partition P by a uniform probability distribution over P, faces a number of apparent difficulties: Bertrand-style paradoxes, counterintuitive prescriptions in Ellsberg’s urn decision problem, and so on. Whilst I do share the referee’s worries here, a constraint of preservation under conservative refinement remains the best that can be achieved within the orthodox Bayesian framework that was assumed from the outset of this paper.

  21. One should note the strong constraint imposed on the relation of model refinement by our endorsement of a linguistic, rather than algebraic, framework. Indeed, the fact that model expansion must proceed here by addition of atomic sentences enforces what one might call a ‘homogenous’ refinement. This would correspond, in the algebraic setting, to a ‘splitting’ of each of the atoms in the algebra of the coarser model into equal numbers of ‘daughter’ atoms. Without this equicardinality restriction, it is easy to see that the conjunction of Lottery-Proofness and even the weakened version of Preservation under Refinement, restricted to conservative refinements, would lead us into trouble.

  22. This is the definition given in Spohn (2009). Just as the standard ratio definition for conditional probability precludes conditionalisation on probability 0 sentences, it precludes conditionalisation on rank \(\infty\) sentences. There are however alternative accounts of both conditional probabilities and conditional ranks that, rightly or wrongly, waive this prohibition. The former will presumably be well-known to the reader. Regarding the latter, we have the following proposal from Huber (2009):

    $$ \kappa(\psi \mid \varphi)= \left\{\begin{array}{ll} \kappa (\psi \wedge \varphi) - \kappa (\varphi) & \hbox{if }\psi\,\nvdash\bot;\\ \infty & \hbox{if }\psi\vdash\bot.\end{array}\right. $$

    The reason for setting the conditional rank to \(\infty\) in case \(\psi\vdash\bot\) is presumably to prevent ψ from receiving a rank of 0 upon conditionalisation on a sentence \(\varphi\) of rank \(\infty\) (since we would then have \(\kappa(\psi \wedge \varphi) - \kappa (\varphi) = \infty - \infty = 0\)), and having \(\kappa(\cdot\mid\varphi)\) violate clause (i) of Definition 5.1.

    Somewhat curiously, however, note that here, contrary to what was the case in Definition 5.2, we no longer have the result that \(\kappa(\psi\mid\neg\psi)=\infty\). Indeed, let \(\kappa(\neg\psi)=\infty\) and \(\psi\,\nvdash\bot\). By the above proposal, \(\kappa(\psi\mid\neg\psi)= \kappa(\psi\wedge\neg\psi) - \kappa(\neg\psi) = \infty - \infty = 0\). This result could however be avoided by simply swapping \(\psi\vdash\bot\) (resp.\(\psi\,\nvdash\bot\)) for \(\psi\wedge\varphi\vdash\bot\) (resp. \(\psi\wedge\varphi\,\nvdash\bot\)) in the above definition.

  23. In Definition 5.1, we took the range of κ to be \({\mathbb{R}^{+}\cup \{\infty\}}\). In some publications, however, including the one from which this quote was taken, the range is stated to be \({\mathbb{N}\cup \{\infty\}}\). But this second option obviously doesn’t square with the conjecture that ranks are logarithms of potentially real-valued probabilities.

  24. Similar comments apply to D log’s normalised counterpart, mentioned in footnote 13 above.

  25. I owe both this example and the general observation that it supports to an anonymous referee.

References

  • Cariani, F., Pauly, M., & Snyder, J. (2008). Decision framing in judgment aggregation. Synthèse, 163, 1–24.

    Article  Google Scholar 

  • Chandler, J. (2010). The lottery paradox generalised? British Journal for the Philosophy of Science, 61(3), 667–679.

    Article  Google Scholar 

  • Dietrich, F., & List, C. (2008). Judgment aggregation without full rationality. Social Choice and Welfare, 31(1), 15–39.

    Article  Google Scholar 

  • Douven, I., & Romeijn, J.-W. (2007). The discursive dilemma as a lottery paradox. Economics and Philosophy, 23, 301–319.

    Article  Google Scholar 

  • Douven, I., & Williamson, T. (2006). Generalizing the lottery paradox. British Journal for the Philosophy of Science, 57(4), 755–779.

    Article  Google Scholar 

  • Hilpinen, R. (1976). Approximate truth and truthlikeness. In M. Przelecki et al. (Eds.), Formal methods in the methodology of the empirical sciences (pp. 19–42). Dordrecht: Reidel.

    Chapter  Google Scholar 

  • Huber, F. (2009). Belief and degrees of belief. In F. Huber, & C. Schmidt-Petri (Eds.), Degrees of belief (pp. 1–33). New York: Springer Synthèse Library.

    Chapter  Google Scholar 

  • Konieczny, S., Lang, J., & Marquis, P. (2004). DA2 merging operators. Artificial Intelligence, 157(1–2), 49–79.

    Article  Google Scholar 

  • Levi, I. (1996). For the sake of the argument: Ramsey test conditionals, inductive inference and non-monotonic reasoning. Cambridge: Cambridge University Press.

    Book  Google Scholar 

  • Levi, I. (1967). Gambling with truth. New York: Knopf.

    Google Scholar 

  • Miller, D. (1974). On the comparison of false theories by their bases. British Journal for the Philosophy of Science, 25, 178–188.

    Article  Google Scholar 

  • Miller, M., & Osherson, D. (2009). Methods for distance-based judgment aggregation. Social Choice and Welfare, 32, 575–601.

    Article  Google Scholar 

  • Pigozzi, G. (2006). Belief merging and the discursive dilemma: An argument-based account to paradoxes of judgment aggregation. Synthèse, 152, 285–298.

    Article  Google Scholar 

  • Rudner, R. (1953). The scientist qua scientist makes value judgments. Philosophy of Science, 20(1), 1–6.

    Article  Google Scholar 

  • Smith, M. (2010). A generalised lottery paradox for infinite probability spaces. British Journal for the Philosophy of Science, 61(4), 821–831.

    Article  Google Scholar 

  • Spohn, W. (2009). A survey of ranking theory. In F. Huber, & C. Schmidt-Petri (Eds.), Degrees of belief (pp. 185–228). New York: Springer Synthèse Library.

    Chapter  Google Scholar 

  • Tichý, P. (1974). On Popper’s definitions of verisimilitude. British Journal for the Philosophy of Science, 25, 155–160.

    Article  Google Scholar 

  • Tichý, P. (1976). Verisimilitude redefined. British Journal for the Philosophy of Science, 27, 25–42.

    Article  Google Scholar 

  • Zwart, S. D. (2001). Refined verisimilitude. New York: Springer Synthèse Library.

    Google Scholar 

Download references

Acknowledgments

I am grateful to the members of the Formal Epistemology Project, KU Leuven, and to the audience of PROGIC 2009, Groningen for useful feedback on earlier versions of this paper. I am also indebted to two anonymous referees for this journal for the time and trouble that they took to provide exceptionally detailed and insightful reports. The work carried out by one of these referees, in particular, went way beyond the call of duty. Part of the research for this article was funded by a Research Foundation—Flanders (FWO) postdoctoral research grant.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Jake Chandler.

Appendix

Appendix

Proof of Theorem 2.1

Assume Non-Unanimity. It follows that there exists a model \(\mathcal{M}=\langle\mathcal{L},\hbox{Pr}\rangle\) and sentence \(\varphi\in\mathcal{L}\) such that \(\hbox{Pr}(\varphi)=p<1\) but \(\hbox{Acc}(\mathcal{M},\varphi)=1\). Since, as is well-known, there exists a rational number between any two distinct real numbers, there exists a rational number q = m/n (with \({m, n \in \mathbb{N}}\)), such that p < q < 1. Let \(\mathcal{M}^{\ast}=\langle \mathcal{L}^{\ast}, \hbox{Pr}^{\ast}\rangle\) be a uniform probability model, such that the cardinality of the set of valuations of \(\mathcal{L}^{\ast}\) is equal to m. Let ψ denote an arbitrary sentence validated by exactly n of the valuations. Now by Lottery-Proofness, \(\hbox{Acc}(\mathcal{M}^{\ast},\psi)=0\), since \(\hbox{Pr}(\varphi_{\{w_{1},\ldots,{w_{m}}\}})=q<1\). By Monotonicity, however, \(\hbox{Acc}(\mathcal{M}^{\ast}, \varphi_{\{w_{1},\ldots,{w_{m}}\}})\geq \hbox{Acc}(\mathcal{M},\varphi)=1\). Contradiction. \(\square\)

Proof of Theorem 2.2

Let \(S:=\{\varphi_{W}: W\subseteq V\}\) and \(\varphi_{W}\succeq\varphi_{W^{\ast}}\) iff \(W^{\ast}\subseteq W\). Let L be the lattice \(\langle S, \succeq \rangle\). Let \(B\subseteq S\) denote the set of \(\varphi\in S\) s.t. \(\hbox{Acc}(\mathcal{M}, \varphi)=1\). Zero-Normalisation is true iff \(\varphi_{\emptyset}=\bot\notin B\). Unit-Normalisation and Deductive Closure are true iff B is a filter of L. So the three conditions are true iff B is a proper filter of L. But every proper filter is the intersection of a set of ultrafilters. For each of these ultrafilters u, there is some \(v\in V\) s.t. \(u=\{\varphi\in S : v(\varphi)=1\}\). \(\square\)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chandler, J. Acceptance, Aggregation and Scoring Rules. Erkenn 78, 201–217 (2013). https://doi.org/10.1007/s10670-012-9375-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10670-012-9375-6

Keywords

Navigation