Supergrading: how diverse standards can improve collective performance in ranking tasks

Abstract

The method of supergrading is introduced for deriving a ranking of items from scores or grades awarded by several people. Individual inputs may come in different languages of grades. Diversity in grading standards is an advantage, enabling rankings derived by this method to separate more items from one another. A framework is introduced for studying grading on the basis of observations. Measures of accuracy, reliability and discrimination are developed within this framework. Ability in grading is characterized for individuals and groups as the capacity to grade reliably, accurately and at a high level of discrimination. It is shown that the collective ability of a supergrading group with diverse standards can be greater than that of a less diverse group whose members have greater ability.

Appendix

This section has proofs for technical claims in the main text, after repeating definitions of relevant notions introduced there.

1.1 Lemma 1

Let $L_{i}$ and $L_{j}$ be grade languages for $\langle V,\ge \rangle$. Then $L_{i} \circ L_{j}$ is a grade language for $\langle V,\ge \rangle$.

Definitions$\langle T, \succeq , I\rangle$ is a grade language for $\langle V,\ge \rangle$ if $\langle T, \succeq \rangle$ is a grade vocabulary (that is, $T \ne \emptyset$ is finite and $\succeq$ is a linear ordering of T); and $I:T \rightarrow \wp (V)$ is an interpretation of $\langle T, \succeq \rangle$ in $\langle V,\ge \rangle$, that is:

• $\forall e\in T$, I(e) is convex: $\forall u,v,w \in V$, if $u,w\in I(e)$ and $u \ge v \ge w$, then $v \in I(e)$;

• IpartitionsV: $\forall e \in T, I(e) \ne \emptyset$; $\forall e, f \in T$, if $I(e)\cap I(f) \ne \emptyset$ then $e = f$; and $\bigcup \{ I(e): e \in T\} = V$; and

• I is orderly: $\forall e, f \in T$, if $e \succ f$ then $I(e)> I(f)$ (that is: $\forall u \in I(e)$ and $\forall v \in I(f)$, $u>v$).

Where $L_{i} = \langle T_{i}, \succeq _{i}, I_{i}\rangle$ and $L_{j} = \langle T_{j}, \succeq _{j}, I_{j}\rangle$ are grade languages for a common $\langle V,\ge \rangle$, define:

• $T_{ij} = \{ \langle e,f \rangle : e \in T_{i}, f \in T_{j}$ and $I_{i} (e) \cap I_{j} (f) \ne \emptyset \}$,

• $\langle e,f \rangle \succeq _{ij} \langle g,h \rangle$ if both $e \succeq _{i} g$ and $f \succeq _{j} h$,

• $I_{ij}\langle e,f \rangle$ = $I_{i} (e) \cap I_{j}(f)$, for any $\langle e,f \rangle \in T_{ij}$, and

• $L_{i} \circ L_{j} = \langle T_{ij}, \succeq _{ij}, I_{ij}\rangle$.

Proof of lemma 1

To be shown are that (1) $\langle T_{ij}, \succeq _{ij}\rangle$ is a grade vocabulary and (2) $I_{ij}$ is an interpretation of $\langle T_{ij}, \succeq _{ij}\rangle$ in $\langle V,\ge \rangle$.

(1) $T_{ij} \ne \emptyset$ since $\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle \in T_{ij}$, for any $v \in V.$ ($I^{-1}_{i}(v)$ is the unique $e \in T_{i}$ such that $v \in I_{i}(e)$, and similarly for $I^{-1}_{j}(v)$.) $T_{ij}$ is finite since $T_{i}$, $T_{j}$ are. To see that $\succeq _{ij}$ is a linear ordering of $T_{ij}$, note that antisymmetry and transitivity of $\succeq _{ij}$ follow immediately from the corresponding properties of $\succeq _{i}$ and $\succeq _{j}$. To see that $\succeq _{ij}$ is a total ordering of $T_{ij}$, suppose $\langle e,f \rangle \nsucceq _{ij} \langle g,h \rangle$, i.e. either $e \nsucceq _{i} g$ or $f \nsucceq _{j} h$. Take the first case (the two are analogous). Then, since $\succeq _{i}$ is total, $g \succ _{i} e$. Since $\langle g,h \rangle , \langle e,f \rangle \in T_{ij}$ it is possible to choose $u \in I_{i}(g) \cap I_{j}(h)$ and $v \in I_{i}(e) \cap I_{j}(f)$. Since $I_{i}$ is orderly, $u > v$. Since $I_{j}$ is orderly, $f \nsucc _{j} h$, and so, because $\succeq _{j}$ is total, $h \succeq _{j} f$. But then, as required, $\langle g,h \rangle \succeq _{ij} \langle e,f \rangle$.

(2) To be shown are that (i) $\forall \langle e,f \rangle \in T_{ij}, I_{ij}(\langle e,f \rangle )$ is convex; that (ii) $I_{ij}$ partitions V; and that (iii) $I_{ij}$ is orderly.

(i) Suppose $u,v,w \in V$, $u,w\in I_{ij}(\langle e,f \rangle )$ and $u \ge v \ge w$. By definition of $I_{ij}$, $u,w \in I_{i}(e) \cap I_{j}(f)$. Since both $I_{i}(e)$ and $I_{j}(f)$ are convex, $v\in I_{i}(e) \cap I_{j}(f)= I_{ij}(\langle e,f \rangle )$.

(ii) Consider any $\langle e,f \rangle , \langle g,h \rangle \in T_{ij}$. First, $I_{ij}(\langle e,f \rangle ) \ne \emptyset$ by choice of the vocabulary $T_{ij}$; second, if $I_{ij}(\langle e,f \rangle )\cap I_{ij}(\langle g,h \rangle ) \ne \emptyset$ then, expanding using the definition of $I_{ij}$, we have $I_{i} (e) \cap I_{i} (g) \ne \emptyset$, and also $I_{j}(f) \cap I_{j}(h) \ne \emptyset$. Because $L_{i}$ and $L_{j}$ are grade languages for $\langle V,\ge \rangle$, $I_{i}$ partitions V and so does $I_{j}$. So $e=g$, $f=h$ and, as required, $\langle e,f \rangle =\langle g,h \rangle$; finally, $\bigcup \{ I_{ij}(\langle e,f \rangle ): \langle e,f \rangle \in T_{ij}\} = V$ because for any given $v \in V$, $\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle \in T_{ij}$ and $v \in I_{i}(I^{-1}_{i}(v)) \cap I_{j}(I^{-1}_{j}(v)) = I_{ij}(\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle ).$

(iii) Consider any $\langle e,f \rangle$, $\langle g,h \rangle$$\in T_{ij}$. Suppose $\langle e,f \rangle \succ _{ij}\langle g,h \rangle$. Consider any $u \in I_{ij}(\langle e,f \rangle )$ and $v \in I_{ij}(\langle g,h \rangle )$. Since $\langle g,h \rangle \nsucceq _{ij} \langle e,f \rangle$, either $g \nsucceq _{i} e$ or $h \nsucceq _{j} f$. $\succeq _{i}$ and $\succeq _{j}$ are total relations, so either $e\succ _{i}g$ or $f\succ _{j}h$. Suppose $e\succ _{i}g$. By definition of $I_{ij}$, $u \in I_{i}(e)$ and $v \in I_{i}(g)$. Since $I_{i}$ is orderly, $u>v$. This follows similarly in case $f\succ _{j}h$. So $I_{ij}(\langle e,f \rangle )> I_{ij}(\langle g,h \rangle )$. $\square$

1.2 Fact 3

$\circ (\vec {L})$ is as precise as each of its composing languages.

Definitions Let languages $\vec {L} = \langle L_{1}, \dots L_{n} \rangle$ measure some common dimension. Their superlanguage $\circ (\vec {L})$ is defined recursively: $\circ (\langle L_{1} \rangle ) = L_{1}$; $\circ (\langle L_{1},\dots , L_{m+1}\rangle )$ = $\circ (\langle L_{1} \dots L_{m}\rangle )$$\circ$$L_{m+1}$. The composing languages are $L_{1}, \dots L_{n}$. $\langle T_{1}, \succeq _{1}, I_{1} \rangle$ is as precise as$\langle T_{2}, \succeq _{2}, I_{2} \rangle$ if for each $e \in T_{2}$ there is $T_{e} \subseteq T_{1}$ such that $I_{2} (e) = \bigcup \{I_{1}(t): t \in T_{e}\}$.

Proof of Fact 3

An induction on the length of $\vec {L}$, of which the induction step uses: Lemma. Let $L_{i}$, $L_{j}$ and $L_{i} \circ L_{j}$ be as in the statement of lemma 1. Then $L_{i} \circ L_{j}$ is as precise as $L_{i}$ and $L_{i} \circ L_{j}$ is as precise as $L_{j}$. Proof of the lemma. To be shown is that $L_{i} \circ L_{j}$ is as precise as $L_{i}$. (The similar demonstration that it is as precise as $L_{j}$ is omitted.) Required is that for any $e \in T_{i}$ there is $T_{e} \subseteq T_{ij}$ such that $I_{i} (e) = \bigcup \{I_{ij}(t): t \in T_{e}\}$. For any given $e \in T_{i}$ set $T_{e} = \{\langle e, f \rangle : \langle e, f \rangle \in T_{ij}\}.$ That $I_{i} (e) \supseteq \bigcup \{I_{ij}(t): t \in T_{e}\}$ is easily seen since for any $\langle e, f \rangle \in T_{ij}$ we have $I_{i} (e) \supseteq I_{i}(e) \cap I_{j}(f) = I_{ij}(\langle e, f \rangle )$. To see that furthermore $I_{i} (e) \subseteq \bigcup \{I_{ij}(t): t \in T_{e}\}$, consider any $v \in I_{i} (e)$. Let f be $I^{-1}_{j}(v)$. Then $v \in I_{i}(e) \cap I_{j}(f)$. So $\langle e, f \rangle \in T_{ij}$, and $\langle e, f \rangle \in T_{e}$. Now $v \in I_{i}(e) \cap I_{j}(f) = I_{ij}(\langle e, f \rangle ) \subseteq \bigcup \{I_{ij}(t): t \in T_{e}\}$. This completes the proof of the lemma.

Note before the proof of Fact 3 itself that comparative precision is both reflexive (L is as precise as L) and transitive (if $L_{1}$ is as precise as $L_{2}$, and $L_{2}$ as precise as $L_{3}$, then $L_{1}$ is as precise as $L_{3}$. Now we have:

Base step: $\vec {L} = \langle L_{1} \rangle$. Then $\circ (\vec {L})$ = $L_{1}$. It is as precise as the only composing language, $L_{1}$, because comparative precision is reflexive.

Inductive step: $\vec {L} = \langle L_{1}, \dots , L_{m+1} \rangle$. By the lemma, $\circ (\vec {L}) = \circ (\langle L_{1}, \dots , L_{m}\rangle ) \circ L_{m+1}$ is as precise as $\circ (\langle L_{1}, \dots , L_{m}\rangle )$. By induction hypothesis, $\circ (\langle L_{1}, \dots , L_{m}\rangle )$ is as precise as each of $L_{1}, \dots , L_{m}$. By transitivity of comparative precision therefore $\circ (\vec {L})$ is as precise as each of $L_{1}, \dots , L_{m}$. By the lemma, furthermore, $\circ (\vec {L})$ is as precise as the single remaining composing language, $L_{m+1}$. $\square$

1.3 Fact 4

Let $S_{1}, \dots , S_{n}$ be sets of standards for languages $L_{1}, \dots , L_{n}$. Then $\bigcup \{ S_{1}, \dots , S_{n}\}$ is a set of standards for the superlanguage $\circ (\langle L_{1}, \dots , L_{n}\rangle )$.

Definitions. Let $L = \langle T,\succeq , I\rangle$ be a language for $\langle V,\ge \rangle$. s is the standard for einL if:

• $e \in T$,

• $s \in I(e)$,

• $I^{-1}(v) \succeq e$ for any $v \in V$ such that $v \ge s$, and

• $I^{-1}(v) \prec e$ for any $v \in V$ such that $v < s$.

S is a set of standards for L if for each $s \in S$ there is some $e \in T$ such that s is the standard for e in L.

Proof of Fact 4

An induction on n, of which the induction step uses the following Lemma. Let $L_{i}$ and $L_{j}$ be languages for $\langle V,\ge \rangle$. Let $S_{i}$ be a set of standards for $L_{i}$ and let $S_{j}$ be a set of standards for $L_{j}$. Then $S_{i} \cup S_{j}$ is a set of standards for $L_{i}\circ L_{j}$. Proof of the lemma. Consider any $s \in S_{i} \cup S_{j}$. There are two cases to consider: $s \in S_{i}$ and $s \in S_{j}$. We consider just the first case (the argument for the second case is a mirror image of that for the first). With $s \in S_{i}$, let $e \in T_{i}$ be such that s is the standard for e in $L_{i}$. We will see that s is the standard for $\langle e, I^{-1}_{j}(s) \rangle$ in $L_{ij}$. Required are (1) $\langle e, I^{-1}_{j}(s) \rangle \in T_{ij}$, (2) $s \in I_{ij}(\langle e, I^{-1}_{j}(s) \rangle )$, (3) $I^{-1}_{ij}(v) \succeq _{ij} \langle e, I^{-1}_{j}(s) \rangle$ for any $v \in V$ such that $v > s$, and (4) $I_{ij}^{-1}(v) \prec _{ij} \langle e, I^{-1}_{j}(s) \rangle$ for any $v \in V$ such that $v < s$.

(1) Because s is the standard for e in $L_{i}$, $s \in I_{i}(e).$ Furthermore, $s \in I_{j}(I^{-1}_{j}(s))$, so $I_{i}(e) \cap I_{j}(I^{-1}_{j}(s)) \ne \emptyset$ and $\langle e, I^{-1}_{j}(s) \rangle \in T_{ij}.$

(2) $s \in I_{i}(e) \cap I_{j}(I^{-1}_{j}(s))$ as in (1). Now, by definition of $I_{ij}$, $I_{i}(e) \cap I_{j}(I^{-1}_{j}(s)) = I_{ij}(\langle e, I^{-1}_{j}(s) \rangle ).$

(3) Notice first that for any $v \in V$ we have $I^{-1}_{ij}(v) = \langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle$. This is equivalent to $v \in I_{ij}(\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle )$, which follows by unpacking the relevant definitions. Now, consider any $v \in V$ such that $v > s$. Since both $I_{i}$ and $I_{j}$ are orderly, $I^{-1}_{i}(v) \succeq _{i} I^{-1}_{i}(s)$ and also $I^{-1}_{j}(v) \succeq _{j} I^{-1}_{j}(s)$. So as required $I^{-1}_{ij}(v)$ = $\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle$$\succeq _{ij}$$( I^{-1}_{i}(s), I^{-1}_{j}(s)) = \langle e, I^{-1}_{j}(s) \rangle$. The last equality is due to the fact that since s is the standard for e in $L_{i}$ we have $s \in I_{i}(e)$ or, equivalently, $I^{-1}_{i}(s) = e$.

(4) Consider any $v \in V$ such that $v < s$. Because s is the standard for e in $L_{i}$, $I^{-1}_{i}(v) \prec _{i} e = I^{-1}_{i}(s)$, so $\langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle$$\nsucceq _{ij}$$( I^{-1}_{i}(s), I^{-1}_{j}(s))$. Since $\succeq _{ij}$ is a total ordering of $T_{ij}$ we have as required $I^{-1}_{ij}(v) = \langle I^{-1}_{i}(v), I^{-1}_{j}(v)\rangle$$\prec _{ij}$$(I^{-1}_{i}(s), I^{-1}_{j}(s)) = \langle e, I^{-1}_{j}(s) \rangle$. This completes the proof of the lemma, and of Fact 4. $\square$

1.4 Lemma 5

Let $\langle G_{i}, L_{i}\rangle$ and $\langle G_{j}, L_{j}\rangle$ be solutions to problem P, each accurate by its own standards, where $L_{i}$ and $L_{j}$ measure a common dimension. Then

$$\begin{aligned} \langle G_{i} \circ G_{j}, \quad L_{i} \circ L_{j}\rangle \end{aligned}$$

is an accurate solution to P, by its own standards.

Definitions. Let $L = \langle T,\succeq , I\rangle$ be a grade language for $\langle V,\ge \rangle$ and let $P = \langle X, \alpha \rangle$ be a grading problem. Then

• L is suitable for P if $\forall x \in X$, $\alpha (x) \in V$,

• $\langle G, L \rangle$ is a solution to P if G is an assignment from P into L (that is, $\forall x \in X, G(x) \in T$) and L is suitable for P,

• $\langle G, L \rangle$ is an accurate solution to P, by its own standards if $\langle G, L \rangle$ is a solution to P and $\forall x \in X$, $\alpha (x) \in I(G(x))$.

Furthermore,

• $T_{ij}, I_{ij}$ and $L_{i} \circ L_{j}$ are as in the superlanguage lemma,

and for mappings $G_{i}: X \rightarrow T_{i}$ and $G_{j} : X \rightarrow T_{j}$, we define $G_{i} \circ G_{j}: X \rightarrow T_{i} \times T_{j}$ by putting, for each $x \in X$,

• $G_{i} \circ G_{j}(x) = \langle G_{i}(x), G_{j}(x)\rangle$.

Proof of lemma 5

To be shown are that (1) $\langle G_{i} \circ G_{j}, \quad L_{i} \circ L_{j}\rangle$ is a solution to P, and (2) for each $x \in X$, $\alpha (x) \in I_{ij}(G_{i} \circ G_{j}(x))$.

(1) Required are that (i) $G_{i} \circ G_{j}$ maps each $x \in X$ to some term in $T_{ij}$, the grade terms of $L_{i} \circ L_{j}$, and (ii) $L_{i} \circ L_{j}$ is suitable for P.

(i) Consider any $x \in X$. Since $\langle G_{i}, L_{i}\rangle$ and $\langle G_{j}, L_{j}\rangle$ are accurate by their own standards, both $\alpha (x) \in I_{i}( G_{i}(x))$ and $\alpha (x) \in I_{j}( G_{j}(x))$. So $\alpha (x) \in I_{i}( G_{i}(x)) \cap I_{j}( G_{j}(x)) \ne \emptyset$. Therefore, by definition of $G_{i} \circ G_{j}$ and of $T_{ij}$, $G_{i}\circ G_{j}(x) = \langle G_{i}(x), G_{j}(x)\rangle \in T_{ij}.$

(ii) By lemma 1, $L_{i} \circ L_{j}$ measures the same dimension as $L_{i}$ (and $L_{j}$). Suitability of $L_{i} \circ L_{j}$ for P follows immediately from that of $L_{i}$ (or that of $L_{j}$).

(2) Consider any $x \in X$. As in (1) (i), $\alpha (x) \in I_{i}( G_{i}(x))$ and $\alpha (x) \in I_{j}( G_{j}(x))$. So, by definition of $I_{ij}$ and $G_{i} \circ G_{j}$, $\alpha (x) \in I_{i}( G_{i}(x)) \cap I_{j}( G_{j}(x)) = I_{ij}(\langle G_{i}(x), G_{j}(x)\rangle ) = I_{ij}(G_{i} \circ G_{j}(x))$. $\square$

1.5 Fact 7

Let individual scope $\Phi$ be truth-compatible for P, and let L be suitable for $\Phi$ in P. Then $\langle \mathcal {G}_{L}, L \rangle$ is a reliably accurate solution to P with scope $\Phi$ if and only if L masks $\Phi$ in P.

Definitions. Let $L = \langle T, \succeq , I \rangle$ measure $\langle V,\ge \rangle$, and $P = \langle X, \alpha \rangle$. Let $\Phi$ be an individual scope, and let $\langle \mathcal {G}, L\rangle$ be an individual signaled solution to P with scope $\Phi$. Define

• L is suitable forP if $\forall x \in X, \alpha (x) \in V$,

• L is suitable for$\Phi$inP if $\forall \varphi \in \Phi , \forall x \in X$, $\varphi (x) \in V$,

• $\mathcal {G}_{L}: \Phi \times X \rightarrow T$ is the signaled grade assignment from $\Phi$ and P into L such that $\mathcal {G}_{L}(\varphi , x) = I^{-1}(\varphi (x))$ (defined only in case L is suitable for $\Phi$ in P),

• $\Phi (x)$ = $\{\varphi (x): \varphi \in \Phi \}$, for any given $x \in X$,

• Lmasks$\Phi$inP if for all $x \in X$ and all $e,f \in T$, if $\Phi (x) \cap I(e) \ne \emptyset$ and $\Phi (x) \cap I(f) \ne \emptyset$, then $e=f$,

• $\Phi$ is truth-compatible for P if for each $x \in X$, $\alpha (x) \in \Phi (x)$,

• A signaled solution $\langle \mathcal {G}, L\rangle$ to P with scope $\Phi$ is a reliably accurate solution toP with scope $\Phi$ if for every $\varphi \in \Phi$, $\langle \mathcal {G}^{\varphi }, L \rangle$ is an accurate solution to P, by its own standards.

Proof of Fact 7

For the if half, consider under the assumptions of the theorem the signaled solution $\langle \mathcal {G}_{L}, L \rangle$ to P with scope $\Phi$. Suppose $L = \langle T, \succeq , I \rangle$ masks $\Phi$ in $P = \langle X, \alpha \rangle$. Take any $\varphi \in \Phi$. First, $\langle \mathcal {G}_{L}^{\varphi }, L \rangle$ is a solution to P. For each $x \in X$ we have $\mathcal {G}_{L}^{\varphi }(x) = \mathcal {G}_{L}(\varphi , x) = I^{-1}(\varphi (x)) \in T$, so $\mathcal {G}_{L}^{\varphi }$ is a grade assignment from P into L. Furthermore, L is suitable for P: since by assumption $\Phi$ is truth-compatible for P, for any given $x \in X$ there is some $\varphi \in \Phi$ such that $\alpha (x) = \varphi (x)$. Now, since L is suitable for $\Phi$ in P, $\alpha (x) = \varphi (x) \in V$.

To see that $\langle \mathcal {G}_{L}^{\varphi }, L \rangle$ is accurate, by its own standards, consider any $x \in X$. Because $\Phi$ is truth-compatible for P, $\alpha (x) \in [\Phi (x) \cap I (I^{-1}(\alpha (x)))]\ne \emptyset$. We have, by definition of $\Phi (x)$, also $\varphi (x) \in [\Phi (x) \cap I (I^{-1}(\varphi (x)))]\ne \emptyset$. Since L masks $\Phi$ in P, therefore $I^{-1}(\alpha (x)) = I^{-1}(\varphi (x))$. By definition of $\mathcal {G}_{L}$ furthermore $I^{-1}(\varphi (x)) = \mathcal {G}_{L}(\varphi , x) = \mathcal {G}_{L}^{\varphi }(x)$. Putting these identities together within the scope of I: $\alpha (x) \in I (I^{-1}(\alpha (x)) = I (I^{-1}(\varphi (x))) = I(\mathcal {G}_{L}^{\varphi }(x)).$ This argument is good for any $x \in X$, so $\langle \mathcal {G}_{L}^{\varphi }, L \rangle$ is an accurate solution to P, by its own standards. It is good for any $\varphi \in \Phi$, so $\langle \mathcal {G}_{L}, L \rangle$ is a reliably accurate solution to P with scope $\Phi$.

For the only if part, suppose L does not mask $\Phi$ in P. Choose some $x \in X$ and $e \ne f \in T$ such that $\Phi (x) \cap I(e) \ne \emptyset$ and $\Phi (x) \cap I(f) \ne \emptyset$. Because $\Phi$ is truth-compatible, without loss of generality $\alpha (x) \in I(e).$ (If $\alpha (x)$ is “covered” by neither the chosen e nor f then, by the properties of interpretations, there has to be some other $g \in T$ such that $\alpha (x) \in I(g)$. By truth-compatibility of $\Phi$, $\alpha (x) \in \Phi (x) \cap I(g) \ne \emptyset$, so we can choose this g instead of e.) Now, choose any $v \in \Phi (x) \cap I(f)$, and any $\varphi \in \Phi$ such that $\varphi (x) = v$. We have $\mathcal {G}_{L}^{\varphi }(x) = I^{-1}(\varphi (x)) = I^{-1}(v) = f$. Because I is an interpretation and $e \ne f$, $I(e) \cap I(f) = \emptyset$, so because $\alpha (x) \in I(e)$, $\alpha (x) \not \in I(f) = I(\mathcal {G}_{L}^{\varphi }(x))$. So $\langle \mathcal {G}_{L}^{\varphi }, L\rangle$ is not an accurate solution to P, by its own standards, and $\langle \mathcal {G}_{L} , L \rangle$ is not a reliably accurate solution to P with scope $\Phi$. $\square$

1.6 Theorem 8

Let $\langle \mathcal {G}_{1}, L_{1}\rangle ,$$\dots ,$$\langle \mathcal {G}_{n}, L_{n}\rangle$ be reliably accurate individual solutions to P with respective scopes $\Phi _{1},$$\dots ,$$\Phi _{n}$, where $L_{1},$$\dots ,$$L_{n}$ measure some common dimension. Then

$$\begin{aligned} \langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{n}\rangle ), \quad \circ (\langle L_{1}, \dots L_{n} \rangle )\rangle \end{aligned}$$

is a reliably accurate solution to P with scope $\Phi _{1} \times \dots \times \Phi _{n}$.

Definitions. Let $L = \langle T, \succeq , I \rangle$ and $P = \langle X, \alpha \rangle$, and let $\Phi$ be a scope for problem P.

$\mathcal {G}$ is a signaled grade assignment from$\Phi$andPintoL if

• $\mathcal {G}$ maps each $(\vec {\varphi }, x) \in \Phi \times X$ to a grade in L (that is, $\mathcal {G}(\vec {\varphi }, x) \in T$), and

• $\mathcal {G}(\vec {\varphi }, x) = \mathcal {G}(\vec {\varphi }, y)$ whenever for each component $\varphi _{i}$ of $\vec {\varphi }$, $\varphi _{i}(x) = \varphi _{i}(y)$.

Replace $\vec {\varphi }$ by $\varphi$ for an individual signaled grade assignment.

$\langle \mathcal {G}, L\rangle$ is an (individual) signaled solution to Pwith scope$\Phi$ if $\mathcal {G}$ is an (individual) signaled grade assignment from $\Phi$ and P into L.

A signaled solution $\langle \mathcal {G}, L\rangle$ to P with scope $\Phi$ is a reliably accurate solution toP with scope $\Phi$ if for every $\vec {\varphi } \in \Phi$, $\langle \mathcal {G}^{\vec {\varphi }}, L \rangle$ is an accurate solution to P, by its own standards. Replace $\vec {\varphi }$ by $\varphi$ for a reliably accurate individual solution.

Let $\langle \mathcal {G}_{1}, L_{1}\rangle$, $\dots$, $\langle \mathcal {G}_{n}, L_{n}\rangle$ be reliably accurate individual solutions to P with respective scopes $\Phi _{1}$, $\dots$, $\Phi _{n}$. A function $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{n} \rangle )$ is defined by recursion that maps each pair $(\vec {\varphi },x)$, with $\vec {\varphi } \in \Phi _{1} \times \dots \times \Phi _{n}$ and $x \in X$, to a grade in $\circ (\langle L_{1}, \dots , L_{n} \rangle )$. Where $\varphi _{i} \in \Phi _{i}$ and $x \in X$, put

• $\circ (\langle \mathcal {G}_{1}\rangle )(\langle \varphi _{1}\rangle ,x) = \mathcal {G}_{1}(\varphi _{1},x)$, and

• $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , x)$ =

     $\langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x), \quad \mathcal {G}_{m+1}(\varphi _{m+1},x) \rangle$.

Proof of Theorem 8

An induction on n. The base case is trivial, since $\circ (\langle \mathcal {G}_{1}\rangle )$ = $\mathcal {G}_{1}$ (modulo identifying any singleton $\langle \varphi \rangle$ with $\varphi$), and $\circ (\langle L_{1} \rangle ) = L_{1}$. The induction step is more involved. There are two main things to be shown: (1) $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )$ is a signaled grade assignment from $\Phi _{1} \times \dots \times \Phi _{m+1}$ and $P = \langle X, \alpha \rangle$ into $\circ (\langle L_{1}, \dots , L_{m+1} \rangle )$; and (2) for any $\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle \in \Phi _{1} \times \dots \times \Phi _{m+1}$,

$$\begin{aligned} \langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle }, \quad \circ (\langle L_{1}, \dots L_{m+1} \rangle )\rangle \end{aligned}$$

is an accurate solution to P, by its own standards.

(1) There are again two requirements. The first is that (i) for any given $\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle \in \Phi _{1} \times \dots \times \Phi _{m+1}$, and for any $x \in X$,

$$\begin{aligned} \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , x ) \end{aligned}$$

is a grade in $\circ (\langle L_{1}, \dots , L_{m+1} \rangle )$. The second is that (ii) for any $x, y \in X$,

$$\begin{aligned}&\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle ) ( \langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , x )\\&\quad =\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle ) (\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , y ) \end{aligned}$$

if for each $1 \le i \le m+1$, $\varphi _{i}(x) = \varphi _{i}(y)$. We verify (i) and (ii) in turn.

(i) Consider any particular such $\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle$ and x. Since $\mathcal {G}_{m+1}(\varphi _{m+1},x)$ is a grade in $L_{m+1}$, by the definition of $\circ (\langle L_{1}, \dots , L_{m+1} \rangle )$ it is sufficient that (a)

$$\begin{aligned} \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x) \text { is a grade in } \circ (\langle L_{1}, \dots L_{m} \rangle ) \end{aligned}$$

and, letting $I_{1,\dots , m}$ and $I_{m+1}$ be the interpretation functions of $\circ (\langle L_{1}, \dots L_{m} \rangle )$ and $L_{m+1}$, that (b)

$$\begin{aligned} \begin{array}{c} I_{1,\dots , m}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x))\\ \cap \\ I_{m+1}(\mathcal {G}_{m+1}(\varphi _{m+1},x)) \end{array} \end{aligned}$$

is non-empty.

By induction hypothesis, $\langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle ), \quad \circ (\langle L_{1}, \dots L_{m} \rangle )\rangle$ is a reliably accurate solution to P with scope $\Phi _{1} \times \dots \times \Phi _{m}$. It follows immediately that (a) $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x)$ is a grade in $\circ (\langle L_{1}, \dots L_{m} \rangle )$. Furthermore,

$$\begin{aligned} \langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m} \rangle }, \quad \circ (\langle L_{1}, \dots L_{m} \rangle )\rangle \end{aligned}$$

is an accurate solution to P, by its own standards, and so by the assumptions of the theorem is $\langle \mathcal {G}_{m+1}^{\varphi _{m+1}}, \quad L_{m+1}\rangle$. This secures (b), since

$$\begin{aligned} \alpha (x) \in I_{1,\dots , m}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m} \rangle }(x)) \end{aligned}$$

and

$$\begin{aligned} \alpha (x) \in I_{m+1}(\mathcal {G}_{m+1}^{\varphi _{m+1}}(x)). \end{aligned}$$

(ii) Suppose for given $x, y \in X$ that for each $1 \le i \le m+1$, $\varphi _{i}(x) = \varphi _{i}(y)$. By induction hypothesis $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )$ is a signaled grade assignment from $\Phi _{1} \times \dots \times \Phi _{m}$ and P into $\circ (\langle L_{1}, \dots , L_{m} \rangle )$, and by the assumptions of the theorem $\mathcal {G}_{m+1}$ is a signaled grade assignment from $\Phi _{m+1}$ and P into $L_{m+1}$. Therefore $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x)$ = $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , y)$ and $\mathcal {G}_{m+1}(\varphi _{m+1},x)$ = $\mathcal {G}_{m+1}(\varphi _{m+1},y)$. Thus we have as required:

$$\begin{aligned}&\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )( \langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , x )\\&\quad = \langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x), \quad \mathcal {G}_{m+1}(\varphi _{m+1},x)\rangle \\&\quad = \langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , y), \quad \mathcal {G}_{m+1}(\varphi _{m+1},y)\rangle \\&\quad = \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )( \langle \varphi _{1}, \dots , \varphi _{m+1} \rangle , y ) \end{aligned}$$

(2) Take any such $\langle \varphi _{1}, \dots , \varphi _{m+1} \rangle$, and any $x \in X$. Letting $I_{1,\dots , m+1}$ be the interpretation function of $\circ (\langle L_{1}, \dots , L_{m+1}\rangle )$, to be shown is that $\alpha (x) \in I_{1,\dots , m+1}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m+1}\rangle } (x))$. Reasoning as in (1) (i) (b) above, by induction hypothesis and assumptions of the theorem $\alpha (x) \in$

$$\begin{aligned} I_{1,\dots , m}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m} \rangle }(x)) \cap I_{m+1}(\mathcal {G}^{\varphi _{m+1}}_{m+1}(x)), \end{aligned}$$

which can be rewritten

$$\begin{aligned} I_{1,\dots , m}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x)) \cap I_{m+1}(\mathcal {G}_{m+1}(\varphi _{m+1}, x)). \end{aligned}$$

By definition of $I_{1,\dots , m+1}$ this is equal to

$$\begin{aligned} I_{1,\dots , m+1}(\langle \circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m} \rangle , x), \quad \mathcal {G}_{m+1}(\varphi _{m+1}, x)\rangle ), \end{aligned}$$

which by definition of $\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )$ is just

$$\begin{aligned} I_{1,\dots , m+1}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )(\langle \varphi _{1}, \dots , \varphi _{m+1}\rangle , x)). \end{aligned}$$

Finally, this can be rewritten

$$\begin{aligned} I_{1,\dots , m+1}(\circ (\langle \mathcal {G}_{1}, \dots , \mathcal {G}_{m+1}\rangle )^{\langle \varphi _{1}, \dots , \varphi _{m+1}\rangle } (x)). \end{aligned}$$

This completes the proof of (2) and of Theorem 8. $\square$

1.7 Theorem 9

Let $\langle \mathcal {G}, L \rangle$ be a reliably accurate solution to $\langle X, \alpha \rangle$ with scope $\Phi = \Phi _{1} \times \dots \times \Phi _{n}$. Let $x, y \in X$ be such that $\Phi _{i}(x) \cap \Phi _{i}(y) \ne \emptyset$, for each $1 \le i \le n$. Let x and y be compatible in each $\Phi _{i}$. Then for each $\vec {\varphi } \in \Phi$, $\mathcal {G}(\vec {\varphi }, x) = \mathcal {G}(\vec {\varphi }, y)$.

Comment. From the proof it is clear that with $n=1$ we have, replacing $\vec {\varphi }$ by $\varphi$, the corresponding theorem for reliably accurate individual solutions as a special case.

Definition.x and y are compatible in individual scope $\Phi _{i}$ if $\forall \varphi , \psi \in \Phi _{i}$$\exists \chi \in \Phi _{i}$ such that $\chi (x) = \varphi (x)$ and $\chi (y) = \psi (y)$.

Proof of Theorem 9

Consider any reliably accurate solution $\langle \mathcal {G}, L \rangle$ to $\langle X, \alpha \rangle$ with scope $\Phi = \Phi _{1} \times \dots \times \Phi _{n}$. Take any i, $1 \le i \le n$. Under the assumptions of the theorem, choose $v \in \Phi _{i}(x) \cap \Phi _{i}(y)$. There are $\varphi _{i}, \psi _{i} \in \Phi _{i}$ such that $\varphi _{i}(x) = v = \psi _{i}(y)$. Because x and y are compatible in $\Phi _{i}$ there is $\chi _{i} \in \Phi _{i}$ such that $\chi _{i}(x) = \chi _{i}(y)$. Do this n times to obtain $\vec {\chi } = \langle \chi _{1}, \dots \chi _{n} \rangle \in \Phi = \Phi _{1} \times \dots \times \Phi _{n}$ such that $\forall i$, $\chi _{i}(x) = \chi _{i}(y)$. Because $\mathcal {G}$ is a signaled grade assignment, $\mathcal {G}(\vec {\chi },x) = \mathcal {G}(\vec {\chi },y)$. Since $\langle \mathcal {G}, L \rangle$ is reliably accurate it is, as shown in the paper, reliable: for every $\vec {\varphi }, \vec {\psi } \in \Phi$ and every $z \in X$, $\mathcal {G}(\vec {\varphi },z) = \mathcal {G}(\vec {\psi },z)$. For every $\vec {\varphi } \in \Phi$ therefore $\mathcal {G}(\vec {\varphi },x) = \mathcal {G}(\vec {\chi },x) = \mathcal {G}(\vec {\chi },y) = \mathcal {G}(\vec {\varphi },y)$. $\square$