Foundation models such as GPT-4 are fine-tuned to avoid unsafe or otherwise problematic behavior, so that, for example, they refuse to comply with requests for help with committing crimes or with producing racist text. One approach to fine-tuning, called reinforcement learning from human feedback, learns from humans' expressed preferences over multiple outputs. Another approach is constitutional AI, in which the input from humans is a list of high-level principles. But how do we deal with potentially diverging input from humans? How (...) can we aggregate the input into consistent data about "collective" preferences or otherwise use it to make collective choices about model behavior? In this paper, we argue that the field of social choice is well positioned to address these questions, and we discuss ways forward for this agenda, drawing on discussions in a recent workshop on Social Choice for AI Ethics and Safety held in Berkeley, CA, USA in December 2023. (shrink)

If K is an index of relative voting power for simple voting games, the bicameral postulate requires that the distribution of K -power within a voting assembly, as measured by the ratios of the powers of the voters, be independent of whether the assembly is viewed as a separate legislature or as one chamber of a bicameral system, provided that there are no voters common to both chambers. We argue that a reasonable index – if it is to be used (...) as a tool for analysing abstract, ‘uninhabited’ decision rules – should satisfy this postulate. We show that, among known indices, only the Banzhaf measure does so. Moreover, the Shapley–Shubik, Deegan–Packel and Johnston indices sometimes witness a reversal under these circumstances, with voter x ‘less powerful’ than y when measured in the simple voting game G1 , but ‘more powerful’ than y when G1 is ‘bicamerally joined’ with a second chamber G2 . Thus these three indices violate a weaker, and correspondingly more compelling, form of the bicameral postulate. It is also shown that these indices are not always co-monotonic with the Banzhaf index and that as a result they infringe another intuitively plausible condition – the price monotonicity condition. We discuss implications of these findings, in light of recent work showing that only the Shapley–Shubik index, among known measures, satisfies another compelling principle known as the bloc postulate. We also propose a distinction between two separate aspects of voting power: power as share in a fixed purse (P-power) and power as influence (I-power). (shrink)

In this note, we show that a partition of a cake is Pareto optimal if and only if it maximizes some convex combination of the measures used by those who receive the resulting pieces of cake. Also, given any sequence of positive real numbers that sum to one (which may be thought of as representing the players' relative entitlements), we show that there exists a partition in which each player receives either more than, less than, or exactly his or her (...) entitlement (according to his or her measure), in any desired combination, provided that the measures are not all equal. (shrink)

Consider an election in which each of the n voters casts a vote consisting of a strict preference ranking of the three candidates A, B, and C. In the limit as n→∞, which scoring rule maximizes, under the assumption of Impartial Anonymous Culture (uniform probability distribution over profiles), the probability that the Condorcet candidate wins the election, given that a Condorcet candidate exists? We produce an analytic solution, which is not the Borda Count. Our result agrees with recent numerical results (...) from two independent studies, and contradicts a published result of Van Newenhizen (Economic Theory 2, 69–83. (1992)). (shrink)

We characterize some large cardinal properties, such as μ-measurability and P 2 (κ)-measurability, in terms of ultrafilters, and then explore the Rudin-Keisler (RK) relations between these ultrafilters and supercompact measures on P κ (2 κ ). This leads to the characterization of 2 κ -supercompactness in terms of a measure on measure sequences, and also to the study of a certain natural subset, Full κ , of P κ (2 κ ), whose elements code measures on cardinals less than κ. (...) The hypothesis that Full κ is stationary (a weaker assumption than 2 κ -supercompactness) is equivalent to a higher order Lowenheim-Skolem property, and settles a question about directed versus chain-type unions on P κ λ. (shrink)

We characterize some large cardinal properties, such as $\mu$-measurability and $P^2(\kappa)$-measurability, in terms of ultrafilters, and then explore the Rudin-Keisler (RK) relations between these ultrafilters and supercompact measures on $P_\kappa(2^\kappa)$. This leads to the characterization of $2^\kappa$-supercompactness in terms of a measure on measure sequences, and also to the study of a certain natural subset, $\mathrm{Full}_\kappa$, of $P_\kappa(2^\kappa)$, whose elements code measures on cardinals less than $\kappa$. The hypothesis that $\mathrm{Full}_\kappa$ is stationary (a weaker assumption than $2^\kappa$-supercompactness) is equivalent to (...) a higher order Lowenheim-Skolem property, and settles a question about directed versus chain-type unions on $P_\kappa\lambda$. (shrink)