On the Conception and Design of the PhilPapers Survey
by David Bourget and David Chalmers
Introduction
The PhilPapers Survey was a survey of professional philosophers and
others on their philosophical views. (It was originally called "The
Philosophical Survey", but we have retrospectively retitled it for
reasons given below.) What follows are some thoughts on the
conception and design of the survey, including responses to some
feedback regarding the survey. We will discuss the results
separately.
Why a survey of philosophers' philosophical views? We decided to do
this in part because like many philosophers, we have an interest in
the sociology of philosophy, and we were interested to see some hard
data about this sociology. We are also interested in the experimental
use of online tools as a method of philosophical communication. Using
the PhilPapers technology to execute a survey of philosophical views
plays into both of these interests.
One motivation for the Survey (and the Metasurvey) comes from the idea
that philosophers may often have false sociological beliefs about the
profession. Most of us have had the experience of reading
philosophical papers that make sweeping sociological claims about the
field that seem quite questionable. It is arguable that the "received
wisdom" in a field (roughly, what people in a field take as a default
view, one that can sometimes be presupposed) is not based on what most
people in the field think, but rather is based on what most people
think most people think. If the received wisdom is grounded in false
sociological beliefs, that is worth knowing. It occurred to us that
it would not be all that difficult to actually gather data about these
matters, and that doing so might be an interesting and informative
exercise.
We designed the Philosophical Survey at first through conversations
with a number of other philosophers, and then refined it through three
rounds of beta testing by many professional philosophers and graduate
students. Of course we had to make many somewhat arbitrary choices
along the way, in settling on a survey format, a set of questions, and
so on.
Now that the Survey is completed, we have had the benefit of feedback
by around 600 professional philosophers and others, across a wide
variety of fields. (We encouraged respondents to leave feedback, and
around 20% of them did so.) The feedback was roughly evenly divided
between positive, negative, and neutral. Many philosophers liked the
idea of a survey, with some saying they had been hoping for something
like this for years, and many liked the design and the execution. At
the same time, many thought that there is something odd about the
whole idea of the survey, and many had criticisms of the design. In
what follows we will offer some thoughts mainly on these critical
remarks.
It should be noted up front that the Survey was never intended to be a
definitive survey of all areas and traditions of philosophy, giving a
definitive portrait of the views of the philosophical profession.
Rather, we intended it as an initial study of a few aspects of the
sociology of the profession, one that is likely to be refined by later
work, and one that (like most surveys) is biased in various ways
toward the interests of its designers. In retrospect, calling the
Survey "The Philosophical Survey" may have encouraged some people to
read pretensions of definitiveness into the survey. To help avoid
that reading, we should have called it "The PhilPapers Survey" (or
perhaps "A Philosophical Survey"). We will use that label from now
on.
Criticisms
Critical remarks fell mainly into four categories, which we might
summarize as follows: (i) The whole idea of a philosophical survey is
unphilosophical, or philosophically irrelevant, or misrepresents
philosophy, or is plain silly; (ii) The orientation of the survey is
biased in various respects, and in particular neglects non-analytic
traditions; (iii) The format of the questions and answers is
problematic, and in particular the answer options are too brief; (iv)
Various specific questions are problematic in various specific
respects. The most common criticisms were of type (iv), followed by
types (ii) and (iii) roughly equally, followed by type (i). We also
received numerous emails from nonrespondents making comments of type
(i), giving reason to think that this attitude is well-represented in
the overall population. In what follows we offer some thoughts on
criticisms of each type.
1. The idea of a philosophical survey is unphilosophical, or irrelevant
to philosophy, or misrepresents philosophy, or is plain silly.
In response: The survey concerns philosophy, but it is not philosophy.
If anything, it is an exercise in the sociology of philosophy.
Sociology should not be confused with philosophy, but many
philosophers have an interest in the sociology of philosophy all the
same.
As for relevance: Even if the survey were wholly irrelevant to the
practice of philosophy, we think the results would be interesting to
many philosophers. And certainly, survey votes are likely to be a
poor guide to philosophical truth. Still, beliefs about the sociology
of philosophy are often given a role in the practice of philosophy, in
various ways.
One notable role of sociological beliefs is in determining which views
one can presuppose, attend to, or ignore. We do not especially
condone this practice, but we think that insofar as sociological
beliefs play this role, it is better for them to be accurate. For
example: suppose that a philosopher accepts the analytic-synthetic
distinction and thinks the arguments against it fail. Suppose that
she is writing a paper in which she thinks that (sociology aside) an
appeal to the distinction would strengthen the paper. Suppose that she
nevertheless does not appeal to the distinction in a paper, solely on
the grounds that she thinks a large majority of philosophers reject
the distinction. Suppose that in fact, a large majority of
philosophers accept the distinction. Then her decision will have been
grounded in a false sociological belief, and the paper will be weaker
by her own lights as a result. True sociological beliefs would put
her in a position to write a better paper by her own lights.
The sociology of philosophical belief is also relevant to philosophy
in other ways. For example, in arguing for a philosophical position,
philosophers often make claims about the psychology of philosophical
beliefs (for example, arguing that people only accept p because they
accept q), and survey data may in principle be relevant to assessing
such psychological claims. Finally, the history of philosophy is
often directly concerned with sociological matters, and data about
today's sociology may well be relevant to tomorrow's history.
As for misrepresentation: We certainly agree that a philosopher's
views cannot be captured in a set of survey answers. The survey
answers should not be seen as any sort of definitive representation of
philosophers' views. At best, they capture a few dimensions or
aspects of those views. Likewise, the survey should not be taken as
any sort of definitive representation of the state of the
philosophical profession. Like any survey, it just captures a few
limited aspects of the state of the profession.
Finally, we recognize that there is something comical about the idea
of doing philosophy by multiple choice. But we think that the results
are of significant interest all the same, and multiple choice is by
far the most feasible way to get the relevant data.
2. The survey is biased toward certain traditions and areas. In
particular, it is biased toward analytic philosophy as opposed to
non-analytic traditions, including continental traditions, non-Western
traditions, and others.
In response: Yes, it is. We recognized this in the survey's
information page. We considered incorporating questions drawn from
non-analytic traditions, but it proved difficult to find questions
that would be accessible enough to a predominantly analytic audience.
So we decided to explicitly orient the survey toward analytic
philosophy, which is the tradition within which our own expertise
lies.
A number of respondents suggested that we should have incorporated
questions drawn from continental traditions, Asian traditions, the
pragmatist tradition, and from the history of philosophy, among
others. There were relatively few concrete questions suggested here.
We appreciated the concrete suggestions that were made (e.g. "Madness:
historical or ahistorical", "Hegelian dialectic or Marxist dialectic",
"Early and late Wittgenstein: complementary or contradictory?"), but
they tended to reinforce our initial sense that such questions would
not have worked in context.
It's also worth noting that we kept in a few questions that some
analytic philosophers regarded as problematic -- e.g. the
empiricism/rationalism question and the question about the nonliving
philosophers with whom one most identified -- in part because we
thought that questions of these sort would be accessible and engaging
to philosophers from a variety of traditions, and that they would
acknowledge the history of philosophy in a way that is otherwise hard
to acknowledge in a survey like this.
Still, it is clear that the survey was somewhat alienating for some
philosophers from non-analytic traditions. We could have done more to
note the orientation explicitly up front. We may have other surveys
in the future and input from philosophers from all traditions would be
welcomed. We also encourage other philosophers oriented toward other
traditions to design surveys of their own.
A few respondents suggested that the survey is biased toward certain
areas within analytic philosophy. Comments included "Too much
metaphysics and epistemology", "Too much ethics", "Too much
epistemology and mind, not enough metaphysics", "Not enough
aesthetics", "The survey is just analytic philosophy of mind", and so
on.
As we said in the information page, we deliberately focused on the
"core" areas of metaphysics, epistemology, mind, language, and ethics,
because these are the most accessible to philosophers outside the
area, though we did include questions in numerous other areas
(aesthetics, decision theory, logic, metaphilosophy, philosophy of
action, philosophy of science, political philosophy) as well. There
ended up being a bit more epistemology, ethics, and mind (around five
questions each) than language (three) or metaphysics (three to six,
depending on whether one counts free will and personal identity), in
part because the most natural further questions in the latter areas
tended to be specialized.
There is also a slight bias toward philosophy of mind arising from the
fact that the survey designers are both philosophers of mind. In
particular, we allowed ourselves one "pet question" each that would
not have made it onto the survey otherwise (too specialized). These
were the questions about perceptual experience (Bourget) and zombies
(Chalmers). That bias might be considered survey-designers'
prerogative.
3. The answer options are too brief and should have been further
explained.
In response: Again, this is a decision that we made explicitly up
front. The trouble is that any longer explanation would inevitably be
tendentious and would involve arbitrary choices. We tried designing
some fuller questions, but it becomes a very difficult process and it
is hard not to bias the questions in various ways. Furthermore, the
"clarified" questions always call for further clarification in turn.
In addition, survey results would inevitably be reported and discussed
under simple labels (n% of respondents favored physicalism, atheism,
etc), so we thought it made sense to have the answer options match
those labels.
Of course there are downsides to this decision: especially ambiguities
in various questions, and unfamiliarity with what various labels mean.
We provided "other" options to help to handle these downsides. And we
think that the survey results are clearly informative despite these
problems. For future surveys, though, we will consider trying a
different sort of question format, involving more fleshed-out
questions.
The brevity and ambiguity of the labels means that caution must be
used in interpreting the results. Of course one cannot simply assume
that all respondents adopted the interpretation of "naturalism" or
"empiricism" that one favors. But one can reasonably report that n%
of the respondents reported accepting naturalism, or leaning toward
empiricism, as long as one does not put forward a specific and
contested interpretation of "naturalism" or "empiricism" in doing so.
4. The specific questions are problematic in respect X.
In response: The choice of questions is an imperfect art. We sought
widely-known questions within analytic philosophy that have a
relatively canonical set of associated views, each of which is
associated with a reasonably simple label. We also aimed for
questions that would be accessible to analytic philosophers in a wide
range of areas. We came up with an initial list of twenty questions,
and went through three rounds of beta testing in which about forty
questions were trialed, before settling on the final list of thirty.
Of course no question is perfect, and some are more problematic than
others. For most topics, the labels for views are at least somewhat
ambiguous. For some topics, it is hard to find a short canonical list
of views. For some of the topics, the list of views is far from
exhaustive. For these reasons we provided "other" options (as well as
the comment box) allowing respondents to indicate that they find the
questions problematic in various respects, allowing them to endorse
other alternatives, and so on.
Thoughts on specific questions
Some thoughts on some of the questions and on why we chose them
follow.
Aesthetic value: objective or subjective?
We wanted to include a
question on aesthetics, but it wasn't clear what to include. We
consulted with a number of philosophers working in the area, and there
was a consensus that a question along these lines would be best. Of
course "objective" and "subjective" can be given various
interpretations, but experts preferred this formulation to
formulations in terms of aesthetic realism or relativism.
External world: idealism, skepticism, or non-skeptical realism?
We
asked this one partly because of its centrality in the history of
philosophy, and partly because we were especially interested in data
about how many philosophers accept the "old, dead" positions that
supposedly no-one accepts these days. Skepticism and idealism are
often treated as gateways to reductio in contemporary discussion, for
example, rather than as serious contenders for the truth. We would
have liked to have an option for a view on which the external world is
somehow mind-dependent without this being idealism (e.g. social
constructivism), but we couldn't find a good accessible generic term
here. Of course we expected a big majority for non-skeptical realism,
but we were interested to see whether there would be a good number of
skeptics and idealists out there.
Free will: compatibilism, libertarianism, or no free will?
These
are roughly the standard options. We used "no free will" on the
grounds that it is clearer for nonspecialists than "hard
incompatibilism" and "free will skepticism", and "hard determinism" is
overly narrow in a potentially nondeterministic world. Of course the
three options aren't exclusive, but the answer options allow for that.
We thought about asking two separate questions, one on compatibilism
and one on whether there is free will, but the limited extra
information that would result didn't seem worth the extra question.
Knowledge: empiricism or rationalism?
There is arguably no
consensus interpretation of what these labels mean. "Rationalism"
clearly isn't universally understood as the view that there is a
priori knowledge, given the results of the two questions. Some take
rationalism to be the claim that there is a priori knowledge of
synthetic or substantial or contingent truths, but these
interpretations aren't universal either. It is perhaps best to think
of the question as capturing something about identification with two
broad and ill-defined philosophical "camps", divided roughly over the
role of a priori knowledge.
Logic: classical or non-classical?
Various respondents wondered how
to interpret this questions, given that it's not obvious that there
has to be a fact of the matter about whether classical or nonclassical
logic is correct. Still, plenty of people think there is a fact of
the matter, and those that don't (including ourselves) seem to have by
and large chosen an "other" option.
Metaphilosophy: naturalism or non-naturalism?
As with empiricism
and rationalism, there isn't a single standard interpretation of
"naturalism." It has metaphysical readings (roughly involving
rejection of non-natural entities) and epistemological readings
(roughly involving the role of science in philosophical and other
knowledge). The "metaphilosophy" tag tends to bias the interpretation
toward the epistemological reading, but probably not universally. But
again, "naturalism" seems most often used to label a broad camp or
orientation, rather than a specific thesis, and data about
identification with these camps is interesting.
Moral judgment: cognitivism or non-cognitivism?
This dichotomy is
canonical but nevertheless somewhat problematic, given the recent
prevalence of views that take moral judgments and sentences to be
non-cognitive at root while still taking moral judgments and sentences
to be truth-apt. Perhaps "expressivist" would have been slightly
better than "non-cognitivist" here. Someone also suggested making the
tag "Moral judgment (at base)".
Moral motivation: internalism or externalism?
Of course there are
many meta-ethical internalisms, but the tag was supposed to focus
attention on the connection between moral judgment and motivation.
Even so there are various internalisms here, e.g. depending on whether
the issue is whether moral judgments are always motivating, or whether
they are always motivating for rational agents.
Normative ethics: deontology, consequentialism, or virtue ethics?
Yes, there are other options, as well as combined views. But clearly
these are the canonical "big three".
Perceptual experience: disjunctivism, qualia theory,
representationalism, or sense-datum theory?
Likewise, these are the
canonical "big four" views in contemporary discussions. "Qualia
theory" is perhaps ambiguous for nonspecialists, though is reasonably
well-understood in the field to refer to a view on which experience
consists at least in part in instantiating nonrelational
nonintentional properties (the most common forms of adverbialism can
be seen as qualia theories).
Personal identity: biological view, psychological view, or
further-fact view?
It's not straightforward getting the names right
here, especially for the first option. We considered various
combinations of "brain view", "physical view", and "animal view". But
after consulting with experts "biological view" seemed best. Of
course the options here aren't exhaustive and may not be exclusive
either.
Politics: communitarianism, egalitarianism, or libertarianism?
This
was the hardest question to design. We wanted a question on political
philosophy, but it is very unclear what that question should be.
Perhaps the most central view in the field is liberalism, but after
trialing questions involving this term, it became clear that one can't
use it as it is ambiguous between a US political meaning, a European
political meaning, a contemporary philosophical meaning, and more,
sometimes labeling views that are diametrically opposed. There's also
the question of what it should be opposed to. In much popular
discourse liberalism is opposed to conservatism, but this doesn't
straightforwardly reflect philosophical discourse. Left-wing
vs. right-wing isn't really philosophy per se, either. A question
about the relative importance of freedom and liberty seemed natural,
but it's hard to phrase this question well. We tried "Egalitarianism
vs libertarianism", but found relatively few taking the latter option
(perhaps because it's often taken to stand for a relatively extreme
view). A couple of people suggested including "communitarianism" too
(thereby covering liberty, equality, fraternity!), and it polled well,
so we decided that this was a good balance. The list is obviously
imperfect and there are obviously other views that could have been
included. But as far as we can tell, there is no really good
canonical set of options here.
Proper names: Fregean vs Millian?
Obviously not exclusive, but
still canonical. We tried "descriptivism vs direct reference", but
that is even less exclusive, due to all the non-descriptive Fregeans.
We also tried "Language: Fregean vs Russellian", but complications
about Russell's own descriptivist views make this tricky. So this
seemed best in the end.
Time: A-theory or B-theory?
Along with Newcomb's problem, this got
the highest quota of "Insufficiently familiar with the issue" answers.
We thought about fleshing out the labels -- e.g. "A-theory (passage is
real) vs B-theory (block universe)" -- but any such labels are
problematic. Still, in retrospect, we probably should have tried
something like this to increase the response.
Trolley problem (five straight ahead, one on side track, turn
requires switching, what ought one do?): switch or don't switch?
We
started with "Trolley problem: straight or turn", but many complained
that this was too ambiguous and needed details. So we went with this.
The original trolley problem concerns permissibility rather than
obligation, but still, asking about obligation not to switch gets at
that reasonably well, and we found that when the question explicitly
concerned permissibility the numbers were even more strongly tilted
toward not switching. Of course the trolley case works best when
conjoined with various other related cases, but we didn't have
questions to spare for them this time around. Maybe in a future
survey.
Truth: correspondence, deflationary, or epistemic?
Choosing the
options here was tricky. We tried "deflationism vs nondeflationism",
but it seemed a shame not to get data on correspondence too. But
"correspondence vs deflationary" seemed too far from exclusive.
Adding "epistemic" (our experts' consensus about the most popular
cluster of nondeflationary views) helps a bit, though of course there
are still missing options.
Zombies: inconceivable, conceivable but not metaphysically possible,
or metaphysically possible?
Pet question. Of course "conceivable"
is ambiguous, e.g. between prima facie and ideal conceivability. And
one can allow that zombies are possible with holding that entire
zombie worlds are possible, e.g. if one is a phenomenal externalists.
Still, it's interesting to see the results.
For which nonliving philosophers X would you describe yourself or your
work as X-ian or the equivalent?
Some philosophers (especially
those working on contemporary analytic philsophy) complained about
having to answer this question, while others said we should have asked
about influence rather than identification. But the survey was about
philosophical views, and the phrasing in terms of identification seems
best to get at that. And having the question allows an element of
engagement with the history of philosophy which is otherwise missing.
The listed philosophers were largely based on Brian Leiter's polls
concerning the "most important" philosophers in various historical
eras. We included the top 21 from the all-time list (down to
Berkeley) and the remainder of the top 17 from the last-200-years list
(down to Husserl and Heidegger). Because the resulting list was
all-male, we added Anscombe (the highest-ranked woman on the
last-200-years list). Of course the choice was somewhat arbitrary.
The highest-polling write-in choices were Dewey, Peirce, Sellars,
James, and Merleau-Ponty, suggesting that if anything we might have
gone further down the last-200-years list.
Consent questions:
We explored various configurations here (consulting with beta
testers and the ANU human ethics board) before settling on basic
consent to use results and consent to make them public. In the end,
about 2% of those who otherwise completed the survey did not give
consent to use the results (75 out of 3301 overall, and 16 out of 947
in the target group). We suspect that this is because at the consent
point (and on the information page) we suggested that we might release
potentially identifying institution-based statistics such "100% of
individuals at institution X chose option Y". In fact, we intend to
be cautious about releasing such statistics, but we thought that it
would be best to overstate rather than understate the risks at the
point of consent.
Conclusion
In light of the feedback from 3000+ survey respondents, we
might have done a few things differently in retrospect. We might have
used the alternative title from the start. To improve the reception
of the survey among non-analytic philosophers, we could have more
clearly labeled the analytic orientation up front. For similar
reasons, we might have dropped some of the more specialized questions
(e.g. perceptual experience, contextualism, one or two of the
internalism/externalism questions) and some of the thought-experiments
(e.g. zombies, trolley problems). We might also have changed the
wording of a few questions (e.g. time, moral judgment, politics). We
probably would have stayed with the dichotomy format and the brief
labels, but for future surveys we may well try other formats.
That being said, we are happy with the survey overall, and we think
that it produced interesting and informative results.
loading ..