PhilPapers Surveys

Back   

2009-12-08
How did you do on the Metasurvey?
David Chalmers

Australian National University
New York University

All Survey and Metasurvey respondents will receive an email giving a link to a page with their responses, including an assessment of how well they did on the Metasurvey.  We can't post those results publically, as participants did not consent to that, but people should feel free to post about their own Metasurvey results.

I took the Metasurvey unofficially by making predictions at the start of the Survey.  I didn't take it officially, as even by that point I'd seen results from beta testing the Survey.  Even so, a few of my predictions were off by a long way.  For example, I wrongly predicted a substantial majority for Humeanism, aesthetic subjectivism, Platonism, and invariantism.  I did better on the physicalism and analytic-synthetic distinction questions, predicting 60-20-20 in both cases (compared to 56-27-17 and 65-27-18), and was reasonable close on the zombie question, predicting 40-20-20-20 for CMI, MP, IC, other as opposed to 36-23-16-25.   I also accidentally predicted majorities for theism and scientific anti-realism by putting the predictions in the wrong order (I think this was an artifact of having worked so much with the questions that a particular order was fixed in my mind).  Overall, my average absolute error was 11%, placing me (unofficially) 31st out of 728 respondents.  It would be interesting to hear from others.
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Berit Brogaard

University of Missouri, St. Louis

I wrongly predicted a majority for Humeanism, non-classical logic, moral anti-realism, aesthetic value (subjective), analytic-synthetic distinction, proper names (Fregeanism), time (A-theory), and the trolley problem (don't switch).  My best scores were on: truth, God and politics (!).  I did reasonably well on a priori knowledge, personal identity, Newcomb's problem and mental content.  But overall I was way off.  My average absolute error per answer option was 18.6%.
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Robert Williams

University of Leeds

Fun stuff! My worst prediction was on Newcomb---I thought that 2-boxing was far more widespread. I was surprised it only beat out 1-boxing 3:2. It would be interesting to see the predictions matched against results ignoring both predicted and actual "other" and scaling appropriately----basically, accuracy conditional on venturing an opinion. Anyway, I was 140---in the top quarter, I think....
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Jah Bil

University of Victoria

Average absolute error per answer option: 13.5%
Standard deviation: 10.9%
Your rank by average absolute error: 212 of 728

I wrongly predicted a high percentage of political libertarians, a-theorists, relativists, consequentialists, trolly non-switchers, and analytic-synthetic conflaters.








Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Andrea Weisberger

University of North Florida

Well I did not do so well on the metasurvey -- average absolute error per answer option: 17.8%

I made a conscious effort to answer what I thought other people might select, and had difficulty when that started to coincide with what I thought. So I erred on the side of caution by presuming that if I selected a position, it might be due to my own preference, not necessarily what the majority might opt for, and in many cases deliberately skewed in favor of an opposite approach. 

For example, as an avowed atheist, I assumed that there are more theistic philosophers than the survey indicated.  At least it seems that way in the normal world, atheists being the minority everywhere and few people, even in academia, admitting to that culturally unpopular view.  I see now my approach to the metasurvey didn't work out so well.  Happily there are more identifiable atheists than I thought (at least private ones).

I must admit that I also got a bit frustrated with the sliders and how difficult it was to get an exact rendering of the desired percentages.  After a while, I jettisoned trying to include the "other" category and split results between positive and negative responses.

Overall, it was an enlightening exercise.  Thanks to all who put in the hard work in presenting this to the philosophical community.
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Eric T. Kerr

National University of Singapore

I thought there would be more support for sense-datum theory and was surprised by the number of compatibalists. Overall I had an AAE of 11.6%. It would be interesting to see what the average AAE of respondents was as many so far seem to be reasonably accurate. I wouldn't find this surprising as philosophers perhaps spend more time asking each other where they stand on X and making claims about consensus than many other disciplines. I only took a cursory look at the breakdown by AOS but it is interesting that some groups of specialists are much better at predicting results of respondents in their own AOS than others.
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Helen Daly

Colorado College

I did better on the Metasurvey than I expected, so my MetaMetasurvey score would be quite low. I think this surprise is due to the fact that I skipped a few questions. I don't have a very good idea about where people stand on some issues, so I opted not to respond about those. Having skipped them appears not to have counted against me, and so my average absolute error per answer is probably higher than it would have been had I answered all of the questions.

Average absolute error per answer option: 11.5%
Standard deviation: 9.6%
Your rank by average absolute error: 52 of 728
Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Pablo Stafforini

University of Oxford

Average absolute error per answer option: 11.4%
Standard deviation: 8.4%
Your rank by average absolute error: 46 of 728

I did worst on
  • aesthetic value (subjective: 34.5 | 70; objective: 41 | 20)
  • logic (classical: 51.6 | 20; non-classical: 15.4 | 40)
  • moral judgment (cognitivism: 65.7 | 35; non-cognitivism: 17 | 45)
I did best on
  • free will (no free will: 12.2 | 15; compatibilism: 59.1 | 50; libertarianism: 13.7 | 15)
  • normative ethics (consequentialism: 23.6 | 30; deontology 25.9 | 35; virtue ethics: 18.2 | 15)
  • zombies (inconceivable: 16 | 20; metaphysically possible: 23.3 | 25; conceivable but not metaphysically possible: 35.6 | 30)
(the first is the actual score; the second my predicted score)

Reply

2009-12-09
How did you do on the Metasurvey?
Reply to David Chalmers
Antony Eagle

University of Adelaide

My metasurvey results were: 

Average absolute error per answer option: 14.2%
Standard deviation: 12.1%
Your rank by average absolute error: 266 of 728

I did okay on a few. I did worst on the trolley problem: 

don't switch     switch
Predicted: 70% 25%
Actual: 7.6%     68.2%

But my main problem seemed to be that I systematically underestimated 'other', in every debate. I thought that I was representative of philosophers in having firm opinions about nearly every topic. But maybe there are lots of philosopher who aren't quite so opinionated? (The other alternative, of course, is that I have one of two crude views on every topic, while many others have sophisticated views that don't fall neatly into the categories offered. I'm not so persuaded this alternative is correct.)
Reply

2009-12-10
How did you do on the Metasurvey?
Reply to Antony Eagle
Richard Corry

University of Tasmania

My results:

Average absolute error per answer option: 15.5%
Standard deviation: 11.1%
Your rank by average absolute error: 377 of 728

I had the same problem as Antony: consistently underestimating 'other'.  I guess we shouldn't underestimate the ability of philosophers to make a straightforward question complex! 
Reply

2009-12-10
How did you do on the Metasurvey?
Reply to David Chalmers
Abuzaid Samir

I viewed the survey as asking one general question divided into 30 sub-questions, classical positions against newly developed positions. For example, he who denies a priori knowledge, Nominalist in abstract objects, externalist in epistemic justification, realist in external world, Humean in laws of nature, classical in logic, naturalist in metaphilosopy, physicalist in Mind, externalist in moral motivation, ...etc subscribe in general to the classical position. Those who take the counter positions subscribe to the newly developed positions.

 Naturally, things are much more complicated, many may advance answers that include elements of both positions. However, my view is that the philosophical community is split relatively equal between the two positions. Hence, except in some specifically clear cases I predicted a 45 : 40 :5.

Responses, in general were closer to 60 : 30 : 10 which means that the classical position is still significantly prevailing. As a result my metasurvey results were:

Average absolute error per answer option: 18.6%
Standard deviation: 13.4%
Your rank by average absolute error: 575 of 728

As I see it, the classical position is not so prevailing. Possible reason may be that the "new positions" is flourishing within specific communities (younger professors, non-leading institutes, or else) who are not sufficiently presented in the survey.  

Reply

2009-12-10
How did you do on the Metasurvey?
Reply to David Chalmers
Michael Bishop

University of Chicago

@Someone working with the data:I hope you be analyzing the relationship between score on the metasurvey and philosophical views?
Reply

2009-12-11
How did you do on the Metasurvey?
Reply to David Chalmers
Eric Schwitzgebel

University of California, Riverside

I am inordinately and completely inappropriately proud of the fact that I was ranked number 3 overall in the metasurvey, though I'm not sure I'd want to do a rematch with Dave.

But I wonder what it indicates about someone that they're ranked well.  In Dave's case, I suspect it's largely attunement to the Zeitgeist and lots of experience with informal polling.  But I spend half my time reading psychology and the weirder corners of the history of philosophy, so I doubt my Zeitgeist-attunement is above average for a faculty member at at PhD-granting institution.  My guess -- this could checked -- is that most people oversteered, while I played it flatter except in obvious cases like atheism-theism, including always setting aside a healthy chunk for "other".
Reply

2009-12-12
How did you do on the Metasurvey?
Reply to Eric Schwitzgebel
David Chalmers

Australian National University
New York University

Eric: You're really #2, as the official #2 answered only 12 questions, while you and #1 answered all thirty.  We should have had a separate ranking for no-skippers or a penalty factor for unanswered questions, but that seemed too complicated.

The "winner" has given me permission to reveal his identity: it's Alex Byrne of MIT.  Congrats, champ!  You have a fine sociological nose, indeed.  Alex and Eric were a long way ahead of the pack, with average absolute errors of 8.07 and 8.62, respectively, compared to 9.82 for #3 on the no-skip list (then a cluster down to 10.45 for #10).

I do note that both Alex and Eric were beta testers for the Survey, as were #5 and #6.  It's possible that the testing process in effect leaked some information about the distribution of answers.  I didn't give specific figures to beta testers, but a few bits of qualitative information were passed along.  So I ought also to give special congrats to #3 (a well-known philosopher of language) and #4 (an ex-ANU graduate student), both of whom answered all thirty and neither of whom were beta testers.
Reply

2009-12-12
How did you do on the Metasurvey?
Reply to David Chalmers
Richard Brown

LaGuardia Community College, CUNY

The metasurvey was costly to me as I lost a bet on how many philosophers would self-identify as dualists. I bet Dave $100.00 that it would be less than 10% and it actually turned out to be something like 27%! It urns out the only category where I got it right was among people who explicitly identify Philosophy of Cognitive Science as their AOS…coincidentally these are just the people that I usually associate with…I wonder if other people who took the meta-survey noticed that their meta-survey guesses reflected the numbers filtered for their (main) AOS/friends in philosophy? If so then the mistake we might be making may lie in (wrongly) thinking that the group of philosophers one interacts with regularly is a representative sample of the landscape whereas people who did well will probably tend to be those who interact with a large group of philosophers from many different areas. 
Reply

2009-12-12
How did you do on the Metasurvey?
Reply to Richard Brown
Berit Brogaard

University of Missouri, St. Louis

I don't think interacting with a large number of people will necessarily improve your results.  I take myself to be interacting with a large number of people in the target group.  I also was part of the beta-testing.  I still misfired.  Why?  Here is one reason.  When you talk to people, they will tend to lean towards certain views.  So, it's impossible to predict how many people will say "other".  I highly suspect it was pure puck who got this right and who got it wrong (sorry, Alex).  What we should be looking at is how many people got the majorities wrong.  That's interesting!
Reply

2009-12-14
How did you do on the Metasurvey?
Reply to David Chalmers
John T. Maier

Peking University

Like many others, I underestimated the extent of "other." (My results: 13.0% absolute error, rank 163 of 728).

Here is the case that puzzles me most: the actual for Newcomb's problem was 31.4% two-boxers and 21.3% one-boxers. So 47.3% of respondents fell into the "other" category here! I don't understand this. A response that I made several times, that the question is too unclear to answer, is simply not plausible here. Nor is it plausible that many professional philosophers are ignorant of the issue. There are some sophisticated views one could have that would make one reject both options (some sort of non-factualism about rationality, for example) but I find it very hard to believe that such a large number of philosophers hold this sort of view. Is there a breakdown of the data for the "others" available? Alternate explanations?
Reply

2009-12-14
How did you do on the Metasurvey?
Reply to David Chalmers
John T. Maier

Peking University

Someone pointed me to the "fine-grained" data, and that answers my question, though it remains surprising. The most popular answer was unfamiliarity, while the third-most popular was agnosticism. Based on a quick look through the fine-grained data, that gives Newcomb's problem the highest percentage of agnosticism and the second-highest of unfamiliarity (after time).  
Reply