General forum:

Back    All discussions

2009-09-18
Suggestion: A comment and rating system
Hi! I would like to suggest a special rating and comment system that may work very well with this website. I think that if there were an efficient rating system, it would be a valuable aid for readers who are looking for the best content, and for authors who are looking for some extra exposure of their work. This would become specially valuable if the website grows enough to make it impossible for the readers to keep track of the papers, even in the most specific areas.

The main problem one may consider when it comes to ratings and comments, is that users could vote and comment without giving it much thought, like it usually happens in YouTube. If this happened, even in a small degree, then it would certainly make a lot of people angry (for this is not YouTube, where one bad rating doesen't really matter). In order to avoid this problem, I suggest the following: first, that users are only able to leave a rating, if they also leave a comment (but maybe not viceversa). That is, every rating would be associated with a comment. Second, and most important, that only users who have already submitted a paper are made able to vote and leave comments. This is kind of a filter (which is not something I like a lot, really). It is probably true that if somebody has already made the effort of writing a paper, he or she is probably a serious philosopher. But there is an extra reason for making this requirement: when leaving a comment and a rating, the author of that comment and rating will know that if he or she doesen't take it seriously, then he or she will probably get an angry review back. This fear should be enough to prevent most if not all of the users from taking a rating lightly. And if this is true, then it could be expected that the general rating of the papers actually says something reliable about their quality. The reason for not allowing anybody to leave a rating without a comment, is that even in the case of the most serious rating, if the author of the article being rated is not able to read the reasons for that rating, then he or she will not know if it was a serious rating or not.

Users who have not yet submitted an article could very well be allowed to comment. In the end, bad comments don't hurt: only bad ratings do.

Also, consider that in many websites with ratings and comments, users manage to convey things like "UR8-IR8" (that is: you rate, I rate), meaning that if you leave a serious comment and rating, I will do the same. I think that in this website, that kind of behavior could be expected, and it would boost the rating and comment system.

Anyway, that is my suggestion. I hope you like it and consider incorporating it.

Congratulations for the website. You are helping a lot to bring philosophy into the 21st century.

2009-09-18
Suggestion: A comment and rating system
Thanks for the thoughtful suggestion, Luis.

We've been thinking about setting up a system which plays roughly the role of ratings. There are major concerns, however, some of which go beyond the issues you address. Perhaps the most fundamental issue is that a simple good-bad scale would not take into account the various  respects in which a work can be valuable. Some works are good introductions to a topic but not good research contributions. Others are canonical readings on a topic for historical reasons but perhaps completely wrong. Among the variables that can be of interest to a reader are a) quality qua introduction, b) accuracy / truth, c) pertinence to current debates, d) canonical status, and probably others that don't come to mind right now. With the number of people using PhilPapers as their primary source of research content always growing, we have a responsibility not to create a ranking system that would introduce a new bias in reading patterns. In brief, we'd have to ask users to rate entries along multiple dimensions.

We're also concerned about the quality / pertinence of votes. Your suggestion here risks backfiring: people might upload mediocre papers just to be allowed to vote! This is to be avoided at all cost. We don't want to be swamped with undergraduate term papers.  Leaving comments may be a good idea. But the "peer pressure" moderation you expect would require that ratings not be anonymous, and this is highly problematic. In many cases the pressure would be too great. Philosophy is a very small world. It doesn't take long for one to meet a fairly high percentage of people working on one's favorite topics. A lot of people won't want to express their judgements of others' works so publicly because of this. This would probably repel many of the most active researchers (who tend to travel a lot).

Having said this, I'm acutely aware of the need for some means to find the best articles on PhilPapers, in all relevant respects. I think a system with the following features might be best:
  • Several rating dimensions (good intro, canonical, cutting-edge, truth, etc)
  • Anonymous ratings
  • Weighting of ratings by some "authority factor". Authority factors could be computed from users' CV (number of publications weighted by "prestige" of venue, other credentials, etc, as stored in their profiles). We could also give higher authority to PhilPapers editors, who would have a duty to participate actively in the rating process and would be briefed to ensure they use the system properly. There's a risk that some might be tempted to fake publications to boost their voting powers, but this is relatively easy to counter. If someone submits a term paper and labels it as a phil review paper, it won't be long before someone else finds out and corrects the record. If we add a "report fake citation" option this could be a high risk activity.
I'm not saying we're going to implement a system like this, but I'm curious to know what you and others think about it.


2009-09-28
Suggestion: A comment and rating system
Reply to David Bourget
David, thanks for your answer. I've been thinking about the problems you mentioned, and I would like to suggest a weighting system for ratings, which can be easily combined with what you mentioned about the multiple rating dimensions and anonymous ratings.

Suppose that every rating is associated with a comment (but not necessarily the other way round). You could then add an option next to each comment-rating, allowing every user (except maybe the author of the paper being commented-rated) to rate the comment with a "thumbs up" or "thumbs down", like in YouTube. Every comment-rating will have, then, a positive or negative rating, which is basically the number of thumbs up minus the number of thumbs down, again like in YouTube. Now, the idea would be, that the balance of thumbs ups and thumbs up of every comment-rating determines the weight of that comment-rating. This way, if the comment is bad and the rating unfair, the negative rating on the comment-rating will turn it harmless, whereas if its the other way round, it will boost its influence on the final ranking of the paper.

There is, of course, a risk of friends joining forces to rise the influence of a paper by rating it positively and then also rating positively all the other comments-ratings. But this will hardly become a mayor issue, for as soon as one paper rises high enough on the rankings to attract some extra readers, it will only be a matter of time until those readers put it down to where it belongs, if it deserves it, or not, if it happens to be a good paper. Of course, I take it for granted that users will not be allowed to rate more than once, neither papers nor comments-ratings. That makes it difficult to counter the trend of the masses (unless you create puppet users, but this would be thinking too far ahead).

Another possible solution to this problem would be to forbid any user who has made a comment-rating, to rate other comments-ratings. This doesen't sound very attractive, but it doesen't hurt to consider it.

I would also like to mention another, very different, strategy to create a ranking among papers. Allow every user to create a personal "top list" of papers. This list could be public or not, but anyway, based on that list, it would be possible to create the list of everyones "favorite papers". This is a very rough idea, which would need some developing, but I just wanted to mention it, in case you see any potential in it.

Thanks again for the website.