A NOVEL SOLUTION TO ACADEMIC PUBLISHING Human decision-making from an alternate perspective. by E. Garrett Ennis _________________________________________________________________________ The entertainment industry devotes lots of time to the process of choosing manuscripts and screenplays for production as books and movies. Billions of dollars are regularly risked in attempts to find and create the next major media franchise. As a result, nothing's more desired than the ability to judge how audiences will receive a story before money is spent. But there's a massive disconnect there, where hundred-million-dollar properties fail constantly, for reasons that can only be guessed. The most popular saying, by the screenwriter William Goldman, is that "nobody knows anything," and there's fear in the voice of even high-level executives when they greenlight a film. If you work at a production company or as a literary analyst, who helps make those decisions, you'll hear it. This confusion effects not only audiences, who get worse product, but everyone who creates it as well. It gives wealth to people who seem to lack ability, and ignores or ruins the careers of those who clearly have it, with no apparent cause, which is intolerable to our instincts. We naturally want those who give us the most to receive the most as well, so they continue doing it. (Image: Gaetano Chierici) Meanwhile, there's academic publishing, where papers are chosen to be mass distributed in journals to the scientific community. One might assume, as I did, that this wouldn't occur there. After all, science often deals in concrete numbers, so it should be clear which ideas produce better results, in contrast to entertainment, where feelings are the only measure. But the same problems seem to exist. A recent paper on the matter by Jonathan Tennant and Charlotte Wien states that "researchers are evaluated primarily based on the journals that we publish in, rather than any intrinsic merit," and "evaluations are baseless, senseless, and harmful." Even Einstein's "Annus Mirabilis" papers were ignored by physicists when they were published, and would take over a decade to receive public recognition and the Nobel Prize. The Nobel committee itself has been accused of the same style of decision-making. Speaking 1 of its literature category, the journalist Andrew Brown said that "the pursuit of excellence in art [is] entangled with the pursuit of social prestige. The academy behaved as if the meals in its clubhouse were as much an accomplishment as the work that got people elected." The ideal is for a great idea to rise of its own merit, regardless of the notoriety of its author or the biases of others. But if the most prestigious judges and the greatest scientific papers display the same biases and odd treatment, and the pattern matches exactly what we see in entertainment publishing, it's clear that this ideal has likely never been realized. But this may not be for the reasons we think. Perhaps this is not because people are bad, but because people are people. It's simply the result of natural and reasonable human behavior, and we haven't fully understood it. We can see this by looking through academic lectures and papers on this topic. Several common explanations tend to be offered. First, that evaluators have limited time, so they must use shortcuts to recognize whether a paper is worthwhile. Second, that evaluators use poor measures because they don't know enough about the field in which they're judging, and third, that people in academia are selected for conscientiousness instead of creativity. In other words, their ability to follow rules instead of understanding things. Which suggests that they can only judge a paper by how well it fits into the system they've been taught. These are logical conclusions that a smart person would find. But when compared to enough examples, it's clear that none of them are fully accurate. Reading staffs like the one I worked on have no deadlines for many submissions. But their judgment remains disconnected from public opinion. So much so, actually, that experiments found that staff readers will reject their own company's published books, repeatedly, when the author's name is changed. The problems in literary publishing seem to match scientist's frustrations with academic journals. (Image: Discovery Channel) But if you think these people simply aren't competent enough to know good work, consider that Vincent Van Gogh and Leonardo Da Vinci didn't care for what are now considered their greatest works. Van Gogh called "Starry Night" a failure in his letters, and Da Vinci never 2 turned the "Mona Lisa" in for his commission and didn't even mention it in his will. It's safe to say that both were experts in their field, had plenty of time to judge the paintings while making them, and had creative instead of conscientious minds. Likewise, the same conscientious academics who seem stuck in the system when evaluating papers, toss that system aside to hand honorary degrees to unqualified celebrities. So something else must be happening. On top of that, the most common solution in academia, hiding names of authors and referees to, in theory, force impartial evaluation of the work, doesn't succeed either. A 1985 essay by journal editor Richard Watson said that double-blind refereeing "fosters at least as much irresponsibility and unfairness as it is meant to prevent" and, among other things, "allows a referee to read a paper quickly and accept or reject it on superficial grounds." So this is a far more vicious problem than we may realize, and with so many failed theories and solutions, it's tempting to throw up our hands, use a catch-all phrase like "life is hard," and move on. But let's not. We have centuries of examples of valuable people starving, being ignored, and even, like Van Gogh, taking their own lives due to this situation. Combine that with the knowledge of how many we must have never seen at all, and the problem was serious enough to me to study this for over a decade in an effort to truly find out what was happening. Which, it now seems, has been happening in academia as well. This is an overview of the results. Evaluators supposedly lack knowledge or creativity, but expert creative artists shelve their own work in similar situations. (Image: TIFF) The first important thing is that the human decision-making apparatus doesn't work the way we think it does. It's assumed that emotion and reasoning are separate functions, and that the emotions are an ignorant, animalistic thing that must be turned off for ideal reasonable thinking. But emotions are actually a key part of reasoning. They allow our instinct, which is extremely sophisticated but inflexible, to work in concert with our intellect, which is 3 comparatively simple but able to adjust to situations that aren't already coded in. The instinct, instead of dictating our actions, uses emotions to adjust the likelihood of what our intellect decides, based on what the instinct subconsciously observes and calculates may happen. So if we are in the dark, our instinct detects that we are more likely to be taken by surprise and thus are in greater danger, and gives us the feeling of fear, making us more likely to run away from any sign of a threat, and so on. Thus, these emotions are fundamental to our choices, and neuroscience research by Antonio Damasio found that people with nonfunctioning emotional systems actually can't make decisions at all. Which means that reason can't be separated from our feelings. But there's much more to this, because we haven't fully understood how we process those feelings. We respond to the emotions we feel at a given moment, but our brain actually doesn't track the causes of those emotions. And indeed, multiple things cause the same chemicals to be released. Things that threaten us all release a single chemical that puts us in a defensive state and makes us avoid them. Things that have potential benefit all release a single chemical that makes us want to pay attention. Things that confirm our status or beliefs all release the same chemical that makes us feel "warm and fuzzy," and things that improve our understanding or status all cause a single powerful high of their own. These correspond to specific brain chemicals. But what's important is that there are only a handful of them (for what we're discussing), and the causes of this small number of emotions are indiscrete, or undivided in our minds, like buckets of water dumped into a bathtub. And when processing situations, the brain simply takes in the total amount of things that release feelings, the potential threats, confirmation of beliefs, improvements to our status and so on, and releases a corresponding wave of chemicals. We then respond to that overall resulting feeling, the total amount of water in the tub, without knowing its specific causes. Award voters have been said to value social dinners as much as merit. This is because they literally produce the same feelings. (Image: James Linton) This understanding leads to two important things. First, the defensive, threat-avoiding chemical can stop the whole process. If it goes in at any point, i.e., detecting a danger to our status or well-being, we will disconnect from the situation and not feel any other good things. 4 The reason for this is simple. Our reputation is more important than our rationality. There's no point in participating in something or helping the group if it will harm us personally. It actually defeats the purpose of doing the thing in the first place. Likewise, it doesn't matter how right our ideas are if we end up rejected from the group or looking foolish, and can't mate. So people at all levels instinctively only pursue the truth when it won't hurt their own status, which is logical. Expecting otherwise is like asking them to paint the house when they can't breathe. Secondly, the human decision-making process, as we see it here, actually takes into account every potential result of its choice when it considers a course of action. This includes how it effects its social group's efficiency and survival, how it effects its reputation within the group, and how it might help the person understand the world and make predictions. These are what's represented by the overall feeling that it produces, which dictates its decision. Confusing results are often a sign of incorrect assumptions. (Image: MIT) But one thing there is crucial. As we said, two major contributors to our overall feeling are our social group's efficiency, meaning our desire to find, follow and reward the most capable people, and our own status and acceptance within that group. This actually means that we have inherent emotional responses to encountering people who have great ability or status, and to things that have great effect on or interest from the rest of our group. All of these trigger the feelings of curiosity and "warm" belonging that make us want to follow, pay attention and participate, and thus contribute to the overall feeling we can get from something. This is thus a powerful factor when we decide if a paper is good enough to publish. Mainly because we choose based on whether it passes a certain threshold of feeling for us. In social dynamics, a similar concept is used to explain what makes someone say yes to a date, which is called "buying temperature." In this case, that bar is set by how we feel about other good ideas we've known or learned about in our lives, which, actually, were each accompanied by these social feelings. After all, we each spend our formative years learning from parents and 5 teachers, who are more powerful than we are, and often surrounded by attentive students. The ideas we're taught in that period are the best that society thinks it has. Likewise, in entertainment, we tend to be part of an audience, or online comment section, taking in the supposed best entertainment, with an author, actor, or other powerful figure as the center of attention. So we're conditioned to expect the accompanying feelings when we see a "good" thing. Note though, that what we come to expect is not the situation, being in an audience and so on, but the stimulation, the brain chemicals we feel. This matters most when a person considers a new submission. A screenplay, manuscript, or, as it seems, scientific paper with no reputation. We're supposed to judge it by the quality on the page. But we actually decide in the only way we can, by consulting how reading the thing makes us feel, and since that feeling lacks the social elements that we've learned come with a good idea, the new submission automatically fails the test. But our instinct doesn't tell us why we feel what we do. And thus, the evaluator essentially guesses at why the submission isn't good enough to them. This is why researchers, as we quoted earlier, find the feedback their papers receive to be "baseless" and "senseless." Furthermore, this occurs no matter how good the submission may be. Because, as we said, all the great ideas we remember combine the ultra-high quality of the idea itself, since it's usually being taught in school, with feelings of social momentum and a powerful person as the source. We remember the things that effect us the most, and on a large enough sample, the high water mark for something will always involve all the variables being at or near their peak. So our measuring stick, our waterline, inevitably requires a level of emotion that an unknown idea can never reach. History confirms this. As we said, the "Annus Mirabilis" papers sat in obscurity for years. Likewise, Beethoven wrote in an early letter that "people will not believe in me because I have not yet made a wide reputation." Luciano Pavarotti, now widely considered the best opera singer ever, worked odd jobs for six years while singing in small unpaid recitals. Similarly, J.K. Rowling's manuscript for the first "Harry Potter" novel was rejected twelve times, and the publisher who picked it up was "unenthusiastic" and advised her to get a job. And lest you think she had to improve, Rowling was rejected again years later when she sent a new book to publishers under a pseudonym, and was even told to take a writing course. Hopefully the point is established. A submission without reputation or social momentum, in the mind of any evaluator, will register as not good enough to be accepted, no matter what it is. But this does, of course, bring us to another issue. This suggests, essentially, that nothing will ever be published. Which is slightly different than what we observe. So when do unknown submissions actually get picked for publishing? Well, since the social feelings are missing from any unknown paper, it happens when the paper carries an extra source of emotion for the evaluator that gets it to the necessary level. Such as, for example, if it's about farming, and the evaluator grew up on a farm. Or if the author of the submission is related to the evaluator, or many other random things. 6 This can be seen everywhere in entertainment. The screenplay of the film "Alien," for one, only got produced because the decision-maker in question had just seen "Star Wars" and was looking for anything that happened in space. But the same is true for our earlier examples. Pavarotti got his first real opera job because a singer was looking for a tall tenor to stand beside her. "Harry Potter and the Philosopher's Stone" was actually picked up by the original publisher because his eight-year-old daughter, who was too young to have finished this conditioning process, liked it. Our interest in Beethoven, despite his talent, is largely from the dramatic irony of him being a deaf musician. Then, on the scientific side, we return to Einstein. Did he become famous because the brilliance of relativity won out? Actually, in the year his eclipse tests were conducted, the editor of the New York Times happened to be an amateur mathematician with a noted distrust of Isaac Newton's theories. Now let's apply this to academic publishing, and particularly double-blind refereeing. As we've discussed, it's been found to do little for unknown authors, since removing normal social feelings from papers doesn't actually make unknown material have more appeal to evaluators. Instead, it makes everything they see have zero appeal. Our normal, seemingly merit-based preferences are actually the choice between things with sufficient social momentum. (Image: UPenn) But as we said, in that situation, random extra sources of feeling become disproportionately powerful. They are, in fact, required to get an evaluator to feel enough about something to decide that it's good. This is why the editor we quoted earlier concluded that double-blind evaluation allowed a paper's referee to "accept or reject it on superficial grounds." So while we see here that double-blind evaluation can address some goals in improving publishing, like reducing the impact of personal squabbles, in terms of the larger goal of having pre-publication evaluation match post-publication group opinion, double-blind refereeing cannot work. But is there anything that could? I think so. 7 The problem to this point has, of course, resisted the attempted solutions. But a clearer understanding of a problem can often suggest new answers. In this case, the key thing we have to be aware of is that levels of feeling reached by quality combined with outside factors such as social momentum and our desire to associate with high-status people are part of our instinctive requirements to recognize a good idea. But to fully understand this situation, we must recognize what this means about our normal opinions of popular things. In entertainment, at a glance, they look purely merit-based. We appear to like and buy the books and movies that are the most entertaining to us in and of themselves, and people who work at high levels in those fields see a clear connection between the entertainment value of their work and their success. I suspect this also transfers, in some degree, to how people seem to judge popular scientific ideas and papers, and the experience of people who are at high levels in various scientific fields. The best ideas appear to succeed most of the time, students are taught that the quality of their work will determine their fate, and I suspect that this is what led us to the belief that there's a purely rational version of science and decision-making that we could uncover if we just get rid of other influences. But of course, as we've been discussing, that seemingly rational world, that exists at the high levels of these fields, evaporates when it comes to unknown work or authors. This seems like a paradox, but if we investigate it deeply, we can find a logical answer. It turns out that when two things generate the same basic feelings for us, we decide what we like more by whatever extra feelings one creates over the other. This means that when we're presented with many potential films to watch (or, I suppose, competing scientific ideas to accept), and each has been published and has social momentum behind it, we decide what we like based on what creates the most emotional reaction for us above that. Sometimes this is due to extra outside factors, like one author having a better life story than another, but, most commonly, the difference-maker is the actual entertainment value of the movie itself. Or, in the case of publishing, likely the accuracy and intellectual value of the paper itself. So, when dealing with things that are already popular, it seems obvious that we simply choose the highest-quality options with no other criteria. It's an illusion though, because stripping away those subconscious elements will cause us to totally overlook the same things. But of course, that situation is still closer to rational decision making than anything else. And things would indeed be much better if we could extend it to the judgment of new and unknown papers. Fortunately, with the things we've discussed in mind, a possible method does present itself. Which is not to hide the names and present papers as unknown, but to do the opposite. You might call this method a "common context" evaluation. We can't remove the emotional attachments that allow the human decision-making apparatus to function in its most rational way, so instead we can do what hackers might call "spoofing" it. Which means presenting all 8 papers that are under consideration as coming from high-status authors, or being the subject of existing interest, or coming from reputable institutions and so on. All of these may have to be done simultaneously, as most of the social elements that boost papers are naturally present in academic organizations, and since this default level of emotion comes with nearly everything they deal with, they have no reason to be aware of it. We can, in fact, only see where things break down when we go to the extreme ends of the spectrum, such as with incredibly valuable outsiders like Einstein. So it's not clear how much reputational boosting would be required to match normal social acceptance. A sneaky computer access trick could help us activate our most rational thinking. (Source: Forcepoint.com) This means that the things which accompany each paper would likely have to be tweaked over time. Too much positive context would likely cause evaluators to approve everything they're shown, except for things that personally threaten them or are so wrong they'd be afraid to say yes. And, as we see, too little context makes evaluators reject everything except in random circumstances. Once the right spot is found though, while other small outside feelings could still influence choices, this could produce much better judgments of unknown work than we've had before. The concepts in this paper also lead to far more than what we've covered. I've written other papers about them, and discuss them regularly on my Youtube, Twitter, and Facebook pages. I can be contacted at those or e-mailed at EGarrett01@yahoo.com. Thanks for your time. -Garrett Ennis