Who Determines What Science Is?

At Dartmouth, I’ve been told by a professor that I do not know how to do science. I’ve been told I can’t continue with a research idea, which applied a scientific study my peers and I completed to active resource management, because it “wasn’t science.” However, I’ve also been told by a professor that if I don’t know what science is, than neither does he. All of this has led me to wonder who determines what science is and thus explore the often unquestioned process through which science research gets funded. What I’ve found has led me to wonder whether or not our prevailing methods of choosing what scientific research is worth funding may actually limit the amount that we as a society can learn and accomplish through scientific research.

Resources for scientific and technical advance are scarce in today’s society, so the institution of peer review has been developed to identify the most “meritorious” ideas and give those ideas resources for further research. Peer review ensures that the most well-published, successful scientists assess the potential success and impact of projects proposed on a topic that they are knowledgeable about. The process is meant to be an objective way to assess proposed research and identify the most impactful studies. However, a working paper coming out of Harvard Business School has suggested that peer review may actually create an inherent bias against novel ideas, decreasing the variability of ideas that are funded and thus the potential for new, breakthrough ideas that are necessary for advancement. 

The study (Boudreau et al, 2012) examined how 142 peer reviewers within a leading medical research university allocated the first $1 million in grant money given to research on a major endocrine system disease. The reviewers had to collectively rank 150 different research proposals based on potential impact and quality of the proposed research project. The study then measured the novelty of each proposal, based on the amount of descriptive keywords which were included that had not been used previously in the published medical sciences literature. Boudreau et al., 2012, found that the peer review evaluators possessed a bias against novel projects; with one standard deviation increase in novelty for a research proposal, the proposal decreased an average of 4.5% in rank. The study found that this predicted decrease in rank from novelty was not related to the quality or merit of the proposal. 

This bias against novel projects that peer review presents may arise from the fact that peer reviewers are looking for projects that are most likely to be successful, or get published, and projects that are most like the published literature are more likely to be successful than those that stray from previously published literature. Indeed, a study by Fleming, 2001, showed that in patent proposals, more novel proposals had on average less success. However, this study also showed that novel proposals had higher variability, leading to more “breakthrough” ideas leading to shifts in scientific paradigms, despite the overall lower success rates. 

The problems with peer review have been discussed studies other than Boudreau et al., 2012. A paper published in Jama 1990 by Harrobin (1990) discussed how in biomedical literature, many reviewers assume that peer review is just “quality control,” when in fact it should create an appropriate balance between both quality control and the encouragement of innovation to ensure new types of medicine are introduced to improve patient care. Another paper by Travis and Collins (1991) discussed how cognitive particularism in reviewers, or decision-making that is highly influenced by a particular school of thought, can limit the opportunities for interdisciplinary research and frontier science. 

In our society today, peer review to a certain extent thus seems to present a fundamental trade-off between choosing scientific research projects that are likely to succeed through being published and more diverse, potentially revolutionary projects with higher failure rates. However, I believe that such a drastic trade-off should not exist. Yes, we need peer review to ensure that the most well thought-out, meritorious projects are chosen, but this should not result in an inherent bias against novel or interdisciplinary ideas regardless of their merit. Novel ideas are necessary for the advancement of science, as they cause shifts in scientific thinking that could not occur if only projects similar to already existing research are funded. Without novel ideas, we would not have Max Plank’s quantum principle, Einstein’s theories of special and general relativity, Heisenberg’s uncertainty principle, and many of the other foundations of modern science. And without interdisciplinary research, how are we going to address the pressing and complex environmental and social challenges of our century? 

We do not need to get rid of the peer review process to encourage more diverse research, but instead those who are part of the peer review process must not immediately dismiss meritorious research proposals because they are “not science,” as is defined by previously published research. Perhaps, we need more than just the experts providing input into the review process or more journals that encourage innovative, interdisciplinary research to marry traditional success and novelty. Here at Dartmouth, we like to say, “Lest the old traditions fail,” but when it comes to advancing science and addressing the pressing issues of our generation, I believe we must move beyond those traditions.