Peer review, the process by which articles and applications are vetted by experts in a researcher’s field, is a central part of scholarly publishing and grants in fields from physics to classics. For many scientists, it is the crucial hurdle to clear before publishing new research, one that is meant to provide a measure of validation and quality control. Indeed, it is often considered one of the markers of objective scientific inquiry. Peer review is so central to the way science works that many observers assume that, like modern science, the system had its origins in the Scientific Revolution of the seventeenth century. That turns out not to be the case. In fact, the “referee system” did not exist before the nineteenth century and the term “peer review” was a creation of the late twentieth century. Furthermore, systems of review were not intended to be a mark of scientific legitimacy; that expectation only arose during the Cold War. Peer review’s history, like its future, is contested as scientists continue to argue over its place in the scientific enterprise.
The prehistory of peer review
The origins of peer review are usually explained with an appealing, but mostly incorrect, myth. In 1665, the Royal Society of London gave Henry Oldenburg (1619–1667) permission to compile the Philosophical Transactions of the Royal Society, a new periodical that would collect important scientific findings and news. Oldenburg wisely saw that he would need expert opinions to decide which of the available papers he should publish, and thus the expert referee has had a central role in scientific publishing ever since the founding of the first scientific journal.
Image 1: Philosophical Transactions of the Royal Society. Source: Archive of the Royal Society, used under the Creative Commons Attribution 4.0 International license.
That origin story appears to have arisen from a 1971 sociology paper, “Patterns of Evaluation in Science: Institutionalization, Structure and Functions of the Referee System” by Harriet Zuckerman and Robert Merton.1 Zuckerman and Merton’s article about refereeing in the sciences was widely read and extremely influential, and their account of peer review’s history has been repeated in many scholarly books and papers since the 1970s.2 When historians such as Adrian Johns and Aileen Fyfe took a closer look at the early Philosophical Transactions, however, it became clear that Zuckerman and Merton had misunderstood or misdated some of Oldenburg’s correspondence, and that Oldenburg did not in fact employ any system at the Philosophical Transactions that could reasonably be compared to modern refereeing.3 Although the members of the Royal Society were nominally tasked with reviewing the contents of the Transactions, in practice Oldenburg maintained such tight control over the journal that he was sometimes referred to as its “author” instead of its editor.4
In recent years historians have re-analyzed the historical development of specialist scientific journals, and it has become clear that neither peer review nor the scientific journal sprang full-formed from the seventeenth century, as the Zuckerman-Merton story would have it.5 Throughout the eighteenth century and well into the nineteenth, journal articles were just one way among many for researchers to communicate recent scientific findings. Monographs, literary essays, public presentations, and pamphlets were all considered valid methods of disseminating scientific knowledge—and very few works in those formats went through anything like external refereeing.
Pre-publication review was most common at scientific societies, where members were anxious to ensure that anything issued with the society’s name on it did credit to the society as a whole. For example, all publications issued by the Académie des Sciences in Paris had to be vetted by the author’s colleagues in the Académie.6 In 1731, the Edinburgh Medical Society informed readers of its Medical Observations and Essays that
Memoirs sent by correspondence are distributed according to the subject matter to those members who are most versed in these matters. The report of their identity is not known to the author. Nothing is printed in this review which is not stamped with the mark of utility.7
Those systems of internal review and critique, however, do not seem to map onto or presage the external refereeing system that we know today. Aileen Fyfe and Noah Moxham have called this period the “pre-history” of peer review—a useful term that reminds us we shouldn’t conflate formal refereeing systems with the more general idea of having scholars comment on each others’ work.8 The purpose of those older systems was to safeguard the reputation of the society affiliated with the publication. Internal discussion created a communal sense of investment and responsibility in a project, and there was no expectation of anonymity for anyone involved in the process. Many of those older systems of internal critique were also meant to ensure that nothing seditious or heretical was printed under a society’s name. Furthermore, most vetting within a learned society would have been somewhat informal and haphazard; a member of sufficient standing would expect that his work would be approved by his colleagues with minimal fuss. (The gendered language here is intentional, as most early learned societies refused to accept women as members.)
The concept of having peers evaluate each others’ work, in other words, is far from new. But it would be a mistake to look back at informal, unsystematic critiques within elite learned societies and equate them with the modern system of peer review. In purpose and in form, that kind of internal vetting was very different. For systematic review by anonymous experts who are supposed to act as gatekeepers, we need to look to the nineteenth century.
Referees: From adviser to gatekeeper
The origins of systemic external reviewing are tightly linked to the nineteenth century transformation of the academic journal into the dominant form of scientific communication.9 Around 1800, a physicist or naturalist could choose to announce research results through talks at scientific meetings, university lectures, newspaper articles, pamphlets, essays for general-interest literary magazines, and monographs—and all of those methods would be considered valid ways to both communicate their findings and establish priority for their ideas. By 1900, that was no longer the case; researchers were expected to publish their findings in a reputable scientific journal in order to receive credit for their work. But “reputable” in this period did not necessarily mean “refereed.” Many influential journals—and arguably the most influential ones of all—did not use refereeing at any point during the nineteenth century.
In the early nineteenth century, French journals such as the Annals de Chimie sometimes included essays evaluating recent papers released by the Academie des Sciences. In 1831, inspired by those French essays, the English polymath William Whewell (1794–1866) made a proposal to his fellow members of the Royal Society of London: that papers submitted to the Philosophical Transactions should be sent to two Fellows, who would then write a report about the paper. The signed report would be published in the Royal Society’s new journal, the Proceedings of the Royal Society of London. The reports would not only generate scientific debates but also material for the Royal Society to publish in its new periodical.10
Whewell’s scheme quickly ran into trouble when the first two referees—Whewell himself and the mathematician John Lubbock (1803–1865)—disagreed vehemently about the claims of the paper they’d been sent and had trouble drafting a report they were both willing to sign. Only a few referee reports were ever printed in the Proceedings. But the practice of soliciting written reports about Transactions submissions endured, with a twist: instead of asking referees to publish their comments and make their identities public, referee reports became confidential documents that helped the Society decide whether or not to print a paper. The author of the original paper would never know who his referees were or what they had said about his work. It was not until the early twentieth century that journals started sending their authors either paraphrased or verbatim copies of referee reports.
By the mid-nineteenth century, one of the major responsibilities of the Secretaries of the Royal Society—there were two, one in the physical and one in the biological sciences—was to find people to referee submissions to the Transactions. If one of the Royal Society Secretaries wished to communicate concerns about the paper to the author, he would write the author a personal letter and suggest some changes that the author might want to make before the paper went to print.11 Other learned societies followed the Royal Society’s example and began consulting anonymous referees about papers submitted to their journals. Great Britain’s Geological Society and Royal Society of Chemistry, for instance, both adopted refereeing systems for their publications in the nineteenth century. In the US, the American Physical Society and the American Sociological Association adopted refereeing at their journals in the early twentieth century, but not for every paper. Papers the editor liked were often not sent out for external opinions; referees were most often employed when a paper seemed questionable and the editor wanted help justifying its rejection.12
In the late nineteenth and early twentieth centuries, ideas about referees and their purpose started to undergo some important changes. In the 1840s and 1850s, the referee’s job was to advise the editor. They were supposed to give a paper a fair hearing and tell the journal’s editor if anything about it was obviously embarrassing and might discredit the publication; referees were not necessarily expected to make judgments about the truth of the findings. As the number of journals expanded, however, some observers started to complain that there were too many scientific articles, and too many of them were terrible—“veritable sewage thrown into the pure stream of science,” as physiologist Michael Foster put it. Those concerns about the quality of the scientific literature led members of the scientific community to view the referee not as a mere advisor, but as a gatekeeper—someone who was responsible for the quality of the scientific literature as a whole. In practice, that vision of the referee as a guardian of the scientific literature led many journals to rely on a small number of prominent scientists for their refereeing in the late nineteenth century, meaning that an elite inner circle had enormous power to shape a journal’s contents.13
Though it is tempting to take this as the model for the modern system of peer review, there were many important journals in the nineteenth century that didn’t use referees at all. For-profit scientific journals like the Philosophical Magazine and Nature relied entirely on their editorial staffs to make decisions well into the twentieth century.14 Many of those commercial journals were run by dynamic editors who strongly preferred to evaluate submissions themselves and accept or reject them based on the editors’ own authority. Furthermore, learned societies had the luxury of issuing their publications whenever they had enough content. Commercial journals, on the other hand, were expected to appear on a regular weekly or monthly schedule. Editors were usually trying to recruit content, not reject it, and they did not want to have to wait for referees to return their comments before moving forward with a promising article. But since refereeing at a journal was not seen as a sign of scientific rigor or respectability during this period, the lack of referees did not prevent commercial journals for being venues for many important scientific papers. The choice to use referees or not was essentially a logistical one, not an epistemological one.
Image 2: Cover of Nature from first issue in 1869. Source: Nature (https://www.nature.com/articles/d41586-018-07844-6)
In the nineteenth century, refereeing was almost entirely a British and American practice. Outside the English-speaking scientific world, very few periodicals consulted referees about their contents. In 1835, when the French Académie des Sciences founded the Comptes rendus hebdomadaires des séances de l'Académie des sciences, they made the deliberate decision not to solicit reports about papers before moving forward with publication. The Académie wanted to make quick publication a priority and thought consulting outside opinions would just gum up the works.15 In Germany, prominent journals like the Annalen der Physik und Chemie were usually run by powerful editors, along similar lines to the British and American commercial journals.16 Scientists who weren’t used to outside refereeing did not necessarily embrace the practice with enthusiasm when they encountered it for the first time. Famously, in 1936 Albert Einstein was incensed when John Tate, the editor of the American journal Physical Review, sent him a referee report about a paper he had recently submitted. Einstein informed Tate that he and his coauthor Nathan Rosen
had not authorized you to show [our manuscript] to specialists before it is printed. I see no reason to address the—in any case erroneous—comments of your anonymous expert. On the basis of this incident I prefer to publish the paper elsewhere.17
Journals were not the only institutions meant to evaluate the quality of science. At funding bodies, external refereeing of grant proposals was quite rare prior to World War II. Private grant organizations such as the Rockefeller Foundation, for example, generally left funding decisions in the hands of their employees, who were trusted to evaluate a scientist’s worthiness for a grant internally whether or not they understood the science at stake.18 Even government funding bodies tended to let their employees or a small number of trusted advisors make funding decisions with little or no input from experts who were not affiliated with the grant organization. When the Emergency Association for German Science (later the German Research Foundation) was created in 1920, its founders asked a small number of elite scientists to evaluate proposals, and the process judged the applicants’ personal qualities as much as their science.19
A few grant organizations did rely on referees to provide input on proposals, mostly ones affiliated with learned societies or governments. For example, the American National Research Council, the private research arm of the National Academy of Sciences, developed increasingly formalized refereeing procedures during the 1920s and 1930s, in part to ensure fairness to researchers without established national reputations.20 But many government funding bodies, including the American National Institutes of Health and the British Medical Research Council, preferred to create an in-house panel of researchers to evaluate grants instead of reaching out to experts outside the funding body.21
At both journals and funding bodies the use of external referees was entirely optional prior to the Second World War. If a journal prized fast publication, or if a funding body trusted its employees to make decisions, the organization could eschew systematic refereeing with no harm to its reputation. There was little to suggest that refereeing would become an essential part of scientific knowledge-making.
Peer review in the Cold War
Journals and funding bodies both began placing more emphasis on external refereeing following the Second World War. One reason for the shift towards refereeing at journals was the increasing burden on editors as the number of scientists—and scientific papers—expanded due to generous Cold War funding. At the American weekly Science, for instance, the Editorial Board handled all refereeing in-house for the first half of the twentieth century. In the 1950s, however, members of the editorial board complained that “the job of refereeing and suggesting revisions for hundreds of technical papers is neither the best use of their time nor pleasant, satisfying work,” and agreed to begin sending papers to outside experts.22 Similarly, when the American Journal of Medicine was founded in 1946, its editor Alexander Gutman wanted to offer his authors fast publication and decided to handle acceptances and rejections almost entirely on his own. However, as the journal became more popular, Gutman was unable to keep up with the number of submissions, and by the 1960s he too had begun sending papers out for external opinions.23
But the growing workload at scientific journals does not explain how refereeing became crucial to the very idea of scientific rigor. The link between refereeing and scientific legitimacy seems to have followed the dramatic rise in government-sponsored research in the postwar United States. Between 1948 and 1953, federal spending on scientific research in the US increased by a factor of 25.24 The massive expansion of government funding led to more public attention on scientists—and to suggestions that science should be more accountable to the public and to members of the US Congress. Scientists, on the other hand, were less than enthusiastic about the idea that their grant proposals should be evaluated by laymen with no scientific training.
The tension between accountability to the public and scientific autonomy came to a head in the mid-1970s during a controversy over research grants awarded by the National Science Foundation (NSF). The early 1970s were a time of growing economic crisis for the US, and three legislators—Republican Congressmen John Conlan (1930–) and Robert Bauman (1937–), and Democratic Senator William Proxmire (1915–2005)—launched a series of very public attacks on specific grants the NSF had awarded. All three men accused the NSF of awarding frivolous grants—largely in the social sciences—and wasting taxpayer money on projects such as a middle-school sociology curriculum and a study of stress in rats and monkeys. Bauman and Proxmire argued that the NSF’s poor decision-making justified far more Congressional control over the grants they awarded.25
Conlan took his criticisms a step further: he accused the NSF of employing a shoddy, secretive review process. Unusually for a grant organization, the NSF had employed external referees to evaluate proposals since its foundation in 1950. According to Conlan, however, its refereeing process was a sham. He claimed that NSF employees had the power to make whatever decision they wanted, and that they frequently ignored referee opinions. Conlan argued that the full text of all referee reports, plus the names of the authors, needed to be available to both the grant applicants and the public at large. The NSF, however, insisted that referees would not be able to offer candid feedback unless their identities were kept confidential. By this time, referees for journals and for the few funding bodies who used them systematically had grown to expect that their identities would remain a secret from the authors, and NSF Director H. Guyford Stever cast Conlan’s request as a major break with scientific protocol.
Image 3: Rep. John Conlan (Arizona). Source: Congressional Pictorial Directory (1973), 93rd US Congress, p. 12
The controversy eventually became so heated that Congress convened a hearing on the NSF’s review procedures, held in July 1975. NSF leaders, including director Stever and deputy director Richard Atkinson, argued that their reviewer system was basically sound, that NSF employees did not have undue power, and that grant review had to be left in scientific hands in order to be valid. They agreed to place more weight on referee opinions and created a new audit office to confirm that both positive and negative referee reports were taken into account when making a final decision about a grant.
Meanwhile, almost everyone called to testify at the hearings, from scientists to public servants, agreed that review by colleagues was a crucial part of science. Witnesses called it “a fair and effective system,”26 “an indispensable component of the decisionmaking process for allocating funds,”27 and “an integral feature of ‘the scientific method.’”28 In the face of a challenge to their autonomy, scientists now insisted refereeing was more than an optional bureaucratic function: it had become “indispensable” for the practice of science. That shift helped the NSF resist major changes to its grant-awarding powers, and also created the expectation that scientific quality would be assessed by scientists—creating a sense of intellectual autonomy for the scientific community even as scientists were ever more dependent on support from the federal government.
Interestingly, the mid-1970s were also the moment when Americans increasingly called refereeing by a new term: “peer review.” The term first came into use in the US to describe the ways physicians evaluated each other under the Professional Standards Review Organization (PSRO) associated with Medicaid. By the mid-1960s the term “peer review” was used to describe grant review procedures at the National Institutes of Health, where small study sections of in-house experts evaluated proposals; by the 1970s “peer review” was synonymous with all funding body and journal refereeing in the US.29 The shift from “refereeing” to “peer review” is not just a linguistic oddity. Renaming the refereeing process “peer review” implies that only a very narrow range of people—the expert peers of the person who wrote the paper or grant proposal—can be permitted to weigh in on the paper or proposal’s worthiness. It limits the range of acceptable reviewers. The term “peer review” itself helped connect claims about the quality of scientific research to the people allowed to be involved in its publication.
Peer review in the 21st century
The new emphasis on peer review in the United States was soon reflected in journals as well; American scientists increasingly regarded journals with unsystematic peer review systems as shoddy or unreliable. Relying on editorial judgment or the opinions of funding body employees was no longer acceptable; a respectable scientific institution needed to listen to its referees in order to make decisions about which science deserved publication or funding. The judgment of the peer reviewers—experts offering their time anonymously in order to serve the cause of improving science—came to represent the judgment of the scientific community as a whole. Peer review was thus elevated from an optional bureaucratic process to a system that was supposed to ensure the quality and trustworthiness of science.
Countries and organizations outside the purview of American governmental oversight tended to embrace systematic external refereeing later, and did not necessarily see it as a practice central to science until after the 1970s. As late as 1989, the editors of the prominent British medical journal The Lancet complained that “in the United States far too much is being asked of peer review” and proudly assured readers that at The Lancet, “reviewers are advisers not decision makers.”30 The influence and sheer size of the American scientific community, however, ensured that the vision of peer review as the key to scientific respectability spread beyond US borders. Even the Lancet began using more and more referees out of concern that without formal peer review, they could no longer attract American authors or readers.31 The experience of The Lancet was typical: though perhaps initially lamenting the emphasis American journals and funding put upon peer review, its increasing centrality in the US combined with the scientific importance of American authors, universities, and funding bodies ensured the practice spread quickly. The impact spread beyond the sciences; disciplines in the humanities also came to rely more on peer review to signal scholarly legitimacy in the late 1970s and early 1980s. By the 1990s, few journals or funding bodies in any nation were peer review holdouts.
More recently, peer review seems have faced its own moment of crisis. Critics have argued that the peer review process is not doing a good job of distinguishing good science from bad. Several high-profile papers have been published in top journals after having passed through peer review, only to be heavily criticized after publication or retracted amid allegations of fraud.32 Some studies have indicated that women and underrepresented minorities are more likely to receive unfavorable referee reports than their colleagues.33 Other observers have argued that peer review suppresses innovative research and rewards more familiar, safer projects.34 In 2011, Great Britain’s House of Commons commissioned a report on the state of peer review, and concluded that while peer review “is crucial to the reputation and reliability of scientific research,” many scientists believe the system stifles progress, is often biased, and that “there is little solid evidence on its efficacy.”35 In the 1970s, peer review was recast as the system that rewarded good science and corrected bad science; in the 2010s, scientists are now grappling with the fact that it doesn’t seem to do either of those things particularly well.
Members of the scientific community have offered several different visions for how peer review might change in the future. One popular suggestion is that referees should sign their reports—a suggestion that unconsciously echoes both Whewell’s initial vision for referee reports in the 1830s and Conlan’s plan for the NSF in the 1970s. Proponents of that change argue that if reviewers have to sign their comments, those comments will be more thoughtful and more constructive than if the reviewers are allowed to remain anonymous. Other observers argue that scientists should eschew peer review entirely and allow researchers to simply post their papers for feedback from the whole community, as physicists do on www.arxiv.org. So far, none of these suggestions has gained much traction, or had much of an influence over publishing or grant refereeing practices. Some journals, however, have begun experimenting with smaller-scale changes to the peer review system. The journal eLife, for example, solicits opinions from multiple referees, but does not send their reports directly to the author. Instead, the referees and editor discuss the paper with each other to arrive at one unanimous decision about whether the paper should be accepted, revised, or rejected.36
One of the reasons that it has been difficult to imagine substantial change to the peer review system may be the persistence of the myth claiming Henry Oldenburg invented peer review during the Scientific Revolution. If peer review is framed as an unchanging and time-honored system, intimately connected to the origins of modern science itself, reform seems impossible. But peer review has a far messier and more complicated history than its origin myth suggests, and its role in the scientific community has never been static. Its form and purpose have been shaped and reshaped according to what scientists needed from the practice—credibility for a new scientific society, improvement in the level of scientific literature, or assurance to patrons that money was being spent responsibly. Refereeing first emerged during a major shift in scientific publishing—the rise of a journal-centric system—and peer review became elevated to a central place in scientific practice as one resolution to a shifting relationship between scientists and funding sources. As scientific publishing grapples with the rise of online publishing, and as scientists attempt to navigate a political moment that has seen deep skepticism of fields like vaccine research and climate science, we may be standing on the brink of big changes for peer review as we know it.
Bibliography
National Science Foundation Peer Review Special Oversight Hearings: Hearings before the Subcommittee on Science, Research, and Technology, 1995. Washington, D.C.: U.S. Government Printing Office.
“Peers Reviewed,” 1989. The Lancet 333: 1115–1116.
Philip Abelson Papers, American Association for the Advancement of Science Archives, Washington, DC.
“Science and Technology Committee, Eighth Report: Peer Review in Scientific Publications,” 2011. www.Parliament.uk, United Kingdom Parliament, London, accessed 29 July 2016, <http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/ 856/856.pdf >.
Bakanic, Von, Clark McPhail, and Rita Simon, 1987. “The Manuscript Review and Decision-Making Process.” American Sociological Review, 52: 631–642.
Baldwin, Melinda, 2014. “Tyndall and Stokes: Correspondence, Referee Reports, and the Physical Sciences in Victorian Britain.” In Bernard Lightman and Michael Reidy, eds., The Age of Scientific Naturalism: John Tyndall and his Contemporaries, 171–186. London: Pickering and Chatto.
Baldwin, Melinda, 2015. Making Nature: The History of a Scientific Journal. Chicago: University of Chicago Press.
Baldwin, Melinda, 2018. “Scientific autonomy, public accountability, and the rise of ‘peer review’ in the Cold War United States,” Isis 109: 538–558.
Barany, Michael, 2018. “A Postwar Guide to Winning a Science Grant.” Physics Today Online, doi:10.1063/PT.6.4.20180320a.
Boas Hall, Marie, 2002. Henry Oldenburg: Shaping the Royal Society. Oxford: Oxford University Press.
Bornmann, Lutz, Rüdiger Mutz, and Hans-Dieter Daniel, 2007. “Gender differences in grant peer review: A meta-analysis,” Journal of Infometrics 1: 226–238.
Burnham, John, 1990. “The Evolution of Editorial Peer Review.” Journal of the American Medical Association 263: 1323–1329
Burnham, John, 1992. “How Journal Editors Came to Develop and Critique Peer Review Procedures,” in Research Ethics, Manuscript Review, and Journal Quality: Proceedings of a Symposium on the Peer Review-Editing Process, ed. H. F. Maryland and R. E. Sojka, 55–62. Madison, Wis.: ACS Miscellaneous Publications.
Chubin, Daryl and Edward Hackett, 1990. Peerless Science: Peer Review and U.S. Science Policy. Albany: State Univ. New York Press.
Clarke, Imogen, 2016. “The Gatekeepers of Modern Physics: Periodicals and Peer Review in 1920s Britain.” Isis 106: 70–93.
Clarke, Imogen and Jim Mussell, 2015. “Conservative attitudes to old-established organs: Oliver Lodge and Philosophical Magazine.” Notes and Records: The Royal Society Journal of the History of Science 69: 321–336.
Csiszar, Alex, 2018. The Scientific Journal: Authorship and the Politics of Knowledge in the Nineteenth Century. Chicago: University of Chicago Press.
Csiszar, Alex, 2016. “Peer Review: Troubled from the Start.” Nature 532: 306–308.
Frodeman, Robert and Adam Briggle, 2012. “The Dedisciplining of Peer Review.” Minerva 50: 3–19.
Fyfe, Aileen, Julie McDougall-Waters and Noah Moxham, 2015. “350 Years of Scientific Periodicals.” Notes and Records: The Royal Society Journal of the History of Science 69 no. 3: 227–239.
Ginther, Donna, Walter Schaffer, Joshua Schnell, Beth Masimore, Faye Liu, Laurel Haak, and Raynard Kington, 2011. “Race, Ethnicity, and NIH Research Awards,” Science 333: 1015–1019.
Goldstein, Raymond E, 2016. “Commentary: A biology journal provides a lesson in peer review.” Physics Today 69 no. 12: 10-11.
Goodrich, Dorris West, 1945. “An Analysis of Manuscripts Received by Editors of the American Sociological Review from May 1, 1944 to September 1, 1945.” American Sociological Review 10: 716–725.
Gross, Alan G., Joseph E. Harmon, and Michael Reidy, 2002. Communicating Science: The Scientific Article from the 17th Century to the Present. Oxford: Oxford University Press.
Ingelfinger, Frank, 1974. “Peer Review in Biomedical Publication.” The American Journal of Medicine 56: 686–692.
Johns, Adrian, 1998. The Nature of the Book: Print and Knowledge in the Making. Chicago: University of Chicago Press.
Kaiser, David, 2014. “Shut up and calculate!” Nature 505: 153–155.
Kennefick, Daniel, 1999. “Controversies in the History of the Radiation Reaction Problem in General Relativity.” In Hubert Goenner, Jürgen Renn, Jim Ritter, and Tilman Sauer, eds., The Expanding Worlds of General Relativity, 207–234. New York: Springer-Verlag.
Kohler, Robert, 1991. Partners in Science: Foundations and Natural Scientists, 1900–1945. Chicago: University of Chicago Press.
Kronick, David, 1990. “Peer Review in 18th-Century Scientific Journalism.” Journal of the American Medical Association 263 no. 10: 1321-1322.
Lacour, Michael and Donald Green, 2014. “When contact changes minds: An experiment on transmission of support for gay equality,” Science 346: 1366–1369.
Lalli, Roberto, 2016. “‘Dirty work,’ but someone has to do it: Howard P. Robertson and the refereeing practices of Physical Review in the 1930s.” Notes and Records: The Royal Society Journal of the History of Science 70: 151–174.
Lock, Stephen, 1994. “Does Editorial Peer Review Work?” Annals of Internal Medicine 121: 60–61.
Mandel, Richard, 1996. A Half Century of Peer Review, 1946–1996. Bethesda, MD: National Institutes of Health.
McClellan III, J.E., 2003. Specialist Control: The Publications Committee of the Académie Royale des Sciences (Paris), 1700–1793. New York: American Philosophical Society.
Moxham, Noah and Aileen Fyfe, 2017. “The Royal Society and the Prehistory of Peer Review, 1665–1965.” Historical Journal: doi:10.1017/S0018246X17000334.
Nightingale, Paul and Alister Scott, 2007. “Peer Review and the Relevance Gap: Ten Suggestions for Policy-Makers.” Science and Public Policy 34: 543–553.
Pyenson, Lewis, 1985. “Physical Sense in Relativity: Max Planck Edits the Annalen der Physik, 1906–1918.” In Lewis Pyenson, ed., The Young Einstein: The Advent of Relativity, 194–214. Bristol: Adam Hilger.
Rennie, Drummond, 1999. “Editorial Peer Review: Its Development and Rationale.” In Fiona Godlee and Tom Jefferson, eds., Peer Review in Health Sciences, ed. Fiona Godlee and Tom Jefferson, 1–13. London: BMJ.
Siler, Kyle, Kirby Lee, and Lisa Bero, 2015. “Measuring the effectiveness of scientific gatekeeping,” Proceedings of the National Academy of Sciences, 112: 360–365.
Thomson, A. Landsborough, 1973. Half a Century of Medical Research. London: Her Majesty’s Stationery Office.
Wagner, Patrick, 2013. “‘Preserve of Full Professors’—On the History of the German Research Foundation between 1920 and 1970.” In Mark Walker, Karin Orth, Ulrich Herbert and Rüdiger vom Bruch, eds., The German Research Foundation 1920-1970: Funding Posed between Science and Politics, 23–40. Berlin: Franz Steiner Verlag.
Wakefield, Andrew et al., 1998. “Ileal-Lymphoid-Nodular Hyperplasia, Non-Specific Colitis, and Pervasive Developmental Disorder in Children.” The Lancet 351: 637–641.
Weller, Anne, 2001. Editorial Peer Review: Its Strengths and Weaknesses. Medford, N.J.: Information Today.
Wennerås, Christine and Agnes Wold, 1997. “Nepotism and sexism in peer-review,” Nature 387: 341–343.
Wolfe-Simon, Felisa et al., 2011 “A Bacterium That Can Grow by Using Arsenic Instead of Phosphorus,” Science 332: 1163–1166.
Zuckerman, Harriet and Robert Merton, 1971. “Patterns of Evaluation in Science: Institutionalization, Structure, and Functions of the Referee System.” Minerva 9: 66–100.
Zuckerman and Merton 1971.↩
E.g. Bakanic et al. 1987; Chubin and Hackett 1990; Frodeman and Briggle, “The Dedisciplining of Peer Review”; Ingelfinger 1971; Lock 1994.↩
See, e.g., Fyfe et al. 2015; Moxham and Fyfe 2017; Johns 1998, chapter 7.↩
Boas Hall 2002.↩
See, e.g., Baldwin 2015; Baldwin 2018; Burnham 1990; Burnham 1992; Clarke 2016; Clarke and Mussell 2015; Csiszar 2018; Fyfe et al. 2015; Gross et al. 2002; Lalli 2016; Moxham and Fyfe 2017.↩
McClellan 2003.↩
Kronick 1990.↩
Moxham and Fyfe 2017.↩
Csiszar 2018.↩
Csiszar 2016; Csiszar 2018, pp. 142–148.↩
Baldwin 2014.↩
Lalli 2016; Goodrich 1945.↩
Foster quoted in Csiszar 2016; see also Clarke 2016.↩
On refereeing systems at these periodicals, see Baldwin 2015; Clarke and Mussell 2015.↩
Csiszar 2018, 110-117.↩
Pyenson 1985.↩
Albert Einstein, quoted in Kennefick 1999, p. 208.↩
Kohler 1991, 68–70; Barany 2018.↩
Wagner 2013.↩
Mandel 1996, pp. 4–5.↩
Thomson 1973, pp. 147–161.↩
“Minutes of the Joint Meeting of AAAS Editorial Board and Publications Committee, 10 June 1955,” Abelson Papers, Box 1.↩
Ingelfinger 1974.↩
Kaiser 2014.↩
Baldwin 2018.↩
National Science Foundation Peer Review Special Oversight Hearings 1975, p. 1092.↩
National Science Foundation Peer Review Special Oversight Hearings 1975, p. 460.↩
National Science Foundation Peer Review Special Oversight Hearings 1975, p. 390↩
Baldwin 2018.↩
The Lancet 1989.↩
Rennie 1999.↩
See, e.g., Wolfe-Simon 2011, which faced vigorous criticism of its conclusions, and Lacour and Green 2014 and Wakefield 1998, which were retracted by their publishing journals after allegations of fraud.↩
Bornmann et al. 2007; Ginther et al. 2011; Wennerås and Wold 1997.↩
Nightingale and Scott 2007; Siler, Lee, and Bero 2015.↩
“Science and Technology Committee, Eighth Report: Peer Review in Scientific Publications,” 2011.↩
Goldstein 2016.↩