Peer review is overwhelmed—can it survive in the AI era?

“I struggle like anything to get peer reviewers,” said Haseeb Irfanullah, who is on the editorial board of Wiley’s Learned Publishing but whose comments represent his own views as an independent consultant. Steven Mack, an editor for Human Immunology, has recently had to email around thirty researchers to find just one who would review a paper. Five years ago, he estimates it would have taken only five to 10 emails to find three willing reviewers. In this landscape, it’s not surprising that editors are assigning reviews to people who aren’t qualified or don’t have the time to concentrate on them. In most fields, researchers need peer-reviewed publications to secure jobs and grants, so they continue to slog through the system even though it’s becoming “extremely protracted and painful,” in the words of microbiologist Sebastian Lourido from the Whitehead Institute. Increasingly, many are advocating for change. Meanwhile, a few bold researchers have decided that the traditional publishing system just isn’t worth it anymore—AI research in particular has seen a shift toward communicating impactful findings through very informal means, like blogs. Peer review is “the bedrock of science,” Semprini said. But at this moment, “it’s not standing strong.” Researchers were independent gentlemen rather than university professors, so publishing wasn’t tied to career progression. Moreover, working for a journal was a fun thing to do with friends, so they were usually happy to donate their time as reviewers. Peer review became universal almost accidentally, as a way to solve a branding problem. In the 1970s, the US economy was tanking. Inflation was up, employment was down, there was a gas crisis, and the National Science Foundation found itself under public pressure to justify its spending. The pressure followed a period of post-World War II enthusiasm for science during which federal funding increased 25-fold. The National Science Foundation had used external reviewers to assess grant applications since the 1950s—which was unusual at the time—and that turned out to be its saving grace. Because panels of independent reviewers decided how to allocate funding, their decisions had an air of legitimacy. Ultimately, that legitimacy was what kept lawmakers from putting the foundation’s funding decisions under their direct political control. “It was really, in a lot of ways, a sales pitch from scientists to other stakeholders that made peer review into this cornerstone of science,” said Melinda Baldwin, a historian of science from the University of Maryland. The United Kingdom went through a similar moment in the late 1980s and early 1990s when a number of controversial scientific issues—from HIV to the cause of autism to cold fusion—garnered a great deal of press attention. Suddenly, the public had an interest in distinguishing legitimate science from quackery, and a number of scientific organizations responded by saying, “We can tell what proper science looks like because proper science is peer reviewed,” Fyfe said. Today, peer review is standard. Journal editors ask university professors to write reviews alongside a multitude of other professional responsibilities, and those requests often come at a quick clip. Lourido estimates he gets about 10 requests to review manuscripts each month, when realistically, he has time for only one or two. Mack receives a review request every day or two as part of his role as an immunogeneticist at the University of California, San Francisco (which he holds in addition to being a journal editor). That cadence comes in part because more people are submitting studies to journals than ever before. But journals also proliferated between 1960 and 2020, and some regularly run “special issues,” for which researchers generate custom content. Many journals are also publishing more manuscripts because they’re now online-only, which frees editors from having to fit an issue onto a limited amount of paper. Research is also increasingly interdisciplinary, meaning many studies require more knowledge and time to review. And AI contributes to the overload by making it easier for researchers to write up their work and easier for people from around the globe to submit to English-language journals. Then there are paper mills, or services that charge desperate researchers a fee to put their names on fabricated (but often legitimate-looking) publications. All of this has made it difficult not only to maintain the quality of peer review but to ensure it happens at all. “Sometimes I think we need to start talking about the de-growth of publishing,” Irfanullah said. “It has grown like anything. It’s extremely unhealthy!” AI researchers are feeling the content explosion acutely. Computer scientists typically publish in conference proceedings, and the number of papers submitted to top AI conferences has increased between two- and 10-fold since 2019. As a result, it’s common to receive reviews from people who “just don’t understand the field enough to evaluate, or give good feedback,” says Haewon Jeong, a computer scientist from the University of California, Santa Barbara. AI researchers are well-positioned to experiment with something different. Their field is new, it’s moving lightning-fast, and much of the work is taking place outside academia, so many researchers don’t need formal publications for their career progression. Against this backdrop, blogs have taken off. These days, “maintaining a good blog is almost like having a good podcast channel,” Jeong said. “Your voice gets really amplified, and you get known.” The colloquial writing style used on many blogs can also draw attention, which comes with pros and cons. For Qu, it means she doesn’t have to worry as much about someone else getting credit for her ideas when she publishes on her blog because nobody will put quite the same personal slant on the work that she will. And Jeong thinks the accessible style can make complex concepts easier to understand. “There’s value in that,” she said. But philosopher David Thorstad from Vanderbilt University cautions that accessibility could come at the expense of rigor and that colloquial writing could let charismatic voices rise to the top regardless of the quality of their work. Moving away from peer review can also isolate your work, Thorstad said. He studies longtermerism: how to best protect the people of the future. Lately, many longtermists have been concerned about how to prevent AI from bringing about the downfall of humanity, and many have published their work on the AI Alignment Forum. That poses a problem for Thorstad, who is a strong believer in peer review and publishes only in venues that use it. Citing work that wasn’t peer-reviewed is sure to raise red flags, given that his own work has to go through the review process in those venues. That means he can’t cite a large segment of the work in his field. “It’s incredibly limiting,” he said. Five years ago, this was a bigger problem than it is today, Habryka said. Now, reviewers in most disciplines accept that there are various ways to publish legitimate work. “I think less than 50 percent of things you really want to cite in [a machine learning] paper end up actually being properly peer reviewed,” he said. With peer review still taking up a lot of time, plenty of researchers are looking for shortcuts to speed up the process. A survey conducted by staff at the Frontiers journals found that over half of peer reviewers already incorporate AI into the process in some way, probably at least partly for this reason. One day, AI-produced reviews may be seen as routine and beneficial. But at the moment, AI is often used in ways researchers find annoying and unethical. Linguist Marijn van Putten from Leiden University, for instance, was initially excited to receive a review suggesting he cite a particular medieval Arabic manual in his publication about which ancient religious leader put forward a particular interpretation of the Qur’an. “I was like, ‘That’s great, I’m going to look it up!’” he said. Other ideas focus on the review process itself. One obvious step is to redistribute the review workload to reflect the source of submissions. Editors tend to disproportionately assign reviews to male researchers from the United States and a handful of other countries rather than distributing the workload equitably. Some researchers also advocate for paying peer reviewers, though others worry that the practice would increase the burdensome publication fees researchers already face. There is evidence that compensation in the $100–$300 range per review speeds up the process without compromising quality. That could be because adding money to the equation makes it feel like “there’s actually a hard deadline,” said ChoGlueck. Without that motivation, some reviewers might view deadlines as “magical,” he suggested, “because you can just change them.” Others are pushing for more structural change. One option, which is already in use on a limited basis, is to divorce the peer-review process from journal submission. Rather, a community of researchers reviews papers before they’re formally submitted, and researchers can then choose to submit their papers to a number of journals that have committed to publishing studies reviewed this way. If the first journal decides not to publish an article, the authors can approach a second journal without having to undergo a new round of peer review. Meanwhile, scholarly communication advocate Cameron Neylon has long been on the record as questioning whether all journal articles really need peer review, given how few papers end up having an impact on the world. Maybe, he suggests, studies should have to meet a threshold of interest before they’re sent out for peer review. With all these pressures and propositions, is peer review nearing a breaking point, and will it ultimately fail? It’s hard to envision a wholesale shift away from peer review because the system is so firmly entrenched in most fields, both as part of career progression and as a way of garnering public trust. At the same time, if academic publishing keeps growing at the rate it has been, the system simply won’t be able to keep up. “So something’s going to break there at some point,” Neylon said. Instead of dropping peer review, he suggests that maybe research fields should find ways to encourage quality publications over quantity. It’s just as hard to envision researchers voluntarily publishing less, given current incentives. But maybe both will happen to some degree. With the pool of reviewers being tapped out, journals will have to curb their proliferation, which could temper the rise in publications. Meanwhile, peer review may become a less rigorous, more fragmented version of what it once was, with the practice taking on different meanings in different fields. There’s already evidence of these last two things happening. It used to be standard for three reviewers to review manuscripts, but Mack says he’s lately seen the average in immunology move closer to two. In economics, some prominent journals already pay peer reviewers. Meanwhile, researchers in many scientific fields routinely publish preprints. Peer review is resilient, if not unflappable. The practice will probably stick around in one form or another, but it might not be the same peer review we’re used to.