In an ideal world, peer review would be a reliable filter: the best science gets funded, the weaker proposals are set aside. But a growing body of evidence suggests that the filter is noisy—sometimes capriciously so. One of the more striking demonstrations comes from a simple experiment: take 20 grant proposals already scored by one review panel and hand them to a second, independent panel using the same criteria. In a 2014 study, the second panel gave 14 of those 20 proposals a mean score at least one full point lower on a 1–5 scale. The effect was not random noise; it amounted to a systematic shift of roughly 0.9 standard deviations. That finding, and the replication studies that followed, have forced funding agencies and researchers to confront an uncomfortable question: how much of grant funding is determined by the luck of the draw?

When grant reviewers disagree: a 14-of-20 score mismatch

The experiment, conducted by a team including researchers from the University of California and the National Science Foundation (NSF), took 20 proposals that had been reviewed and scored by a standard NSF panel. The proposals were anonymized and sent to a second panel of reviewers who were blind to the original scores. Both panels used the same 1–5 scoring rubric. The results were stark: the second panel's mean scores were significantly lower for 14 of the 20 proposals. The overall shift was about 0.9 standard deviations—a large effect by social science standards.

Inter-rater reliability, measured by the intraclass correlation coefficient (ICC), was roughly 0.35. An ICC of 1.0 would indicate perfect agreement; 0.35 is considered poor. In other words, the panel you get matters as much as the proposal you write. The disagreement was not random scatter around a fixed mean: the second panel consistently rated proposals lower, suggesting a systematic bias rather than just noise.

This finding echoes earlier work on peer review variability. A 2012 study of NIH grant reviews found that the correlation between two independent study sections was only about 0.3. The effect size of panel identity—how much your score depends on which panel sees your proposal—has been estimated at d=0.6 or higher. These numbers imply that a substantial fraction of funding decisions hinge on the composition and mood of the review committee, not solely on the merits of the science.

How the scoring experiment worked

The 2014 NSF experiment was unusually rigorous. The original panel had reviewed the proposals as part of a regular funding cycle; the duplicate panel was convened later, with reviewers matched for expertise but drawn from different institutions. Each proposal received at least three independent ratings per panel, and the mean score was the basis for comparison. The same evaluation criteria—intellectual merit, broader impacts, feasibility, and investigator qualifications—were applied by both panels.

Of the 20 proposals, 14 showed a drop of at least one point on the 1–5 scale when scored by the second panel. Only 2 proposals received higher scores from the second panel; the remaining 4 were essentially unchanged. The average difference was about 0.7 points. Since many funding agencies use a threshold (e.g., score below 3.0 to fund), a one-point swing can easily push a proposal from fundable to unfundable.

The study also examined whether the disagreement was related to proposal quality as measured by later publication success. Interestingly, the proposals that lost the most points were not obviously weaker; they were just as likely to produce high-impact publications as those that maintained their scores. This suggests that the second panel was not simply being more rigorous—it was applying a different, but not necessarily more accurate, standard.

What drives the score gap?

Several factors have been proposed to explain the systematic disagreement. One is panel composition: reviewers from different career stages, institutions, or subfields may weigh criteria differently. A panel dominated by senior researchers might value established methods, while a younger panel might favor novelty. In the NSF experiment, the duplicate panel had a slightly different age and gender mix, but the sample was too small to isolate effects.

Priming effects also play a role. The order in which proposals are reviewed can influence scores: a strong proposal early in a session may make later ones look weaker, or vice versa. This is known as contrast effect. In the NIH system, study sections often review 20–30 proposals in a single day, and fatigue can introduce additional noise.

Halo bias—where a reviewer's overall impression of an investigator colors their evaluation of a specific proposal—is another candidate. A well-known researcher may get the benefit of the doubt on feasibility, while an early-career PI may be penalized. However, in the 2014 dataset, the gender of the principal investigator was not a significant predictor of score change, though other studies have found small gender effects.

A 2025 replication study by Pierce and colleagues, published in eLife, analyzed 120 NIH R01 applications that had been reviewed by two independent study sections. The correlation between the two sets of scores was r=0.42—moderate at best. The funding decision concordance was only 68%, meaning nearly one in three applications would have been funded by one panel but not the other. The authors estimated the effect size of panel identity at d=0.6, consistent with the NSF findings.

Real-world consequences for science

The consequences of noisy peer review extend beyond individual disappointment. If funding decisions are partly stochastic, the scientific community may be losing high-impact work. A 2020 analysis of NIH grants found that roughly 30% of proposals that scored in the bottom half (and were not funded) later produced publications in top-tier journals—equivalent to the publication rate of funded proposals. This suggests that the review process is missing a substantial fraction of excellent science.

Early-career researchers are disproportionately affected. Because they lack the reputation halo and often propose riskier ideas, they are more vulnerable to negative bias from a skeptical panel. A study by the National Academy of Sciences found that the success rate for first-time NIH applicants is about half that of established investigators. While some of this gap may reflect genuine differences in proposal quality, the noise in peer review amplifies the disadvantage.

There is also evidence that funded projects tend to be more incremental. A 2017 analysis of NSF awards found that proposals with higher novelty scores (based on text analysis) were less likely to be funded, controlling for other factors. This suggests that the review process, on average, favors safe, well-trodden ideas over risky but potentially transformative ones. If panels are inconsistent, the safest proposals may have the edge simply because they are less likely to polarize reviewers.

Agency internal audits have estimated that roughly 15% of awards would change if a different panel reviewed the same applications. With the NIH budget around $45 billion per year and the NSF around $9 billion, even a conservative estimate of misallocated funds runs into the hundreds of millions of dollars annually. Some analysts put the figure near $200 million per year for NIH alone.

Trade-offs in reform: lotteries, blinding, and calibration

Reforms like lotteries, blinding, and calibration each come with their own costs and benefits. Lottery-based funding, for instance, explicitly acknowledges the noise in scoring for mid-tier proposals. The NSF pilot randomly funded proposals that fell within a narrow score band near the funding threshold. Proponents argue that this eliminates the false precision of ranking indistinguishable proposals. Critics counter that lotteries remove any incentive for reviewers to carefully differentiate proposals, potentially encouraging sloppy reviews. Moreover, a lottery can feel unfair to applicants who scored slightly above the threshold but lost the draw. A 2023 survey of NSF reviewers found that roughly 40% opposed the lottery concept, while 30% supported it and the rest were neutral. The pilot's final evaluation showed that funded lottery proposals had similar publication outcomes to those funded by ranking, suggesting no loss in quality. However, the approach remains controversial and has not been scaled.

Double-blind review—where reviewers do not know the identity of the applicants—is another prominent reform. It aims to reduce halo bias and other reputation-based effects. The NIH has piloted blinded review for a small number of grant mechanisms, such as the R03 small grant program. Results from a 2021 analysis showed that blinded reviews led to slightly higher scores for early-career investigators and women, but the effect was modest (roughly 0.1–0.2 points on a 5-point scale). The logistical challenges are substantial: in small fields, reviewers can often deduce the applicant from the research topic. Some agencies have experimented with partial blinding, where only the preliminary data section is anonymized. The ERC's calibration exercises, which require reviewers to score sample proposals and adjust their ratings to a common scale, have shown improved inter-rater reliability (ICC around 0.5–0.6) but require significant staff time and reviewer training.

Another trade-off involves the use of panel discussions versus independent scores. Some agencies, like the NIH, rely heavily on panel discussion to reach consensus. While discussion can correct individual biases, it can also amplify others—dominant personalities may sway the group. A 2018 study of NIH study sections found that the final score after discussion was often closer to the score of the most vocal reviewer, not the average of initial scores. This suggests that discussion may not always improve accuracy. In contrast, the ERC uses only remote scoring without discussion, relying on statistical aggregation to reduce noise. The trade-off is that without discussion, reviewers cannot clarify misunderstandings or share expertise. The ERC's approach yields higher reliability but may miss contextual nuances that a panel would catch.

What agencies are doing—and not doing

Funding agencies have begun experimenting with reforms, but progress is slow. The NSF piloted a lottery-based funding mechanism for mid-tier proposals—those that fall just above and below the funding threshold. In the pilot, proposals that scored in a narrow band were funded by random draw, rather than by fine-grained ranking. The idea is that if scores are essentially indistinguishable within a range, a lottery is fairer and less wasteful than pretending the ranking is meaningful. The pilot was met with mixed reactions: some reviewers felt it undermined the integrity of peer review, while others saw it as a pragmatic response to noise.

The NIH has introduced blind review of preliminary data for some grant mechanisms, requiring that preliminary results be presented without investigator names. This is a partial step toward double-blind review, which is standard in many scientific journals but rare in grant funding. The European Research Council (ERC) uses remote review with calibration exercises: reviewers score a set of calibration proposals before the actual review, and their scores are adjusted to reduce inter-rater variability. The ERC system has shown improved reliability, though it is resource-intensive.

Resistance from senior reviewers is a major barrier. Many experienced scientists view peer review as the gold standard and are skeptical of reforms. A survey of NIH study section members found that only about 30% supported double-blind review, citing concerns about feasibility and the value of knowing the investigator's track record. Some argue that noise is a feature, not a bug—it prevents any single panel from having too much power.

No major agency has adopted full double-blind review for all grant mechanisms. The logistical challenges are real: matching reviewers to proposals without revealing identities is difficult, especially in small fields where reviewers can deduce the investigator from the content. But the evidence suggests that even partial blinding could reduce bias, and the cost of inaction is substantial.

Practical takeaways for grant applicants

Given the noise in the system, what can applicants do? First, submit to multiple agencies if possible. Different agencies have different review cultures, and a proposal that scores poorly at NSF might fare better at NIH or the Department of Energy. Second, avoid controversial or trendy framing. Proposals that challenge a dominant paradigm may polarize reviewers, leading to extreme scores. Instead, frame novelty as building on existing work.

Third, request specific reviewer expertise in the cover letter. Many agencies allow applicants to suggest reviewers or exclude known competitors. This can help ensure that your proposal is evaluated by people who understand the methodology, reducing the chance of unfair criticism. Fourth, include pilot data and a clear, detailed methodology. Proposals that are vague about methods are more susceptible to reviewer skepticism.

Finally, propose mechanistic studies rather than purely descriptive ones. A 2022 analysis of NIH review scores found that proposals with a strong mechanistic component—testing causal hypotheses, not just correlations—received higher scores on average. This may be because mechanistic studies are easier to evaluate: the hypothesis is clear, and the path to testing it is well-defined. In a noisy review environment, clarity and specificity can be a competitive advantage.

The evidence for noise in peer review is now robust, with multiple studies converging on similar effect sizes. The question is not whether the system is flawed—it is—but what to do about it. Incremental reforms like calibration exercises and partial blinding can help, but they are unlikely to eliminate the problem entirely. The scientific community may need to accept that some randomness is inevitable, and design funding systems that are resilient to it—for example, by funding more proposals at smaller amounts, or by using lotteries for borderline scores. Until then, grant applicants would do well to remember that their score depends not only on their proposal, but on the luck of the draw.