An experimental study of simulated grant peer review: Gender differences and psychometric characteristics of proposal scores.

Peer review is a decisive factor in selecting research grant proposals for funding. The usefulness of peer review depends in part on the agreement of multiple reviewers' judgments of the same proposal, and on each reviewer's consistency in judging proposals. Peer reviewers are also instructed to disregard characteristics that are not among the evaluation criteria.

A new approach to grant review assessments: score, then rank.

Background: In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently.

Gender differences in peer reviewed grant applications, awards, and amounts: a systematic review and meta-analysis.

Background: Differential participation and success in grant applications may contribute to women's lesser representation in the sciences. This study's objective was to conduct a systematic review and meta-analysis to address the question of gender differences in grant award acceptance rates and reapplication award acceptance rates (potential bias in peer review outcomes) and other grant outcomes.

Methods: The review was registered on PROSPERO (CRD42021232153) and conducted in accordance with PRISMA 2020 standards.

Peer review: Risk and risk tolerance.

Peer review, commonly used in grant funding decisions, relies on scientists' ability to evaluate research proposals' quality. Such judgments are sometimes beyond reviewers' discriminatory power and could lead to a reliance on subjective biases, including preferences for lower risk, incremental projects. However, peer reviewers' risk tolerance has not been well studied.

Grant Review Feedback: Appropriateness and Usefulness.

The primary goal of the peer review of research grant proposals is to evaluate their quality for the funding agency. An important secondary goal is to provide constructive feedback to applicants for their resubmissions. However, little is known about whether review feedback achieves this goal.

Grant reviewer perceptions of the quality, effectiveness, and influence of panel discussion.

Background: Funding agencies have long used panel discussion in the peer review of research grant proposals as a way to utilize a set of expertise and perspectives in making funding decisions. Little research has examined the quality of panel discussions and how effectively they are facilitated.

Methods: Here, we present a mixed-method analysis of data from a survey of reviewers focused on their perceptions of the quality, effectiveness, and influence of panel discussion from their last peer review experience.

Corrigendum: The Extended Specimen Network: A Strategy to Enhance US Biodiversity Collections, Promote Research and Education.

[This corrects the article DOI: 10.1093/biosci/biz140.].

Asking the Right Questions About Secondhand Smoke.

Introduction: Despite knowledge about major health effects of secondhand tobacco smoke (SHS) exposure, systematic incorporation of SHS screening and counseling in clinical settings has not occurred.

Methods: A three-round modified Delphi Panel of tobacco control experts was convened to build consensus on the screening questions that should be asked and identify opportunities and barriers to SHS exposure screening and counseling. The panel considered four questions: (1) what questions should be asked about SHS exposure; (2) what are the top priorities to advance the goal of ensuring that these questions are asked; (3) what are the barriers to achieving these goals; and (4) how might these barriers be overcome.

The Participation and Motivations of Grant Peer Reviewers: A Comprehensive Survey.

Scientific peer reviewers play an integral role in the grant selection process, yet very little has been reported on the levels of participation or the motivations of scientists to take part in peer review. The American Institute of Biological Sciences (AIBS) developed a comprehensive peer review survey that examined the motivations and levels of participation of grant reviewers. The survey was disseminated to 13,091 scientists in AIBS's proprietary database.

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study.

: Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return. : We examined the peer reviewers' scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006.

The Influence of Peer Reviewer Expertise on the Evaluation of Research Funding Applications.

Although the scientific peer review process is crucial to distributing research investments, little has been reported about the decision-making processes used by reviewers. One key attribute likely to be important for decision-making is reviewer expertise. Recent data from an experimental blinded review utilizing a direct measure of expertise has found that closer intellectual distances between applicant and reviewer lead to harsher evaluations, possibly suggesting that information is differentially sampled across subject-matter expertise levels and across information type (e.

A retrospective analysis of the effect of discussion in teleconference and face-to-face scientific peer-review panels.

Objective: With the use of teleconferencing for grant peer-review panels increasing, further studies are necessary to determine the efficacy of the teleconference setting compared to the traditional onsite/face-to-face setting. The objective of this analysis was to examine the effects of discussion, namely changes in application scoring premeeting and postdiscussion, in these settings. We also investigated other parameters, including the magnitude of score shifts and application discussion time in face-to-face and teleconference review settings.

Frequency and Type of Conflicts of Interest in the Peer Review of Basic Biomedical Research Funding Applications: Self-Reporting Versus Manual Detection.

Despite the presumed frequency of conflicts of interest in scientific peer review, there is a paucity of data in the literature reporting on the frequency and type of conflicts that occur, particularly with regard to the peer review of basic science applications. To address this gap, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of conflict of interest data from the peer review of 282 biomedical research applications via several onsite review panels. The overall conflicted-ness of these panels was significantly lower than that reported for regulatory review.

The validation of peer review through research impact measures and the implications for funding strategies.

There is a paucity of data in the literature concerning the validation of the grant application peer review process, which is used to help direct billions of dollars in research funds. Ultimately, this validation will hinge upon empirical data relating the output of funded projects to the predictions implicit in the overall scientific merit scores from the peer review of submitted applications. In an effort to address this need, the American Institute of Biological Sciences (AIBS) conducted a retrospective analysis of peer review data of 2,063 applications submitted to a particular research program and the bibliometric output of the resultant 227 funded projects over an 8-year period.

Teleconference versus face-to-face scientific peer review of grant application: effects on review outcomes.

Teleconferencing as a setting for scientific peer review is an attractive option for funding agencies, given the substantial environmental and cost savings. Despite this, there is a paucity of published data validating teleconference-based peer review compared to the face-to-face process. Our aim was to conduct a retrospective analysis of scientific peer review data to investigate whether review setting has an effect on review process and outcome measures.