A Randomized Controlled Trial on Anonymizing Reviewers to Each Other in Peer Review Discussions
(2403.01015)Abstract
Peer review often involves reviewers submitting their independent reviews, followed by a discussion among reviewers of each paper. A question among policymakers is whether the reviewers of a paper should be anonymous to each other during the discussion. We shed light on this by conducting a randomized controlled trial at the UAI 2022 conference. We randomly split the reviewers and papers into two conditions--one with anonymous discussions and the other with non-anonymous discussions, and conduct an anonymous survey of all reviewers, to address the following questions: 1. Do reviewers discuss more in one of the conditions? Marginally more in anonymous (n = 2281, p = 0.051). 2. Does seniority have more influence on final decisions when non-anonymous? Yes, the decisions are closer to senior reviewers' scores in the non-anonymous condition than in anonymous (n = 484, p = 0.04). 3. Are reviewers more polite in one of the conditions? No significant difference in politeness of reviewers' text-based responses (n = 1125, p = 0.72). 4. Do reviewers' self-reported experiences differ across the two conditions? No significant difference for each of the five questions asked (n = 132 and p > 0.3). 5. Do reviewers prefer one condition over the other? Yes, there is a weak preference for anonymous discussions (n = 159 and Cohen's d= 0.25). 6. What do reviewers consider important to make policy on anonymity among reviewers? Reviewers' feeling of safety in expressing their opinions was rated most important, while polite communication among reviewers was rated least important (n = 159). 7. Have reviewers experienced dishonest behavior due to non-anonymity in discussions? Yes, roughly 7% of respondents answered affirmatively (n = 167). Overall, this experiment reveals evidence supporting an anonymous discussion setup in the peer-review process, in terms of the evaluation criteria considered.
Overview
-
The study conducted during UAI 2022 examines the impact of reviewer anonymity on peer review discussions, focusing on engagement, decision-making, politeness, and preferences.
-
An empirical comparison revealed a marginally higher discussion engagement in anonymous settings, challenging the assumption that anonymity reduces participation.
-
Final decisions in non-anonymous discussions were more likely influenced by senior reviewers, suggesting identity visibility can skew decision-making towards seniority.
-
No significant difference in politeness between anonymous and non-anonymous discussions was found, contradicting the belief that anonymity leads to less civil discourse.
An Empirical Examination of Anonymity in Peer Review Discussions
Introduction
The peer review system, while not without its flaws, remains a cornerstone of academic integrity and quality assurance in scientific research. An often debated aspect of this system is whether reviewers should remain anonymous to each other during discussions. This study, conducted during the Conference on Uncertainty in Artificial Intelligence (UAI) 2022, employs a randomized controlled trial to explore the implications of reviewer anonymity on discussion engagement, decision-making influence, politeness, and participant preferences.
Experiment Design
The UAI 2022 conference served as a live testing ground for the experiment, with submissions and reviewers randomly assigned to anonymous or non-anonymous discussion conditions. This setup allowed for an empirical comparison across several dimensions, including discussion participation rates, influence of reviewer seniority on final decisions, and perceptions of discussion politeness. A supplementary survey provided additional insights into reviewers' preferences and perceptions regarding anonymity.
Discussion Engagement
One of the primary findings of the study is a marginally higher rate of discussion posts in the anonymous condition, although the difference, at a p-value of 0.051, skirts the edge of statistical significance. Interestingly, there was no significant difference in posting behavior when dissected by reviewer seniority, contrasting with the hypothesis that junior reviewers might be more vocal when their identities are concealed.
Influence of Reviewer Seniority
A key concern within peer review discussions is the potential for dominant influences, particularly from senior reviewers. The study uncovered that final decisions in the non-anonymous condition were more likely to align with the initial scores given by senior reviewers, an observation supported by a p-value of 0.04. This finding suggests that the visibility of reviewer identities can indeed skew decision-making toward more senior participants.
Discussion Politeness
Contrary to the assumption that anonymity might lead to less polite discourse, the study found no significant difference in the politeness levels of discussion posts across both conditions. This outcome challenges the notion that removing anonymity necessarily improves the civility of peer review discussions.
Reviewer Preferences and Experiences
Survey responses indicated a weak preference for anonymous discussions among reviewers, with no significant differences in self-reported experiences related to comfort, understanding, or perceived responsibility in discussions. Notably, 7% of respondents reported witnessing dishonest behavior in previous non-anonymous review settings, highlighting a potential risk associated with visible reviewer identities.
Implications and Future Directions
This study provides valuable empirical evidence on the impacts of reviewer anonymity in peer review discussions. The findings on seniority influence and the lack of difference in politeness levels challenge some commonly held beliefs about the benefits of non-anonymous reviews. However, the observed preference for anonymity, albeit slight, suggests that the academic community may lean towards more private discussion environments.
Moving forward, it will be crucial for conference organizers and journal editors to consider these findings when designing or refining their review processes. Further research could explore additional factors, such as the quality of review content and long-term effects on publication quality, to build a more comprehensive understanding of optimal peer review practices.
Create an account to read this summary for free: