Thursday, 27 September 2012

Paper: Opening The Black-Box of Referee Behaviour. An Agent-Based Model of Peer Review

Opening The Black-Box of Referee Behaviour. An Agent-Based Model of Peer Review
by Flaminio Squazzoni and Claudio Gandelli

ABSTRACT. This paper investigates the impact of referee behaviour on the quality and efficiency of peer review. We focused especially on the importance of reciprocity motives to ensure cooperation between everyone involved. We modelled peer review as a process based on knowledge asymmetries and subject to evaluation bias. We built various simulation scenarios where we tested interaction conditions and manipulated author and referee behaviour. We found that reciprocity per se can have a negative effect on peer review as it tends to increase evaluation bias. It can have a positive impact only when purged by self-interest motivation and accompanied by disinterestedness and fairness standard.

KEYWORDS. Peer review; referees; referee behaviour; reciprocity; agent-based model.


Paper: A simulation model of scientists as utility-driven agents

A simulation model of scientists as utility-driven agents
Melanie Baier

ABSTRACT. Agent-based simulations of science that account for the linkage between micro-level behavior of scientists and macro-level results of scientific competition are rather scarce. The approach of this simulation model is to link the motivation and behavior of scientists to knowledge growth and scientific innovations via the emergence of new knowledge fields. A new knowledge field is considered both to be a result of scientific competition and a representation of scientific advancement. This paper takes a closer look at the scientists’ motivation and how they coordinate and add to scientific progress as utilitydriven agents. Accounting for stylized facts of scientific competition, selected simulation results show how deep the processes of knowledge generation, reputation and scientific innovations are intertwined. As scientists are assumed to be of different utility types and have different aspiration levels, this approach is able to account for adaptive behavior of agents.

KEYWORDS. Agent-based modeling (ABM), coordination, knowledge generation, reputation, scientific advancement, scientific competition, status competition, utility function


Paper: A simulation of disagreement for control of rational cheating in peer review

A simulation of disagreement for control of rational cheating in peer review
by Francisco Grimaldo and Mario Paolucci

ABSTRACT. We present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The systems is implemented on a BDI platform (Jason) that allows us to define a rich model of scoring, evaluating and selecting papers for conferences. Some of the reviewers apply a strategy (called “rational cheating”) aimed to prevent papers better than their own to be accepted. We show how a programme committee update based on disagreement control can remove them.

KEYWORDS. Artificial social systems, Peer Review, Agent-based simulation, Trust reliability and reputation

Available at:

Paper: Social Dynamics of Science

Social Dynamics of Science

The birth and decline of disciplines are critical to science and society. However, no quantitative model to date allows us to validate competing theories of whether the emergence of scientific disciplines drives or follows the formation of social communities of scholars. Here we propose an agent-based model based on a \emph{social dynamics of science,} in which the evolution of disciplines is guided mainly by the social interactions among scientists. We find that such a social theory can account for a number of stylized facts about the relationships between disciplines, authors, and publications. These results provide strong quantitative support for the key role of social interactions in shaping the dynamics of science. A "science of science" must gauge the role of exogenous events, such as scientific discoveries and technological advances, against this purely social baseline.

Tuesday, 25 September 2012

Paper: Modeling peer review: an agent-based approach

Modeling peer review: an agent-based approach

Stefano Allesina, Chicago University

Abstract: The peer review system is under severe strain. Corrections have been
proposed, but experiments to determine effective measures are difficult to
perform. I propose a framework in which alternatives to the current peer
review system can be studied quantitatively using agent-based modeling. I
implement three possible systems. I show how, all other things being equal,
these alternatives produce different results in terms of speed of
publication, quality control, reviewers' effort, and authors' impact. This
modeling framework can be used to test other solutions for peer review,
leading the way to an improvement of how science is disseminated.

Keywords: peer review; agent-based modeling; publishing; editorial rejection;

Tuesday, 11 September 2012

Paper: "Positive assortment for peer review" by Aktipis & Thompson-Schill

Positive assortment for peer review

  1. C Athena Aktipis
    1. Department of Ecology and Evolutionary Biology, University of Arizona,
  1. Sharon L Thompson-Schill
    1. Department of Psychology, University of Pennsylvania


We suggest that the introduction of positive assortment (the pairing of individuals with similar characteristics) to the peer review process would increase the speed of reviewing, improve the quality of reviews, and decrease the burden on reviewers. In assortative reviewing, each reviewer is given a score based on speed of reviewing, the usefulness of the review, the rate of reviewing, or any other priority of the journal editor. Authors submitting manuscripts are then paired with reviewers who have similar scores to themselves. This is a no-cost solution that aligns reviewers’ incentives by accounting for the benefits provided to the scientific community and returning them in kind. This assortative reviewing system can promote rapid, high quality, and high volume reviewing at a benefit to the scientific community at no financial cost.