Opening The Black-Box of Referee Behaviour. An Agent-Based Model of Peer Review
by Flaminio Squazzoni and Claudio Gandelli
ABSTRACT. This paper investigates the impact of referee behaviour on the quality and efficiency of peer review. We focused especially on the importance of reciprocity motives to ensure cooperation between everyone involved. We modelled peer review as a process based on knowledge asymmetries and subject to evaluation bias. We built various simulation scenarios where we tested interaction conditions and manipulated author and referee behaviour. We found that reciprocity per se can have a negative effect on peer review as it tends to increase evaluation bias. It can have a positive impact only when purged by self-interest motivation and accompanied by disinterestedness and fairness standard.
KEYWORDS. Peer review; referees; referee behaviour; reciprocity; agent-based model.
AVAILABLE: http://www.scs-europe.net/conf/ecms2012/ecms2012%20accepted%20papers/sdcb_ECMS_0022.pdf
Thursday, 27 September 2012
Paper: A simulation model of scientists as utility-driven agents
A simulation model of scientists as utility-driven agents
Melanie Baier
ABSTRACT. Agent-based simulations of science that account for the linkage between micro-level behavior of scientists and macro-level results of scientific competition are rather scarce. The approach of this simulation model is to link the motivation and behavior of scientists to knowledge growth and scientific innovations via the emergence of new knowledge fields. A new knowledge field is considered both to be a result of scientific competition and a representation of scientific advancement. This paper takes a closer look at the scientists’ motivation and how they coordinate and add to scientific progress as utilitydriven agents. Accounting for stylized facts of scientific competition, selected simulation results show how deep the processes of knowledge generation, reputation and scientific innovations are intertwined. As scientists are assumed to be of different utility types and have different aspiration levels, this approach is able to account for adaptive behavior of agents.
KEYWORDS. Agent-based modeling (ABM), coordination, knowledge generation, reputation, scientific advancement, scientific competition, status competition, utility function
AVAILABLE AT: http://www.scs-europe.net/conf/ecms2012/ecms2012%20accepted%20papers/sdcb_ECMS_0104.pdf
Melanie Baier
ABSTRACT. Agent-based simulations of science that account for the linkage between micro-level behavior of scientists and macro-level results of scientific competition are rather scarce. The approach of this simulation model is to link the motivation and behavior of scientists to knowledge growth and scientific innovations via the emergence of new knowledge fields. A new knowledge field is considered both to be a result of scientific competition and a representation of scientific advancement. This paper takes a closer look at the scientists’ motivation and how they coordinate and add to scientific progress as utilitydriven agents. Accounting for stylized facts of scientific competition, selected simulation results show how deep the processes of knowledge generation, reputation and scientific innovations are intertwined. As scientists are assumed to be of different utility types and have different aspiration levels, this approach is able to account for adaptive behavior of agents.
KEYWORDS. Agent-based modeling (ABM), coordination, knowledge generation, reputation, scientific advancement, scientific competition, status competition, utility function
AVAILABLE AT: http://www.scs-europe.net/conf/ecms2012/ecms2012%20accepted%20papers/sdcb_ECMS_0104.pdf
Paper: A simulation of disagreement for control of rational cheating in peer review
A simulation of disagreement for control of rational cheating in peer review
by Francisco Grimaldo and Mario Paolucci
ABSTRACT. We present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The systems is implemented on a BDI platform (Jason) that allows us to define a rich model of scoring, evaluating and selecting papers for conferences. Some of the reviewers apply a strategy (called “rational cheating”) aimed to prevent papers better than their own to be accepted. We show how a programme committee update based on disagreement control can remove them.
KEYWORDS. Artificial social systems, Peer Review, Agent-based simulation, Trust reliability and reputation
Available at: http://www.scs-europe.net/conf/ecms2012/ecms2012%20accepted%20papers/sdcb_ECMS_0088.pdf
by Francisco Grimaldo and Mario Paolucci
ABSTRACT. We present an agent-based model of peer review built on three entities - the paper, the scientist and the conference. The systems is implemented on a BDI platform (Jason) that allows us to define a rich model of scoring, evaluating and selecting papers for conferences. Some of the reviewers apply a strategy (called “rational cheating”) aimed to prevent papers better than their own to be accepted. We show how a programme committee update based on disagreement control can remove them.
KEYWORDS. Artificial social systems, Peer Review, Agent-based simulation, Trust reliability and reputation
Available at: http://www.scs-europe.net/conf/ecms2012/ecms2012%20accepted%20papers/sdcb_ECMS_0088.pdf
Paper: Social Dynamics of Science
Social Dynamics of Science
(Archived on 22 Sep 2012)
The birth and decline of disciplines are critical to science and society. However, no quantitative model to date allows us to validate competing theories of whether the emergence of scientific disciplines drives or follows the formation of social communities of scholars. Here we propose an agent-based model based on a \emph{social dynamics of science,} in which the evolution of disciplines is guided mainly by the social interactions among scientists. We find that such a social theory can account for a number of stylized facts about the relationships between disciplines, authors, and publications. These results provide strong quantitative support for the key role of social interactions in shaping the dynamics of science. A "science of science" must gauge the role of exogenous events, such as scientific discoveries and technological advances, against this purely social baseline.At: http://arxiv.org/abs/1209.4950
Tuesday, 25 September 2012
Paper: Modeling peer review: an agent-based approach
Modeling peer review: an agent-based approach
Stefano Allesina, Chicago UniversityAbstract: The peer review system is under severe strain. Corrections have been
proposed, but experiments to determine effective measures are difficult to
perform. I propose a framework in which alternatives to the current peer
review system can be studied quantitatively using agent-based modeling. I
implement three possible systems. I show how, all other things being equal,
these alternatives produce different results in terms of speed of
publication, quality control, reviewers' effort, and authors' impact. This
modeling framework can be used to test other solutions for peer review,
leading the way to an improvement of how science is disseminated.
Keywords: peer review; agent-based modeling; publishing; editorial rejection;
simulations
http://library.queensu.ca/ojs/
Tuesday, 11 September 2012
Paper: "Positive assortment for peer review" by Aktipis & Thompson-Schill
Positive assortment for peer review
- C Athena Aktipis
- Department of Ecology and Evolutionary Biology, University of Arizona, aktipis@alumni.reed.edu
Abstract
We suggest that the introduction of
positive assortment (the pairing of individuals with similar
characteristics) to the peer
review process would increase the speed of
reviewing, improve the quality of reviews, and decrease the burden on
reviewers.
In assortative reviewing, each reviewer is given a
score based on speed of reviewing, the usefulness of the review, the
rate
of reviewing, or any other priority of the journal
editor. Authors submitting manuscripts are then paired with reviewers
who
have similar scores to themselves. This is a
no-cost solution that aligns reviewers’ incentives by accounting for the
benefits
provided to the scientific community and returning
them in kind. This assortative reviewing system can promote rapid, high
quality, and high volume reviewing at a benefit to
the scientific community at no financial cost.
Tuesday, 10 July 2012
CfP: EOPS V - Epistemological Perspectives on Simulation 2012
=============================
CALL FOR PAPERS OR EXTENDED ABSTRACT
Epistemological Perspectives on Simulation 2012 (EPOS V)
San Antonio, Texas, USA
October 10-12, 2012
trinity.edu/epos
CALL FOR PAPERS OR EXTENDED ABSTRACT
Epistemological Perspectives on Simulation 2012 (EPOS V)
San Antonio, Texas, USA
October 10-12, 2012
trinity.edu/epos
Extended Deadline for Manuscript Submission: August 20, 2012.
==============================
Epistemological Perspectives on Simulation 2012
EPOS V Call for Papers
The Fifth Edition of Epistemological Perspectives on Simulation promises to be a productive meeting of minds. Held of the first time outside of Europe, this meeting aims to bring together researchers working on common issues of Computer Simulation, Methodology, and Philosophy of Science from many different angles, making for a truly interdisciplinary opportunity for discussion on the state of the art in this blooming field.
==============================
Epistemological Perspectives on Simulation 2012
EPOS V Call for Papers
The Fifth Edition of Epistemological Perspectives on Simulation promises to be a productive meeting of minds. Held of the first time outside of Europe, this meeting aims to bring together researchers working on common issues of Computer Simulation, Methodology, and Philosophy of Science from many different angles, making for a truly interdisciplinary opportunity for discussion on the state of the art in this blooming field.
The meeting will take place on the grounds of Trinity
University, in San Antonio, Texas, on October 10 – 13, 2012.
Like its predecessors the workshop wants to provide
a forum
for researchers from various disciplines, such as the social sciences,
economics, computer science, engineering or the natural sciences, who
are interested in discussing epistemological aspects of simulation
across disciplinary boundaries.
Philosophers are highly welcome, too.
Topics to be addressed in the workshop include, but are not restricted to issues such as the epistemology of simulation, credentials for model building, and standards for presenting and analyzing simulation results. EPOS V will bring together simulation experts, modelers, scientists and philosophers of science to share the latest advancements in simulation technology, its applications, as well as potential benefits and drawbacks. Like its predecessors, the workshop wants to provide a forum for researchers from various disciplines, such as the social sciences, computer science, engineering or the natural sciences, who are interested in discussing epistemological aspects of simulation across disciplinary boundaries.
Topics to be addressed in the workshop include, but are not restricted to issues such as the epistemology of simulation, credentials for model building, and standards for presenting and analyzing simulation results. EPOS V will bring together simulation experts, modelers, scientists and philosophers of science to share the latest advancements in simulation technology, its applications, as well as potential benefits and drawbacks. Like its predecessors, the workshop wants to provide a forum for researchers from various disciplines, such as the social sciences, computer science, engineering or the natural sciences, who are interested in discussing epistemological aspects of simulation across disciplinary boundaries.
Following
the tradition of previous EPOS workshops, we will publish a selection of the
accepted papers in a special issue of a journal after a further reviewing
process. This time, the venue will be a special issue of the International Journal of Agent Technologies
and Systems (IJATS).
Important deadlines are as follows:
August 20, 2012: Submission of papers or extended abstracts
(Extended Deadline)
September 10, 2012: Notification of acceptance
September 28, 2012: Receipt of full papers
October 10-12, 2012: Workshop, San Antonio, Texas, USA
October 13, 2012: Complementary field trip for participants
in the beautiful Texas Hill Country
Hosts: Prof. Dr. Dante Suarez and Prof. Dr. Yu Zhang
(Trinity University), Prof. Dr. Klaus G. Troitzsch (University of
Koblenz-Landau) and Prof. Dr. Manuel Castañon-Puga (University of Baja
California).
Subscribe to:
Posts (Atom)