Interesting data about paper submission trajectories (Science 2012)

Vincent Calcagno, an ecologist at the French Institute for Agricultural Research in Sophia-Antipolis, and his colleagues tracked the submission histories of 80,748 scientific articles published among 923 bioscience journals between 2006 and 2008, based on information provided by the papers’ authors for 18 biological/ecological topics. They then analysed the data in the paper:

Calcagno, V. et al. (2012) Flows of Research Manuscripts Among Scientific Journals Reveal Hidden Submission Patterns. Science 23 November 2012: 338 (6110), 1065-1069. DOI: 10.1126/science.1227833

• Abstract
• Full Text
• Full Text (PDF)
• Supplementary Materials
• Podcast Interview

(Free access to paper when you register with Science/AAAS)

 

Some results were unsurprising:

• the number of times a journal was first chosen for submission increases with impact factor
• about 75% of published articles were submitted first to the journal that would publish them, indicating that many academics do no resubmit papers that are rejected
• the impact factor of the publishing journal was generally less the one previously attempted
• journals with low impact factors would more often receive and publish manuscripts previously rejected by higher-impact journals 
• even Nature and Science are far from publishing 100% of first-intents because each often publishes manuscripts rejected by the other

Others more interesting findings included:

• high-impact journals publish more that are resubmitted to them from another journal
• none of the journals was just recycling manuscripts rejected from other journals 
• low-impact journals are more specialized, with lower rejection rates in it receiving proportionally fewer resubmissions from their neighbours
• resubmissions were significantly more cited than first-intents published the same year in the same journal and these were more cited irrespective of their going up or down in impact factor as a result of resubmission
• resubmissions occurring between two journals from the same journal community were significantly more cited than those between two different communities

My interpretation of this is as follows. The higher citation from resubmitted papers may partly just a trickle down effect from Nature/Science/Cell Biology to second tier high-impact journals.  Authors often learn from the submission process and either improve their papers or have faith in them and resubmit them elsewhere (as indicated by the higher citations from these papers).  Maybe authors have a good feel for their paper's worth and tend to submit to an appropriate journal, with the authors of not so interesting/good papers being more reluctant to resubmit.  It does reinforce the evidence for an oligopoly of the top journals (science, nature) as bemoaned by a recent Nobel prize winner (see my post here).

The Economist Examines the Self-Checking Processes of Science

In a leader, an article, a discussion and a poll, the Economist asks "Has Science Gone Wrong?"

The critiques are as follows:

• The bias in journals and the pressure to publish novel results results in many results that do not stack up when replicated or really checked
• Peer review is not doing its job in weeding these out
• Replication of results is thankless, time-consuming and usually not chosen by funding agencies
• Negative results are difficult to publish and hence this is not usually done
• Estimations of the number of errors in published results is probably underestimated
• Much data, methods and program code is effectively not openly shared
• Scientists downplay rather than admit their mistakes

Although mostly concentrated on bio-medical research, the critiques have a lot of traction elsewhere in science.




A lot of interesting issues on which simulation could help understand -- given you admit when they don't work, of course ;-)

A paper exposing the world of dodgy "open access" journals

Who’s Afraid of Peer Review?

A spoof paper concocted by Science reveals little or no scrutiny at many open-access journals

 

Accessible at: http://www.sciencemag.org/content/342/6154/60.full.pdf

Alongside very respectable open-access journals, who are committed to keeping published research open-access, are a world of journals who are little more than a vanity press, publishing anything in return for cash.  The journal "Science" has done an investigation, and found that many of these journals accepted a spoof paper they concocted.  Of course, Science, has a vested interest in stressing the virtues of the old "pay wall" model of publication, but there is no doubt that many unscrupulous "open access" journals exist (along side older "conferences" with basically the same purpose).

The official registration form for the Lorentz workshop on "Simulating the Social Processes of Science" is now open

The form for requesting registration is at:
  http://www.lorentzcenter.nl/lc/web/2014/607/info.php3?wsid=607&venue=Oort

This allows anyone to request a place at the Lorentz workshop.  For those who have already expressed an interest rest assured their registration will be accepted, for others that are interested and request registration we will probably make a decision sometime by the end of November 2013 (on those making requests up to that point).  We are committed to encourage women and young researchers to take up any extra places, and those who are developing or have developed a relevant simulation.

Associated workshop on modelling innovation, Budapest, May, 22-23 2014

A workshop at Eötvös Loránd University, Budapest,Hungary, May, 22-23 2014
This workshop will bring together two communities to join forces in research on innovation policy modelling as it intersects the areas of complexity science and social simulation. Broadly speaking it aims to show how complexity models and simulation such as the Simulating Knowledge dynamics in Innovation Networks (SKIN) model can be used to improve and inform the innovation policy making process.

The workshop will focus on three key overlapping themes:

• Modelling, understanding and managing innovation policy using the SKIN model
• Large scale data and scalability for research and innovation policy modelling
• SKIN between complexity science and social science: mechanisms and components

Places are limited and priority will be given to those offering presentations or posters.

Full details at: http://cress.soc.surrey.ac.uk/SKIN/events/third-skin-workshop

ESSA support for 2 PhD students to attend the Lorentz workshop on "Simulating Social Processes of Science"

There are two scholarships available for ESSA members who are still PhD students to give them financial support for them to attend the Lorentz workshop on "Simulating the Social Processes of Science", that will occur 7-11 April 2014 at the Lorentz Centre in Leiden, the Netherlands.

This would cover:

• 6 nights hotel: €77 x 6 = €462
• 5 days lunch: €11 x 5 = €55
• €300 (max) travel costs

Total per student = €817

To apply, please send an explanation of why you are worthy to receive this and a short CV to Bruce Edmonds <bruce@edmonds.name> by 24th November 2013.

Job: postdoc on Crowdsourced conceptualization of complex scientific knowledge and discovery of discoveries”. Leiden

PostDoc position (3 years, starting 1st November 2013 or as soon as possible thereafter)

The PostDoc position is funded by the Swiss Sinergia Grant “Crowdsourced conceptualization of complex scientific knowledge and discovery of discoveries”. The goal of this project is to make a significant step towards the semi-­‐automated conceptualization of scientific knowledge and its large scale analysis. This includes developing a participatory platform for knowledge elicitation as well as resolving a number of deep theoretical problems that have to be tackled prior to designing such a system. The results of these theoretical investigations will be implemented in the participatory platform, in order to increase its usability and data-­‐harvesting power and at the same time providing a validation framework for tuning the methods and algorithms. The project will be based on the ScienceWISE system (http://sciencewise.info), already used by many scientists for semantically importing, storing and searching scientific data, and will develop it further.

Within the project, the specific contribution of the PostDoc position funded in Leiden will be the development and validation of algorithms for “networks of networks” based on scientific information (including the proper definition of their topological properties; multi-­‐level community detection and hierarchical clustering; coarse-­‐graining; studying the time evolution of multi-­‐networks, etc.). The ScienceWISE platform will provide significant opportunity and user feedback for real-­‐time validation of such methods.

As all postdoctoral positions in the Netherlands, the position is a regular employment contract and is renewed for 2 years after the first one, for a total of 3 years. Interested candidates should send an email with their CV and motivation to Alexey Boyarsky (boyarsky@lorentz.leidenuniv.nl) as soon as possible.

The simulation of Squazzoni & Gandelli's peer review simulation (downloadable)

Squazzoni, Flaminio, Gandelli, Claudio (2012, December 25). "Peer Review Model" (Version 2). CoMSES Computational Model Library.
Downloadable from: http://www.openabm.org/model/3145/version/2

The associated papers are:

Squazzoni F. and Gandelli C. (2013) Opening Black-Box of Peer Review. An Agent-Based Model of Scientist Behaviour, Journal of Artificial Societies and Social Simulation, 16 (2) 3: <http://jasss.soc.surrey.ac.uk/16/2/3.html>

and

Flaminio Squazzoni (2013) Can Editors Compensate for the “Luck of the Reviewer Draw” Effect in Peer Review? An Agent-Based Model. ESSA 2013 conference, Warsaw. (paper)

The papers from the session at ESSA 2013 on "Social simulation of science processes"

9th Conference of the European Social Simulation Association (ESSA), Warsaw, Sept. 2013
Parallel session: Social simulation of science processes
Session Chair: Flaminio Squazzoni
Location: Aula II, Warsaw School of Economics, "C" Building
Tuesday, 2013/09/17 16:00-17:45

• Warren Thorngate and Wahida Chowdhury: By the numbers: Track record, flawed reviews, journal space, and the fate of talented authors (abstract) (full paper on SpringerLink)
• Petra Ahrweiler, Nigel Gilbert and Andreas Pyka: Modelling research networks (extended abstract)
• Melanie Baier: Heterogeneous, satisficing scientists on the road to scientific consensus (extended abstract)
• Bulent Ozel: A Multi-agent Simulation Model on Individual Cognitive Structures and Collaboration in Sciences (extended abstract)
• Flaminio Squazzoni, Francisco Grimaldo and Juan Bautista Cabotà: Can Editors Compensate for the “Luck of the Reviewer Draw” Effect in Peer Review? An Agent-Based Model (extended paper)

An implementation of Gilbert's "A simulation of the structure of academic science" model

A netlogo version of the model described in:

Gilbert, Nigel. (1997). A simulation of the structure of academic science. Sociological Research Online, 2(2)3, http://www.socresonline.org.uk/socresonline/2/2/3.html

Which, as far as I am aware is the first multi-agent simulation of inter-scientist processes.

The model is accessible at the CoMSES model archive (openabm.org):
     http://www.openabm.org/model/2296/version/2/view

Paper: Bibliometric Evidence for a Hierarchy of the Sciences

Fanelli D, Glänzel W (2013) Bibliometric Evidence for a Hierarchy of the Sciences. PLoS ONE 8(6): e66938. http://dx.doi.org/10.1371/journal.pone.0066938

Abstract

The hypothesis of a Hierarchy of the Sciences, first formulated in the 19^th century, predicts that, moving from simple and general phenomena (e.g. particle dynamics) to complex and particular (e.g. human behaviour), researchers lose ability to reach theoretical and methodological consensus. This hypothesis places each field of research along a continuum of complexity and “softness”, with profound implications for our understanding of scientific knowledge. Today, however, the idea is still unproven and philosophically overlooked, too often confused with simplistic dichotomies that contrast natural and social sciences, or science and the humanities. Empirical tests of the hypothesis have usually compared few fields and this, combined with other limitations, makes their results contradictory and inconclusive. We verified whether discipline characteristics reflect a hierarchy, a dichotomy or neither, by sampling nearly 29,000 papers published contemporaneously in 12 disciplines and measuring a set of parameters hypothesised to reflect theoretical and methodological consensus. The biological sciences had in most cases intermediate values between the physical and the social, with bio-molecular disciplines appearing harder than zoology, botany or ecology. In multivariable analyses, most of these parameters were independent predictors of the hierarchy, even when mathematics and the humanities were included. These results support a “gradualist” view of scientific knowledge, suggesting that the Hierarchy of the Sciences provides the best rational framework to understand disciplines' diversity. A deeper grasp of the relationship between subject matter's complexity and consensus could have profound implications for how we interpret, publish, popularize and administer scientific research.

New article on peer review in Nature Scientific Reports: "How important tasks are performed: peer review" by T. Hartonen and M.J. Alaya

Abstract

The advancement of various fields of science depends on the actions of individual scientists via the peer review process. The referees' work patterns and stochastic nature of decision making both relate to the particular features of refereeing and to the universal aspects of human behavior. Here, we show that the time a referee takes to write a report on a scientific manuscript depends on the final verdict. The data is compared to a model, where the review takes place in an ongoing competition of completing an important composite task with a large number of concurrent ones - a Deadline -effect. In peer review human decision making and task completion combine both long-range predictability and stochastic variation due to a large degree of ever-changing external “friction”. http://www.nature.com/srep/2013/130417/srep01679/full/srep01679.html?WT.ec_id=SREP-20130423

COST action on knowledge maps launched: KNOWeSCAPE

The following COST action has been launched today, on developing new maps (or ways of navigating knowledge).  It will be organising workshops, conferences, visits and funding costs of eligible participants to attend these.  In particular it will fund some of the costs of people attending the Lorentz Workshop on "Simulating the Social Processes of Science", to be held 7-11 April 2013.

Anyone from any COST countries can apply to this network project to be funded to attend such, or even run such an event. 

Their website is at: knowescape.org.
The official COST page for this is: http://www.cost.eu/domains_actions/mpns/Actions/TD1210?parties

This action is described as follows:

There is no escape from the expansion of information, so that structuring and locating meaningful knowledge becomes ever more difficult. This project will tackle this urgent problem using the unique networking and capacity-building features provided by the COST framework. For the first time, a platform will be created where information professionals, sociologists, physicists, digital humanities scholars and computer scientists collaborate on problems of data mining and data curation in collections. The main objective is advancing the analysis of large knowledge spaces and systems that organize and order them. The combination of insights from complexity theory and knowledge organization will improve our understanding of the collective, self-organized nature of human knowledge production and will support the development of new principles and methods of data representation, processing, and archiving. To this end, the knowledge organization in web-based information spaces such as Wikipedia as well as collections from libraries, archives, and museums will be studied. KnowEscape aims to create interactive knowledge maps. Their end users could be scientists working between disciplines and seeking mutual understanding; science policy makers designing funding frameworks; cultural heritage institutions aiming at better access to their collections; and students seeking a first orientation in academia.

Paper: Bollen et al. "Collective allocation of science funding: from funding agencies to scientific agency"

Collective allocation of science funding: from funding agencies to scientific agency

Johan Bollen, David Crandall, Damion Junk, Ying Ding, Katy Boerner

(Submitted on 3 Apr 2013)

Public agencies like the U.S. National Science Foundation (NSF) and the National Institutes of Health (NIH) award tens of billions of dollars in annual science funding. How can this money be distributed as efficiently as possible to best promote scientific innovation and productivity? The present system relies primarily on peer review of project proposals. In 2010 alone, NSF convened more than 15,000 scientists to review 55,542 proposals. Although considered the scientific gold standard, peer review requires significant overhead costs, and may be subject to biases, inconsistencies, and oversights. We investigate a class of funding models in which all participants receive an equal portion of yearly funding, but are then required to anonymously donate a fraction of their funding to peers. The funding thus flows from one participant to the next, each acting as if he or she were a funding agency themselves. Here we show through a simulation conducted over large-scale citation data (37M articles, 770M citations) that such a distributed system for science may yield funding patterns similar to existing NIH and NSF distributions, but may do so at much lower overhead while exhibiting a range of other desirable features. Self-correcting mechanisms in scientific peer evaluation can yield an efficient and fair distribution of funding. The proposed model can be applied in many situations in which top-down or bottom-up allocation of public resources is either impractical or undesirable, e.g. public investments, distribution chains, and shared resource management.

 Available at: http://arxiv.org/abs/1304.1067

Paper: Martin et al "Coauthorship and citation in scientific publishing"

Coauthorship and citation in scientific publishing

Travis Martin, Brian Ball, Brian Karrer, M. E. J. Newman

(Submitted on 1 Apr 2013)

A large number of published studies have examined the properties of either networks of citation among scientific papers or networks of coauthorship among scientists. Here, using an extensive data set covering more than a century of physics papers published in the Physical Review, we study a hybrid coauthorship/citation network that combines the two, which we analyze to gain insight into the correlations and interactions between authorship and citation. Among other things, we investigate the extent to which individuals tend to cite themselves or their collaborators more than others, the extent to which they cite themselves or their collaborators more quickly after publication, and the extent to which they tend to return the favor of a citation from another scientist.

Available at: http://arxiv.org/abs/1304.0473

Lorentz workshop on “Simulating the Social Processes of Science” 7-11 April 2014 Accepted!

The proposal for a  workshop on:

“Simulating the Social Processes of Science”

has been approved by NIAS and the Lorentz Center. Congratulations! We have been able to schedule the workshop for the period 7 – 11 April 2014 at the venue Lorentz Center@Oort in Leiden in the Netherlands.

More information when it is available.

New Paper: Opening the Black-Box of Peer Review: An Agent-Based Model of Scientist Behaviour

Squazzoni, Flaminio and Gandelli, Claudio (2013) 'Opening the Black-Box of Peer Review: An Agent-Based Model of Scientist Behaviour' Journal of Artificial Societies and Social Simulation 16 (2) 3 <http://jasss.soc.surrey.ac.uk/16/2/3.html>.

Abstract

This paper investigates the impact of referee behaviour on the quality and efficiency of peer review. We focused on the importance of reciprocity motives in ensuring cooperation between all involved parties. We modelled peer review as a process based on knowledge asymmetries and subject to evaluation bias. We built various simulation scenarios in which we tested different interaction conditions and author and referee behaviour. We found that reciprocity cannot always have per se a positive effect on the quality of peer review, as it may tend to increase evaluation bias. It can have a positive effect only when reciprocity motives are inspired by disinterested standards of fairness

 

Keywords:

Peer Review, Referees, Referee Behaviour, Reciprocity, Fairness

 

• Available at: http://jasss.soc.surrey.ac.uk/16/2/3.html