Monday, 5 December 2016
If you are near Chicago... A Science of Science meeting, today!
See http://www.science-of-science-chicago.org/
More evidence for the class divide in UK universities
...from an article in the Guardian.
See Guardian article at: https://www.theguardian.com/education/2016/dec/05/poorer-white-pupils-underperform-in-later-academic-choices-study
This is some more evidence that the role of the top universities is substantially about class and not just academic achievement. The top universities have simply not taken effective measures to address this lack of access.
Researchers found that as much as half of the gap in admissions to highly selective Russell Group universities between children on free school meals (FSM) and their better-off peers could be a result of factors beyond academic ability.From an analysis published by the Social Mobility Commission, headed by former Labour minister Alan Milburn.
See Guardian article at: https://www.theguardian.com/education/2016/dec/05/poorer-white-pupils-underperform-in-later-academic-choices-study
This is some more evidence that the role of the top universities is substantially about class and not just academic achievement. The top universities have simply not taken effective measures to address this lack of access.
Thursday, 1 December 2016
Friday, 11 November 2016
Understanding peer review and the social processes of science mentioned in EuroScience article
Self-organised scientific crowds to remedy research bureaucracy
Read it at:
Saturday, 24 September 2016
Papers on Peer Review and other aspects of Simulating Science @ Social Simulation 2016, Rome
Five papers on simulating Peer Review!
- Simulating Peer-review processes as social practices by Dignum Virginia, Dignum Frank
Indirect Reciprocity and the Miracle of Peer Review by Righi Simone, Takacs Karoly - Quality versus Sexiness: The rival qualities of papers in the competition for academics’ attention by Watts Christopher, Payette Nicolas
- The shadow of reciprocity on confidential vs. open peer review: An agent-based model by Bianchi Federico, Squazzoni Flaminio
- Peer review as a reputation system by Sabater-mir Jordi, Grimaldo Francisco, Paolucci Mario
- Complex systems approach to scientific publication and peer-review system: development of an agent-based model calibrated with empirical journal data by Kovanis Michail, Porcher Raphael, Ravaud Philippe, Trinquart Ludovic
- Simulating German Extra-University Research Institutes Using Ego Network Samples by Schilperoord Michel, Schrempf Benjamin, Dirlam Arno, Ahrweiler Petra
- Towards the Discovery of Scientific Revolutions in Scientometric Data by De Langhe Rogier
Sunday, 24 July 2016
Citation Cartels and Parocialism in Science
There is hard evidence that academics and journals are manipulating citations to (unfairly) improve their reputation. There is a summary of this and a call for more evidence in the post "What do we know about journal citation cartels? A call for information" at https://www.cwts.nl/blog?article=n-q2w2b4
However, these deliberately fraudulent cases are simply the extreme end of a spectrum of practices which need examination, understanding and (of course) simulation modelling. It is well known that many referees demand that papers cite certain papers (e.g. their own) as a condition of publication. It is well known that in some fields, there is a strong social norm that one should spend the first 2-5 slides of any presentation citing previous work (whether relevant or not). This demand that outsiders should learn the 'key' references before being allowed to present their research has advantages in terms of not repeating past debates/research and to aid the coherence of the field, but it is also an effective means of excluding outsiders and ensuring insiders are well cited.
There have now been a stream of simulations that touch on the processes of peer review, but a wider set of simulations and analysis of science are needed that focus upon the tendencies of fields to become inward-looking to the extent that some look more like cartels than academic discussions.
However, these deliberately fraudulent cases are simply the extreme end of a spectrum of practices which need examination, understanding and (of course) simulation modelling. It is well known that many referees demand that papers cite certain papers (e.g. their own) as a condition of publication. It is well known that in some fields, there is a strong social norm that one should spend the first 2-5 slides of any presentation citing previous work (whether relevant or not). This demand that outsiders should learn the 'key' references before being allowed to present their research has advantages in terms of not repeating past debates/research and to aid the coherence of the field, but it is also an effective means of excluding outsiders and ensuring insiders are well cited.
There have now been a stream of simulations that touch on the processes of peer review, but a wider set of simulations and analysis of science are needed that focus upon the tendencies of fields to become inward-looking to the extent that some look more like cartels than academic discussions.
Sunday, 10 July 2016
A 1982 paper: " Peer-review practices of psychological journals: The fate of published articles"
I have just come across this.
Petersa, DP & Cecia, SJ (1982) Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Science, 5(2):187- 195. http://dx.doi.org/10.1017/S0140525X00011183
Abstract
A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.
The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.
With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.This is followed by an extensive and interesting open-commentary, making a very interesting issue, available at: http://journals.cambridge.org/action/displayIssue?jid=BBS&volumeId=5&seriesId=0&issueId=02
Coherency Models of Review Judgements
I am at a nice interdisciplinary workshop in Berlin on "Coherency-Based Approaches to Decision Making, Cognition and Communication.
The basic ideas come from Thagard (1989), that human 'reasoning' happens in a way to ensure coherency between beliefs rather than the classical logicist picture of reasoning from evidence to conclusions. For example, as well as forward inference, backward inference from conclusions to the evaluation of evidence is common. This theory has now a reasonable amount of evidence to support it and has been extended to include emotions and goals (Thagard 2006)
As well as Thagard there was a nice talk by Dan Simon of University of South Carolina, explaining many identified 'biases' in human reasoning using the coherency framework. If you want details see his paper on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=439984. One of the biases he talked about was confirmation bias and cited a nice paper that I did not know about about Peer Review. This is:
The coherency model of thought seems to be a good basis for modelling the judgement of reviewers within simulations of the peer review system.
The basic ideas come from Thagard (1989), that human 'reasoning' happens in a way to ensure coherency between beliefs rather than the classical logicist picture of reasoning from evidence to conclusions. For example, as well as forward inference, backward inference from conclusions to the evaluation of evidence is common. This theory has now a reasonable amount of evidence to support it and has been extended to include emotions and goals (Thagard 2006)
As well as Thagard there was a nice talk by Dan Simon of University of South Carolina, explaining many identified 'biases' in human reasoning using the coherency framework. If you want details see his paper on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=439984. One of the biases he talked about was confirmation bias and cited a nice paper that I did not know about about Peer Review. This is:
Mahoney, M (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2):161–175. http://link.springer.com/article/10.1007/BF01173636.In this the author sent papers to a selection of reviewers where it was known whether the reviewer agreed or disagreed with the conclusion of the paper (on a controversial issue). The results were that the judgement of the reviewers on the quality of the paper were substantially affected by whether the reviewer agreed with the conclusion.
The coherency model of thought seems to be a good basis for modelling the judgement of reviewers within simulations of the peer review system.
Friday, 8 July 2016
CfP special issue on "Academic Misconduct & Misrepresentation: From Fraud and Plagiarism to Fake Peer Reviews, Citation Rings, Gaming Rankings, Dodgy Journals, “Vacation” Conferences, and Beyond"
in the journal: Research Policy
Edited by: Mario Biagioli and Martin Kenney
This Special Issue solicits social scientific articles examining not only traditional forms of misconduct, but also modalities of misconduct that are meant to “game” the modern metrics-based regimes of academic evaluation. If traditional misconduct – fabrication, falsification, and plagiarism – concerned fraudulent ways to produce scholarly publications, much of the new misconduct targets the publication system itself, for example by producing fake peer reviews, citation rings among authors and journals, or by publishing articles in dubious journals.
...
See http://www.journals.elsevier.com/research-policy/call-for-papers/academic-misconduct-misrepresentation-from-fraud-and-plagiar for more details.
Edited by: Mario Biagioli and Martin Kenney
This Special Issue solicits social scientific articles examining not only traditional forms of misconduct, but also modalities of misconduct that are meant to “game” the modern metrics-based regimes of academic evaluation. If traditional misconduct – fabrication, falsification, and plagiarism – concerned fraudulent ways to produce scholarly publications, much of the new misconduct targets the publication system itself, for example by producing fake peer reviews, citation rings among authors and journals, or by publishing articles in dubious journals.
...
See http://www.journals.elsevier.com/research-policy/call-for-papers/academic-misconduct-misrepresentation-from-fraud-and-plagiar for more details.
Thursday, 30 June 2016
Nature paper: Interdisciplinary research has consistently lower funding success
See http://www.nature.com/nature/journal/v534/n7609/full/nature18315.html
x axis is the proportion of applications funded (Australian research council) and the y-axis is the estimated bias towards interdisciplinary proposals (estimated using a logit regression model).
Environmental, physical, social science and economics were (unsurprisingly) most averse to interdisciplinary proposals.
"...we show that the greater the degree of interdisciplinarity, the lower the probability of being funded. The negative impact of interdisciplinarity is significant even when number of collaborators, primary research field and type of institution are taken into account."
x axis is the proportion of applications funded (Australian research council) and the y-axis is the estimated bias towards interdisciplinary proposals (estimated using a logit regression model).
Environmental, physical, social science and economics were (unsurprisingly) most averse to interdisciplinary proposals.
Thursday, 3 March 2016
Repost: Why publishing negative findings is hard
From retraction watch, the story of a professor who tried to submit his reanalysis of two papers to journals and the obstacles he faced.
At: http://retractionwatch.com/2016/02/17/why-publishing-negative-findings-is-hard/
At: http://retractionwatch.com/2016/02/17/why-publishing-negative-findings-is-hard/
Abstract of Paper: Job Insecurity in Academic Research Employment: An Agent-Based Model
Abstract:
This paper presents an agent-based model of fixed-term academic employment in a competitive research funding environment. The goal of the model is to investigate the effects of job insecurity on research productivity. Agents may be either established academics who may apply for grants, or postdoctoral researchers who are unable to apply for grants and experience hardship when reaching the end of their fixed-term contracts. Results show that in general adding fixed-term postdocs to the system produces less total research output than adding half as many permanent academics. An in-depth sensitivity analysis is performed across postdoc scenarios, and indicates that promoting more postdocs into permanent positions produces significant increases in research output.More details at: https://drericsilverman.wordpress.com/2016/03/03/paper-submitted-to-alife-xv/
Monday, 15 February 2016
Network analysis paper: Weaving the fabric of science: Dynamic network models of science's unfolding structure
Weaving the fabric of science: Dynamic network models of science's unfolding structure
By Feng Shi, Jacob G. Foster, and James A. EvansHighlights
- • Our hypergraph framework captures the multi-mode, higher-order complexity of science.
- • Our random walk model powerfully predicts how science evolves.
- • Our approach reveals intriguing modal dispositions behind the advance of science.
- • We find that entities of one type typically connect through entities of another type.
- • We find a special bridging role for methods and chemicals in the fabric of science.
- • We find that adding more node types leads to superlinear improvements in prediction.
Abstract
Science
is a complex system. Building on Latour's actor network theory, we
model published science as a dynamic hypergraph and explore how this
fabric provides a substrate for future scientific discovery. Using
millions of abstracts from MEDLINE, we show that the network distance
between biomedical things (i.e., people, methods, diseases, chemicals)
is surprisingly small. We then show how science moves from questions
answered in one year to problems investigated in the next through a
weighted random walk model. Our analysis reveals intriguing modal
dispositions in the way biomedical science evolves: methods play a
bridging role and things of one type connect through things of another.
This has the methodological implication that adding more node types to
network models of science and other creative domains will likely lead to
a superlinear increase in prediction and understanding.
Keywords
Link prediction; Hypergraphs; Random walks; Multi-mode networks; Science studies; MetaknowledgePaper (open access) available at: http://www.sciencedirect.com/science/article/pii/S0378873315000118
Friday, 8 January 2016
Modelling academic research funding as a resource allocation problem
OK this is 5 years old but I have only just come accross it. From 2010 but modified 2014.
Geard, Nicholas and Noble, Jason (2010) Modelling academic research funding as a resource allocation problem. In, 3rd World Congress on Social Simulation, University of Kassel, Germany, 06 - 09 Sep 2010. (Submitted).
Abstract:
For code and paper see: http://eprints.soton.ac.uk/271374/
Geard, Nicholas and Noble, Jason (2010) Modelling academic research funding as a resource allocation problem. In, 3rd World Congress on Social Simulation, University of Kassel, Germany, 06 - 09 Sep 2010. (Submitted).
Abstract:
Academic research funding is allocated through a competitive bidding process that may lead to inefficiency as excessive time is spent on proposal writing. We develop a simple agent-based model of the process and find that current systems are indeed likely to be inefficient. Alternative allocation schemes involving either a cap on individual effort or appraisal from the centre are indicated as improvements.
For code and paper see: http://eprints.soton.ac.uk/271374/
Subscribe to:
Posts (Atom)