Sunday, 10 July 2016

Coherency Models of Review Judgements

I am at a nice interdisciplinary workshop in Berlin on "Coherency-Based Approaches to Decision Making, Cognition and Communication.

The basic ideas come from Thagard (1989), that human 'reasoning' happens in a way to ensure coherency between beliefs rather than the classical logicist picture of reasoning from evidence to conclusions.  For example, as well as forward inference, backward inference from conclusions to the evaluation of evidence is common. This theory has now a reasonable amount of evidence to support it and has been extended to include emotions and goals (Thagard 2006)

As well as Thagard there was a nice talk by Dan Simon of University of South Carolina, explaining many identified 'biases' in human reasoning using the coherency framework. If you want details see his paper on SSRN at http://papers.ssrn.com/sol3/papers.cfm?abstract_id=439984. One of the biases he talked about was confirmation bias and cited a nice paper that I did not know about about Peer Review.  This is:
Mahoney, M (1977) Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1(2):161–175. http://link.springer.com/article/10.1007/BF01173636
 In this the author sent papers to a selection of reviewers where it was known whether the reviewer agreed or disagreed with the conclusion of the paper (on a controversial issue). The results were that the judgement of the reviewers on the quality of the paper were substantially affected by whether the reviewer agreed with the conclusion.

The coherency model of thought seems to be a good basis for modelling the judgement of reviewers within simulations of the peer review system.


No comments:

Post a Comment