Description of the topic: The role of peer review in SSH is to give scholars feedback on un-published manuscripts so that they can improve their research. Often it is referred to as the ‘gold standard’ of evaluation; yet, as a vetting process, it is time consuming, difficult to reproduce, subject to bias, and understudied (Campanario, 1998; Lee et al., 2013; Sayer, 2014). Little is known about the ‘quality’ of reviewing techniques and the effect that it is having on research products. In fact “there can be a huge variation” in how reviews are written, “from deep and insightful to shallow and misinformed” (Flier, 2016).2 Across the humanities in particular, scholars generally believe that “there is a lack of consensus on quality criteria” and that “comparing or assessing research quality is impossible” (Hug et al., 2013, p. 370).3 Guidelines are usually available to scholars concerning review practices; however without a sound body of research related to this, focused on large datasets of review texts, we have no evidence-base proxies for what we might call review ‘quality’ in the SSH.
The aim of this STSM is to investigate the peer review process both in terms of authors’ observations concerning manuscript reviews, as well as the thematic content of review texts themselves (i.e., manuscript referee reports). The research is designed to operate inductively (i.e., identifying themes/concepts as they occur in full-text) rather using a hypothetico-deductive approach to the study of peer review. Our project will be guided; therefore; by three major research questions: 1) What ‘proxies’ of quality are prevalent in referee reports produced for journal articles/books in the SSH? 2) Do these proxies of ‘quality’ differ for various disciplinary/subject areas? 3) Which ‘proxies’ of quality do SSH scholars perceive to be helpful in terms of stimulating specific types of actions?
Objectives: The first stage of data collection will involve the development and implementation of a questionnaire survey, with a specific crowd sourcing function: scholars from the SSH will be asked to respond to questions concerning their experience of a manuscript review process, and then submit copies of their received referee reports. Use of this data will involve tagging, coding, and extracting concepts from the referee reports, and comparing them to the questionnaire responses. It is based on the notion that “text mining [via natural language processing] and qualitative research are epistemologically compatible” (Yu et al., 2011).6 The second stage will involve analysing the survey results and evaluating them in connection with the referee reports using a combination of statistics, computational linguistic methods, with the integration of hermeneutics and social simulation to the mixed-methods design..
Special criteria for this STSM: The applicant must have knowledge of SSH research in his or her country and ideally possess the following:
- Masters degree or Ph.D. in social science/humanities research, relevant to the study description and objectives outlined above;
- A sound scientific background relevant to peer review research;
- Excellent skills related to the development of and analysis of questionnaire-based surveys;
- Excellent skills in quantitative or computational research methods;
- High interdisciplinary openness;
- Excellent English language skills (oral and written);
- High communicative and inter-cultural competence.
Results: The proposed STSM will a) contribute to new insights and indicators related to peer review and SSH research, b) generate empirical evidence for developing better approaches to peer review, and c) result in co-authored publications, as well as further applications for project funding with European granting agencies.
Practical details:
Working group: WG1 (conceptual frameworks for SSH research evaluation);
Duration and timing: between 2 weeks and 1 month from June to November 2018;
Location: University of Copenhagen (Denmark) – http://www.ku.dk/ENGLISH/;
Contact: Alesia Zuccala (mailto:a.zuccala@hum.ku.dk).