ENRESSH Newsletter

N° 6 May 2019

Welcome

Emanuel Kulczycki
Chair of ENRESSH

Dear readers,

Welcome to the sixth ENRESSH newsletter!

It is my first foreword to an ENRESSH newsletter. Ioana Galleron—the founding-mother of ENRESSH— has taken a new professional challenge and now I have this exciting opportunity to be the chair of ENRESSH that gathers participants from 37 Europeans countries.

In its final year, ENRESSH is progressing full steam towards the realization of its milestones and objectives. As we saw in the MC & WG meeting in Podgorica, we are on track with the numerous of tasks and subprojects, among others, preparing the recommendations for evaluation agencies on the construction of national labelled lists of journals and publishers, classification of journals and publishers,  an overview of peer-review practices, and the recommendations and guidelines for proof-based impact narratives. Hence, the upcoming grand period will be crucial for completing the action goals. We are going to organize Work Groups meeting in September 2019, the WG3 training school for research information systems developers, library managers, bibliometrist on October 2019, and the final meeting of the action on the beginning of 2020.

Do not hesitate to look at our website for more, or even to drop us a line!

Wishing you a pleasant reading,

Emanuel Kulczycki
Chair of ENRESSH

People

ENRESSH Collaboration

Nataša Jermen

My name is Nataša Jermen fromThe Miroslav Krleža Institute of Lexicography in Zagreb. I visited Rupert Gatti from the Open Book Publishers in Cambridge and Tom Mowlam from Ubiquity Press in London for two weeks in February 2019.

The purpose of my visit was to get an insight into the HIRMEOS (High Integration of Research Monographs in the European Open Science Infrastructure) project, especially to confront the HIRMEOS aim to prototype innovative metrics service for Open Access (OA) scholarly monographs – a set of measures for tracing their usage on the HIRMEOS publishing platforms. One of the goals was to stimulate further research concerning scholarly monographs, as they represent a significant research output in the SSH and are vital for the scholarly communication landscape. Furthermore, this STSM tackled the ENRESSH’s aim to develop metrics and indicators that can better assess the performance and impact of SSH.

The HIRMEOS project aims at developing several innovative services for OA scholarly monographs by providing the additional data, links and interactions to the documents, at the same time paving the way for new potential tools for research assessment. It brings together 9 partners from 6 countries. The main objective of HIRMEOS is to enhance five European digital platforms for the publication of OA monographs (OpenEdition Books, OAPEN Library, EKT Open Book Press, Ubiquity Press and Göttingen University Press) and to ensure their interoperability by adopting common standards. The publishing platforms will be enriched with services and tools that enable identification, certification, information enrichment and entity extraction, annotation, and gathering usage and alternative metrics data. 

My main task was the qualitative analysis of the governance and designing of the HIRMEOS service aimed at collecting and visualising usage metrics and  altmetrics for OA scholarly monographs.  The HIRMEOS metrics service, which is still at the developmental stage in Open Book Publishers and Ubiquity Press, will provide metrics for views and downloads, altmetrics (tweets, Wikipedia quotes and annotations) and citation metrics, and a widget to display all metrics on the partners’ website. Data disseminated across a range of different platforms hosting a book will be collected, aggregated and presented to the public in the standardised way.

During my stay at Open Book Publishersand Ubiquity Press, I monitored the development of the HIRMEOS metrics service, as well as the current designing, tools and data presented on the two aforementioned digital platforms. I discussed the difficulties associated with aggregation and interpretation of usage data with her hosts, as well as the challenges associated with tracing the usage of books in OA.

It is expected that this visit will pave the way for the future collaboration between ENRESSH and HIRMEOS, which will enable the assessment of the HIRMEOS metrics service and its ability to measure influence and impact of scholarly monographs. Once the HIRMEOS project ends, ENRESSH could continue the collaboration with OPERAS, the research infrastructure project for the development of open scholarly communication, particularly in the SSH, and tackle the common goal of supporting the OA publishing.  

New Members

Laura James

Introducing Laura James

I am an Associate Professor of Tourism and Regional Change in the Department for Culture and Global Studies at Aalborg University, Denmark. My academic background is in the field of economic geography and regional studies, and my research has covered a variety of topics in the areas of learning, innovation, and regional development policy. In particular, my work has focused on the study of regional knowledge dynamics and territorial innovation models, looking at the ways in which different kinds of knowledge are circulated between universities, firms and civil society. I worked on the EU FP6 project EURODITE: Regional trajectories to the Knowledge Economy (2005-2010) and led the ESRC-funded Knowledge, Innovation and Learning Network (2012-2015). I have also worked in a variety of applied research settings, especially in relation to unemployment, labour market, innovation and education policies in the UK, Sweden and Denmark, which is where my interest in the conceptualisation and evaluation of SSH research and its impact began.

I joined ENRESSH in order to get a broader perspective on the evaluation of SSH research and  learn more about the conceptualisation of SSH value and impact in different countries as a member of working group 2. I hope that I can contribute with my multidisciplinary experience and am looking forward to learning from other network members who come from different disciplinary backgrounds and traditions. I hope also to help build synergies with the NORSSHVALUE project (Towards a Nordic Perspective on the Public Value of Social Sciences and Humanities Research), of which I am also a member.

Message to ENRESSH

Hugues Dumont
Vice-Rector for Research
Université Saint-Louis – Bruxelles

Université Saint-Louis – Bruxelles was very pleased to host and co-organize this first COST ENRESSH event about the challenges of SSH research evaluation.

The subject matters a lot. Indeed, as we have to manage careers, manage resources and be accountable for them, academic authorities, researchers, funders, all, we need to assess and to be assessed. And, all, we face the inability of finding simple and universal answers.

Evaluating science is intricate for various reasons, especially when we talk about evaluating SSH.

For example, speaking as a vice-rector for research, in universities we often feel like we are caught between two incompatible trends. On one hand, the norms and assessment standards required by the outside and not fully adapted to SSH: e.g., bibliometrics to measure scientific excellence and patents or spin-offs to measure its engagement with society. On the other hand, our wish to assess our researchers with criteria which acknowledge not so easily what scientific excellence also requires: criteria which acknowledge that in-depth research request a lot of time, which criteria open to the risk-taking character of interdisciplinary research, criteria which give credit to those who generously devote themselves to research, without obsessing about a personal benefit for their CV, in order to spark off and stir collective research in their institution.

Plus, we need assessment criteria which take into consideration professional and personal life cycle of researchers, a topic that COST ENRESSH also address.

We would like to assess our researchers with truly asses the quality of their work, but we also have the responsibility to prepare our researchers for the competition that they will encounter everywhere, and that our Universities also encounter.

Because of all those questions and concerns, I wanted to express my satisfaction about this seminar which gave us a first preview of the COST ENRESSH research results. My satisfaction is even greater since the day gathered, it is quite rare, researchers, funders and academic authorities.

Prof. Hugues Dumont,
Vice-Rector for Research,
Université Saint-Louis – Bruxelles

News

Evaluation Frameworks

Ethics of research evaluation should be made transparent.

While the need for clear ethical principles in research evaluation is known for a while, it seems that the discussion of ethics in research evaluation has not progressed much. This is why Aldis Gedutis and Maria Teresa Biagetti set out to investigate what ethical principles are employed in research evaluation. “The field is underdeveloped, even though it is obvious that judging others’ work involves ethical issues. Everyone does evaluate research, but nobody really bothers to provide clear rules and explicate the values that are employed”, Aldis says.

However, there are works on ethics of research and ethics of evaluation. “These works can be used to develop an ethics of research evaluation” the researchers conclude. After having analysed the existing scientific and grey literature including many guidelines on research ethics and evalution ethics, Aldis Gedutis and Maria Teresa Biagetti will continue with interviewing SSH scholars and stakeholders and develop guidelines for ethics of research evaluation by combining ethical theories, research ethics, evaluation ethics and interview data.

The goal of the study is to stimulate a reflection on ethical principles among people active in research evaluation, be it SSH scholars, evaluators or policy makers, to make efforts to make these principles transparent. It goes without saying that not engaging in research evaluation (e.g. boycott of evaluation procedures) comes equally with ethical principles. Persons interested in sharing their thoughts on this issue are warmly invited to contact Aldis Gedutis and Maria Teresa Biagetti.

Relevance and Impact

The big news from Working Group 2 in Spring 2019 has been the progress in the CARES project, looking at the impact of impact research evaluation on the career choices of Early Career researchers in the social sciences and humanities.

There were in total three short-term scientific missions, focusing on different aspects of those effects.

• Carina Balaban considered the effects of impact evaluation on SSH ECRs’ motivations and identities, and identified that the greatest problem was the destabilising effect that impact evaluation had on their careers,  in detracting from the time available for publication whilst demanding additional learning and expertise.  This research was covered recently in an LSE Impact Blog.
• Marc Vanholsbeecke looked at the effects of social context on impact evaluation, highlighting that the great diversity of social structures across Europe affected the ways that impact should be evaluated to more reasonably reflect what could be achieved.
• Brad Good looked at where the blockages lay in the creation of impact, and identified that they predominantly were a consequence of academic structures rather than a lack of receptiveness from society.

Results from these STSMs were presented at the Podgorica working group meeting along with two other papers.

• Stefan de Jong extended his earlier work and identified that there were differences the way impact evaluation affected ECRs between so-called high-performing and low-performing countries, and in particular the fact that it is typically not something taken as seriously by universities in LPCs, further undermining incentives to develop impact expertise.
• Julia Olmos Peñuela submitted research that showed that there were different combinations of factors associated with early career researchers creating impact, but that more training and more supportive universities were necessary to give ECRs better chances to create impact.

This research is still ongoing, and we will present more complete versions of these research tracks at the forthcoming RESSH conference in Valencia in September. We look forward to seeing you there!

Databases and Metrics

Upcoming training school on national bibliographic databases

From October 21^st to 25th we are organizing the third ENRESSH training school in Poznan, Poland. The topic of the training school concerns national bibliographic databases and the uses of data for understanding SSH research. The training school will build on the experience of the workshop Working with national bibliographic databases for research output which took place September 2018 in Antwerp. The main observations have been discussed on the LSE Impact of Social Sciences blog.

If you or someone you know wants to participate in the upcoming training school, please send an email. A formal call for applications will be launched soon, and confirmation of participation will be send before mid-July.

Meanwhile, a new paper on How to identify peer-reviewed publications : Open-identity labels in scholarly book publishing is published in PlosOne. In the paper we show that the open-identity label that Polish publishers use is a type of peer-review label like those used in Flanders and Finland, and as such it can be used to identify peer-reviewed scholarly books. Also, our contribution Guidelines for data sharing and data citation in social sciences and humanities originally presented at a workshop organized by the DARIAH EU Working group sustainable publishing of metadata has now been published online.

Special Interest Group: Early Career Investigators

Trainers and Participants of the Vilnius Training School

Training School “Evaluation procedures and their impact on SSH careers” 

The Training School “Evaluation procedures and their impact on SSH careers” was organised  from 7th to 11th January 2019 at the Vilnius University in Vilnius, Lithuania. The feedback of the Training School by the participants showed that the information on evaluation is dearly needed, and that early career investigators generally lack opportunities to gain knowledge about how to use and how to prepare themselves for evaluations. Also, there is a lack of training how to write and interpret reviews as well as how to use metrics.

The Training School was organised in lectures and hands-on sessions. The hands-on sessions were held in fixed groups of 5 trainees. In the morning, the lectures prepared the trainees for the hands-on session in the afternoon. After a kick-off where everyone provided spontaneous associations with research evaluation and filled in a questionnaire for Early Career Investigators, the trainees were introduced into the topic of research evaluation by lectures on quality criteria and measurement in research evaluation, differences in national research evaluation systems and the relation between scientific work, reporting and effects of regulations on research. The trainees also received detailed information on Bibliometrics and Altmetrics. A third focus topic were peer review practices, where the students learnt about peer review in the SSH, about the issues of peer review and how to write and how to interpret reviews.

In the hands-on session, the trainees developed a fictive interdisciplinary research proposal for a call entitled “Culture & Innovation”. Using this proposal, they applied what they learned using the references in their proposals for exercises on bibliometrics and peer review. Each group wrote reviews for the proposal of two other groups, which served to train writing and interpreting reviews. Finally, the students and trainers discussed on how to improve the situation for early career investigators.

The school included 29 participants and 7 teachers from 14 countries (and even more nationalities), representing 24 disciplines, composed of 15 men and 21 women formed a highly diverse and very active group. Two of the participants and six of the trainers were members of ENRESSH. The trainers were Aldis Gedutis, Agnė Girkontaitė, Thed van Leeuwen, Michael Ochsner, Jolanta Šinkūnienė, Wojciech Sowa and Alesia Zuccala.

The Training School materials can be accessed here.

Publications

ENRESSH members published two new papers:

Emmanuel Kulczycki, Ewa A. Rozkosz, Tim. C.E. Engels, Raf Guns, Marek Hołowiecki & Janne Pölönen (2019). ‘How to identify peer-reviewed publications: Open-identity labels in scholarly book publishing.’ In: PlosOne (Open Access).

Muhonen Reetta, Benneworth Paul & Julia Olmos-Peñuela (2019). ‘From productive interactions to impact pathways: Understanding the key dimensions in developing SSH research societal impact.‘ In: Research Evaluation, 1–14.

a working paper:

Wróblewska, M. N. (2019). ‘Impact evaluation in Norway and in the UK: A comparative study, based on REF 2014 and Humeval 2015-2017’. ENRESSH working paper series, 1. 

Glossary

Alexis Dewaele

Policy recommendation

At a certain moment in my career I basically stopped writing policy recommendations. As, I was working for the Policy Research Center on Equal Opportunities, it made sense to write them yet I soon became aware of the fact that they were rarely taken serious or implemented. Often they were not even read by those at the policy ‘steering wheel’. Policy making is a complex multifactorial process that involves a plethora of actors and is often quite unpredictable. I do believe that researchers can (and should) have an important role in policy development but it takes more than writing a one-pager as part of an often lengthy research report. Although many researchers are passionate about their topic and willing to talk with policy makers to make things happen, they sometimes lack what in literature has been referred to as ‘policy acumen’: developing an awareness of the policy world and understand that priorities can change quickly. The gap between policy makers and researchers is too large. Also, the latter often refrain from taking research results a step forward towards policy because they are afraid of losing their neutral or objective position. Research results are rather factual. Yet political decisions based on these ‘facts’ are seldomly ‘objective’ since they have great impact on people’s lives, economy and social systems. However, evidence-based policy is of utter importance and this is especially true in these fragile times of fake news. A transparent and democratic debate about taking results to action with involvement of all stakeholders (citizens, academia, policy makers and industry) is needed to tackle the important challenges of tomorrow.

Calendar

Events

5-7 June

European Forum for Studies of Policies for Research and Innovation

Rome, Italy

13-14 June

Nordic STS conference 2019

Tampere, Finland

4-6 July

European Group for Organizational Studies

Edinburgh, UK

2-5 September

International Conference on Scientometrics & Informetrics (ISSI)

Rome, Italy

4-7 September

4S Annual Meeting of the Society for Social Studies of Science

New Orleans, US

9-10 September

Association for Studies in Innovation Science and Technology (ASSIST) conference

Manchester, UK

9-11 September 

Triple Helix Conference

Cape Town, South Africa

10-12 September

Herrenhausen Symposium – Positioning the humanities in the 2020’s

Hanover, Germany

19-20 September

Research Evaluation in the Social Sciences (RESSH) conference 2019

Valencia, Spain

October 14-17

Atlanta Conference on Science and Innovation Policy

Atlanta, GA USA

October 17-18

Impact of Social Sciences and Humanities on Society

Washing DC, USA