N° 1 July 2017
Welcome to the first newsletter of ENRESSH, the COST Action dedicated to research evaluation in social sciences and humanities!
Born from the EvalHum initiative (www.evalhum.eu), ENRESSH gathers participants from 35 Europeans countries. They are researchers in scientometrics, information sciences, sociology of organisations, or ‘simply’ history, literature and languages, but also policy makers and research administrators. This variety of profiles allows a confrontation of approaches towards the sensitive subject of social sciences and humanities (SSH) research evaluation. And it allows cross fertilisation between bibliometric and non-bibliometric methods, between ‘excellence’ and ‘societal value’, between the points of view of evaluators and of evaluated, and so on.
This newsletter is an invitation to discover and participate in this discussion. Going beyond the statement of the ‘difference’ of the SSH when it comes to research evaluation, ENRESSH explores perceptions about evaluation and attempts to understand quality representations in the SSH, identifies paths to impact and reflects about a judicious use of data on SSH output. The ultimate goal is to enable SSH to demonstrate their value, for academia as well as for society at large.
You have received this newsletter because one or more of the ENRESSH participants believes the action is relevant to you. We hope this assessment was correct and that you keep following us! (We will send a maximum of four newsletters a year.) If not, you can unsubscribe using the link at the bottom.
Wishing you an interesting read,
Chair of ENRESSH
Reetta Muhonen: Short term scientific mission to identify societal impact pathways
Dr. Reetta Muhonen from the Research Center for Knowledge, Science, Technology and Innovation Studies, TaSTI, University of Tampere, Finland, visited the Centre for Higher Education Policy Studies (CHEPS) at the University of Twente, The Netherlands, in January and February 2017 as STSM candidate. The main person responsible for her stay at CHEPS was the leader of the ENRESSH Working Group 2, dr. Paul Benneworth. In addition, co-leader of the WG2, dr. Julia Olmos-Peñuela, commented on the progress of the STSM’s work.
Reetta got inspired by the compelling opportunity to collaborate internationally on one of her favourite research topics. The main aim of the STSM was to develop a systematic typology of SSH pathways to societal impact by analyzing qualitative impact cases from 17 different countries in the field of social sciences and humanities (SSH).
ENRESSH and STSM also offers an agenda for strengthening research collaborations. Responsibility of the data gathering and analysis was a great opportunity for the candidate to network with Working Group 2 other from ca. 20 countries.
So far the results of STSM have already been presented in two conferences, two stakeholder meetings and four workshops and seminars. A paper on SSH Pathways to societal impact will be published next autumn in the CHEPS working paper series. The working paper will be developed into the journal publication by the end of this year.
Reetta feels that she got a great opportunity to strengthen the international aspect of her research and she wants to pay it forward. Thus, she is going to host two STSM candidates of the next STSM round of ENRESSH WG2 with the support of Cost ENRESSH network in the University of Tampere in the late summer and early autumn this year.
Introducing Nelius Boshoff
I am a Senior Lecturer at the Centre for Research on Evaluation, Science and Technology (CREST) at Stellenbosch University, South Africa. CREST is both an academic department and a research centre, and is the host of a national Centre of Excellence in Scientometrics and Science, Technology and Innovation Policy (SciSTIP). I teach Research Uptake and Societal Impact in the postgraduate programme in Science and Technology Studies, which is offered by CREST. My research on the topic of research impact is mainly located in two broad fields – SSH and agriculture – but I am also active in other fields. In addition to specialising in studies of research impact, I also have a track record in bibliometrics, focussing on research collaboration in the African context. In terms of academic training, I have a PhD in Science and Technology Studies on the topic of knowledge utilisation in the South African wine industry and a master’s degree in Research Psychology.
I joined ENRESSH in late 2016 and have since participated in the activities of Work Group 2 in both Sofia (Bulgaria) and Antwerp (Belgium). Three reasons motivated my participation. The first was the need for direct exposure to the prevailing European discourse on research evaluation in the SSH, particularly the research impact agenda. Secondly, by listening attentively to that discourse, I am hoping to creatively expand my knowledge and identify areas of mutual interest. Thirdly, and perhaps most importantly, I am seeking international research collaborations to contribute to future studies of research evaluation in SSH in the African context. I am looking forward to co-publish with other ENRESSH researchers on the broader impacts and other dimensions of research in the SSH, should such opportunities arise. There is a need to introduce an African perspective to the largely European-based research literature (email: firstname.lastname@example.org).
Other new members
- Agne Girkontaite, PhD candidate in Sociology at Vilnius University, Lithuania
- Andrea Petö, Professor, Department of Gender Studies, Central European University, Budapest
- Tobias Wolbring, Assistant Professor, University of Manheinem, Germany
- Gemma Elisabeth Derrick, Lecturer, Centre for Higher Education Research and Evaluation, Educational Research, Lancaster University
Message to ENRESSH
Recently the evaluation of science has become a rather topical problem worldwide. In the hard sciences these processes had already started in the 19th century, but only in 21st century did the scholars from social sciences and humanities (SSH) start to be involved in scientometrics in an uneven lyricists-physicists fight with the scientists, where the winners seem to be the physicists.
Next to this, Horizon 2020 put SSH in horizontal position.
Thus although in the beginning the resistance of the scholars against evaluation was enormously slow they have started to realise the advantages of measuring their work. At least in the sake of better worldwide visibility and for the purposes of funding.
The things are not easygoing.
The main problems pertain to the specifics of the different social sciences and humanities, which have the mission to preserve national identities, to safeguard and study national cultural heritage and values, to maintain the local in the global and to popularize it abroad. In this respect the STEM evaluation methods could not be simply used for SSH evaluation without adapting them, or even creating new ones.
And here in 2016, just in time, started European Network for Research Evaluation in the SSH (ENRESSH) COST Action 15137 with its main objectives “to enable social sciences and humanities to better demonstrate their true place in academia”. The ambition is to develop adapted and transparent methods of evaluation, calling in turn to improve the understanding of how SSH fields generate knowledge, what kind of scientific and societal interactions characterize different SSH disciplines, and what are the patterns of dissemination of research results in the SSH. ENRESSH aims “to benefit to European and international scholars in research evaluation and in sociology of sciences; to research managers and policy makers at all levels; to research data managers and information system designers; and, last but not least, to researchers in the SSH fields themselves”.
In January 2017 I was invited in my capacity of Scientific secretary of the Bulgarian Academy of Sciences in the field of SSH and national contact person of ERIH PLUS NSD to contribute to the meeting of members of ENRESSH, working group leaders and national experts from different European countries on the procedures for evaluating the research activities as well as the possibilities and the need to differentiate scientific research in various fields of science.
Discussions were provided on the opinion of the national scientific communities about the science-based indicators, about the impact of scientific research to society and the state; on the methods of collecting databases of the scientific work; on the influence of the evaluation of the scientific results on the career development of scientists and especially of the young scientists; on the ways to promote interdisciplinary research and partnership between the different scientific disciplines and in particular between the social sciences and humanities on the one hand and the natural and technical sciences on the other. A lot of questions were put to be resolved.
Prospectives for future reconsideration of the criteria for evaluation of the SSH scientific work on local, European and global level in order to improve communication and visibility of SSH in the European Research Area were outlined. Later in March there was another meeting in Sofia Bulgaria.
I would like to conclude that I find ENRESSH Cost Action rather necessary for the European scholars and SSH development.
Prof. Joanna Spassova-Dikova, PhD
Scientific secretary of the Social Sciences and Humanities at the Bulgarian Academy of Sciences
The objective of the working group on conceptual frameworks for SSH (social sciences and humanities) research evaluation (WG1) is to further our understanding of the SSH knowledge production processes as well as dissemination practices, as a basis for developing evaluation procedures that adequately reflect the research practices, goals and aims of the SSH. We will tackle many dialectic issues of potentials and drawbacks, e.g., metric approaches and peer review; international exchange and the local rootedness of SSH; the need for interdisciplinary exchange and disciplinary expertise.
Furthermore, the Work Group investigates SSH scholars’ quality perceptions and links them to evaluation procedures and analyses values regarding research evaluation as well as the effects evaluation has on scholars’ behavior and SSH knowledge production.
In the first year, the Work Group has amongst others collected projects that investigate SSH scholars’ notions of quality in the participating countries; has developed a scheme of national evaluation systems, along which countries will be classified; has investigated bottom-up procedures for evaluating SSH research in Europe; and worked on methodologies for comparing internationally qualitative interviews with scholars. WG1 members have submitted two papers to journals and presented their work on three international conferences (STI, OpenEval and RESSH).
In the running second year, we will create a typology of national evaluation systems classifying the participating countries along a variety of dimensions developed in the first grant period. We will expand the work on SSH scholars’ perceptions of quality using qualitative and quantitative techniques and analyse the peer review processes in the SSH from different perspectives. In cooperation with the Special Interest Group on Early Career Investigators, we will work on the importance and effects of evaluation on their work and career in order to prepare a Training School scheduled in the third grant period.
Call to action
The work from members of the work group show that evaluation criteria commonly applied in SSH assessments do not adequately reflect research quality in the disciplines. We suggest reflecting on what criteria are put forward by the scholars themselves in order to apply better adapted evaluation procedures. An overview on bottom-up approaches to research evaluation in the SSH is presented in the WG1-article published in open access in Palgrave Communications.
Relevance and Impact
The recent months from the working group on societal impact and relevance of the SSH (social sciences and humanities) research (WG2) have been devoted to finishing off the development of the typology of modes of engagement by which SSH research creates impact. This comes out of the Short Term Scientific Mission that Reetta Muhonen undertook to Twente in spring this year, in which a twelve-fold typology was developed from the 65 research impact fiches that WG2 members had submitted in winter 2016. An overview of this activity is provided in Reetta’s STSM report which is available to download here. As a result of the close orientation of this STSM to the activities for the spring of this year, Reetta has been co-opted as a Vice Chair of this WG and was responsible for presenting the findings to the Sofia meeting.
The basis for the typology is to analyse the ways in which impact is created along pathways and networks that deviate in various ways from a straightforward linear model, where a researcher publishes something, a user reads it and then does something differently thereby creating impact. In some cases, the deviations are relatively slight, involving various kinds of interaction between researcher and user around the finished product. But other pathways involve deeper interactions with the user, and have also potentially more significant implications in terms of creating scientific impact, offering new potentially interesting research questions.
Since then, we have been busy at presenting the findings (the typology) into a range of communities, both scientific and societal. The research has been presented to policy-makers in Norway at a symposium of the Nordic SSH research council, to the Luxembourg Institute of Socio-Economic Research and the Finnish Ministry of Education and culture. Scientific papers have been presented in Finnish meetings on sociology and Science & Technology Studies, as well as to the recent Research on Evaluation of SSH (RESSH) conference in Antwerp earlier this year.
At the recent Antwerp ENRESSH meeting, the WG held a very productive pair of sessions seeking to sharpen up the wicked issues facing SSH impact generation from scientific and practical perspectives. First, there was a concern to identify the main similarities and differences of practices at both national and disciplinary levels recognising that there will also be some similarities. Second, there was a concern to better consider ‘ordinary’ impact rather than just heroic tales of academics solving complex social problems. Thirdly was a need to understand the balance between allowing academics to use their creativity whilst still ensuring all academics recognise their societal duty. Fourth was a desire to better sketch out the networks and communities within which impact arises. And finally, and perhaps most interestingly, was the growing demand by academics to retain ethical control over shaping their impact to avoid potentially stoking unwelcome societal undercurrent at this time of growing ‘fake news’.
Databases and Metrics
The working group on databases and the uses of data for understanding databases and uses of SSH (social sciences and humanities) research (WG3) three tasks are (1) the analysis of dissemination patterns and characteristics, especially through bi- and multilateral work, (2) recommend rules and procedures for databases that captures SSH-outputs and design a roadmap for a European database of SSH-outputs, and (3) develop alternative metrics for the SSH.
On May 2nd Elea Giménez-Toledo, Jorge Mañana-Rodríguez, and Gunnar Sivertsen published their paper ‘Scholarly book publishing’ in the journal Research Evaluation, thus realizing the first peer reviewed paper that is the result of the collaboration within the working group.
Then on July 5th Linda Sīle presented the report ‘European databases and repositories for social sciences and humanities output’, which was the result of an intense process of surveying 75 experts from 39 countries about the state of the art in their country in terms of data collection practices for SSH output.
In the coming months Janne Pölönen and Emanuel Kulczycki will also be presenting their comparative work regarding dissemination patterns at the STI and ISSI conferences in Paris and Wuhan, respectively. Gradually their work will expand into multilateral comparisons with participants from more than 10 countries that have set up comprehensive databases for the SSH.
Under the leadership of Hanna-Mari Puuska the working group is also progressing with a proof of concept for a European integrated data infrastructure for SSH-outputs, the first results of which will be presented during our upcoming meeting on November 8th in Helsinki.
The working group is also exploring the possibilities of collaborative analysis of newer publication formats, predatory publishing, and data publishing in the SSH. Should you have any recommendations for topics to be addressed, please let us know.
Early Career Investigators
The goal of the transversal Special Interest Group composed of Early Career Investigators (ECI) is to gather experiences and reflections of ECI from different countries and across different disciplines of the social sciences and humanities (SSH), to identify the most problematic areas in SSH evaluation policies from ECI perspective, to voice ECI interests in SSH evaluation, to identify success factors in ECI career paths as well as the most compelling challenges they face, as well as to address other issues relevant for young scholars in SSH.
Based on best practices, this manifesto by ENRESSH underlines principles for designing and applying evaluation protocols for the social sciences and humanities.
More ENRESSH publications
- Jack Spaapen & Leonie van Drooge, 2017. ‘Evaluation and monitoring of transdisciplinary collaborations’ in Journal of Technology Transfer
- Elea Giménez-Toledo, Jorge Mañana-Rodríguez, and Gunnar Sivertsen, 2017. ‘Scholarly book publishing’ in Research Evaluation
- Linda Sīle, Raf Guns, Gunnar Sivertsen and Tim Engel, 2017. ‘European databases and repositories for social sciences and humanities output’
As any lexicographer will tell you, words only have meaning in context. So, when looking at ‘impact’ first stop must be a good corpus based dictionary, as the Oxford Dictionary of English, not to be confused with the OED, a historical dictionary. ODE tells us that impact refers to “the action of one object coming forcibly into contact with another”. This may not help us much in research evaluation, except to visualise a unhappy humanist throwing a weighty tome at a policy officer.
The second definition is more useful in our case, as it defines impact as being “a marked effect or influence”. Add a context word, such as ‘societal’ and we can imagine impact as being how academic work effects or influences society. Add an evaluation protocol, and impact becomes about how policies stimulate, influence, reorient or hinder research. Impact can be positive or negative, since all man’s activities can be highly beneficial or detrimental to himself, the environment and society.
This is where the social sciences and humanities play a key role as they allow us to understand and react to man’s behaviour towards himself and his fellows. Also, social and cultural outlooks change over time and space; understanding them is in itself beneficial, it is about reflecting on ourselves. For all these reasons, SSH research is essentially positive, and thus having impact by its very existence.
Societal impact should never be about justifying the value of research, it is about demonstrating the multitude of ways in which different fields contribute to a rich, stable democratic society. Impact cannot be measured and cannot be produced to order, but its many forms can be studied. This is precisely what ENRESSH does, so as to better understand research processes, research evaluation methodologies and their links, and to encourage the adoption of best practice.
The next keyword is ‘Metrics’. Dr. Aldis Gedutis Klaipeda University is cordially invited to reflect on this keyword in the next newsletter.
28-30 August 2017
Annual Consortium on Higher Education Researchers Conference
University of Jyväskylä, Jyväskylä, Finland
6-8 September 2017
Science, technology and innovation indicators conference
École Supérieure d’Ingénieurs en Électrotechnique et Électronique, Paris, France
18-19 September 2017
SATORI Final conference. Ethics assessment of Research and Innovation : Looking to the Future
MCE Conference & Business Centre, Brussels, Belgium
21-22 September 2017
The 4th Conference on Scholarly Publishing in the Context of Open Science (PUBMET 2017)
University of Zagreb, Zagreb, Croatia
20-21 September 2017
Optimising the impact of the SSH
University of Cardiff, Cardiff, United Kingdom
26-29 September 2017
4th Altmetrics conference
Ryerson University, Toronto, Canada
10-11 October 2017
Atlanta Conference on Science and Innovation Policy
Georgia Tech, Atlanta, United States of America
9-10 November 2017
22nd Nordic Workshop on Bibliometric and Research Policy
The House of Science and Letters, Helsinki, Finland