- 1 By research governance system I mean the processes, systems and organisational structures (both exp (...)
- 2 All the information and references used to compile the empirical basis for this article are or have (...)
1This article addresses the challenges that the reform of research evaluation poses in research governance systems1 dominated by centralised management structures. Using the example of Spain, I show the difficulties of adjusting evaluation practices within such governance structures and emphasize the systemic nature of the change needed. First, I summarise the rationale for research evaluation reform, its main objectives as laid out by the Agreement on Reforming Research Assessment, and the changes that would be necessary to achieve them. Next, I present an example of an attempt to introduce Agreement principles in Spain and the difficulties it faced. I use this example to show the systemic nature of the changes required, and in particular the weight of legal provisions and the role that the judiciary plays through the enforcement of such provisions in the Spanish evaluation system. I conclude with a discussion of the implications of this experience for the scope and focus of research evaluation reform strategies2.
2The Coalition on the Reform of Research Assessment (CoARA) (2022) is an initiative promoted by the European Commission with the support of several scientific organisations to further a change in the dominant way in which research evaluation is being conducted. CoARA is a coalition of diverse research agents whose goal is to promote the principles and pursue the commitments defined in the Agreement on Reforming Research Assessment, published on 20 July 2022 (henceforth Agreement). This document is the result of a broad process of collaboration and consultation in which more than 350 organisations participated. It proposes a set of reforms to address the problems posed by the current evaluation system, which stem from the consolidated practice to make evaluation revolve around a very narrow set of publication-based indicators:
Assessment processes relying predominantly on journal – and publication – based metrics are known to result in a ‘publish or perish’ culture that falls short of recognising diverse approaches and could come at the expense of quality.
3In response, the Agreement proposes 10 “overarching principles” and 10 “commitments”. Particularly relevant for their profound implications for the Spanish evaluation system are the two first commitments:
-
«Recognise the diversity of contributions to, and careers in, research in accordance with the needs and nature of the research»;
-
«Base research assessment primarily on qualitative evaluation for which peer review is central, supported by responsible use of quantitative indicators».
4Importantly, the diversity of contributions refers not only to «contributions to science» but also «for the benefit of society». Accordingly, the Agreement mentions among the contributions to be recognised several related to the diffusion and application of scientific knowledge: «public outreach, science diplomacy, science advice and science communicator roles to name a few».
5The publication of the Agreement was followed by an institution-building process. Its first step was the setting up of CoARA, whose first General Assembly took place on 1 December 2022. This first Assembly approved the Coalition’s governance structure. The Coalition’s main goal is to «enable systemic reform on the basis of common principles within an agreed timeframe, and to facilitate exchanges of information and mutual learning between all those willing to improve research assessment practices». The Coalition is «global in scope» and can be joined by any organisation involved in research: «research funding organisations, research performing organisations, national/regional assessment authorities and agencies, as well as associations of the above organisations, learned societies and other relevant organisations». Any of these organisations can sign the Agreement and can apply to become members of the Coalition. Members commit to drive a systemic reform in order to pursue the common principles, and to report on the measures their organisations have taken to progress towards the objectives set up in the Agreement.
- 3 In addition, the public research evaluation infrastructure includes some much smaller regional eval (...)
- 4 AEI manages the evaluation of research proposals and the monitoring of the projects for the largest (...)
6The main research evaluation agencies in Spain are the Agencia Nacional de Evaluación de la Calidad y Acreditación (ANECA) and the Agencia Estatal de Investigación (AEI)3. While AEI focuses mainly on the selection and monitoring processes of several large research funding programmes and initiatives4, ANECA is in charge of individual performance evaluations, accreditation of university teaching programme and the evaluation of the main Spanish doctoral grants programme. There are two main centralised individual performance evaluations:
-
The “sexenios”: every six years Spanish academics can submit their research outputs to an Aneca Commission (the National Commission for the Evaluation of Research Activity-CNEAI) and if they are positively evaluated they are granted one sexenio. A sexenio is a mark of recognition associated to a salary increase. In addition, a number of sexenios or having a sexenio granted within the last six years are often required to perform some academic tasks (like being part of a doctoral examination boards). ANECA assesses around 12.000 sexenio applications per year. The sections below will analyse ANECA’s attempt at aligning the sexenio evaluation process with CoARA principles.
-
Accreditations of university teaching staff. Before being able to apply for a tenured job in Spanish public universities, or to apply for a promotion to Chair, candidates must be accredited by ANECA. The accreditation guarantees that the candidate fulfils some basic performance conditions. Yearly, ANECA assesses more than 3.000 accreditation applications.
- 5 For instance, although figures change substantially from panel to panel, a disciplinary sexenios ev (...)
7ANECA and AEI coordinate the tasks of assembling academic panels who become responsible for the evaluations they have been entrusted. Although they thus mobilise large numbers of peer reviewers, it is still common that each of these panels needs to assess a large number of submissions5.
- 6 See for instance, Ràfols, Molas-Gallart (2022). This point has also been made, in relation to the t (...)
8Spain has become one of the leading countries by the number of CoARA signatories. These include many key organisations including ANECA, the Spanish National Research Council (CSIC), CERCA (the institution that coordinates more than 40 Catalan research centres employing close to 9.000 researchers), academies, private research foundations, and many universities. It has been argued that the large number of Spanish signatories can be explained by the extreme nature of the problems that the current evaluation system is posing in Spain and the consequent interest in reform. The centralised nature of many evaluation processes, their placement within a large bureaucratic system and the related judicialization of disagreements over evaluation results, combined with the large throughput of assessments that the central evaluation agencies have to manage, have generated a rigid evaluation system revolving around strictly defined, and measurable indicators that should preferably be based on auditable data. The system has effectively posed a barrier to incoming international mobility6, penalised interdisciplinarity, and reduced diversity and has even been argued to be detrimental to the prestige of leading university departments. The small set of indicators used are only able to value a specific set of academic profiles (focused on research able to generate a constant stream of publishable research outputs), working within the disciplinary boundaries that provide the structure of the evaluation panels.
9Yet, and despite the appeal of CoARA among many members of the academic community, the CoARA proposals were met with suspicion when not straightforward opposition from other parts of this same community. First, that an organisation has signed CoARA does not mean that there is internal consensus about the appropriateness of doing so. For example, as we will see below, ANECA’s joining of CoARA had to wait to a change of director, with the previous director being more cautious about the adequacy of the principles stated in Agreement. Second, there are some important organisations within the Spanish system that have not signed, more conspicuous among them the AEI. Therefore, only one of the two most important Spanish research evaluation agencies has joined CoARA, and this difference emerged when the two agencies were organically dependent from two different ministries: ANECA from the ministry in charge of universities and AEI from the ministry responsible for research and innovation investments. Different sensibilities about the priority of the CoARA process had emerged in different ministries.
- 7 https://blog.caixaresearch.org/el-uso-de-herramientas-de-inteligencia-artificial-puede-ayudarnos-a- (...)
10There are several points for debate. First the centrality of peer review and the use of indicators only as a tool to inform the peer review process. Critics argue that peer review will suffer from the biases that the individuals conforming the panels will unavoidably convey, and the biases injected by the panel mechanisms themselves. These biases, seen as subjectivity, are confronted with the “objectivity” of quantitative measures. It follows that the response to the problems identified by CoARA and in particular the need to value different types of scientific contribution has to be addressed by more complex and sophisticated quantitative assessment techniques, incorporating broader sets of indicators and more sophisticated quantitative analysis techniques like those afforded Artificial Intelligence methods. Thus, for instance, the AEI organised in March 2024, a meeting to discuss «New Methods of Scientific Assessment», focusing exclusively on the potential of Artificial Intelligence to improve “significantly” «scientific assessment systems»7.
11Second, CoARA’s goal to broaden evaluation criteria to cover the variety of forms in which contributions to science can be made has opened another line of criticism. A system might develop in which “anything goes”, and therefore unable to discriminate between substantial, “excellent” contributions to the advancement of science from more routine, marginal or incremental contributions. Limiting the opening up of evaluation criteria, while at the same time defending the pre-eminence of quantitative impact indicators as the ultimate measures of value can be seen as a means to identify and value “excellence”.
12In addition to the critical arguments the reform faces severe institutional barriers in the existent organisations and rules defining the evaluation system. The case that we will review below focuses on these barriers and the institutional inertia they have generated.
13Among ANECA’s responsibilities, one of the most noteworthy is the assessment of the sexenios applications. Within ANECA, the organism responsible for this task is the National Commission for the Evaluation of Research Activity (CNEAI). The sexenio was initially conceived as a recognition of research activity, to be linked to a reward in terms of a small, but permanent, salary increase. Its main “target” are the academics who hold tenured civil servant positions (lecturers in public universities and researchers at CSIC), but potential applicants have been expanded to include some academics contracted by public and private universities. Every six years, eligible academics can apply to receive the recognition and present evidence of their research results (based mainly on publications). Yet, from their initial intended focus on activity, the sexenios and the number of sexenios obtained through an academic career started to develop into a badge of quality associated with a broader set of rewards and opportunities. Importantly, from 2012, tenured university lecturers with an “active” sexenio (granted within the previous six years) hold a teaching load that is 25% lower than the rest of lecturers. Moreover, in some Spanish universities to be part of doctoral examining boards, teach in masters’ courses or supervise doctoral students requires to have a number of sexenios and the holding of an “active” sexenio. What had started as a tool to recognise and offer a reward for research activity, turned into a nationwide evaluation system to certify a given base level of research achievement. The system has, therefore, adopted an accreditation nature, and this is reflected in the effort that it is invested in verifying the applicants’ claims. The result of a process that is geared towards bureaucratic oversight of applicants’ claims reinforces its “controlling” function, and requires the support of clear quantitative criteria that can be used efficiently to deal with the hundreds of appeals that ANECA will receive every year (see below). Although the system is formally structured as a bibliometrics-informed peer review process, in practice it became a bibliometrics-led decision system one of whose goals is to limit, as much as possible, the influence of individual peer judgement. In other words, the opposite of COARA’s commitment to «Base research assessment primarily on qualitative evaluation for which peer review is central».
- 8 In 2023 there were a total of 15.112 applications.
14Further, the kind of bibliometric indicators that could be used in practice were very much limited by the conditions under which the system operated: thousands of applicants presenting five “contributions” each, published over six-year periods8. Article citation counts, which arguably could reflect somewhat more accurately the scientific impact of a contribution, would need to be field-weighted, yet different articles even by the same author could fall under different fields, and article citations would not reflect the impact of recent publications (publications up to the date of submission of the applications could be included). The only practical, straightforward solution to these problems was to use the publication outlet (the journal or the publisher in case of books) as the quality criterion. The quartile of the journal in which the contribution was published became de facto the main, if not only, criteria to establish quality, and this was formally laid out in official grading scales (“baremos”). For each research field a scale is published, establishing the number of articles that had to be published and the quartiles of the journals in which they were published in order to be granted a sexenio. Although in some cases the official criteria recognised that the citations received by the contribution should also be a consideration, the perception in the scientific community was clear that what counted was the Clarivate Impact Factor (IF) of the journal where the applicants had published their selected contributions. The system could be said to guarantee transparency but ran against the recommendations of most evaluation experts and the agreements laid out by the main reform guiding documents including the CoARA agreement. This was this situation that the ANECA’s reform process attempted to address.
15The limitations of the sexenios evaluation system had been recognised by ANECA long before the publication of the Agreement. On 7 October 2021 ANECA published the «general principles and guidelines […] for the development of research evaluation criteria» to be applied in several of its evaluation activities, including the sexenios. The guidelines represented a move towards the principles stated later in the Agreement: they incorporate «for the first time» «references to multidisciplinarity», they endorsed the use of «scientific production indicators that are not centred solely on quantitative assessments or on the exclusive use of the journals’ Impact Factor», and stated that in no case would ANECA blacklist journals. The consideration of multidisciplinarity was informed by the same state of opinion that was, at the time, driving the drafting of the Agreement. Yet, it must be noted that the guidelines were not against the use of the IF, but only its “exclusive” use. Further, they stressed the need to «improve the objectivity and transparency» of the evaluations as a key goal, but this is a goal that in the debates that accompanied the reform process has been associated with the use of quantitative indicators. Thus, for instance, the then ANECA Director Mercedes Siles Molina was reported to state in a speech to the XXI General Assembly of the Association of Women Researchers and Technologists that, in relation to the ongoing debate about the value of quantitative vs. qualitative approaches, both needed to be considered, but:
We have to be careful with the trend to discard everything that has been done previously and to start with a clean slate, forgetting the acquired knowledge. Then I do not consider it appropriate to shift suddenly to the qualitative. Because we know a lot about quantitative [approaches], we know what works and what does not, and pure qualitative is what we had twenty years ago, when the decision about the suitability of a candidate to a tenured position was decided without objective criteria.
16This speech, made in October 2022, in the middle of the period between the publication of the Agreement and the General Assembly that set up the Coalition, was referring without mentioning it by name to the debate generated by the principles laid out in the Agreement. It summarises the arguments that were to be repeated by those that held sceptical or outright opposed views to the CoARA reform process. First, the position acknowledges that qualitative indicators need to be considered, but then presents the qualitative approaches as being more limited than the more modern quantitative techniques; in this quote, it is presented as an old superseded way of carrying out assessments that is equated with the absence of objective criteria. The association of quantitative with objectivity and qualitative with subjectivity is very common in Spanish debates to the point that the equivalence is seldom questioned. Yet, CoARA principles see quantitative indicators just as a source of information to be added to an assessment process which is ultimately the responsibility of expert panels. By placing the qualitative assessment of expert panels at the core of the evaluation process, CoARA principles and objectives enter in conflict with views that see quantitative assessments as inherently superior.
17This view, that confers to quantitative indicators a special place in the assessment process is implicitly endorsed by the 2021 ANECA guidelines when they state that research assessment should seek to add «indicators to validate the quality of the contributions, and not only of the publication outlet». The goal seems to be to complement the JIF with new, yet to be specified quantitative indicators, which, on their own can be used to “validate” the quality of the contributions. Although the press release stated that «The Agency also studies the use of indicators for the evaluation of scientific production that are not entirely centred on quantitative assessment», the tentative tone is consistent with the positions expressed by the Agency’s Director. Importantly, ANECA was explicitly proposing to retain the use of IFs in research assessment, while CoARA decried its use.
18The debate in Spain was on, with officials and academics expressing a variety of opinions on the adequacy of the principles of the Agreement to the Spanish context. After the formal constitution of CoARA in December 2022, Spanish institutions took time to consider their joining the Coalition. In the case of ANECA the decision to join had to wait until a new Director was appointed in February 2023. Shortly after, on 22 March a new Organic Law of the University System (LOSU) was published. In the Spanish system, Organic Laws are foundational legislation of the highest rank that develop fundamental rights and establish the principles by which key State institutions will be ruled. The LOSU established, as a “general norm”, that interdisciplinarity was to be considered a merit in the evaluation of academic staff (art. 11.7); and stated that that the evaluation of the merits in accreditation processes had to be both quantitative and qualitative (art. 69.2.c) and be «based on the specificity of the area of knowledge» (art. 69.2.d).
19These developments led to a shift in ANECA’s position: it took less than two months for ANECA to join CoARA (and sign DORA) and it immediately set to work to implement the Agreement principles in its evaluation practice. In parallel, the LOSU was developed through a Royal Decree (RD) published on 18 July (RD 678/2023) regulating the accreditation system and the process to access the tenured, civil servant, lecturer positions. Although this RD did not address the sexenio evaluation, it is relevant as it opened another front in the introduction of the Agreement principles. The RD stated among its motivations the incorporation to Spanish evaluation practice of the «the orientations of the international movement to reform research evaluation, promoted by the European Commission, among others» followed by a restatement of the Agreement main principles:
the need to evaluate more adequately the plurality of functions of academic activity; to prevent the undesired incentives of a bad use of bibliometric indicators that lead to the overproduction of low-risk research and the standardisation of the system; and to foster creativity and innovation, reward the quality of scientific contributions over quantity, strengthen the link between production and the transfer and exchange of knowledge and the needs of society, as well as fostering collaborative research environments (RD 678/2023).
20The Agreement principles had, therefore, made it explicitly to Spanish legislation. Yet, the same text continued immediately to identify the main difficulty that these intentions faced:
These principles must also be adapted to the characteristics and challenges of the Spanish university system, as well as to the two-stage evaluation system that governs the selection and promotion processes for university teaching staff: while the accreditation procedure, due to its scale and centralisation, requires a higher level of standardisation, the entrance examination procedure allows greater scope for a qualitative and contextualised evaluation, which integrates a greater diversity of criteria (RD 678/2023).
21This statement recognises the main difficulty faced by the reform of the Spanish research evaluation system: the «scale and centralisation» of some of the procedures (the “sexenios” could be added to the group of centralised procedures) require high levels of “standardisation” and provide less scope for «qualitative and contextualised evaluation». In other words, the Spanish legislation recognises that the evaluation system, and specifically those procedures characterised by their centralisation and high “throughput” pose a barrier to the implementation of the central tenets driving the CoARA movement; and yet there is a desire to join the movement and reform the system. In support of these goals and to help address the problems identified, the RD established a «Commission for the evaluation and monitoring of the State accreditation system» to help ANECA in the development of evaluation criteria, guidelines and processes.
22The problem was even more substantial in that the main tool to provide the desired standardisation was the Journal Impact Factor. It was difficult to encounter a simple, alternative “measure” that could be used for rapid and easy comparison with a set of clear benchmarks. How to modify the processes to approximate the CoARA principles, while at the same time retaining an evaluation system that requires the yearly assessment of thousands of applicants became a central difficulty.
23In December 2023, with the help of the above-mentioned Commission and after an extensive consultation process, ANECA published a set of principles and assessment criteria to be used in the sexenios evaluation. The principles and criteria would allow for a recognition of a wider diversity of profiles and would progress towards the goal of building a more qualitative and better contextualised assessment. The proposals included the introduction of a new interdisciplinary area to be added to the structure of disciplinary panels organising the evaluation process, and a set of differentiated principles and criteria for each of these panels.
- 9 Based on my own notes of the event and discussions. References to this meeting can be found at http (...)
24A problem emerged in that the introduction of a qualitative perspective in the assessment system could arguably endanger the «Legal Certainty» («Seguridad Jurídica») of the applicants. Legal Certainty is considered a fundamental principle of the Rule of Law and states the need for uniform criteria in the judicial application of the Law (Rosado Iglesias, 2006). This need for uniform criteria is at tension with the need for contextual assessment that CoARA is endorsing. This tension is particularly relevant given the extent to which the Spanish sexenio evaluation system has become judicialized. A 2009 article published in a legal journal stated, in a rather matter of fact way, that thousands of negative assessments had been appealed in Court and consequently thousands of verdicts had been issued at different levels of the judiciary system (Martos Martínez, 2009). The author did not see anything amiss in the fact that judges without any knowledge of the scientific field under assessment were placed in a position to decide on the validity of the decisions reached by specialised peer review panels. This was because judges would not enter discussions about the scientific value of the cases presented, but rather assess whether due process had been observed; yet, the definition of such due process necessarily requires assumptions on the validity of the criteria employed. For instance, Martos Martínez (2009) points out that the Sexenios evaluation process is “oriented” towards assessing «research activity exclusively through indications of quality, in other words, it is not the work that is analysed directly, but the indications of quality evidence that are provided» (my emphasis). When we comprehend the weight of judicial decisions on the final outcome of many evaluation processes we can understand why the evaluation needs to be based exclusively on the indications of quality, rather than the direct assessment of such quality, on which a court judge would not be able to pass a verdict. Yet, this system is at odds with CoARA objectives, specifically “Commitment 3” of the Agreement which explicitly counts «relying exclusively on author-based metrics (e.g. counting papers, patents, citations, grants, etc.) to assess quality and/or impact» as a case of «inappropriate use of indicators». Further, in practice, the «indications of quality» (indicios de calidad) have revolved mainly around the impact factors of the journals in which the work is published or the quality or prestige of the publishers in the case of books or book chapters. That research evaluation can be conducted without addressing the research results themselves but relying only on indicators represents a significant decision regarding evaluation practice, adopted in Spain not because of explicit considerations about evaluation methods, but determined by the practicalities imposed by a centralised and judicialized system. When individual and panel judgements based on qualitative considerations are equated with “subjectivity”, they emerge as a threat to the “legal certainty” that the legal system requires. Legal certainty can only be achieved by quantitative indicators, that are transparent and equally applied to all evaluands. Again, however, this generates a conflict with the Agreement insistence on considering the diversity of practices across scientific disciplines and the diversity of ways through which scientific contributions can be made. This is a difficult conflict to address and can lead to a reluctance to engage with the CoARA process. In the meeting of the National Forum for the Reform of Research Evaluation which took place in Barcelona, on 20th January 2023 the then Minister for Universities, Joan Subirats, stressed the need for reform and this was followed by a presentation by CoARA’s President, Rianne Letschert, that ended in a passionate call to engagement and commitment to the CoARA process. Yet after a morning of discussions and debates, in the closing panel, José Manuel Pingarrón, then Secretary General for Universities, emphasised the need for prudence lest the principle of “legal certainty” in evaluation practice be upset9. Coming at the end of a day of presentations about Spain’s commitment to CoARA and from such a high-ranked official, this note of caution was significant. Regardless of the perceived need for reform and the political commitment in favour of such reform, the governance structures of the Spanish science system were emerging as a considerable, if often understated, barrier to the reform process.
25The reform of ANECA Sexenios evaluation guidelines and criteria provide an illustration of how the difficulties would express themselves in practice. Through the second half of 2023 ANECA worked to develop a set of evaluation criteria that would be aligned with the CoARA commitment. The work resulted in three documents:
-
A Resolution published 5 December 2023 presenting the general evaluation criteria and guidelines10.
-
The “general scale” published on 29 December 202311.
-
- 12 https://www.aneca.es/web/guest/orientaciones-baremos-de-los-criterios-de-la-evaluaci%C3%B3n-de-sexe (...)
The “specific scales” for each one of the disciplinary fields developed by the evaluating committees in accordance with the general criteria and scales12.
26In the first place, the Resolution specifies the general criteria following the above-mentioned Royal Decree 678/2023 published on 18 July and being based on the previous legislation: its introductory section refers one Organic Law (the LOSU), one Law, four Royal Decrees, two Orders, one Statute and one Resolution. Section 1 presenting the general criteria cites again some of this legislation (one Organic Law, one Law, two Royal Decrees and one Order). It is not the objective of this paper to assess the extent to which previous legislation may support or hamper the reform measures, but it is clear that the legal framework is important in the Spanish research evaluation context and its reform.
27The Resolution states the will to incorporate, in «a flexible manner, the orientation of the international research evaluation reform movement, which ANECA joined in 2023 through its adhesion to the San Francisco Declaration on Research Assessment (DORA) and to the agreements and principles of the Coalition for Advancing Research Assessment (CoARA)». This is reflected in one of the innovations that the Resolution introduces: the importance given to the “narrative” that will be used to justify the “indications of quality” presented for each of the contributions that applicants will have to present. In its Section 1, Point 2 it states that the evaluation will be based on the narrative presented and it will apply «qualitative and quantitative evaluation criteria and methods». In its Point 7 we read that, for the case of publications, «their being indexed in a bibliographic selection or database can be used as a complementary argument in the narrative defence of the contribution, but cannot be used to obtain automatically a positive evaluation of the contribution, without taking into account the intrinsic quality of its content». Then, for each of the disciplines a brief statement of evaluation guidelines is presented. For all fields, except Field 0 (Interdisciplinary and multidisciplinary) we can read the same guideline as regards publications:
In the case of articles, preference will be given to contributions published in journals of proven quality or indexed in bibliographic databases of recognised prestige.
28Therefore, it matters where the contribution is published, but in itself does not automatically guarantee a positive evaluation. In what is arguably a departure from the assumption that evaluation will be conducted without direct reference to the content of the contributions but based solely on indications of their quality, the new approach states that the «intrinsic quality of its content» needs to be considered. A further example of the departure from previous practice is the absence of any reference to impact factors, quartiles or other journal impact indicators. The Resolution includes an Appendix detailing some acceptable metrics and sources, labelling the approach as «narrative bibliometrics». It presents different metrics at the article/contribution level but it also includes a reference to the publication outlet thus making it acceptable to include «quantitative bibliometric indicators that assess the impact of a journal or publisher within the scientific community». A list of 11 different types of possible indicators follows, including journal impact indicators (JIF, SJR, Citescore, etc.).
29The Resolution therefore endeavours to comply with the Agreement principles, although it does not rule out the use of impact factors. This, as we will see, is important. In the Resolution, IF is one more of many indicators that can be used as part of a “narrative bibliometrics argument”, but which does not automatically lead to recognition; besides it refers to only one of the expanded list of possible contributions that will be taken into account. Yet, the subsequent documents detailing the implementation of these principles will increase the prominence of the traditional journal impact indicators.
30The next step was the publication of the general “scales” (“baremos” in Spanish). The “scales” are a guide to how the panel assessments will be quantified. Assessment may be qualitative in nature but it is ultimately expressed in a number. The scales guide evaluators in the process of attributing numbers to their decisions. The “general scale” is presented as a one-page template specifying that “research impact” counts for 60% of the score for a specific contribution with all the other attributes (contribution to the area’s scientific knowledge, social impact, contribution to open science, and “preferential contribution”) counting 10% each. The scientific impact can be determined through five criteria including “use and readership” and citations received by the contribution and, significantly, the “scientific impact” of the publication outlet.
31Finally, the evaluation committees for each one of these areas operationalise this template. The specific scales are much more specific on the conditions that are required to fulfil a criterion and the points to be given for satisfying specific conditions. Here is where the journal impact indicator re-enters the scene with force. Criteria vary significantly across disciplines, but, for instance, Field 4 (biomedical sciences) considers citations received by the article submitted, but also includes quite a detailed description of the conditions that an article has to fulfil to be considered:
It will be taken into account if the contribution is published in a journal that appears in the JCR of WoS Clarivate using the Journal Impact Factor (JIF), with the criterion of being in the first 2 quartiles of the journal for its year of publication Exceptionally, a publication will be considered in Q3 position if the contribution has at least 10 citations, excluding self-citations, if its ‘normalised citation’ value is higher than 1 or its position in the percentile of similar citations is higher than 50% (consulting these parameters in SCOPUS, not in other metrics systems, for the same reasons argued for choosing JCR). Contributions with between 15 and 30 authors will have their value reduced by 20 % and those with more than 30 authors will have their value reduced by 40 %. Other scores obtained with platforms such as Scimago Journal Rank (SJR), IDR or others will be assessed with a 60% reduction in value. The decision to choose JCR as the main platform in this field is based on achieving the same objective criteria for all those evaluated, as there are significant variations between the different ways of gauging scientific impact between platforms. If the journal is listed in the JCR in several areas, the one most favourable to the applicant will be chosen.
32Therefore, in the journey from CoARA-informed objectives and generic recommendations to operationalisation of the criteria carried out by the evaluation panels, we move from stressing the importance of qualitative assessment and the explicit rejection of journal impact factors, to a set of calculations, that are equated with “objective criteria”, and which include impact factors in order to reach a quantitative valuation of the contribution’s impact. Further, decisions on what data sources to be used are justified in terms of «achieving the same objective criteria for all those evaluated».
33This is not to mean that the CoARA principles and recommendations have been completely overruled. One of the reasons why the scales proposed by the evaluation committees are complex is because they combine article-level, with outlet-level indicators; this combination being explicitly required in the general scales. Article level indicators are much more relevant in this approach than in previous sexenio evaluations. The degree to which Journal Impact Factors play a role in the assessment will also vary across fields. Yet, what we find is that, as we move towards implementation, the perceived need for a system of measurement that can be applied indistinctively to all contributions acquires force and is translated into procedures that divert from the CoARA spirit. The centralised nature of the process and the large number of applications that the system has to process, combines with the complex legal backdrop to result in the persistence of practices that CoARA seeks to overcome.
34The outcome of this first attempt at reform is yet to be seen. How the panels decisions will be received and the extent and scope of future appeals will become arguments to be used in the continuing debate on the merits and drawbacks of the reform process. In the meantime, ANECA persists with its reform programme and, in June 2024 published its CoARA Action Plan. This reaffirms once more the commitment to align the reform of the Spanish evaluation system with the CoARA principles. The plan identifies a series of actions to address a set of challenges including the “essential” need for «collective and aligned action of actors of the national and international science system» and its concomitant challenge to «establish effective coordination and collaboration among interest groups», the importance of «raising awareness among the research community and society in general about the reform that ANECA will develop», and the requirement to «dedicate the necessary resources to implement the plan». The plan includes 18 actions, grouped in 6 action lines aligned with the six Agreement «supporting commitments». It includes the assessment of evaluation models used in different contexts; an analysis of the current ANECA assessment system; the design and development of new assessment criteria for research units and organisations; training and awareness actions; participation in CoARA activities; and the monitoring and evaluation of the reform process including the implementation of new models in ANECA’s assessment commissions.
35As mentioned in the introductory section CoARA’s main goal is to «enable systemic reform» and «to facilitate exchanges of information and mutual learning between all those willing to improve research assessment practices». The Spanish case reaffirms the systemic nature of the reform required and shows how, in this case, reform needs to address also the legislative framework that shapes Spanish evaluation practice.
36Further, formal norms and rules are only part of the governance system. Although, formally the sexenio evaluation approach is based on a peer review system that should consider a variety of criteria, the number of evaluations that need to be conducted and the practices that have evolved in line with the principles of bureaucratic management, have generated a system that still revolves around the establishment of publication-based measurements. It is this persistent, straightforward application of measurements as the core tool of research evaluation that CoARA sees as the main reform target, but constitutes, as well, the central building block of the Spanish centralised research evaluation system and of its research governance structure.
37In Spain, the institutional setup in need of reform extends from the formal rules that legislation embodies to the implicit norms that constitute evaluation practice. The weight of publications and the relevance of their outlet (as shown by the Journal Impact Factor and similar indicators) continues to play a role in the sexenio evaluation system: it offers a relatively simple measure that can be presented as “objective” as it does not depend on the views of experts (although the specific benchmarks used in the “scales” do), and can be applied equally to large numbers of individuals, thus addressing the need to carry out numerous assessments with limited resources and provide some guarantee of “legal certainty”. It is therefore, not surprising, that, while the reform strategy attempts to drive journal impact indicators into the background, as we approach application they make a re-appearance in the scales applied by the disciplinary panels, and the may retain their centrality when the time comes for the panels to make their assessments. To address this situation the reform needs to deal with the more fundamental issues regarding the organization, functions and legal foundations of the overall research evaluation system.
38Therefore, pursuing CoARA principles calls for a systemic change in the research governance system. It is the profound nature of these changes that explains the difficulties discussed in this paper. Not all research governance systems will however face similar challenges. Decentralised systems in which each single research organisation can develop and modify its own evaluation processes have more flexibility and may become more diverse in the application of research evaluation approaches. Here, rather than a systemic change, what will often be required are new evaluation techniques and processes able to account systematically for the diversity of contributions to science that CoARA intends to value. It is here that CoARA is currently placing most of its efforts. Although the Agreement is clear in noting that the changes required are indeed of a systemic nature, the signatories are often pursuing activities focalised on specific piecemeal changes that are expected will contribute to achieve the desired objectives. Yet, the formal assessment procedures, rules and techniques that are constituting the centre of attention of many CoARA efforts are only part of the research governance system, which, in the Spanish case will need to change if the CoARA principles are to be implemented fully. Modifying the evaluation philosophy and principles and adapting the specific scales (baremos) that are used to assess individuals for the sexenios constitute steps forward, but to fully align Spanish practices with CoARA principles will require measures with more profound effects on the organisation, functions and everyday practices that characterise the Spanish evaluation system.