As part of my consultative work for Research Information Networks (RIN), I recently came across proposals for the new “Research Excellence Framework” (REF). This is the successor to the UK’s Research Assessment Exercise (RAE), and will be gradually introduced between 2010 and 2014 to assess funding of research throughout UK higher education institutions. The Higher Education Funding Council for England (HEFCE) recently launched a consultation process on the REF proposal, and it is to this document this short essay refers:

As with the RAE, an institutions evaluation in the proposed REF will determine how much money it receives from their Higher Education Funding Council (HEFC). Thus the REF, like is predecessor will have a major influence over hiring decisions in HE institutions, and shape the UK science base for the next few decades. For this reason it is important to all UK scientists.

A central tenant of REF is that quantitative data (specifically bibliometric indicators) will be used to assess research quality. This will initially apply to the life and physical sciences, though other subjects may follow. On the face of it the increased use of bibliometrics is a positive development. Bibliometrics can be scaled across the published output of all universities and colleges. These measures apply similar criteria to disciplines regardless of the subject, and can be more readily understood than more qualitative measures. Importantly, they can also be calculated automatically from existing databases, notably Thomson’s Web of Science database in the case of the REF proposals. This minimizes the administrative burden of the evaluation process, which was a major problem with REF’s predecessor. Unfortunately, these benefits come at a price.

Firstly, bibliometric indicators do not measure quality. Rather, they measure impact. Any experienced scientist reading or writing for Nature or Science, two of the world’s most highly cited scientific journals, will tell you this. Even Nature and Science’s publishers admit to it. Marketing, publicity (good or bad), and accessibility heavily influence citation rates but in themselves these have nothing to do with research quality. Anybody that thinks the scientific process is above all this is seriously fooling themselves. In some cases impact may be a proxy for quality but this is not always the case. A reasonable analogy would be to compare bibliometric indicators to TV ratings. A high rating simply means you have reached a large audience, but as anyone (in the UK) that watched TV over the recent Christmas period can testify, popularity does not always equate to quality.

The second major failing of bibliometric indicators is that they only measure conventional published output. More specifically in the case of REF they will measure citations of articles indexed in the Web of Science (WoS) database. Notwithstanding deficiencies in WoS (see below), this means that content published elsewhere, or tools, services and applications generated by the UK scientific community, will be completely invisible to REF regardless of their immediate or potential value. Service orientated or data rich sciences involved in information provision will be heavily penalized by the REF, eliminating any incentive for individuals or institutions to generate and sustain them. This is because these resources are typically not bibliographically cited in a manner proportionate to the value they generate.

What are the implications of REF for the future of science funding in the UK? This will vary according to different scientific disciplines, but in some cases the affect of the REF as it is presently proposed, could be quite dramatic. A good case study would be the effect of the Research Assessment Exercise on taxonomic and systematic research in the UK. Arguably the RAE since 1992 has helped kill off this discipline in UK universities, as documented by no less than three House of Lords reports since 1992 (a fourth House of Lords enquiry into taxonomy is currently underway). This is because the output of taxonomic and systematic research has such a low short to medium term impact, that any UK HE institution would have to be mad to hire a taxonomist under their present funding framework. This situation will only be exacerbated by the REF proposals. The typical half-life of research and data generated through taxonomic studies is measured in decades. Rather like a famous painter, the impact of a typical taxonomist may only be appreciated under the REF proposals once the taxonomist is dead! This is because taxonomic research communities are so small, that a taxonomist might require a lifetime’s work to generate enough citations amongst their immediate peers before they become visible through their citations. The discipline of taxonomy is not alone with this problem. Many of the classical descriptive sciences, the so called –ologies and –omies like anatomy, archaeology, astronomy, embryology, palaeontology, stratigraphy etc, are similarly imperiled by REF, as are more contemporary disciplines like genomics and scientific software development. The value of these subjects is measured by the use of the products they generate (data, software etc). However these rarely attract bibliographic citations in the way that traditional research is cited.

To my mind a framework for assessing the value of research should not exclusively focus on either quality or impact as an indicator of their success. Rather we should be measuring outcomes of research, since this is ultimately what the UK science funding is trying to influence. By outcomes I mean the overall difference (positive or negative) that research has made. The problem is that outcomes are complex. Quantitative metrics can help; indeed they are probably essential to measure the diverse outcomes of the UK science base. However, over reliance on a single metric not only imperils those disciplines have not traditionally be measured in this way, but also creates behavioral incentives to influence this metric, where none existed before. In short, over reliance on a single metric is simply an excuse for not thinking.

In summary, here is a less synthesized list of problems with the REF proposals as they currently stand:

• The REF rests on the assumption that the influence of research output in any discipline is proportionate to the number of citations it receives. This assumption is deeply flawed, especially in data rich and service driven sciences which do not accrue traditional bibliographic citations of usage proportional to the value of the output they generate, or in fields that use alternative methods of citation (e.g. taxon names and authority combinations in taxonomy, or Genbank references in molecular biology). Researchers providing tools and services (especially web based services like myself) will be particularly affected.

• REF proposes to normalize citation rates between subject disciplines by classifying researchers and their published output into the 172 subject disciplines defined by Thompson’s Web of Science database. However, many scientists and their work defy ready classification. For example, my published work (circa 25+ papers) spans at least eight of the WoS categories, and these often appear in interdisciplinary journals. As presently described, REF penalizes me for being interdisciplinary, yet most institutions and funding agencies encourage interdisciplinarity!

• Many researchers specialize in sub-disciplines that are very small and it may take these researchers many years (sometimes decades) to accrue significant bibliographic recognition of their work within their broader subject discipline. Specialist scientists in these fields, especially younger scientists with a smaller publication record will be heavily penalized under the REF proposals because their citation rates in these sub-disciplines cannot be adequately normalized. This means HE institutions will have little incentive to hire researchers in these areas, especially those new to a field, or in new subject areas that are beginning to develop.

• The REF conflates quality and impact. The proposal states that the REF “does not entail the grading or ranking or journal titles… or researchers…, we do not consider such approaches helpful for our purposes” (para. 32). However, the document goes on to say “We are commissioning technical work to consider where the boundaries between [the] quality levels might best be set… In particular, we envisage setting a threshold level below which outputs with comparatively few citations will be in the same category as work that has not been cited at all.” (para. 46). Surprisingly the REF consultation document does not even define “research quality”.

• The REF consultation document skims over the behavioral impact of bibliometric driven methods of quality assessment, such as the formation of citation clubs that seek to manipulate article citations over time. Yet the document gives no rational for dismissing this problem. I would argue that at present there is no (or relatively little) incentive for researchers to engage in this behavior. However, implementation of the REF makes citations a valuable commodity. It seems almost inevitable that in time, scientists will start to engage in trading citations, especially if research assessment frameworks like the REF start to be adopted in other countries in addition to the UK.

• The REF will be completely reliant upon data derived from Thompson’s Web of Science. This poses two major problems. It assumes WoS is infallible and complete, neither of which is true (see below). It also places a resource that will be driving the direction of UK science for many decades, in the hands of a single commercial publisher. This dependency is not only a major risk for REF, but puts a great deal of power in the hands Thompson Publishing Group.

• The WoS database includes a minute proportion of the journals (and therefore the articles) actually published; review articles are cited more frequently than original research, thus favoring institutions and individuals that include a disproportionate number of review articles are part of their published output; errors are common in reference lists (occurring in up to a quarter of references). These discrepancies will all affect the value of the REF as it is currently described.

Disclaimer. I work for an organization (The Natural History Museum, London), which is not subject to the RAE and (to the best of my knowledge) is unlikely to be subject to the REF. The NHM has numerous methods of assessing the output and value of the work we generate. Arguably, these suffer from many of the problems common to the RAE and REF as described above. However, I am pleased to say that at least the NHM’s methods are more holistic that those proposed under the REF. If they were not, the NHM probably would not exist.