Before making its recommendations for changes to the next REF, the Stern Review looked at REF 2014 and highlighted some of the problems and issues that arose from it. They are summarised below, but click here if you would like to read the full report.
REF 2014 was an expensive business costing an estimated £246 million – 133% more that RAE 2008 and about 2.5% of the funds distributed. The majority of this cost (£212 million) was borne by the HEIs. The rest was divided between the four UK funding bodies and the cost of the panellists.
There was potential for ‘gaming’ i.e. playing the system in REF 2014. The following practices were highlighted:
- Institutions hiring staff to enhance their REF return as the census date approached
- recruiting overseas staff on fractional contracts who were submitted to REF, but contributed very little to the institution.
In some areas, this led to:
- a highly selective submission to REF that was not a true representation of the overall research activity.
- Exclusion of other good research staff who did not fit the HEI selection strategy
- Salary inflation for those the HEI’s wished to recruit/retain, but not for others.
In order to be submitted to REF 2014, full time staff were expected to return 4 high quality outputs or their HEI had to claim mitigating circumstances for those individuals. This was expensive for the HEIs and stressful for the individuals.
An analysis of staff selected for REF 2014 showed that 67% were men and 51% women. The selection rate for ethnic minorities and disabilities were lower.
- Peer Review
Feedback from REF 2014 confirms the importance of peer review and that metrics cannot capture all dimensions of an output’s quality. However, panels need to have a broad range of expertise and time to look at each output in detail. This inevitably is expensive.
- Effects on Research
A combination of wanting to be included in the REF and pressure from within the Institution may have influenced researchers to favour safe projects, delivering short term success and avoiding risky or multidisciplinary ones. Changing this is a priority for the next REF.
- Interdisciplinarity and Collaboration
Following on from point 5 above, the number of interdisciplinary and collaborative projects submitted to REF 2014 was not representative of the work being done.
- Effects on Careers
Again, following on from the last two points, the need to produce a set number of outputs may have influenced researchers to focus on safer, short term projects and prevented some good research from being included in the REF. It may also have discouraged collaboration within departments as the research quality was closely tied with the individual.
- Capturing the Research Environment
An important element of the REF assessment is to recognise HEIs that develop the vitality and sustainability of their research environment. However, it was felt that the environment statement should be simplified and that metrics could be used for this assessment. This was not so relevant to UCA in REF 2014 as we only submitted to one Unit of Assessment, but we may submit to more in 2021.
In REF 2014, impact case studies were linked to the numbers of individuals submitted to each Unit of Assessment and to key research outputs. Several issues arose from this including:
- Impact on teaching not recognised despite research and teaching being very closely linked
- Impact on industry, public engagement, policy advice etc was not always captured
- It disincentivised universities from recruiting from business and other sectors
- Impact that crossed units of assessment was missed
What should the length of time be between assessments? There is some support to increase it so that the cost is spread over a longer period. However, the counter argument to that is that the accuracy of the exercise would be compromised and that dynamic changes to the research system not be recognised.