Annex A: Questions for consultation
Question 1: Do you have any comments on the proposal to maintain an overall continuity of approach with REF 2014, as outlined in paragraphs 10 and 23?
Unit of Assessment structure
Question 2: What comments do you have about the Unit of Assessment structure in REF 2021?
Question 3a: Do you agree that the submissions guidance and panel criteria should be developed simultaneously?
Question 3b: Do you support the later appointment of sub-panel members, near to the start of the assessment year?
Question 4: Do you agree with the proposed measures outlined at paragraph 35 for improving representativeness on the panels?
Question 5a: Based on the options described at paragraphs 36 to 38, what approach do you think should be taken to nominating panel members?
Question 5b: Do you agree with the proposal to require nominating bodies to provide equality and diversity information?
Question 6: Please comment on any additions or amendments to the list of nominating bodies.
Question 7: Do you have any comments on the proposal to use HESA cost centres to map research-active staff to UOAs and are there any alternative approaches that should be considered?
Question 8: What comments do you have on the proposed definition of ‘research-active’ staff?
Question 9: With regard to the issues raised in relation to decoupling staff and outputs, what comments do you have on:
- The proposal to require an average of two outputs per full-time equivalent staff returned?
- The maximum number of outputs for each staff member?
- Setting a minimum requirement of one for each staff member?
Question 10: What are your comments on the issues described in relation to portability of outputs, specifically:
- Is acceptance for publication a suitable marker to identify outputs that an institution can submit and how would this apply across different output types?
- What challenges would your institution face in verifying the eligibility of outputs?
- Would non-portability have a negative impact on certain groups and how might this be mitigated?
- What comments do you have on sharing outputs proportionally across institutions?
Question 11: Do you support the introduction of a mandatory requirement for the Open Researcher and Contributor ID to be used as the staff identifier, in the event that information about individual staff members continues to be collected in REF 2021?
Question 12: What comments do you have on the proposal to remove Category C as a category of eligible staff?
Question 13: What comments do you have on the definition of research assistants?
Question 14: What comments do you have on the proposal for staff on fractional contracts and is a minimum of 0.2 FTE appropriate?
Question 15: What are your comments on better supporting collaboration between academia and organisations beyond higher education in REF 2021?
Question 16: Do you agree with the proposal to allow the submission of a reserve output in cases where the publication of the preferred output will postdate the submission deadline?
Question 17: What are your comments on the assessment of interdisciplinary research in REF 2021?
Question 18: Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline? If you agree, have you any suggestions for data that could be provided to the panels at output and aggregate level?
Question 19: Do you agree with the proposal to maintain consistency where possible with the REF 2014 impact assessment process?
Question 20: What comments do you have on the recommendation to broaden and deepen the definition of impact?
Question 21: Do you agree with the proposal for the funding bodies and Research Councils UK to align their definitions of academic and wider impact? If yes, what comments do you have on the proposed definitions?
Question 22: What comments do you have on the criteria of reach and significance?
Question 23: What do you think about having further guidance for public engagement impacts and what do you think would be helpful?
Question 24: Do you agree with the proposal that impacts should remain eligible for submission by the institution or institutions in which the associated research has been conducted?
Question 25: Do you agree that the approach to supporting and enabling impact should be captured as an explicit section of the environment element of the assessment?
Question 26: What comments do you have on the suggested approaches to determining the required number of case studies? Are there alternative approaches that merit consideration?
Question 27: Do you agree with the proposal to include mandatory fields (paragraph 96) in the impact case study template to support the assessment and audit process better?
Question 28: What comments do you have on the inclusion of further optional fields in the impact case study template (paragraph 97)?
Question 29: What comments do you have on the inclusion of examples of impact arising from research activity and bodies of work as well as from specific research outputs?
Question 30: Do you agree with the proposed timeframe for the underpinning research activity (1 January 2000 to 31 December 2020)?
Question 31: What are your views on the suggestion that the threshold criterion for underpinning research, research activity or a body of work should be based on standards of rigour? Do you have suggestions for how rigour could be assessed?
Question 32: Evaluation of REF 2014 found that provision of impact evidence was challenging for HEIs and panels. Do you have any comments on the following:
- The suggestion to provide audit evidence to the panels?
- The development of guidelines for the use and standard of quantitative data as evidence for impact?
- Do you have any other comments on evidencing impacts in REF 2021?
Question 33: What are your views on the issues and rules around submitting examples of impact in REF 2021 that were returned in REF 2014?
Question 34a: Do you agree with the proposal to improve the structure of the environment template and introduce more quantitative data into this aspect of the assessment?
Question 34b: Do you have suggestions of data already held by institutions that would provide panels with a valuable insight into the research environment?
Question 35: Do you have any comment on the ways in which the environment element can give more recognition to universities’ collaboration beyond higher education?
Question 36: Do you agree with the proposals for awarding additional credit to units for open access?
Question 37: What comments do you have on ways to incentivise units to share and manage their research data more effectively?
Question 38: What are your views on the introduction of institutional-level assessment of impact and environment?
Question 39: Do you have any comments on the factors that should be considered when piloting an institutional-level assessment?
Outcomes and weighting
Question 40: What comments do you have on the proposed approach to creating the overall quality profile for each submission?
Question 41: Given the proposal that the weighting for outputs remain at 65 per cent, do you agree that the overall weighting for impact should remain at 20 per cent?
Question 42: Do you agree with the proposed split of the weightings between the institutional and submission-level elements of impact and environment?
Proposed timetable for REF 2021
Question 43: What comments do you have on the proposed timetable for REF 2021?
Question 44: Are there proposals not referred to above, or captured in your response so far, that you feel should be considered? If so, what are they and what is the rationale
Q2. Maintain current UoA structure and add sub-profiles where necessary and appropriate.
Q.3a – No – one of the benefits of REF UoA structure is that the panels decide submission guidance and criteria. This has been particularly important in ensuring that research assessment has been appropriately inclusive of the creative and performing arts disciplines and media.
Q.3b – No – there is a danger that the panel would not feel the appropriate ownership of the process that is in fact their responsibility. Paragraph 67 of this consultation argues for the importance of panels in drafting criteria in relation to the submission of outputs and we would support this but emphasize that these must be UoA panels.
Q4 a – Yes
Q5 a – we would recommend widening representation to sector organisations such as mission groups but would caution against individual self-nomination. It is imperative that panel members can demonstrate vested interest and understanding of the processes of research assessment, and it is this that makes it peer review. Self-nomination, whilst possibly evidencing a range of skills and achievements that might merit appointment, would not necessarily guarantee the kind of strategic sector leadership and understanding that we expect of peer review panels. If people are good and appointable they will be sufficiently embedded within their academic communities to be able to garner nomination.
Q5 b – yes.
Q 6. – additional
Q. 7 The HESA staff return is used for a variety of purposes and is in itself potentially open to a degree of gaming. Selection into UoAs is a fundamental aspect of the research assessment that stems from the academic infrastructures of research and academic enquiry. To be placed in a UoA simply according to a HESA staff Return feels utterly instrumental and would, in our view, be a very retrograde step for REF. In addition, this would have highly detrimental implications for those institutions involved in portfolio development and expansion where staff dedicated to new areas of provision, often cognate to existing fields and provision but which falls into new JACs codes / HESA cost centres could be disadvantaged. This would unfairly misrepresent the organic, often research-led nature of academic provision and enquiry.
Q.8 Paragraph 40 makes the case for the status quo of selecting staff on the basis of quality. We believe that the arguments made in Paragraph 41 about the cost burden of selecting staff are largely specious. All institutions make strategic decisions about the level of investment that they make into any process or activity, and we have no doubt that the inclusion of all staff submission will not reduce it in anyway. In fact, in our view, an all staff submission, made on the basis of a reduced or potentially ‘uncoupled’ level of outputs per individual researcher will heighten the need for investment in REF as the supposed burden with selection of researchers will become a much greater burden of selection of outputs. Fundamentally, this is resource that will come at the expense of actually supporting research and will have significantly differential impact on institutions, which is perhaps its purpose.
We have to recognise that the purpose of REF is not simply about the allocation of QR funds but also to provide a moment in the academic cycle when academic communities and organisations take thoroughgoing stock of research achievement. The process of selection through achievement is an essential motivating factor for both individual researchers and their institutions. If concerns have in past been raised in relation to institutional processes of fairness and equality in relation to selection, the prospect of determining ‘research-active’ within the relatively unbounded terms of the consultation would prove incredibly difficult, and would in all probability lead to the wholesale redesignation of staff as teaching only, thus enacting the very separation of teaching and research that the Stern Review stated to be undesirable. If REF2021 demands the inclusion of all staff, it must take the responsibility of providing unequivocal clarity of definition or mechanism for institutional self-definition of what constitutes ‘research active’. Hiding beneath the HESA staff collection will be neither appropriate nor indeed useful.
Q9 a. We would whole-heartedly support the decoupling of outputs from researchers as proposed in paragraph 47 of the consultation but would do so on the basis of selective rather than all staff submission. On this basis we feel an average of 2 outputs per FTE member of staff would seem appropriate. It must be recognised, however, that it is in the combination of all staff submission and decoupled submission of outputs that REF2021 will have its most differential impact in favour of research-intensive institutions. It might be anticipated that it was on the basis of number of outputs rather than the quality of outputs that staff were not submitted to REF2014 by research-intensive HEIs. This measure will have addressed that problem for them.
Q.9 b. On the basis that this is about recognising research excellence, we would support a maximum of 6 outputs per researcher.
Q.9 c. 1 output per researcher seems appropriate.
Q. 10 a. No. As a specialist arts institution we have only a limited number of ‘published’ outputs and as such, publication date would not be appropriate. The notion of ‘acceptance to publish’ relates most precisely to the final decision of an academic journal editorial board. Even with academic publishing, the decision to publish that is made on the basis of a book proposal is not the same as the actual decision to publish the final manuscript. The often significant period of time between these two points would prove extremely difficult for individual staff, institutions and publishers. We do not see any reason why if the principal of non-portability of outputs is taken up by REF2021 that the eligibility of an output should not be determined as previously according to when it entered the public domain.
Q 10 b in a unit such as UoA34 (the second largest unit of REF2014), date of acceptance of publication would prove fairly meaningless. We have colleagues that are scheduled for exhibitions or working on long term project that will be ultimately completed until 2019 and 2020. They have been commissioned and as such there is an ‘acceptance to publish’ of sorts, but their work, and the work of their collaborators does not exist yet and might not exist for a significant period of time to come. We do not know how we would ‘date’ this work and would be extremely concerned about the very difficult positions colleagues would find themselves in.
Q10 d. We would support the sharing of outputs as the least worst outcome of this proposal. Unless it worked straightforwardly on the basis 50/50 FTE as applied to part-time staff submitted by two different HEIs in REF2014, it would not negate the need to determine ‘date of acceptance to publish’. Would a former institution be in a position to lay claim to a research output published after the departure of a member of staff on the basis that the research was undertaken substantially with their support?
Q.11. On the basis that institutions should not be compelled to subscribe to a third party system in order to make a return to HEFCE, we do not support the use of ORCID.
Q12 – None. We didn’t use it.
Q.13 – Every effort should be made to enable the inclusion of research assistants and early career researchers as these are often integral to the research infrastructure of institutions and submitting units. Whilst the recommendation to reduce the quantity of outputs required per individual would support this, the non-portability of outputs could impede it.
Q.14 – We strongly support the continued eligibility of 0.2 FTE members of staff. We do not however support the further proposal, which we feel introduces a degree of potentially subjective judgement that we would find unwelcome. It seems to place an onus upon institutions and individual researchers to demonstrate an institutional engagement and mission that we would not necessarily expect to demonstrate for any other member of staff. The phenomenon of the full-time lone professor with only limited institutional impact beyond reputation and REF is not unheard of. Would s/he be ineligible too?
Q.15 On the basis of the REF2014 we feel that there is already ample opportunity to demonstrate collaborative and partnership research. We do not therefore see the need for any further mechanism.
Q16 – Yes
Q.17 – We don’t really feel this is a priority and are hesitant to support any further mandatory measures. The identification of ‘interdisciplinary champions’ feels slightly problematic, indeed the proposal that they meet within their main panels suggests an understanding that interdisciplinary research might only exist within the bounds of a main panel, which it clearly doesn’t. Again, this feels like an unnecessary complication of what was felt by most institutions and units to be perfectly fine. Where units do not feel that they embraced or understood interdisciplinarity sufficiently, we would suggest that they have the opportunity to address this within their own terms and criteria.
Q.18 – We agree with the proposal in the confidence that our disciplinary academic communities and peers on the assessment panels will roundly reject the use of metrics as they did in the previous consultation. Once again, we would point to the importance of the appointed UoA panel members being integral to the decision-making in this regard.
Q. 19 – Yes.
Q.20 . Broadly speaking we felt confident that we were able to demonstrate and describe a range of impacts in our submission to REF2014. Therefore we do not immediately feel the need to broaden or deepen any definition. We recognise, however, that other subject areas may well have had difficulty interpreting and evidencing ‘reach’, perhaps particularly in regard to public engagement. We would suggest, however, that this could be dealt with at the level UoA or even Main Panel.
Q 21 – The alignment of definitions would be welcome. However, REF2021 would need to be very clear in its guidance about how it valued ‘academic’ and ‘wider’ impact. In REF2014, Impact was viewed largely through the lens of ‘wider’ impact and to be measured beyond the academy. This would need to be clarified by guidance.
Q.22 – Reach and Significance are fine but they only really tell part of the story of impact. In REF2014 successful case studies demonstrated the ‘effect’ of research often as a measure of reach and significance although this was not explicit within the criteria or template. The definition of ‘wider’ impact as ‘an effect on, change to or benefit to’ as proposed in this consultation seems a better description of what UoA’s ultimately recognised and valued.
Q.23 – We would recommend that this would be for Main Panels and UoA’s to determine in relation to their own modes of working. Panel D, for example, might feel the need for further guidance less than other panels?
Q.24 – Yes
Q.25 – Yes, providing that this is at Unit rather than institutional level
Q.26 – We are broadly supportive of this approach and have no concerns regarding the identification of assessment to any particular case study.
Q.27 – Yes
Q.28 – We have no issues with this proposal
Q.29 – We agree with the proposal to encompass research activities and bodies of work.
Q.30 – Yes
Q.31 – Whilst we agree with the analysis, the rigour would best be confirmed by evidence of the actual research output – which would challenge the proposal to broaden the research base from which impact can be measured.
Q 32a – We agree with the proposal to provide evidence to support the case studies
32b – We have reservations about this for the reasons outlined in the consultation document, that it might proscribe particular kinds of evidence. We remain confident that panel members should be able to interpret and judge appropriately the evidence put before them, and that it is the responsibility of individual submissions to endure this is the case.
32c – No but we hope that this isn’t made too prescriptive
Q.33 – We feel that it should be in the spirit of REF to allow the resubmission of case study on the basis that they can demonstrate additional impact beyond that claimed in REF2014. The template could easily include a section where the case for additionality might be made, even in the case of on-going rather than entirely new impact. In light of this, we don’t feel that there is a need for further evidencing beyond what is already required, and have reservations about placing limitations on the amount of impact that might be resubmitted. Impact is either evident or not.
Q34 a – We agree with this proposal only where quantitative data is used to clarify understanding of the submission not the actual assessment of it.
Q 34 b – Greater use could be made of Equality and Diversity data.
Q 35 – In our experience, the narrative aspect of the submission was absolutely essential to making a submission that gave an accurate picture of the vitality and sustainability of our work and activities, collaborative or not. Therefore we remain confident that the existing template affords institutions the opportunity to fully detail any collaborative work they might be doing.
Q 36 – Whilst we are in full sympathy with the sentiment behind the proposal to reward open access, we cannot support it given that the Open Access agenda and regulation that you are pursuing have still not taken satisfactory account of material research outcomes that cannot simply be digitised and made available online. To reward particular kinds of research outputs that are able to fulfil HEFCE’s ambitions according to the strict limitations of its own making would be to further perpetuate a perception that policy was skewed in favour of certain disciplines. Until guidance an Open Access for non text-based outputs is provided, the proposed rewarding of open access seems unfair.
Q.38 we are broadly supportive of an institution-level submission for environment but would be deeply concerned if this limited the role of the UoA submission to any great expense.
Q: 39 we would recommend, once again, that this isn’t made too complicated. In our experience, REF2014 provided ample opportunity to note the paragraphs 122 & 124. We do have concerns about the extent to which forward planning and strategy becomes a measure of assessment. The REF should be concerned with allocated Quality Related funding. In our view, to consider future plans and strategies as part of REF begins to blur the distinction between the dual funding streams, and might further benefit larger and research-intensive institutions which due to their scale and resulting research infrastructure are already favoured by competitive funding
Q: 40 In general we are supportive
Q: 41 Yes
Q: 42 if we are to have institutional environment and impact elements, this seems appropriate.
Q: 43 we are supportive
Q: 44 No