by Sarah Franklin
Introduction
In the next proposed quality assessment of research in higher education (if it takes place) it is planned for the first time to include recognition for ‘impact’. The purpose of this workshop was to explore the potential impact of impact assessment, to critically address the meaning, history, and implications of the turn to impact, and to consider the kinds of evidence that might be used to measure impact. In sum, the aim was to explore how an emphasis on impact is likely to influence research efforts? What can past impact measurement strategies tell us about their future as a knowledge quality assessment tool? What are impact culture’s kinships with audit culture, enterprise culture, ‘evidence led’ policy, and metrication? In the light of already severe funding cuts to higher education, it is certain that the contest for funds allocated through the next REF will be exceptionally competitive. Furthermore, it is likely the impact criterion will be influential whether or not it is formally part of REF. If REF is cut, impact may, ironically, become a more important, if even less clearly specified, criterion. Thus there are strategic as well as intellectual concerns about the various implications of ‘impact’.
Background: the HEFCE Impact Criterion
The inclusion of impact in the HEFCE outline for the next research evaluation exercise is defined in the following way: ‘significant additional recognition will be given where high quality research has contributed to the economy, society, public policy, culture, the environment, international development or quality of life’. HEFCE has designed a pilot impact measurement exercise that is currently in progress, and will report in autumn 2010. (Further details are available at http://www.hefce.ac.uk/ref ). The HEFCE consultation exercise about its next evaluation framework identified ‘widespread support in principle for including an element for the explicit assessment of impact within the REF from higher education institutions, mission groups, the national academies, research user representatives and other funders of research, and from a clear majority of academic subject associations’. In their letter of 25 March 2010 they added that: ‘Many [respondents] qualified their support [for impact evaluation] by emphasising the need to develop a robust method for assessing ‘impact’ and acknowledged that ‘a significant minority of responses objected to our proposals for the assessment of impact’. The proposed weighting for this element within the overall quality assessment is 25 per cent, and HEFCE is currently devising means to assess impact.
To the extent impact recognition is desirable, its potential benefits are thus offset by a number of difficulties, not least defining what counts as impact and determining how it might be measured. These challenges are complicated by the findings of empirical research, such as that recently conducted by BIOS on scientists’ perceptions of impact measurement, demonstrating that there is a perceived conflict between formal measurement of impact and its ‘genuineness’. In other words, bureaucratising impact is perceived by some academic researchers as potentially compromising the very impact activities they are intended to recognise, such as public outreach and engagement activities, which might, paradoxically, be diminished in quality if they are undertaken ‘for impact points’.
HEFCE has set out a long list of ‘key principles and parameters’ that are intended to address the problem of measuring impact.
a. Our aim is to identify and reward the impact that excellent research carried out within UK higher education is already achieving, and to encourage the sector to build on this to achieve the full potential impact across a broad range of research activity in the future.
b. We embrace a wide definition of impact, including benefits to the economy, society, culture, public policy and services, health, the environment, international development and quality of life.
c. We aim to assess historical impacts, not attempt to predict future impacts. The REF will assess impacts that have already occurred, underpinned by excellent research over a sufficiently long timeframe to allow the impacts to become evident.
d. The REF will assess impact in terms of complete submissions covering a body of activity, not at the level of the individual researcher. The assessment will focus on selected case studies of impacts that have arisen from across a broad portfolio of research activity, and will not expect each submitted researcher to demonstrate the impact of their work.
e. Our proposals aim to recognise and reward impacts arising from excellent research of all kinds. Given the evidence provided to us about impacts that have arisen from curiosity driven research, rather than discourage such research the REF will provide full recognition where curiosity-driven research has benefited the economy or society, in addition to rewarding excellent research of all kinds through the assessment of output quality.
2. Key elements in our assessment approach will therefore include:
a. Enabling excellent research in all disciplines to demonstrate impacts according to criteria and indicators that are appropriate to research in each discipline-group, to be applied within a common assessment framework. A key aim of the pilot exercise is to work with experts in a range of disciplines to develop these, and the REF panels will develop them further in consultation with their communities.
b. Producing outcomes of the assessment in the form of an ‘impact subprofile’ for each submission; these will be the product of expert panels’ judgements based on an appropriate mix of qualitative and quantitative evidence.
c. Recognition that multiple units can contribute to an impact, whether through collaborative or cross-disciplinary research, research in one discipline that informed work in another leading to impact, or several units’ independent contributions to building up an influential body of knowledge.
d. A substantive input into the assessment of impact by representatives of the users, beneficiaries and wider audiences of research, working in partnership with academics to develop the criteria and undertake the assessment.
The HEFCE Impact Pilot Steering Committee members are: Simon Denegri, AMRC, Janet Finch, Keele University, Russell Hamilton, Department of Health, Alison Hodge, QinetiQ, Roger Kain, University of Exeter, Ian Leslie, University of Cambridge, Ashley Malster, Department for Business, Innovation and Skills, John Rea, DEFRA, John Stageman, Astrazenaca, Jeremy Watson, ARUP, Marie Williams, RCUK. The list of 29 participating institutions in the impact evaluation exercise is included as Appendix A to this report.
Workshop Presentations
The Impact of Impact workshop was divided into two parts. Before the tea break, our three speakers, Donald Gillies of UCL, Valerie Hey from Sussex, and Mike Power from the LSE each presented 15 minute papers. They were each followed by a 5-minute response from an LSE discussant (Mary Evans, Fran Tonkiss and Don Slater respectively). This left approximately ten minutes for discussion after each paper for 2 or 3 questions from the audience. After the tea break we had open discussion for an hour followed by brief closing comments from the Chairs (Clare Hemmings and Sarah Franklin).
Donald Gillies began his talk by asking what a research assessment system was for? And what could be wrong with the principle of such a system – to determine which research was worth supporting? The main challenges such systems face, he argued, are well known: that research can only be evaluated retrospectively, and that by definition, some of the very best research, because it is counter-intuitive, will only have influence much later. The influence – or impact – of research that is susceptible to ‘delayed recognition’ is also proportionately greater, for the reasons Thomas Kuhn (1962) spells out so clearly in his work on paradigms: some of the most important discoveries are the most ‘delayed’ because they are the most controversial.
Gillies emphasised that, accordingly, some of the most path-breaking research in any given generation will by necessity be denigrated by impact measures. He showed how, if applied to the essential discovery of the role of the Human Papilloma virus in the epidemiology of cervical cancer by a marginal group of researchers, ‘impact assessment’ would have substantially delayed the introduction of a highly successful, and profitable, means of preventing a common and life-threatening disease.
In her response, Mary Evans pointed out that although a precise parallel to such a case might be harder to find in social science, many of the points about how it is assumed impact can be measured exist within social science are similar – such as the notion of the ‘big idea’ from an individual researcher or theorist, when in reality much academic exchange and influence is highly networked, relational, and collaborative. She asked us to consider how ideas and recognition are related, and asked if the percussive analogy of ‘impact’ could capture the mutuality that characterises the intellectual exchange binding together specialist academic communities?
How, Evans asked, does the impact model encourage us to understand the relationship of ideas to understanding? Are some ideas very ‘impactful’ because they are telling us what we already know (an implication of Gillies’ argument)? She used the example of moral panic to illustrate how an idea could be highly influential because it was antiintellectual – pandering to fearfulness. Is impact always positive? Finally, she noted that for the same reasons, we are mistaken to believe that the force of an influential idea necessarily comes from the idea itself: the receipt of ideas can be as ‘active’ a force driving their take-up as their production.
Valerie Hey focussed her remarks more on the culture of research evaluation than its own stated aims. She described this as 'disheartening and disaffecting' and sought to examine alternatives to impact as means of recognising quality and value in the academic community. How can key workers in the HE sector generate their own impact language? How can we use our own resources to understand and improve the conditions of academic labour? How can we ‘regenerate our own intentionality’ as opposed to constantly responding to ‘other peoples’ intentions for us’? She argued that the language of impact suggests we need to account for ourselves because we are underperforming. Citing Sara Ahmed’s (e.g. 2004) work on affect, Hey argued impact represents an 'intensification of audit' experienced as a prescriptive realignment with aims that are largely external to the HE sector. She described audit’s affects as puritanical, endemic, and ‘beyond irony’: ‘we produce paper mountains that we never have time to actually read’. She advocated an effort to reclaim our own commitments, inspiration and liveliness from the ‘cadaverous’ imaginaries of impact and impact culture.
Fran Tonkiss responded with agreement to Hey’s description of ‘the uglification of language’ that accompanies research evaluation. Citing the work of Margaret Wetherell (2008), she drew attention to the performative dimension of ‘impact’ – asking what it orients us toward, and what kind of labour it makes us do. Challenging the what Val Hey calls ‘corralling of the imagination in the service of the mundane’, Tonkiss pointed to the reversed priorities of impact whereby we would produce more mundane documents yet overlook some of the most ‘impactful’ labour we do – most obviously through the ordinary and mundane work of teaching. Giddens, for example, is one of the most cited sociologists, but in no small part this is because of his many textbooks for students – work that is often seen as insignificant, and not counted as impact. A final irony, she noted, was that researchers with high impact are often regarded with some suspicion by their peers. In this sense, external impact is at odds with internal measures of quality.
Mike Power satirised the ‘vulgar narcissism’ of impact culture by discussing the new imperative to minute and record our impact. He described this as both an administrative and cognitive impact of impact culture, for example keeping personal impact files, or impact folders in Outlook. He asked the audience to think instead about connections: how is our work connected to other domains of practice? It turns out these connections are very complex, and possibly not very well understood, even by those of us involved in HE and academic research. He said we should be sceptical of the idea of academic autonomy – as we are not necessarily as autonomous as we may think, or are seen to be. He asked us to think more creatively about what it means for us to demonstrate our benefit to society, and wondered if such a complex relationship could be captured by a simple collision metaphor that derives from physics. In the impact imaginary, the extreme example would be the research paper that has huge impact for users and generates a vast citation trail. However, Power pointed out, ‘impactees’ don’t always accurately cite the origins of their ideas – indeed sometimes they do not even acknowledge they have been impacted at all. Impactees often imitate what they have been influenced by – and not always consciously. This raises the issue of impact’s uncontrollability: ironically, the closer you get to ‘real’ impact, the further you are from it (as in the Gillies case, when the ‘real impact’ is very early, but it is only ‘impactful’ very late). For something to ‘have an impact’ there needs to be a space for impact to occur. It is a relationship contextualised by many other factors.
For these and other reasons Power advocated a wider range of genres of writing about and recognising impact – such as the impact of whole organisations, or schools of thought. He referred to his own work on audit culture (e.g. 1997) and the well-known constraints of the audit analogy. Trying to design the outcome into the product, such as its impactfulness, can diminish, rather than improve, its quality, and can lower, rather than raise, outputs. As in audit culture, where point-scoring for quality ‘tickboxes’ posed a risk to quality production - impact ‘gaming’ would inevitably result from impact evaluation linked to the distribution of scarce economic resources. Circuits of impact recognition and impact generation would emerge and could be detrimental. So the challenge will be to get a better understanding of impact, and a more complex model of what this term means.
Don Slater began by asking to what extent we (academics) are victims of our own failure to manage the impact of impact, or for that matter, the expectations of higher education and academic ‘outputs’ more broadly. Had we perhaps left ourselves open to being evaluated by others because we had not come up with better assessment criteria ourselves? Have we gone too far endorsing knowledge ‘for its own sake’, while at the same time overlooking the complexity of our own engagements with the wider world? He argued the gold standard for evaluating such complexity could be ethnography – a social scientific lens that is perfectly suited to generating productive, evidence-rich understandings of our own roles – and a means to reflect on them. Like others, Slater argued impact was an unhelpful model, based as it is on a collision between two completely separate objects. Citing the work of Steve Woolgar (2002), he used the expression ‘configuring the client’ as an example of the kinds of complex connections between users, consumers, products, services and producers that are routinely imagined in everyday life, and yet belied by the simplistic mechanism of ‘impact’. In response to a question about whether there might be a way to rename what we are already doing so that it fits with the impact model – or some alternative model – Slater confessed to being ‘tired of playing double consciousness’.
Other comments from the audience addressed both the cultural and the practical implications of impact. If we are not going to have impact, we need to be very careful about what we put in its place. To a certain extent, impact is already here to stay, so it may be more a question of adjustment than replacement. Patrick Dunleavy of the LSE prepared a report for the BA in 2008 on impact measurement, and is now preparing a report for HEFCE on metricisation. He argued for a model of a knowledge inventory, which would acknowledge that not all knowledge is immediately useful, but might become useful, at some unspecified future date. Different knowledge outputs will have different contexts of relevance, to diverse communities. Social science is part of what makes advanced societies advanced, he said. We should reject impact measures for individuals, and instead go for groups, showing the time-specific relevance of their models in a complex, multi-faceted, plural society.
His points were echoed by others, who pointed out, for example, that economic contributions are hard to measure, not least as GDP itself continues to become a weaker concept in the context of its triple bottom line – equality, ecology, economy. Sara Ahmed from Goldsmiths pointed out that higher education impacts that are out of line with government priorities are not likely to be counted as impact – precisely because they might have impact if they are not suppressed. She noted the importance of resisting the implicit compliance of the impact model.
Discussion
Two models of the impact of impact dominated both the presentation and discussion sessions of the workshop. One was critical, identifying shortcomings of the impact analogy, and dangers to its use. The other was more creative, imaging alternative definitions of impact, or alternatives to impact altogether. Practical suggestions and possibilities were generated from both strands of debate. The need to be wary of the ‘dumbing down’ and ‘disaffecting’ consequences of implementing impact measurement were well rehearsed, as were some of its inevitable shortcomings, such as delayed recognition. Alternatives to impact measurement, and variations of it, foregrounded a number of strategies, a large number of which could be described as ‘scholarly strategies’. These tended to emphasise the need to better understand what impact is supposed to be measuring, and the potential conflict between the choice of impact as an analogy and the complexity of academic networks and communities. An implication of this was to re-imagine impact as influence, or flow. What are the flows of knowledge production and reception, and how do they work – exactly? Academics might at once be more influential, but less knowledgeable about what our influences are, or where they go and what they do, than we know. A second prominent theme was writing. This came up in a variety of ways – from the question of how we define or describe impact, to the genres in which we write about it, to the contrast between writing things we don’t ever read, and imagining new kinds of ‘livelier’ writing that would enable us to reclaim language, as well as intentionality, as professionals working in higher education, and as academic researchers. Specialised types of writing, such as ethnography, were deemed highly appropriate mechanisms for increasing our understandings of the complex connections linking academic work to ‘the wider world’.
Both the emphasis on writing and documentation, and the challenge to the impact model or analogy, reflected a pervasive sense of the potentially parodic or paradoxical impacts of impact, whereby impact and its measurement co-evolve in such a way that impact becomes ‘impact’ – inseparable from its scare quotes. A sense that this phenomenon, or something similar, was extremely likely to be a consequence of impact measurement drove much of the effort to find better languages to describe what is meant by impact, and how we know it when we see it – as opposed to seeing the after effects of something else (interestingly there is a precisely similar effect in physics when, after a collision, the impacted particles can only confirm or confound the existing theories – they are literally and conceptually only ever after effects).
A somewhat ironic consequence of the workshop was the realisation that impact proved an unexpectedly interesting and lively topic. The effort to redefine the impact model proved far more exciting than had been anticipated. Partly this may reflect the paucity of occasions to reflect on the culture of evaluation that has come to dominate UK higher education. Partly it stems from changing working conditions affecting the relationship between academia and ‘the outside world’ – such as email, Wikipedia, Google, Facebook, Twitter, and You Tube – all of which have created new interfaces between academia and ‘the rest of society’. No doubt part of the energy in the room was generated by the resilience of academic discourse, and its solipsistic tendency always, at some level, to be about itself. That said, the display of thoughtfulness on offer, and of very high calibre presentations and discussion throughout the afternoon, suggested impact is a topic that easily repays the effort to think it through, even if it might appear to signify the reverse – the reduction of thought into deliverables.
This in turn suggested that impact discussions may prove very useful to the research evaluation process, even if impact measures are not. Whereas impact measures run the risk of becoming counter-productive, it would appear that discussion of them can have the opposite effect. For example, thinking through the impact question in terms of what we do when we are ‘producing quality research’, what relationships we form while doing it, how we define the ‘results’, and how these in turn are redistributed, can help us form a more concrete sense of the ‘socially situated’ nature of all knowledge production. This very broad view of impact moves us away from the narrow ‘collision’ idiom (‘the impact of science on society’) and toward a view of impact as dependent on already existing relationships. For example, impact is often described as something that is ‘made’. Making research impact takes many forms – from making impact on students we educate and inspire, to making an impact on a research community by organising a conference, or making an impact in the popular press with the report of a controversial finding. Once we begin to think of research impact as something we are, in a sense, making all the time, the question of evaluating it focuses less on how much of it there is, but what it is, where it is, and how to characterise its many forms. This reminds us of the question posed in the workshop about impact’s uncontrollability, and about undesirable impacts.
Using the idiom of tracking impacts inevitably also brings us to the question of lack of impact. This too is a problem the workshop considered in some depth, as the whole point of using impact as a criterion is to identify a distinctive set of impact interfaces – places where impact can be seen and measured. If impact is imagined as a one-way trajectory that generates significant outcomes, then it will primarily be found in the wake of such outcomes. Many contributions that are crucial to the reproduction of quality research within academia would not necessarily be recognised to have impact within such a model. They would be considered ‘un-impactful’, as it were, and thus as less important than high-impact activities. A different model of impact, which is based on the model of creating more connections and relationships around our research projects, and making these explicit, could include a much wider range of ‘impact factors’. This model would also lessen the risk of impact-parochialism – whereby more senior members of the academic community are seen to have a higher impact, because they are higher status, and therefore ‘carry more weight’.
Returning to the HEFCE criteria (despite the fact they may be irrelevant under ‘the emergency plan’), it would seem that the over-riding aspiration to encourage to researchers ‘to achieve the full potential impact across a broad range of research activity in the future’ will be difficult, since the ‘full potential impact’ can never be known. This difficulty is implicitly acknowledged by HEFCE’s own criteria of seeking to employ the widest possible definition of impact over the longest possible time perspectives, looking both forward and back. Our discussion in the workshop would, consequently, point more favourably toward the counter-strategy proposed by HEFCE to narrow this enormous range of impacts by focussing on a limited number of case studies. Our discussions would also appear to favour the HEFCE strategy of allowing all kinds of research excellence to be considered, according to criteria appropriate for each specialist field. Finally, our workshop suggests that whether or not impact becomes a substantial criteria for the further evaluation of higher education – formally or informally – that it is nonetheless a useful ‘lens’ through which to concentrate more carefully on the work academics do and the many connections and relationships on which it is dependent, and through which its ‘impact’ is both generated and registered – whether it is recognised or not.
References
Ahmed, S. (2004) The Cultural Politics of Emotion. Edinburgh: Edinburgh University Press.
Gillies, D. (2008) How Should Research Be Organised? London: College Publications.
Kuhn, T. S. (1962) The Structure of Scientific Revolutions. Chicago: University of Chicago Press.
Power, M. (1997) The Audit Society: Rituals of Verification. Oxford: Oxford University Press.
Wetherell, M. (2008) Speaking to Power: Tony Blair, Complex Multicultures and Fragile White English Identities. Critical Social Policy. 28 (3) pp. 299-319.
