how to write the research environment

• Search Menu
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

The research excellence framework, topic modelling the research excellence framework, predicting environment scores, general discussion, acknowledgements, data availability, what is a high-quality research environment evidence from the uk’s research excellence framework.

• Article contents
• Figures & tables
• Supplementary Data

Matthew Inglis, Elizabeth Gadd, Elizabeth Stokoe, What is a high-quality research environment? Evidence from the UK’s research excellence framework, Research Evaluation , 2024;, rvae010, https://doi.org/10.1093/reseval/rvae010

• Permissions Icon Permissions

As part of the UK university sector’s performance-related research funding model, the ‘REF’ (Research Excellence Framework), each discipline-derived ‘Unit of Assessment’ must submit a statement to provide information about their environment, culture, and strategy for enabling research and impact. Our aim in this paper is to identify the topics on which these statements focus, and how topic variation predicts funding-relevant research environment quality profiles. Using latent Dirichlet allocation topic modelling, we analysed all 1888 disciplinary ‘unit-level’ environment statements from REF2021. Our model identified eight topics which collectively predicted a surprisingly large proportion—58.9%—of the variance in units’ environment scores, indicating that the way in which statements were written contributed substantially to the perceived quality of a unit’s research environment. Assessing research environments will increase in importance in the next REF exercise and the insights found through our analysis may support reflection and discussion about what it means to have a high-quality research environment.

For the past four decades, higher education institutions in the UK have been subject to evaluations of their research by the higher education funding councils. The first evaluation, the ‘Research Selectivity Exercise (RSE)’ took place in 1986 (for a history, see Bence and Oppenheim 2005 ). Over the years, and with six assessments between 1986–08, the RSE evolved into the Research Assessment Exercise (RAE) and then, in 2014, into the ‘Research Excellence Framework’ (REF). For each evaluation, university disciplines and fields of study were divided into ‘Units of Assessment’ (UoAs). The most recent assessment took place in 2021, with results published in 2022.

As well as name changes, the requirements for submissions have evolved (see Marques et al. 2017 ), from the “quick and dirty” ( Jones and Sizer 1990 ) approach taken in 1986 through to including/excluding particular categories of staff; changing the minimum/maximum numbers of publications per individual; the introduction of research environment statements (RAE 1996), and the introduction of impact case studies (REF 2014). Two constants about RSE/RAE/REF remain: the original principle of peer assessment, despite the rise of publication metrics in other domains, and the use of the results to distribute government funding.

The RSE represented the creation of “the first and most highly institutionalised research evaluation system worldwide” ( Marques et al., 2017 : 822). Since then, the RAE/REF has been widely discussed and used as a model for other countries (e.g. Geuna and Martin 2003 ) or resisted and rejected (e.g. Swedish Government 2016 ), but rarely adopted wholesale in countries internationally ( French, Massy and Young 2001 ; for overviews, see Sivertsen 2017 ; Thomas et al. 2020 ; Pinar and Horne 2022 ). Either way, the discourse of the RAE/REF reaches far beyond the UK.

Analysing the research excellence framework

Unsurprisingly, the RAE/REF has been scrutinized in terms of (i) critiques of the politics and methodologies that underpin the process and, (ii) quantitative and qualitative analyses of submissions, assessment processes, and results themselves. The former comprises a literature too vast to cover substantially here, but includes criticisms of the trend towards a competitive, neoliberal, and commodified higher education system (e.g. Fairclough, 1995 ; Brown and Carasso, 2013 ), of the impact of assessment on individual disciplines and interdisciplinarity (e.g. Pardo-Guerra 2022 ), and of unintended consequences (for overviews, see Gillies 2008 ; Brassington 2022 ; Pinar and Horne 2022 ; Watermeyer and Derrick 2022 ). The RAE/REF has driven both policy and debate in UK higher education, with a series of consultations, evaluations, recommendations, and iterated processes ( Manville et al. 2015 ; Curry, Gadd and Wilsdon 2022 ).

Another approach to evaluating and critiquing the RAE/REF focuses on the actual content of HEIs’ submissions to RAE/REF, using both quantitative and qualitative methods. Several of these studies have used similar text-mining or topic modelling and related methods to those used in this paper. Perhaps because of the increasing significance of research impact over the past decade ( Derrick and Samuel 2016 ; Kellard and Śliwa 2016 ; Sutton 2020 ; Jensen, Wong and Reed 2022 ), several studies have scrutinized the content and composition of case studies. For instance, a report commissioned by the UK research funding councils ( King’s College London and Digital Science 2015 ) used text-mining and qualitative analysis to provide an initial assessment of all REF2014 impact case studies, making observations about the diverse range of impacts, their underpinning research, and their global reach (see also Terämä et al. 2016 ). Reichard et al. (2020) conducted two studies using qualitative thematic and quantitative linguistic analysis of REF2014 impact case studies to identify what individual words and phrases were associated with high and low scores. They identified numerous “lexical bundles” associated with lower (e.g. “involved in”, “has been disseminated”, “the event”) and higher (e.g. “the government’s”, “in the UK”) scores, the former associated with describing activities and pathways to impact and the latter evidence of significant and far-reaching impact itself.

Within the wider literature on REF methodology and submissions, the least scrutinized aspect is the environment statement ( Thorpe et al. 2018a ). We now turn to discuss what we know and do not know about REF environment statements, starting by describing the submission requirements for the 2021 assessment.

The REF environment statement

The environment statement(s) and their submission requirements.

As noted above, an environment statement was introduced relatively early into the UK RAE/REF cycle. In 2021, two main changes occurred: the introduction of a pilot “Institutional-level environment statement”, and the removal of an ‘impact template’ from REF2014 and the incorporation of ‘research and impact’ as one environment statement in 2021.

The aim of REF2021 unit-level environment statements was to provide assessable information about each UoA’s “environment for research and enabling impact” (Guidance on Submissions, p. 82) and especially its “vitality” (“the extent to which a unit supports a thriving and inclusive research culture for all staff and research students, that is based on a clearly articulated strategy for research and enabling its impact, is engaged with the national and international research and user communities and is able to attract excellent postgraduate and postdoctoral researchers”) and “sustainability” (“the extent to which the research environment ensures the future health, diversity, wellbeing and wider contribution of the unit and the discipline(s), including investment in people and in infrastructure”) (Panel Criteria and Working Methods, p. 58). For REF 2021, detailed guidance notes and a template were provided to structure the information in four sections: (1) “unit context, research and impact strategy”; (2) “people, including: staffing strategy and staff development, research students, equality and diversity”; (3) “income, infrastructure and facilities”, and (4) “collaboration and contribution to the research base, economy and society.” The permitted length of the statement varied according to the number of staff in a UoA, from 8,000 (for a submission comprising 1–19.99 FTE) to 12,000 words (plus 800 further words per additional 20 FTE). The environment statement was worth 15% of the funding allocation. For Main Panels A, B, and C, each of the four subsections attracted equal weighting; in Main Panel D, sections (1) and (4) attracted 25%, the ‘People’ section 30%, and ‘income, infrastructure and facilities’ 20%.

Analyses of environment statements

To the best of our knowledge, no close analyses of REF2021 environment statements have yet been published, apart from Manville et al.’s (2021) real-time study of REF as it happened, including focus group research on academic and professional service staff experiences of completing them. However, research has been conducted on REF2014 environment statements. For example, Matthews and Kotzee (2022) analysed both REF (2014 submission) and TEF (Teaching Excellence Framework, 2017 submission) documentation with the aim of investigating links between research and teaching. They found that the term “research-led”, analysed in the context of its collocates, was often used in connection with teaching, and argued that “according to what universities themselves write in institutional texts, teaching and research are not always in a mutually beneficial entanglement, but often rather a one-way relationship in which research expertise and institutional prestige are used to bolster claims of teaching excellence” (p. 578).

Mellors-Bourne, Metcalfe and Gill (2017) also used text-mining to assess the level of engagement with equality and diversity in the ‘People’ section of REF2014 environment statements. This included evidence of participation in schemes such as ‘Athena Swan’ and Stonewall’s, and the relative frequencies of words pertaining to the topic: ‘equality’, ‘diversity’, ‘Athena’, ‘gender’, and ‘ethnicity’. Mellors-Bourne et al. found that statements focused predominantly on gender; that “the word ‘equality’ was used on average between once and twice within each environment statement” (p. 2), and that the Main Panels A and B (the STEM disciplines) mentioned ‘Athena’ more than twice as much as Main Panels C and D. The researchers also found “[e]vidence suggesting a positive relationship between REF research environment subprofiles (scores) and reference to key E&D terms within submissions, overall and at the level of the main panels” (p. 2). In REF2021, the focus on equality, diversity and inclusion in the guidance notes extended well beyond the ‘People’ section, presumably to encourage HEIs to demonstrate how EDI strategies and outcomes were embedded in all areas of research and impact.

Thorpe et al. (2018a , 2018b ) used computer-assisted text analysis to scrutinize the language content and ‘tone’ of REF2014 environment statements in business and management schools. They sought to understand whether the way environment statements were written differed between high and low scoring universities. They found that higher-ranked universities used less passive voice, were more coherent, adopted “a ‘finished article’ discourse rather than a ‘we are developing’ discourse”, cited “specifics rather than generalities”, and were more self-referential ( Thorpe et al., 2018a , p. 582). In terms of tone, the authors found that, perhaps counterintuitively, higher-scoring statements scored lower in terms of ‘activity’ tone “that evokes a ‘safe’, ‘staid’, ‘orthodox’, ‘conservative’, and ‘settled’ environment that is not disturbed (unduly at least) by reform, disruption, or major staff turnover” ( Thorpe et al. 2018b : 60). The authors concluded that “low-ranked universities could have achieved higher scores by reflecting on particular areas of word choice and the potential effects of those choices on assessors” ( Thorpe et al. 2018b : 53).

Like Reichard et al.’s (2020) analysis of impact case studies, Thorpe et al. (2018a , 2018b ) revealed the importance of language as well as content in the production of environment statements. They concluded that “the accompanying narrative played an important role in determining REF2014 environment scores” (2018a: 572). Furthermore, in contrast to impact case studies, which included corroborating evidence as part of submission, “little supporting evidence was required in environment submissions”, meaning that “there is potential for writing quality to have an even larger effect in environment submissions, and for HEIs to use language-related techniques to manage their image” ( Thorpe et al. 2018a : 574).

In June 2023 the ‘Future Research Assessment Programme’ (FRAP) published their Initial Decisions on REF 2028 ( Joint UK HE Funding Bodies 2023 ), although it has since been announced that the exercise will be delayed until 2029. Despite the Institution-Level Environment Panel Pilot Panel (Research England 2022) recommending the removal of unit-level environment statements and focusing instead on institution-level statements, the Initial Decisions propose the retention and assessment of both. In the 2029 exercise, ‘Environment Statements’ will be broadened to become ‘People, Culture & Environment (PCE) Statements’ with a concomitant increase in weighting from 15% to 25%. The Decisions state that “the collection of evidence for the people, culture and environment element will move towards a more tightly defined, questionnaire-style template that will create greater consistency across submissions and focus on demonstrable outcomes ( Joint UK HE Funding Bodies 2023 )”. At the time of writing this is yet to be developed and it is unclear what proportion of the submission will take a narrative format and what proportion will comprise data and evidence. However, given the qualitative nature of the dimensions being assessed, it is likely that there will be a significant narrative element. Furthermore, given the increased weighting of PCE, any such element should have an even greater bearing on overall results.

In sum, the focus on environment statements as a subset of research on the UK REF has, to date, been small. It has either been partial (e.g. has analysed just one UoA ( Thorpe et al. 2018a , 2018b )) or not focused on the most recent exercise. The aim of this paper is to investigate whether thematic patterns may be identified in the 2021 submissions, whether such patterns may be correlated with scores, and what this might teach us about crafting future environment narratives. We begin by introducing the method we adopted, latent Dirichlet allocation topic modelling.

Topic modelling is a computation method that seeks to analyse the content of many texts by identifying a small number of semantically connected themes or topics ( Blei, Ng and Jordan 2003 ). The aim is to take a collection of unstructured texts and identify the topics they cover by studying their words. For example, if a document uses the words ‘water’, ‘sand’, and ‘swimming’ with an unusually high frequency, this may constitute evidence that the document is about beaches. One way to understand the topic modelling approach is to think about how to create documents from a predefined set of topics, defined to be probability distributions over words. For instance, our beach topic might assign very high probabilities to ‘water’, ‘sand’ and ‘swimming’, medium probabilities to ‘inflatable’, ‘spade’ and ‘picnic’, and low probabilities to ‘dioxide’, ‘carpentry’ and ‘veneer’. Similarly, we might define a topic about Greece (which might perhaps have a high probability associated with the words ‘Greek’, ‘Athens’, ‘souvlaki’ and so on). If we wanted to create a new document about beach holidays in Greece, we might choose 30% of the new documents words to be from the beach topic, 30% from the Greece topic, perhaps 30% from a travel topic, and 10% from other topics. A document about Greek beach holidays can then be created by simply sampling words from each topic using the appropriate probabilities. This method makes two simplifying assumptions. First, the ‘bag of words’ model of text is adopted by ignoring word order; and second, so-called ‘stop words’ (words such as ‘the’, which is topic independent) are ignored.

The topic modelling approach can be thought of as carrying out this document construction process in reverse. We start with a large collection of texts, assume that they were created in this way, and then computationally identify the topics that would have been most likely to lead to these documents using a latent Dirichlet allocation (LDA) algorithm ( Blei, Ng and Jordan 2003 ; Grimmer and Stewart 2013 ). Topic modelling adopts a grounded theory mentality: the analyst has no preconceived ideas about what topics will be identified, instead topics/themes emerge from the analysis process. Once topics have been identified, the semantic content of each document can be analysed by studying the topic composition of each document. For instance, we may find that Document 1 contains 6% of words from Topic 1, 30% from Topic 2, 0% from Topic 3, and so on.

We downloaded all 1888 unit-level environment statements (a total of 18.0 m words) from the REF website. These were converted from pdf into plain text using the UNIX pdftotext command ( Poppler 2022 ). We used MALLET (version 2.0.8RC2), a UNIX topic modelling tool ( McCallum 2002 ), to calculate possible models, using MALLET’s default list of stop words.

Topic modelling requires that one specifies how many topics the LDA algorithm should identify. By making different choices researchers can specify the granularity of their analysis. We adopted the perplexity approach to decide on the number of topics. Each model can be assigned a perplexity, which is analogous to a model fit ( Blei, Ng and Jordan 2003 ). Perplexity can be calculated by fitting a model with a specified number of topics to a subset of the documents, and then assessing its fit to the remaining documents. One can always reduce the perplexity (or increase the fit) by fitting a model with a larger number of topics, although at some point the benefit of doing so will be offset by the increased difficulty of interpretation. Jacobi, van Atteveldt and Welbers (2016) suggested using a method similar to the scree test often used in factor analyses: by calculating the perplexity of models with a range of different topic numbers, it is possible to determine if there is a point at which the benefit, in terms of reduced perplexity, of increasing the number of topics appears to level off.

We split the environment statements into a training corpus (80% of statements) and a testing corpus (20% of statements), fitted models with 10, 20, 30, …, 100 topics to the training corpus, and calculated the associated perplexities using the testing corpus. These perplexity figures are shown in Figure 1 . We then fitted a piecewise linear regression to these points, which suggested that the ‘elbow’ of the graph appeared at 41.99 topics. We therefore selected 42 topics for our main analysis.

Perplexities associated with models with 10, 20, 30, …, 100 topics. The dotted lines show a piecewise linear regression line of best fit.

Topic modelling has an important advantage over more traditional qualitative analytical techniques in that it is extremely inclusive. Given that 1888 unit-level environment statements were returned to REF2021, containing ∼18 m words, it would have been impractical for a human analyst to read and analyse each statement. However, topic modelling is not purely quantitative: the LDA algorithm identifies topics which must then be interpreted. One common approach to this task is to conduct careful qualitative analyses of documents that contain a high proportion of words from each topic. We return to this issue later in the paper.

The 42 topics identified are shown in Table 1 . The table shows the characteristic words associated with each topic, the statement with the highest proportion of words from that topic, and the label we gave the topic. These labels were based on our interpretations of studying the characteristic words, the statements with particularly high proportions of words from the topic, and the statements with particularly low proportions of words from the topic.

Descriptive statistics, averaged across each of the four quarters of the experiment, for each of the four indices under consideration

Of the 42 topics, 28 were disciplinary specific. For example, Topic 42 was characterised by words including ‘clinical’, ‘cancer’, ‘medicine’, ‘disease’, ‘MRC’ and ‘NHS’, and the top 10 environment statements in terms of the proportion of words with this topic were all returned to the Clinical Medicine panel. We therefore labelled this topic ‘Clinical Medicine’. In some cases our model combined two or more disciplines. For instance, Topic 2 was characterised by the words ‘philosophy’, ‘religion’, ‘theology’, ‘religious’ and ‘ethics’. Of the top 20 statements with high proportions of words from this topic, 9 were returned to the Philosophy panel and 11 to the Theology and Religious Studies panel. We used the label ‘Philosophy and Religion’.

There were five geographical topics. For instance, Topic 8 was characterised by the words ‘Liverpool’, ‘Leeds’, ‘Manchester’, ‘York’, ‘Sheffield’ and ‘Yorkshire’, all cities/regions in the north of England, and the statements with the highest proportion of words from this topic were from northern universities. There were geographical topics associated with the North, the West Country, Scotland, London and Wales.

We also found a topic, Topic 25, that was used by institutions that organise their academic work through constituent colleges. The topic was characterised by words including ‘university’, ‘faculty’, ‘centre’, ‘college’ and ‘institute’, as well as geographical terms that referenced multi-college universities (‘London’, ‘Oxford’, and ‘Cambridge’). Of the 25 statements with the highest proportions from this topic, 23 were returned by constitute faculties or colleges of the University of Oxford, the University of Cambridge or the University of London. The exceptions were statements from Kingston University (an institution based in London) and Oxford Brookes University (an institution based in Oxford).

Of most interest for our purposes are the remaining eight topics. To identify appropriate labels we followed a similar process, separately for each topic. First, we studied the topic’s characterising words. Second, we read the five environment statements with the highest proportion of words from the topic, and the five environment statements with the lowest proportion of words from the topic. Finally, we conducted concordance analyses to identify how characterising words were used in statements with high and low proportions of words from the topic. This involved using a keyword in context (KWIC) tool from a traditional corpus linguistics package ( Anthony 2022 ). For instance, if ‘faculty’ was a characteristic word for a topic, we would find every occurrence of ‘faculty’ in the five statements with the highest proportion of words from the topic and read the surrounding context to identify how the word was typically being used. We would then do the same for the five statements with the lowest proportion of words from the topic. To illustrate our approach, we first discuss the reasoning behind our naming of Topic 16 in some detail, and then discuss each of the remaining seven topics in turn. Note that the online data associated with this article includes the topic weightings derived from our model for each topic and each environment statement submitted to REF2021, so interested readers can independently verify our analyses and assess for themselves the topic names’ appropriateness.

Although all REF environment statements are publicly available from the Research England website, we opted to redact individuals’ names in the quotes reported below as they are not relevant to the research questions we asked. We have, however, not anonymised at the institution or department level, as these may assist readers interpret the topics.

Topic 16—Immature Research Environment

The proportion of words from Topic 16 used by statements varied from 0.0001 (Imperial College’s Mathematical Sciences statement) to 0.362 (Bedfordshire’s Business and Management Studies statement). The topic was characterised by words such as ‘research’, ‘university’, ‘staff’, ‘international’, ‘REF’, ‘member’, and ‘members’ (see Table 1 ). Unlike with some of the other topics discussed below, we did not find these words very insightful for determining the semantic content of the topic.

Our next step was to carefully read the five statements with the highest proportion of words from the topic and the five statements with the lowest proportion of words from the topic. This revealed that the topic-defining words seemed to be being used in characteristic ways by those statements with a high proportion of words from the topic. Specifically, Topic 16 was characterised by descriptions of how the units were trying to encourage staff to engage in research. For example, the Wrexham Glyndŵr University Computer Science and Informatics statement (Topic 16 proportion 0.327) noted that “Data from October 2020 indicates that 38% of the 13 members of academic staff associated with UoA11 [the Computer Science and Informatics unit] have a doctoral qualification” and that “An encouraging sign is that 38% of UoA11 staff are studying towards a doctorate.” Similarly, the Liverpool John Moores University Business and Management Studies statement (T16 proportion 0.339) noted that 31 members of staff “are being supported in their research-related activities, with a view to them being research active in the next assessment period” and that staff were supported by holding seminars, where the invited external speakers were “editors of peer-reviewed journals with high impact factors” in order to assist staff “target publications in highly respected journals”. The Bedfordshire Business and Management Studies statement (T16 proportion 0.362) emphasised that “Staff members are strongly encouraged to attend international conferences and present their research results” and that their staff “are allocated dedicated research time as part of their workload”. In contrast, the statements with low frequencies of words from Topic 16 seemed to take for granted that their academic staff had doctorates and routinely conducted research.

Next, we conducted concordance analyses comparing how the topic’s defining words were used in the five statements with a high proportion of words from Topic 16 with how they were used in the five statements with a low proportion of words from the topic. For instance, we compared how these statements used the word “research”. In the five high statements there were 19 uses of “research active” (e.g. “continued to be research active”, “support to be research active”, “increased the number of research active staff”, “sought to retain research active staff”, “staff on the cusp of being research-active”) compared to just one in the five low statements, which appeared in a subheading in the University College London’s Law statement (“2.1 Research-active staff and output selection profile”).

As another example, across the five high Topic 16 statements there were 82 instances of “conference”, including numerous examples of conferences that had been attended by staff from these units. In contrast, this word appeared only 29 times in the five low Topic 16 statements and tended to be used as an illustration of a wider point. For example, in the Cambridge philosophy statement (T16 topic proportion 0.00003), the organisation of the Cambridge Platonism conference was given as an example of the unit’s interdisciplinary research (the conference was jointly organised with the Cambridge Faculty of Divinity).

To give one final example, there was also a difference in the way the high- and low-T16-proportion statements used the word “journal”. The five high-T16-proportion statements contained 37 instances of this word. In some cases, these were examples of how members of the unit had written research articles, e.g. Newman University’s Sport and Exercise Sciences, Leisure and Tourism statement (T16 proportion 0.360) noted that “Visiting Professor [anonymised] has produced a manuscript currently in review in the European Respiratory Journal Open”. In other cases these were lists of interactions with journals: Wrexham Glyndŵr University’s Computer Science and Informatics statement explained how one colleague was “on the review panel for a further 6 journals”. In contrast, the five low-T16-proportion statements had only 13 instances of the word ‘journal’. These tended to be examples of how the unit was contributing to the wider academic community. For instance, the University College London law environment statement (T16 proportion 0.0003) discussed how they were progressing towards an open research environment and exemplified this by noting how one member of the unit had “founded Europe and the World: A Law Review as a fully peer-reviewed OA [open access] journal”.

In sum, we concluded that those statements which had a high proportion of words from Topic 16 tended to spend a large proportion of their statement discussing how they were attempting to encourage or support routine research activities. These kinds of discussions were absent from those statements with a low proportion of words from this topic. We therefore named this topic “Immature Research Environment”.

Topic 4—Internal Structure of Research Units

Topic 4 was characterised by the high use of words such as ‘unit’, ‘unit’s’, ‘faculty’, ‘section’, ‘themes’, ‘theme’ and ‘institutional’. The proportion of words from this topic ranged from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.150 (the University of Cumbria’s Business and Management Studies statement). The topic tended to be characterised by detailed descriptions of the internal structure of the units. For instance, the University of Cumbria’s Business and Management Studies statement (T4 proportion 0.150) devoted 1.5 pages of their statement to the “unit context and structure” which noted how, during the assessment period, they had created a new institute and developed three new research themes. Similarly, the University of Winchester’s English Language and Literature statement (T4 proportion 0.137) spent just over a page discussing their unit context and structure, noting how the unit was situated within the University’s department and faculty structure, how it contained a research centre, and how the Centre interacted with other centres across the University. We named this topic “Internal Structure of Research Units”.

Topic 7—Career Development and EDI

Topic 7 was characterised by words such as ‘staff’, ‘support’, ‘training’, ‘including’, ‘access’, ‘career’ and ‘diversity’. The proportion of words from this topic ranged from 0.004 (The Royal Agricultural University’s Agriculture, Food and Veterinary Sciences statement) to 0.399 (Leeds Arts University’s Music, Drama, Dance, Performing Arts, Film and Screen Studies statement). Statements with a high proportion of words from Topic 7 tended to have long sections that discussed how the unit supported staff and student development, and about their equality and diversity processes. For instance, the University of Nottingham’s Politics and International Studies statement (T7 proportion 0.290) included careful statistical analyses of their gender balance at different career stages, as well as analyses of their staff by ethnicity, disability and age profiles. The statement then went on to discuss how these analyses informed “EDI-focused improvements”. For instance, in response to “too few members from underrepresented groups in leadership roles” the statement noted how the unit had reconstituted its EDI committee and “increased leadership by women in major committees”. In contrast, the Royal Agricultural College’s Agriculture, Food and Veterinary Sciences statement (T7 proportion 0.004) devoted just 50 words to EDI issues, and only used the word ‘diversity’ in the context of their research on “the global distribution of earthworm diversity”. We named this topic “Career Development and EDI”.

Topic 18—Staff Ways of Working

Topic 18 was characterised by words such as ‘work’, ‘school’, ‘colleagues’, ‘teaching’, ‘group’, ‘members’, ‘part’ and ‘years’. Some statements contained a very low proportion of words from this topic, e.g. Heriot-Watt University’s Architecture, Built Environment and Planning statement (T18 proportion 0.000), whereas others contained a substantial proportion from it, e.g. the University of East Anglia’s Law statement (T18 proportion 0.262). The statements with a high proportion of words from the topic were characterised by many concrete descriptions of staff working practices. For example, the University of Newcastle upon Tyne’s Classics statement (T18 proportion 0.244) described how “Members who have held a substantial administrative role are entitled to an extra semester of research leave”. Similarly, the University of St Andrews’s Economics and Econometrics statement (T18 proportion 0.255) discussed the process by which academic staff can apply for sabbatical leave: “The HoS considers applications in relation to the general workload allocation process and, if there are doubts about the feasibility of accommodating all applications, the HoS consults a panel of senior colleagues.”

In our concordance analysis we noted that ‘work’ was commonly used as a verb in high-T18-proportion statements to describe concrete examples of how the unit operated (“the University granted [anonymised] two years of unpaid leave to work and develop Impact at the European Central Bank”), whereas in low-T18-proportion statements it was often used as a noun (“our work on public health engineering”). We also found that ‘members’ was more often used in high-T18-proportion statements to describe concrete internal activities (“Individual staff members can request travel funding and leave to attend masterclasses and short courses”), whereas in low-T18-proportion statements it was typically used to describe esteem activities (“Our researchers also chaired or have served as members of important grant panels”). We named this topic “Staff Ways of Working”.

Topic 28—REF-Focused Research Strategy

Topic 28 was characterised by words such as ‘UoA’, ‘REF’, ‘UoA’s’, ‘UoAs’, ‘section’, ‘cycle’ and ‘submitted’. The proportion of words from this topic varied from 0.000 (the University of Edinburgh’s Clinical Medicine statement) to 0.126 (the University of Winchester’s History statement). Statements with a high proportion of words from Topic 28 tended to use REF terminology to describe their research environment. For example, they might characterise their internal structure in terms of UoAs or ‘units’ rather than departments, centres or institutes. For instance, the University of Worcester’s English Language and Literature statement (T28 proportion 0.126) described “The Unit’s strategic research objectives”, “the unit’s impact strategy” and “the unit team”; and the University of Winchester’s History statement (T28 proportion 0.126) described how “the UoA had a devolved budget”, the existence of a “UoA working group” and “the strategic aims of the UoA over the cycle”. In contrast, the joint engineering statement from the University of Edinburgh and Heriot-Watt University (T28 proportion 0.000) contained no instances of ‘UoA’, and only used “unit” in the generic text used in the page header (“unit-level environment template (REF5b)”). Instead, they described how their research was organised into “cross-cutting organisational themes” and “interdisciplinary global research challenge areas”. Similarly, the University College London education statement (T28 proportion 0.000) discussed how their research was organised into departments and research centres, and only used the word ‘UoA’ in a table reporting the submission’s demographic data. We named this topic “REF-Focused Research Strategy”.

Topic 30—Exemplification of Strategy and Processes

Words that characterised Topic 30 included “e.g”, “including”, “funding”, “supported”, “grant” “PGRs”, “impact” and “awards”. Like Topic 16, it was not immediately obvious to us from studying these words what the topic referred to. However, when we compared the high-T30-proportion statements and the low-T30-proportion statements, we concluded that the topic was capturing an increased use of concrete examples to illustrate strategies and processes. To illustrate, the five statements with the highest proportion of words from Topic 30 made liberal use of “e.g.” to give explicit examples of the research strategies being described. For example, the University of Nottingham’s Geography and Environmental Studies statement (T30 proportion 0.278) described how they enable and facilitate impact: “the new Institutional Institute of Policy and Engagement has helped fund pump-priming engagement work (e.g. [anonymised]); fund high-level policy relevant talks (e.g. [anonymised] at Asia House and Chatham House) and aid development of policy briefs (e.g. [anonymised] on water management in the Red River, French on indebtedness and financial exclusion).” In their section on open research, they wrote that “The School developed and hosts online, openly accessible maps, including the Blue-Green Cities multiple benefits toolbox ([anonymised]) and the ‘black presences and the legacies of slavery and colonialism’ online map ([anonymised])”. Similarly, the University of Leicester’s Communication, Cultural and Media Studies, Library and Information Management statement (T30 proportion 0.266) noted that their “strategy of enabling researcher development in Media focuses on supporting ECRs and mid-career academics, to achieve external funding success. For example, 23 of the 40 awards secured in the REF period were to Assistant Professors.”

While low-T30-proportion statements also used exemplification, these tended to be less related to the research strategies and processes described in the statements. For instance, there were 22 instances of “e.g.” in Imperial College London’s Clinical Medicine statement (T30 proportion 0.000), of which 12 were used in front of lists of journals (“Our reach is demonstrated by publishing in specialist journals (e.g. Lancet Infect Dis [11], Nature Immunology [7]”) and a further 4 were used in front of scientific concepts (“…to reveal mechanisms of cardiovascular disease. e.g. identification of titin variants in health and disease”).

We named Topic 30 “Exemplification of Strategy and Processes”.

Topic 34—Industry Partners and Funding

Topic 34 was characterised by words such as ‘award’, ‘awards’, ‘industry’, ‘society’, ‘data’, and ‘international’. The proportion of words from this topic in statements varied from 0.000 (the University of East Anglia’s Area Studies statement) to 0.259 (the University of Bristol’s chemistry statement). Those statements which had a high proportion of words from Topic 34 devoted considerable space to discussing their industrial partnerships and research funding. For example, the Imperial College London Chemistry statement (T34 proportion 0.252) noted that “Collaborations with industry include GSK and Pfizer”, and that “members are involved in industry collaborations e.g. a £3.2M EPSRC BP Prosperity Partnership”. The University of Surrey Physics statement (T34 proportion 0.249) argued that their “world-class research is evidenced by grant awards over the REF period that total more than £19.6 million” and noted that they “work closely with industrial partners”. Given this focus, it was unsurprising that the correlation between the proportion of an environment statement made up of words from Topic 34 was strongly correlated with a submission’s research funding per FTE, r = 0.642, P < 0.001. Notably, however, this correlation was much reduced if research income was standardised within each UoA (to r = 0.275, P < 0.001). In other words, Topic 34 related to overall unstandardised research funding, meaning that statements with a particularly high proportion of Topic 34 words tended to come from highly funded scientific disciplines. Indeed, the mean proportion of words from Topic 34 for statements to Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) were 0.109, 0.139, 0.044 and 0.020 respectively. In other words, statements from scientific disciplines tended to use more words from Topic 34 than statements from non-scientific disciplines, an observation consistent with our conclusion that the topic concerned industrial partnerships and funding. We named this topic “Industry Partners and Funding”.

Topic 40—Early Career Researcher (ECR) Development

The last of our eight general topics was Topic 40. This topic was characterised by words such as ‘development’, ‘develop’, ‘support’, ‘research’, ‘researchers’, ‘strategy’, ‘strategic’, ‘work’, ‘working’ and ‘funding’. Of the eight general topics, Topic 40 was the most common: on average environment statements devoted 18.6% of their content to it, although the proportions of words from Topic 40 ranged from 0.038 (Kingston University’s Philosophy statement) to 0.403 (Canterbury Christ Church University’s Sport and Exercise Sciences, Leisure and Tourism statement). Those statements with a high proportion of words from Topic 40 spoke at length about researcher development, with a particular focus on early career researchers. For instance, the Queen Margaret University Edinburgh Sociology statement (T40 proportion 0.360) discussed how they “support researchers in exploring and preparing for a diversity of careers, for example, through the use of mentors and careers professionals, training, and secondment” and the Solent University Southampton Sport and Exercise Sciences, Leisure and Tourism statement mentioned that a “research mentoring programme organised through [the School Advisory Group for Research] has been implemented to support researchers”. The five statements with the highest proportion of words from Topic 40 made 23 references to the Concordat to Support the Career Development of Researchers compared to 4 references in the five statements with the lowest proportion of words from this topic. We named the topic “Early Career Research (ECR) Development”.

For our main analysis, we asked whether the eight general topics that environment statements focused on were related to the quality profiles they received. Recall that panels assessed each submission’s environment using a five-point scale from ‘unclassified’ (“an environment that is not conducive to producing research of nationally recognised quality or enabling impact of reach and significance”) through to ‘4*’ (“an environment that is conducive to producing research of world-leading quality and enabling outstanding impact, in terms of its vitality and sustainability”). Each submission was awarded a ‘quality profile’ based on its environment statement and associated data (discussed below). For instance, the Open University’s submission to the Classics UoA was rated as having an environment where 25% of activity was 4* (world-leading), 50% was 3* (‘internationally excellent’), 25% was 2* (‘recognised internationally’) and 0% was 1* (‘recognised nationally’) or unclassified. For each submission we calculated a grade point average (GPA), which was a simple linear combination of the percentage of each quality level. So the Open University’s Classics submission obtained an environment GPA of 3.0 (0.25 × 4 + 0.5 × 3 + 0.25 × 2 + 0 × 1 + 0 × 0).

Alongside environment statements, the assessment panels were also provided with additional metrics associated with each submission. These included the full-time equivalent number of staff (FTE) being returned in the submission, the grant income that the unit had received during the assessment period (which could be broken down by source and date), and the number of doctoral degrees that the unit had awarded during the assessment period.

We ran a hierarchical regression predicting each unit’s environment GPA. In the first block we entered each unit’s FTE, their research income per FTE, and the number of doctoral degrees awarded per FTE. Each of these metrics was standardised (using z scores) within each UoA to take account of disciplinary norms (for instance, the mean grant income per FTE in the Clinical Medicine unit was £3.7 m compared to £74k in the English Language and Literature unit). In the second block we entered the proportion of each environment statement from the eight general topics discussed above.

The results of this regression are shown in Table 2 . Together the environment metrics could explain 47.3% of the variance in environment GPAs. When the topic weightings were added, an additional 21.9% of the variance could be explained, bringing the overall R 2 to 69.1%. Thus the weightings of these eight topics explained significant extra variance in environment GPAs, F (8, 1870) = 166, P < 0.001. When the eight topic weightings were used as predictors in the first block (ie before the metrics were entered) they explained 58.9% of the variance in environment GPAs, F (8, 1873) = 336, P < 0.001. In sum, the weightings associated with the eight general topics in our topic model predicted a surprisingly large proportion of the variance in submissions’ environment GPAs, indicating that the topics that environment statements focused upon made a substantial contribution to the perceived quality of each submission’s research environment.

A hierarchical regression analysis predicting environment GPA with various metrics (entered in Block 1) and topic weightings from the eight general topics (entered in Block 2)

P < 0.05.

P < 0.01.

P < 0.001.

As shown in Table 2 , of the eight topics, four were significant negative predictors of environment GPA, two were significant positive predictors and two were not significant predictors. Statements that had higher weightings from the Immature Research Environment, Staff Ways of Working, REF-Focused Research Strategy, and ECR Development topics were associated with lower environment GPAs. Statements that had higher weightings from the Exemplification of Strategy and Processes, and Industry Partners and Funding topics were associated with higher environment GPAs.

Because this regression analysis only assessed whether topic weightings were linearly associated with environment GPAs, we also investigated whether there were nonlinear relationships by inspecting scatterplots of topic weightings against environment GPAs separately for each topic. These are shown in Figure 2 , together with cubics of best fit. There appeared to be a clearly nonlinear relationship between topic weighting and environment GPA for Topic 7 Career Development & EDI. Placing little emphasis on this topic was associated with receiving a low environment GPA, but so was placing too much emphasis on it. The cubic of best fit obtained its maximum when 13.4% of the statement was made up of words from Topic 7 (recall that this figure is a percentage of all words in the statement, after stop words have been removed). Environment statements varied substantially in the extent that they discussed Career Development and EDI—the statement with the lowest emphasis on this issue had just 0.4% of its words from the topic, the statement with the highest had 39.9%. But the highest environment GPAs, on average, were obtained by statements where 13–14% of the statement focused on Career Development and EDI.

Scatterplots showing topic weightings (proportion of each statement made up of words from the given topic) against environment GPA, separately for the eight general topics. Bold lines are cubics of best fit.

Topic 40 ECR Development also showed a possibly nonlinear relationship between topic weighting and GPA, although this was less clearly the case than for Topic 7. For statements where between 0% and 20% of their words came from Topic 40 there was a reasonably flat relationship with GPA. But those statements with higher proportions from this topic showed a negative relationship between topic weighting and GPA.

Next, we explored whether environment statements that discussed more discipline-specific issues scored more highly than those which did not. In other words, we asked whether environment statements returned to, say, the Clinical Medicine UoA scored more highly when if they used more words from the Clinical Medicine topic. To investigate this we calculated the correlation between environment GPA and topic weighting for the disciplinary topic, separately for each UoA. These results are shown in Table 3 . While a large majority of these correlations were positive, indicating that environment statements that contained more discipline-specific language tended to score higher, there were systematic differences between broad subject areas. The mean correlations between the percentage of discipline-specific language and GPA for Main Panels A (medicine, health and life sciences), B (physical sciences, engineering and mathematics), C (social sciences) and D (arts and humanities) respectively were 0.472, 0.217, 0.172 and 0.093 respectively. For all main panels other than D, these means were significantly greater than zero.

Correlations between environment GPA and disciplinary topic weightings, per UoA (e.g. within the Clinical Medicine UoA, the correlation between environment GPA and weightings on Topic 42 was r = 0.560)

To compare the strength of the association between the extent to which submissions discussed disciplinary issues and their GPAs with the strength of the associations between the eight general topics discussed above and GPAs, we ran a regression analysis on submissions to the Business and Management Studies panel. We chose Business and Management as it was the panel which received the largest number of submissions (108), and so offered the greatest statistical power for an analysis of this kind. In this regression we used the eight general topics, plus the Business and Management topic (Topic 20) to predict environment GPAs. This model is shown in Table 4 . Crucially, the Business and Management topic weighting variable was a significant predictor in this model and had a standardised regression coefficient of β = 0.120, larger than that associated with Exemplification of Strategy and Process (Topic 30, β = 0.102), and roughly a third the size of Industry Partners and Funding (Topic 40, β = 0.336). In other words, using language associated with business and management was a stronger predictor of environment GPAs than giving examples of the unit’s strategy, and around a third as strong a predictor as discussing external funding and industrial partnerships.

A regression analysis predicting environment GPA of submissions to the Business and Management Studies panel, with the topic weightings from the eight general topics and the Business and Management topic

In sum, the more an environment statement included content from the relevant discipline, the higher the environment GPA it received, although this effect was more pronounced for medicine, health and life sciences, and less pronounced for the arts and humanities. To illustrate this, we analysed statements with the most and the least discipline-specific language from the ‘Biological Sciences’ and ‘Economics and Econometrics’ panels (the two panels where the relationship between discipline-specific language use and environment GPA was strongest). The differing content and emphasis were clear. For example, University College London’s Economics and Econometrics statement began with a sentence that listed its research strengths in “microeconomics, macroeconomics and econometrics” and went on to link their research focus to “the most pressing national and international socio-economic challenges of our time, such as inequality, migration, globalization, and sustainable growth” as well as international economic policy. By contrast, the opening remarks in the University of Northampton’s Economics statement focused on being a first-time submission, and noted that this new development “has been led in part by structural changes at the University level, but more significantly by the appointment of a new Dean.” Their first paragraph continued to list internal structure rather than discipline-relevant topics of strength and expertise.

Similarly, in the opening paragraph of Birkbeck’s submission to the Biological Sciences UoA, discipline-specific language was used to articulate the “fundamental biological questions” conducted by staff “who are using microbial, plant and animal systems to advance our understanding of the fundamental principles underlying molecular and cellular function, physiology and behaviour” In contrast, the opening paragraphs of the University of Worcester’s Biological Sciences statement described their first-time submission to the panel and articulated their internal structures and strategies (e.g. “the University went through an academic restructure introducing Colleges and Schools”; “The University Research Strategy 2014–19 outlined the key role played by Research Groups in operationalising plans and ambitions for excellent research”). Thus, right from the start of these statements, a focus on disciplinary contribution was much clearer in the higher-scoring submissions than those with a lower GPA.

Summary of main findings

We asked whether the perceived quality of a research environment, as measured in the UK’s Research Excellence Framework, could be predicted by the text used by that unit to describe their environment. By topic modelling the full text of all 1888 unit-level environment statements submitted to REF2021, we settled on a model that included eight specific topics that were distinct from disciplinary or geographical topics. These were related to the Internal Structure of Research Units, Career Development and EDI, Immature Research Environments, Staff Ways of Working, REF-Focused Research Strategies, Exemplifications of Staff Ways of Working, Industry Partners and Funding, and ECR Development. The proportion of words each statement included from these eight topics was surprisingly predictive of the environment score that the unit received in REF2021. Specifically, these topic proportions collectively explained 58.9% of the variance in environment GPAs, and 21.9% of the variance over and above the variance explained by the unit’s (standardised) FTE staff number, the (standardised) number of doctoral degrees it awarded, and its (standardised) grant income. In total, these metrics and the topic proportions from these eight topics collectively explained 69.1% of the variance in environment GPAs. Alongside these main findings, we also identified that environment statements that contained a lot of disciplinary-specific language tended to score higher than those which did not, although this effect was stronger for medical and biological disciplines, and weaker for the arts and humanities.

All the analyses we have reported in this paper are correlational in nature. Clearly, we were not able to experimentally manipulate the environment statements submitted to the REF and then assess the effect that these manipulations had on GPAs. Given this, care must be taken before assuming that the relationships we have reported are causal . Of particular concern is that some of our findings might be attributable to confounding factors. Indeed, in at least some cases this seems quite plausible. For instance, perhaps the reason why a research strategy focused on the REF seems to be negatively correlated with GPAs is that departments which have (relatively) low levels of research activity tend to both have a less mature research environment and also choose to write their environment statements using a higher proportion of REF terminology, as they have fewer pre-existing structures with pre-existing terminology to draw upon. We discuss this issue further below. Given this possibility of confounding factors, caution is required when interpreting our findings. Clearly, we cannot confidently draw causal conclusions in the absence of an experimental study (which would inevitably be of questionable external validity). Nevertheless, we can speculate.

REF environment scores are awarded through a process of human judgement. These are, by necessity given the volume of reading required of REF panellists, produced relatively rapidly. Many theories of human judgement emphasise how judgements are formed by comparing to-be-judged objects against prototypical instances sampled from memory (e.g. Fiedler 2000 , 2008 ; Stewart, Chater and Brown, 2006 ; Unkelbach, Fiedler and Freytag, 2007 ). Such theories would likely conceptualise reaching judgements about REF environment quality as a process which involves storing multiple exemplars of high- and low-quality statements in memory and, when encountering a new statement, generating a quality estimate by matching the features of the to-be-judged statements against those exemplars or prototypes ( Glöckner and Witteman 2010 ). This process need not be conscious, meaning that panellists are unlikely to be fully aware of the features they use to decide upon environment scores. Given this, it perhaps reasonable to suspect that if low-scoring environment statements typically have a given feature, then when panellists encounter a new statement with that feature, there may be a bias towards it receiving a lower score. In other words, if the majority of high-scoring environment statements that a panellist sees avoid using REF-heavy terminology, then in light of the decision-making literature on reasoning from prototypes and exemplars, it seems plausible that their judgements of future environment statements will be influenced by the presence or absence of this terminology, perhaps only unconsciously. If this account is correct, then the correlations between topic weightings and environment GPAs that we have reported above may well be, in part at least, causal.

How to write a ‘good’ environment statement

Assuming there remains a significant narrative element to REF people, culture and environment statements in 2029, what lessons might we learn from this analysis to support the crafting of written submissions that include all those features that are associated with high-scoring statements, and none of those features associated with low-scoring statements? We make eight recommendations.

First, we would avoid stating things that high-quality research environments would consider trivial. For example, we would not mention that most staff in our unit have doctorates, or that our staff attend academic conferences and write articles in academic journals. We would avoid using the phrase “research-active”, especially in an aspirational way, and we would not mention that our staff review articles for academic journals unless they had substantial editorial roles. In short, routine research activities should not be discussed in REF environment statements. Doing this is likely to give readers the impression that research is not a central feature of the unit’s work and reduce perceptions of the vitality of the unit’s research environment.

Second, when discussing research strategy, we would not give the impression that our research strategy is solely driven by the REF. Instead, our strategy would be organised around research centres, research groups, and departments. It would be focused on an academic discipline, not a “UoA”. The staff who led our submission would not be characterised as a “UoA Working Group”, and if we appointed other academic staff to REF leadership roles we would not mention it in our submission. Although we have robustly demonstrated that there is a relationship between conceptualising strategy using REF-centred terminology and receiving lower environment scores, it is less clear why this might be. One possibility is that the most research-intensive universities are sufficiently self-confident, and have a sufficiently long history of conducting research, to define their research activity in their own terms. In contrast, less research-intensive universities may need to create research infrastructure and strategies primarily in order to produce a respectable REF statement. If this were the case, we might expect the whole research enterprise in less research-intensive institutions to be more likely to be conceptualised in REF terms.

Third, we would not go into too much detail about the specific ways in which staff-related processes operate. For instance, we would not explain how decisions about sabbatical leave are informed by input from both the research committee and the teaching allocation committee. Similarly, details about which staff are involved at which stages in approving requests for conference travel funding would be omitted. Why might including detail of this sort be associated with lower GPAs? One plausible explanation is simply that including such details is a waste of space. As noted in the Introduction, REF environment statements are word limited, so including superfluous details might simply prevent the inclusion of content that would be causally associated with higher GPAs. This might be sufficient to generate a small negative relationship with GPAs (even though none would exist if there were no length restriction on submissions).

Fourth, we would not focus too much attention on how we support the career development of our ECRs. This finding is particularly surprising in light of the REF submission guidance’s statement that submissions should include “evidence of how individuals at the beginning of their research careers are being supported and integrated into the research culture of the submitting unit” (REF Panel Criteria and Working Methods 2019: 63). What might explain this apparent contradiction? An inspection of Figure 2 reveals that the negative relationship between discussing ECR development and GPA was driven by statements which included a relatively high proportion of this topic. Perhaps too much discussion of ECR development had the effect of crowding out space which could have been used for content that was more strongly associated with positive GPAs. Another possibility is that an excessive focus on ECR development might indicate to panellists that a unit feels that they have an unusually low proportion of senior established researchers in post.

Fifth, we would take care to make sure that we discussed career development and EDI, but not too much. The REF guidance emphasised that EDI should be discussed throughout submissions but, as shown in Figure 2 , some submissions clearly failed to follow this guidance, and these tended to receive low GPAs. However, some submissions seemed to discuss career development and EDI too much. Our analysis suggested that devoting ∼13% of the statement to this topic was optimal, with scores falling off for submissions with substantially higher or lower figures than this. Explaining why submissions which did not spend much time discussing career development and EDI tended to score poorly is straightforward: they failed to follow the clear instructions provided, and panellists may have concluded that they were poor places for minoritized colleagues to work. But why might have submissions which discussed these topics at length received lower scores? Again, one plausible account involves appealing to the length limitations of the environment statement template. Perhaps discussing career development and EDI was a qualifying criterion: not taking this issue sufficiently seriously would harm a submission, but once a submission successfully demonstrated that career development and EDI was a matter of concern, then further discussions on the topic became unnecessary. Instead, extra details on these matters had the effect of crowding out space that could have been productively used to discuss other issues associated with higher GPAs. A second possibility is that some statements mentioned EDI so often that it conveyed a ‘tick box’ approach rather than an authentic embedding. One final possibility is that an unusually high level of discussion of EDI issues might give the impression to reviewers that the unit felt that they had an unusually high number of issues in this area which required particular attention. Again, this might give the impression of a poor environment for minoritized colleagues.

Sixth, we would illustrate our research strategy by giving as many concrete examples as possible of how it has been implemented in practice. For example, if we provided pump-priming research funding to our staff, we would give an example of someone who had received funding, what they did with it, and what this led to. If we had particular strategies in place to facilitate interdisciplinary work, we would give an example of how this had led to a successful interdisciplinary workshop or funding application. If we had a policy on sharing research data, we might state the proportion of empirical papers in our submission where data had been shared online and give an example of how these datasets had been used by external colleagues in their own work. If we had a particularly generous study leave allowance for colleagues returning from parental leave, we might give an example of outputs or successful grant applications that had been produced as a result of this policy, and so on.

Seventh, we would mention our research funding and industrial partnerships as much as possible. This might seem superfluous, as panellists were provided with each unit’s grant expenditure alongside the written environment statement. However, our data suggest that mentioning funding and partnerships explained significant variance in GPAs over and above standardised grant income per FTE. 1 In sum, having high levels of grant income is insufficient: one must use it to provide evidence of a successful research strategy and environment as well as what the income enabled.

Eighth and finally, we would discuss our discipline as much as possible, particularly if we were writing a statement as part of a submission to a STEM UoA. For example, we would illustrate the success of our research strategy by discussing some of the important research findings it facilitated, we would name our research groups using well-understood disciplinary terms, and we would describe the work that our research funding allowed us to do, rather than merely state grant funding amounts. This finding that the use of discipline-specific language tends to be associated with higher environment GPAs is particularly interesting given the original suggestion of the Institution-Level Environment Panel Pilot that future REF exercises should abandon unit-level environment statements altogether ( REF 2022 ). Our finding that the scores produced by discipline-specific panellists were correlated with the amount of discipline-specific language submissions used, suggests that they may have used their domain expertise to come to decisions about submission quality. Clearly this would not be possible to the same extent, if at all, if environment were assessed at the institutional level by a panel made up of experts from a variety of disciplines—even if, as the REF (2022) pilot panel proposed, brief unit-level narratives were incorporated into the institutional-level statement. In sum, our results indicate that research environment assessed at the institutional level would likely be a different construct from research environment assessed at the unit level.

It has now been communicated that the next Research Excellence Framework will continue to assess an institution’s people, culture and environment at both institution- and discipline level, with the greater weight being given to the discipline-level assessment ( Joint UK HE Funding Bodies 2023 ). It is not yet clear the extent to which this element will look beyond the domains assessed in REF2021 (context, people, income & infrastructure, and collaboration & contribution), nor the extent to which the assessment will consist of quantitative indicators relative to narrative description. Regardless, our analysis may help institutions reflect upon what it means to have a high-quality research environment.

We are grateful to Hugues Lortie-Forgues, Victoria Simms, Steve Rothberg, and two anonymous reviewers for insightful comments on earlier drafts of this manuscript.

This work was partially supported by Research England, via an Expanding Excellence in England grant to the Centre for Mathematical Cognition, and the Economic and Social Research Council [grant number ES/W002914/1].

Conflict of interest statement. None declared.

Data associated with this manuscript are available at https://doi.org/10.17028/rd.lboro.23912499.v1 .

A regression predicting environment GPAs using only two independent variables, standardised grant income per head and the weightings associated with Topic 34 Industry Partners and Funding, could explain 36% of the variance in environment GPA. In this regression the standardised coefficients associated with the grant income metric and Topic 34 were β = 0.476 and β = 0.255 respectively, indicating that the amount of grant income a unit received was only slightly less than twice as important as how much it discussed it in its environment return.

Anthony L. ( 2022 ) AntConc (Version 4.2.0) [Computer Software] . Tokyo, Japan : Waseda University .

Google Scholar

Google Preview

Bence V. , Oppenheim C. ( 2005 ) ‘ The Evolution of the UK’s Research Assessment Exercise: publications, Performance and Perceptions ’, Journal of Educational Administration and History , 37 : 137 – 55 .

Blei D. M. , Ng A. Y. , Jordan M. I. ( 2003 ) ‘ Latent Dirichlet Allocation ’, The Journal of Machine Learning Research , 3 : 993 – 1022 .

Brassington L. ( 2022 ) Research Evaluation: Past, Present and Future . HEPI Report 152. Higher Education Policy Institute.

Brown R. , Carasso H. ( 2013 ) Everything for Sale? The Marketisation of UK Higher Education . Routledge .

Curry S. , Gadd E. , Wilsdon J. ( 2022 ) Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment . < https://rori.figshare.com/articles/report/Harnessing_the_Metric_Tide/21701624/2/files/38515103.pdf > accessed 5 Feb 2024.

Derrick G. E. , Samuel G. N. ( 2016 ) ‘ The Evaluation Scale: Exploring Decisions about Societal Impact in Peer Review Panels ’, Minerva , 54 : 75 – 97 .

Fairclough N. ( 1995 ) Critical Discourse Analysis . London : Longman .

Fiedler K. ( 2000 ) ‘ Beware of Samples! A Cognitive-Ecological Sampling Approach to Judgement Biases ’, Psychological Review , 107 : 659 – 76 .

Fiedler K. ( 2008 ) ‘ The Ultimate Sampling Dilemma in Experience-Based Decision Making ’, Journal of Experimental Psychology: Learning, Memory, and Cognition , 34 : 186 – 203 .

French N. J. , Massy W. F. , Young K. ( 2001 ) ‘ Research Assessment in Hong Kong ’, Higher Education , 42 : 35 – 46 .

Geuna A. , Martin B. R. ( 2003 ) ‘ University Research Evaluation and Funding: An International Comparison ’, Minerva , 41 : 277 – 304 .

Gillies D. ( 2008 ) How Should Research Be Organised? London, UK : College Publications .

Glöckner A. , Witteman C. ( 2010 ) ‘ Beyond Dual-Process Models: A Categorisation of Processes Underlying Intuitive Judgement and Decision Making ’, Thinking & Reasoning , 16 : 1 – 25 .

Grimmer J. , Stewart B. M. ( 2013 ) ‘ Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts ’, Political Analysis , 21 : 267 – 97 .

Jacobi C. , van Atteveldt W. , Welbers K. ( 2016 ) ‘ Quantitative analysis of large amounts of journalistic texts using topic modelling. ’, Digital Journalism , 4 : 89 – 106 .

Jensen E. A. , Wong P. , Reed M. S. ( 2022 ) ‘ How Research Data Deliver Non-Academic Impacts: A Secondary Analysis of UK Research Excellence Framework Impact Case Studies ’, Plos One , 17 : e0264914 .

Joint UK HE Funding Bodies . ( 2023 ) Research Excellence Framework 2028 Initial Decisions and issues for further consultation (REF/2028/23/01). < https://repository.jisc.ac.uk/9148/1/research-excellence-framework-2028-initial-decisions-report.pdf > accessed 5 Feb 2024.

Jones P. , Sizer J. ( 1990 ) ‘ The Universities Funding Council’s 1989 Research Selectivity Exercise ’, Beiträge Zur Hochschulforschung , 4 : 309 – 48 .

Kellard N. M. , Śliwa M. ( 2016 ) ‘ Business and Management Impact Assessment in Research Excellence Framework 2014: analysis and Reflection ’, British Journal of Management , 27 : 693 – 711 .

King’s College London and Digital Science ( 2015 ) The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies . Bristol, United Kingdom : HEFCE .

Manville C. , d’Angelo C. , Culora A. , Gloinson E.R. , Stevenson C. , Weinstein N. , Wilsdon J. , Haddock G. , Guthrie S. ( 2021 ) Understanding Perceptions of the Research Excellence Framework among UK Researchers . Cambridge, UK : RAND Corporation .

Manville C. , Guthrie S. , Henham M.-L. , Garrod B. , Sousa S. , Kirtley A. , Castle-Clarke S. , Ling T. ( 2015 ) Assessing Impact Submissions for REF 2014: An Evaluation . Cambridge, UK : RAND Corporation .

Marques M. , Powell J. J. , Zapp M. , Biesta G. ( 2017 ) ‘ How Does Research Evaluation Impact Educational Research? Exploring Intended and Unintended Consequences of Research Assessment in the United Kingdom, 1986–2014 ’, European Educational Research Journal , 16 : 820 – 42 .

Matthews A. , Kotzee B. ( 2022 ) ‘ Bundled or Unbundled? A Multi‐Text Corpus‐Assisted Discourse Analysis of the Relationship between Teaching and Research in UK Universities ’, British Educational Research Journal , 48 : 578 – 97 .

McCallum A. K. ( 2002 ) MALLET: MAchine Learning for LanguagE Toolkit . < http://mallet.cs.umass.edu > accessed 5 Feb 2024.

Mellors-Bourne R. , Metcalfe J. , Gill A. ( 2017 ) Exploring Equality and Diversity using REF2014 Environment Statements , CRAC LTD Report to HEFCE.

Pardo-Guerra J. P. ( 2022 ) The Quantified Scholar: How Research Evaluations Transformed the British Social Sciences . Columbia University Press .

Pinar M. , Horne T. J. ( 2022 ) ‘ Assessing Research Excellence: evaluating the Research Excellence Framework ’, Research Evaluation , 31 : 173 – 87 .

Poppler ( 2022 ) [Computer Software] < https://poppler.freedesktop.org > accessed 5 Feb 2024.

REF ( 2019 ) REF Panel Criteria and Working Methods . < https://archive.ref.ac.uk/media/1450/ref-2019_02-panel-criteria-and-working-methods.pdf > accessed 5 Feb 2024.

REF ( 2022 ) REF Institutional-level Environment Pilot: Report of the Pilot Panel . < https://archive.ref.ac.uk/media/1908/ref-2021-ilepp-final-report.pdf > accessed 5 Feb 2024.

Reichard B. , Reed M. S. , Chubb J. , Hall G. , Jowett L. , Peart A. , Whittle A. ( 2020 ) ‘ Writing Impact Case Studies: A Comparative Study of High-Scoring and Low-Scoring Case Studies from REF2014 ’, Palgrave Communications , 6 : 1 – 17 .

Sivertsen G. ( 2017 ) ‘ Unique, but Still Best Practice? The Research Excellence Framework (REF) from an International Perspective ’, Palgrave Communications , 3 : 17078 .

Stewart N. , Chater N. , Brown G. ( 2006 ) ‘ Decision by Sampling ’, Cognitive Psychology , 53 : 1 – 26 .

Sutton E. ( 2020 ) ‘ The Increasing Significance of Impact within the Research Excellence Framework (REF) ’, Radiography , 26 : S17 – S19 .

Swedish Government . ( 2016 ) Regeringens Proposition 2016/17:50 . Kunskap i samverkan—för samhällets utmaningar och stärkt konkurrenskraft .

Terämä E. , Smallman M. , Lock S. J. , Johnson C. , Austwick M. Z. ( 2016 ) ‘ Beyond Academia–Interrogating Research Impact in the Research Excellence Framework ’, Plos One , 11 : e0168533 .

Thomas D. A. , Nedeva M. , Tirado M. M. , Jacob M. ( 2020 ) ‘ Changing Research on Research Evaluation: A Critical Literature Review to Revisit the Agenda ’, Research Evaluation , 29 : 275 – 88 .

Thorpe A. , Craig R. , Tourish D. , Hadikin G. , Batistic S. ( 2018a ) ‘ Environment’ Submissions in the Uk's Research Excellence Framework 2014 ’, British Journal of Management , 29 : 571 – 87 .

Thorpe A. , Craig R. , Hadikin G. , Batistic S. ( 2018b ) ‘ Semantic Tone of Research ‘Environment’submissions in the UK’s Research Evaluation Framework 2014 ’, Research Evaluation , 27 : 53 – 62 .

Unkelbach C. , Fiedler K. , Freytag P. ( 2007 ) ‘ Information Repetition in Evaluative Judgements: Easy to Monitor, Hard to Control ’, Organizational Behavior and Human Decision Processes , 103 : 37 – 52 .

Watermeyer R. , Derrick G. E. ( 2022 ) ‘ Affective Auditing: The Emotional Weight of the Research Excellence Framework on Middle Management ’, Research Evaluation , 31 : [rvac041]. 498 – 506 . https://doi.org/10.1093/reseval/rvac041

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

National Research Council (US) and Institute of Medicine (US) Committee on Assessing Integrity in Research Environments. Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct. Washington (DC): National Academies Press (US); 2002.

Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct.

• Hardcopy Version at National Academies Press

3 The Research Environment and Its Impact on Integrity in Research

To provide a scientific basis for describing and defining the research environment and its impact on integrity in research, it is necessary to articulate a conceptual framework that delineates the various components of this environment and the relationships between these factors. In this chapter, the committee proposes such a framework based on an opensystems model, which is often used to describe social organizations and the interrelationships between and among the component parts. This model offers a general framework that can be used to guide the specification of factors both internal and external to the research organization that is relevant to understanding integrity in research.

After its review of the literature, the committee found that there is little empirical research to guide the development of hypotheses regarding the relationships between environmental factors and the responsible conduct of research. Thus, the committee drew on more general theoretical and research literature to inform its discussion. Relevant literature was found in the areas of organizational behavior and processes, ethical cultures and climates, moral development, adult learning and educational practices, and professional socialization. 1

• THE OPEN-SYSTEMS MODEL

The open-systems model depicts the various elements of a social organization; these elements include the external environment, the organizational divisions or departments, the individuals comprising those divisions, and the reciprocal influences between the various organizational elements and the external environment (Ashforth, 1985; Beer, 1980; Daft, 1992; Harrison, 1994; Katz and Kahn, 1978; Schneider and Reichers, 1983). The underlying assumptions of the open-systems model and its various elements are as follows (Harrison, 1994):

External conditions influence the inputs into an organization, affect the reception of outputs from an organization's activities, and directly affect an organization's internal operations.

All system elements and their subcomponent parts are interrelated and influence one another in a multidirectional fashion (rather than through simple linear relationships).

Any element or part of an organization can be viewed as a system in and of itself.

There is a feedback loop whereby the system outputs and outcomes are used as system inputs over time, with continual change occurring in the organization.

Organizational structure and processes are in part determined by the external environment and are influenced by the dynamics between and among organizational members.

An organization's success depends on its ability to adapt to its environment, to tie individual members to their roles and responsibilities within the organization, to conduct its processes, and to manage its operations over time.

• THE OPEN-SYSTEMS MODEL OF RESEARCH ORGANIZATIONS

Figure 3-1 shows the application of the open-systems model to the research environment, which can include public and private institutions, such as research universities, medical schools, and independent research organizations. As noted above, any element or part of an organization can be viewed as a system in and of itself. For research organizations, then, this includes not only the institution itself, but also any of its departments, divisions, research groups, and so on. Figure 3-1 illustrates the research environment as a system that functions within an external environment, whereas Figure 3-2 depicts the specific factors within the external environment and their influence on the research organization. These factors within the external environment are discussed later in this section.

Open-systems model of the research organization. This model depicts the internal environmental elements of a research organization (white oval), showing the relationships among the inputs that provide resources for organizational functions, the structures (more...)

Environmental influences on integrity in research that are external to research organizations. The external-task environment includes all of the organizations and conditions that are directly related to an organization's main operations and technologies. (more...)

An organization's internal environment consists of a number of key elements—specifically, the inputs that provide resources for organizational functions, the organizational structure and processes that define an organization's setup and operations, and the outputs and outcomes that are the results of an organization's activities. The system is dynamic, and, as indicated by the feedback arrow in Figure 3-1 , outputs and outcomes affect future inputs and resources. However, all of these components exist within the context of an organization's culture and specific climate dimensions—that is, the prevailing norms and values that inform individuals within the organization about acceptable and unacceptable behaviors. With respect to the committee's focus on integrity in research, the ethical dimension of the organizational culture and climate is very important.

Structurally, organizations are compartmentalized into various subunits, including work groups or divisions (the research group or team), along with other defined sets of organizational activities and responsibilities (e.g., programs that educate members about the responsible conduct of research, institutional review boards [IRBs], and mechanisms for disclosing and managing conflicts of interest). The operation of these programs and their overall effectiveness influence researchers' perceptions of the organization's ethical climate. Individuals within an organization exist both within and across these defined groups and sets of activities. Given this, it is important to differentiate between an organizational level of analysis (e.g., the research university, medical school, and independent research organization) vis-à-vis the group level of analysis (e.g., the research group or team) and the individual level of analysis (e.g., the individual scientist or researcher).

Inputs and Resources

In its examination of research environments, the committee focused on two input and resource factors of importance: the levels and sources of funding for scientific research, and the characteristics of human resources. These inputs and resources are obtained from an organization's external environment and are used in the production of an organization's outputs.

The research funding that an organization receives is distributed to research groups or teams and to individual scientists. Funding levels may increase and decrease over the years, both for the organization as a whole and for individual research groups. Just as the overall level of funding available for research within society affects the scientific enterprise as a whole, the level of funding coming into a particular research organization or research team also affects behavior.

The impacts that the level of funding and the competition over funding have on the responsible conduct of research are not clearly understood. There is some limited evidence that in highly competitive environments, individuals with a high “competitive achievement striving” are at risk for engaging in misconduct, particularly when they are faced with situations in which their expectations for success cannot be reached by exerting additional effort (Heitman, 2000; Perry et al., 1990). Encouraging a high level of individual integrity in research, despite vigorous competition for funding, presents a significant challenge for research organizations.

Human Resources

The human resources available to a research organization are also important to the analysis of integrity in research. The background characteristics of scientists coming into a research organization influence its structure and processes as well as its overall culture and climate, and these factors, in turn, influence the responsible conduct of research by individual scientists. Scientists (whether they are trainees, junior researchers, or senior researchers) entering into a research organization will have competing professional demands (e.g., research, teaching, practice, and professional service), and thus there are likely to be conflicting commitments. The dynamics of these competing demands and conflicting commitments change as individual scientists become integrated into the research organization, taking on specific roles and responsibilities.

Also, scientists enter into an organization with various educational and cultural backgrounds. They have different conceptions of the collaborative and competitive roles of the scientist, different abilities to interpret the moral dimensions of problems, and different capacities to reason about and effectively resolve ethical problems. These individual differences will influence organizational behavior, in general, and research conduct, in particular, in complex and dynamic ways.

Given this variation in human resource input into the research organization, it is particularly important for institutions to socialize newcomers and provide them with an understanding of the organization and how to act within it. As in any organization, newcomers must learn the logistics of their organization, the general expectations of their roles by peers, the formal and informal norms governing behavior, the status and power structures, the reward and communication systems, various organizational policies, and so on (Katz, 1980). Within research organizations, individual differences are complicated by the international nature of the scientific workforce and the corresponding sociocultural differences. Therefore, it is particularly important for research institutions to create an environment in which scientists are able to gain an awareness of the responsible conduct of research as it is defined within the culture, to understand the importance of professional norms, to acquire the capacity to resolve ethical dilemmas, and to recognize and be able to address conflicting standards of research conduct.

Organizational Structure and Processes

To better understand the impact of the research environment on integrity in research, it is important to focus on the organizational elements that characterize its structure—those elements that are more enduring and less prone to change on a day-to-day basis. These elements include an organization's policies and procedures; the roles and responsibilities of members of the organization; decision-making practices; mission, goals and objectives, including the strategies and plans of the organization; and technology.

Policies, Procedures, and Codes The formalization of policies and practices to support the responsible conduct of research is important in the analysis of research environments and their influence on integrity in research. Chapter 2 identified a number of the practices that are essential to the research environment. Specifically, a research organization should have explicit (versus implicit or nonexistent) procedures and systems in place to fairly (1) monitor and evaluate research performance, (2) distribute the resources needed for research, and (3) reward achievement. These policies and procedures should include criteria related to the responsible conduct of research that are applied consistently. Furthermore, research organizations support integrity in research when they have efficient and effective systems in place to review research involving humans and animals, manage conflicts of interest, respond to misconduct, and socialize trainees and other scientists into responsible research practices. The specification of these policies and procedures helps to regulate and maintain group control and reduce uncertainty about acceptable and unacceptable behaviors (Hamner and Organ, 1978).

Research has shown that strongly implemented and embedded ethical codes of conduct within organizations are associated with ethical behavior in the workplace. McCabe and Pavela (1998) describe the University of Maryland at College Park as one example where implementation of a strong “modified” 2 honor code has proven to be a successful strategy for creating a culture where cheating is viewed as socially unacceptable. Major elements of the Maryland model include (1) involving students in educating their peers and resolving academic dishonesty allegations, (2) treating academic integrity as a moral issue, and (3) promoting enhanced student-faculty contact and better teaching. The mere presence of an honor code, however, is generally not sufficient. Rather, the honor code is used as a vehicle to create a shared understanding and acceptance of the policies on academic integrity among both faculty and students (McCabe and Trevino, 1993).

Corporate codes have a similar effect in the workplace. An original study by McCabe demonstrated that self-reported unethical behavior was lower for survey respondents who worked in a company with a corporate code of conduct (McCabe et al., 1996). Self-reported unethical behavior was inversely correlated with the degree to which the codes were embedded in corporate philosophy and the strength with which the code was implemented (determined by survey questionnaire of employee perceptions).

Roles and Responsibilities The specification of roles and responsibilities within various research groups and teams and relevant research programs (e.g., education in the responsible conduct of research, IRBs, and conflict-of-interest review committees) provides a blueprint for researcher behavior. It is particularly important to clearly define researchers' responsibilities related to the responsible conduct of research. Furthermore, the relative positions of these responsibilities within the organizational hierarchy and the status of persons who operate them will send a clear message to the research community about the importance of such endeavors. For example, if a highly respected scientist with high status spearheads the program of education in the responsible conduct of research, and sufficient resources (in terms of both staff and financial resources) are available to carry out the program's work, then there is a greater likelihood that its efforts will be taken seriously. Again, these factors have great symbolic value within the organization and provide compelling images of the organization's ethical culture, which affects the degree to which members of the organization will internalize the norms associated with the responsible conduct of research (Pfeffer, 1981; Siehl and Martin, 1984).

Decision-Making Practices How an organization reaches decisions and formulates policies will affect individuals' perceptions of these policies and their behavioral compliance with them. Individuals are more likely to accept and adhere to policies and practices when they have played a role in their development and implementation. Hence, scientists are more likely to buy into various research policy decisions that are reached through a collaborative process among key stakeholder groups, rather than being imposed by a top-level centralized authority (Anderson et al., 1995, Saraph et al., 1989). Organizational research that focuses on the pursuit of quality and that explicitly values cooperation and collaboration to achieve maximum effectiveness leads to better decisions, higher quality, and higher morale within an organization (NIST, 1999). Classically, faculty and administrators both have governing roles in academic institutions, and this shared responsibility facilitates the bottom-up establishment of rules of research behavior.

Missions, Goals and Objectives, and Strategies and Plans The mission and goals of an organization specify its desired end states (e.g., becoming a “best-practice” site in terms of the protection of human research subjects). Objectives are the specific targets and indicators of goal attainment (e.g., becoming an accredited program and receiving recognitions and awards through scientific associations). Strategies and plans are the overall routes and specific courses of action (e.g., allocating the resources to comply with the standards for accreditation and ensuring that the program has leadership support) to the achievement of goals. If the responsible conduct of research is a prominent part of the mission and goals of a research organization, along with associated objectives, strategies, and plans, then the prominence of this issue sets the tone for the organization's ethical climate and sends a message to scientists that the responsible conduct of research is important. Research has shown that the most successful organizations are those that have a vision and goals that are clearly defined, consistent, and shared among their members (Anderson et al., 1995; Deming, 1986; Freuberg, 1986; Hackman and Wageman, 1995).

Technology An organization's technology offers the methods for transforming system resources into system outputs. It consists of such aspects of an organization's infrastructure as facilities, tools and equipment, and techniques. These aspects can be mental and social, mechanical, chemical, physical, or electronic. Research environments not only need the necessary tools and equipment for their respective types of scientific research, but they must also establish technologies (e.g., accounting systems and library and information retrieval systems) within the organization for the effective and efficient operation of the research. There may be competition within an organization to acquire the various forms of technology that are of sufficient quantity and quality to facilitate research production. The availability of this technology may, in turn, attract highly skilled scientists who hope to carry out research at the cutting edge of technology. As already mentioned, the effective management of competition—in this case, for technologies—is an important element of promoting the responsible conduct of research.

Organizational processes, as opposed to an organization's more stable and enduring structural elements, are the patterned forms of interaction between and among groups or individuals within an organization. Processes represent the dynamic aspects of an organization. The processes that characterize organizational dynamics are too numerous to mention here. However, in the committee's examination of research organizations, the processes of most interest consist of (1) leadership, (2) competition, (3) supervision, (4) communication, (5) socialization, and (6) organizational learning.

Leadership The level of support for high ethical standards by the leadership of an organization or research group can vary; leaders can be extremely supportive, can show ambivalence, or can be nonsupportive. Leaders at every level serve as role models for organizational members and set the tone for an organization's ethical climate (Ashforth, 1985; OGE, 2000; Treviño et al., 1996). Therefore, when leaders support high ethical standards, pay attention to responsible conduct of research, and are openly and strongly committed to integrity in research, they send a clear message about the importance of adhering to responsible research practices (Wimbush and Shepard, 1994). Considerable evidence from the organizational research literature supports the relationship between supervisor behavior and the ethical conduct of the members of an organization (Posner and Schmidt, 1982, 1984; Walker et al., 1979). Supervisors provide a model for how subordinates should act in an organization. Furthermore, supervisors have a primary influence over their subordinates, an influence that is greater than that of an ethics policy. Even if a company or profession has an ethics policy or code of conduct, subordinates follow the leads of their supervisors (Andrews, 1989).

Competition The extent to which the organization is highly competitive, along with the extent to which its rewards (e.g., funding, recognition, access to quality trainees, and power and influence over others) are based on extramural funding and short-term research production, may have negative impacts on integrity in research. Evidence from organizational research indicates that reward systems based on self-interest and commitment only to self rather than to coworkers and the organization are negatively associated with ethical conduct (Kurland, 1996; Treviño et al., 1996). In addition, the level of unethical behavior increases in organizations where there is a high degree of competitiveness among workers (Hegarty and Sims, 1978, 1979). Given these facts, one might expect that a research environment in which competition for resources is fierce and rewards accrue to those who produce the most over the short term sends a wrong message, a message that says “produce at all costs.”

Creating a reward system and policies that promote being the “best” within the scientific enterprise, and within a context that encourages the responsible conduct of research, represents a challenge in research environments.

Supervision The extent to which research behavior is monitored and quality control systems are operational will affect the level of adherence to ethical standards. Scientists need to see that policies about responsible research behavior are not just window dressing and that the organization has implemented practices that follow up stated policies. Consistency between words and deeds encourages the members of an organization to take policies seriously. Organizations vary widely in terms of their efforts to communicate codes of conduct to members, as well as to implement mechanisms to ensure compliance. When implementation is forceful and the policies and practices become deeply embedded in an organization's culture, there is a greater likelihood that they will be effective in preventing unethical behavior (McCabe and Treviño, 1993; Treviño, 1990; OGE, 2000).

Communication Communication among members of a research organization or research group that is frequent and open, versus infrequent and closed, should have a positive influence on integrity in research. A positive ethical climate is supported by open discussions about ethical issues (Jendrek, 1992; OGE, 2000). Frequent and open communication enhances awareness of issues, encourages individuals to seek advice when faced with ethical dilemmas, and establishes the importance of resolving issues before they become something to be hidden.

Socialization Mentoring relationships between research trainees and their advisers are important in the socialization of young scientists (Anderson et al., 2001; Swazey and Anderson, 1998). These relationships can be characterized by a variety of factors, including the level of trust, communication patterns, and the fulfillment of responsibilities as a mentor or trainee. In addition to mentoring relationships, education in research and professional ethics is an aspect of socialization (Anderson, 1996; Anderson and Louis, 1994; Anderson et al., 1994; Louis et al., 1995; Swazey et al., 1993). Socialization practices can be formal or informal, but they are essential to helping individuals internalize the norms and values associated with the responsible conduct of research. Research that examines the effect of more formalized methods of socialization—for example, education—reveals that interactive techniques (e.g., case discussion, roleplaying, and hands-on practice sessions) are generally more effective in producing behavioral change than are activities with minimal participant interaction or discussion (e.g., lectures or presentations [Davis et al., 1999]). Furthermore, sequenced education has a greater impact than single educational sessions (Davis et al., 1999; OGE, 2000). These findings substantiate the principles of adult education; these principles describe successful practices as being learner-centered, active rather than passive, relevant to the learner's needs, engaging, and reinforcing (Brookfield, 1986; Cross, 1981; Knowles, 1970) ( Chapter 5 ).

Organizational Learning Organizations that learn from their operations and that continuously seek to improve their performance are better able to adapt to a changing environment (Anderson et al., 1994; Deming, 1986; Hackman and Wageman, 1995; Schön, 1983). All organizations change over time, but for some this can be an excruciating and painful process if it comes about through reaction to a crisis situation. For example, when a research subject dies or a researcher is accused of data fabrication, the organization should respond immediately. However, this response is focused on crisis intervention rather than prevention. On the other hand, organizations that have mechanisms in place to continuously evaluate the efficiency and effectiveness of their programs and activities are more likely to build a preventive maintenance system (Fiol and Lyles, 1985; Schön, 1983). Furthermore, if the members of an organization have a voice in the design and implementation of such systems, then they are more likely to accept and be cooperative with the continual evaluative processes.

Culture and Climate

All of the enduring elements and features of an organization's structure and its more dynamic processes exist within the context of an organization's culture and climate. In fact, an organization's structure and processes help to create the culture and climate inasmuch as they are shaped by them (Ashforth, 1985). An organization's culture consists of the set of shared norms, values, beliefs, and assumptions, along with the behavior and other artifacts (e.g., symbols, rituals, stories, and language) that express these orientations.. Culture and climate factors are characteristics of an organization that guide members' thoughts and actions (Schneider, 1975).

The ethical (or moral) climate is one component of an organization's culture and is particularly relevant in the analysis of integrity in research (Victor and Cullen, 1988). This climate is defined as the prevailing moral beliefs (i.e., the prescribed behaviors, beliefs, and attitudes within the community and the sanctions expressed) that provide the context for conduct. The stable, psychologically meaningful, and shared perceptions of the members of an organization are used as indicators of ethical climate, which may exist both at the organizational level and at the research group or team level (Schneider, 1975; Schneider and Reichers, 1983).

An ethical climate that supports the responsible conduct of research is created when scientists perceive that adherence to ethical standards takes precedence and that sanctions for ethical violation are consistently applied. Research in this area has established that the factors within an organization that are most strongly related to ethical behavior are attention to ethics by supervisors and organizational leadership, consistency between policies and practices, open discussions about ethics, and followup of reports of ethics concerns (OGE, 2000). These features of an organization can help establish an ethical climate in which organizational members perceive that the responsible conduct of research is central to the organization's practice and that it is not something to be worked around. It creates an environment in which a code of conduct is strongly implemented and deeply embedded in the community's culture (Treviño, 1990).

Outputs and Outcomes

The outputs of research organizations are produced at all levels—the organizational level, the research group or team level, and the individual scientist level. The outputs are the products produced, the services delivered, and the ideas developed and tested. The most obvious outputs are the number and quality of research projects completed, reports written, publications produced, patents filed, and students graduated.

For the committee's purposes, however, it is important to focus on the outputs of activities or programs related to integrity in research—for example, institutional review boards, conflict-of-interest review committees, and programs that provide education in the responsible conduct of research. Outputs from these programs are generally measured in terms of the quantity and the quality of activities—for example, the number of workshops and seminars offered, the number of scientists who participate, and the number of research proposals reviewed by IRBs and the dispositions of those proposals. Research organizations that design and implement high-quality activities related to integrity in research—and in a quantity that is sufficient to meet their needs—are more likely to achieve the outcomes that they seek (e.g., adherence to responsible research practices). Although these activities will not be the sole factors that determine the responsible conduct of research, their implementation becomes a symbol for the members of an organization, serving as an indicator of the leadership's commitment to the establishment of a culture and a climate that supports the responsible conduct of research.

The outcomes of organizational activities refer to the specific results that reflect the achievement of goals and objectives. As with organizational outputs, outcomes can be associated with the organization as a whole, the research group, or the individual scientist. However, the committee's primary interest is in the individual scientist's level of integrity in research. As discussed in Chapter 2 , the committee defines integrity in research as the individual scientist's adherence to a number of normative practices for the responsible conduct of research.

Adherence to these practices provides a set of behavioral indicators of an individual's integrity in research. However, behavioral compliance is assumed to be associated with an understanding of the norms, rules, and practices of science. In addition, judgments about an individual's integrity are based on the extent to which intellectual honesty, accuracy, fairness, and collegiality consistently characterize the dispositions and attitudes reflected in a researcher's practice. Judgments about a person's integrity are less about strict adherence to the rules of practice and are more about the disposition to be intellectually honest, accurate, and fair in the practice of science (i.e., in the willingness to admit and correct one's errors and shortcomings).

The committee resisted defining integrity in terms of (1) adherence to the normative practices listed in Chapter 2 , (2) the knowledge and awareness of the practices of responsible research, and (3) the attitudes and orientation toward the practices of responsible research (i.e., the degree to which individuals agree with the practices, the level of importance that they attach to them, and the extent to which they are subject to conflicting sets of practices), as has been common in the social sciences. 3 These three conceptually distinct categories of outcomes fail to capture the complexity of the process through which individuals interact with their environment and make ethical decisions. One simply cannot assume that as scientists gain awareness of standards of practice, they will be positively oriented to them or will be more likely to adhere to the behavioral requirements. The committee recognizes that although researchers might be well intentioned, there is truth in what psychologists (Rest, 1983) have observed: that everyone is capable of missing a moral issue (moral blindness); developing elaborate and internally persuasive arguments to justify questionable actions (defective reasoning); failing to prioritize a moral value over a personal one (lack of motivation or commitment); being ineffectual, devious, or careless (character or personality defects, often implied when someone is referred to as “a jerk”); or having ineffectual skills at problem solving or interpersonal communication (incompetence).

For this reason, focusing on the processes that give rise to the responsible conduct of research are important individual-level outcomes of organizational activities within the research environment. Components of the process of ethical decision making include ethical sensitivity, reasoning, moral motivation and commitment, and character and competence (Bebeau, 2001). Educational programs that train scientists in the responsible conduct of research are often premised on the assumption that these essential capacities for ethical decision making are well developed by the time individuals begin their research education, and that one simply needs to teach the rules of the responsible conduct of research. Research on ethical development in the professions demonstrates that even mature professionals show considerable variability on performance assessments that measure ethical sensitivity, moral reasoning and judgment, professional role orientation, and appropriate character and competence to implement action plans effectively.

Therefore, if a research environment implements educational programs to foster integrity in research, then these programs should promote sensitivity to issues that are likely to arise in the research setting by building a capacity for reasoning carefully about conflicts inherent in proposing, conducting, and reporting research; by developing a sense of personal identity that incorporates the norms and values of the research culture; and by building competence in problem solving and interpersonal communication (see Chapter 5 for further discussion).

External Environment

The external environment of a research organization consists of both an external-task environment and a general environment ( Figure 3-2 ). The external-task environment includes all the organizations and conditions that are directly related to an organization's main operations and its technologies. The systems and subsystems of the external-task environment are embedded within the larger sociocultural, political, and economic environment and have a more indirect impact on an organization. It is important to recognize that relationships also exist between and among all elements within the external environment. For example, government policies and regulations can affect the areas and levels of funding. Journal policies can be affected by decisions made within scientific associations, and these decisions can be driven by government regulation (or pending regulation).

External-Task Environment

A number of factors within the external-task environment have a significant impact on scientists' responsible conduct of research. These factors include government regulation, funding for scientific work, job opportunities for trainees and researchers, journal policies and practices, and the policies and practices of scientific societies.

Government Regulation Governmental bodies, particularly at the federal level, have been promulgating regulations concerning the conduct of research for many years. Most widely known and recognized are the regulations regarding the protection of human research subjects (45 C.F.R. § 46, 1999; 21 C.F.R. § 50 and 56, 1998) and the protection of animals in research (7 U.S.C. §§ 2131, 1966, et seq.). Furthermore, regulations have been promulgated regarding the evaluation of allegations and the reporting of scientific misconduct (42 C.F.R. § 50, §§A, 1989; Federal Register , 2000) and the handling and disposal of hazardous chemicals in the laboratory (29 C.F.R. § 1910.1450, 1996), to name just two. As these government regulations come into force, they have direct impacts on a research organization and individual scientists. Specifically, organizations and individuals must be in compliance with the regulations or face sanctions.

Funding for Scientific Work Research organizations are directly affected by both the level and the source of funding that is available for scientific work (e.g., they are affected by the balances between government and corporate support and between industry and foundation support). Most funding sources provide support for specific research proposals rather than particular investigators. Although proposals are usually ranked on a relative scale, more typically they are funded in an all-or-none fashion. At the same time, funding needs always outpace funding opportunities. For instance, only one in three investigator-initiated grant proposals (see http://silk.nih.gov/public/[email protected]. dsncc ) to the National Institutes of Health is successful. In this situation, even investigators who succeed in their research sometimes lose funding, a fate that threatens the very existence of their projects and often threatens their personal incomes.

The task for research organizations is to develop structures that help their scientists deal with this competitive research situation while maintaining the responsible conduct of research. Similarly, when corporate or industry funds are involved, research organizations should require strategies for the management and disclosure of conflicts of interest to reduce problems related to publication rights, ownership of intellectual property, and research involving human subjects.

Job Opportunities When the job market is tight and there is more competition for every research position, researchers will be pressured to achieve higher levels of productivity and recognition. This situation challenges scientists to be the best while maintaining the highest levels of integrity in research. Similarly, research programs must compete for students and postdoctoral fellows, who, in turn, enhance a program's accomplishments and overall status. The ability of researchers to gain recognition often is believed to be the best path to attracting high-quality trainees to a program. The organizational challenge is to help researchers develop competitive programs while maintaining a high level of commitment to integrity in research.

Journal Policies and Practices Journal editors can be more or less rigorous in their implementation of the review process and the extent to which they insist on high levels of adherence to scientific standards. Furthermore, journals may have specific policies in such areas as authorship practices, disclosure of conflicts of interest, duplicate publication, and reporting of research methodologies. The scientific community receives an important message about integrity in research when journal policies and practices regarding these practices are clear and are required as a condition of publication—and when the most prestigious journals adopt such practices. For example, members of the International Committee of Medical Journal Editors recently revised their submission policies related to industry-sponsored research. Authors are now required to sign a statement accepting full responsibility for the conduct of a clinical trial, and they must confirm that they had access to the original data and had full control over the decision to publish (Davidoff et al., 2001).

Policies and Practices of Scientific Societies Scientific societies are in a position to influence the behaviors of their members in ways that could promote integrity in research 4 (AAAS, 2000). The societies vary extensively, however, in their development of codes of conduct, their enforcement of such codes, and their socialization of members with regard to these standards of behavior. To aid in this process, the Association of American Medical Colleges has published a guide to help societies in the development of ethical codes (AAMC, 1997). Other associations develop standards for accreditation—for example, standards for science education programs, research laboratories, and programs for the protection of human and animal research subjects. These accreditation standards generally have specific statements regarding the responsible conduct of research and stipulate the structures within the organization that must be in place to ensure compliance with the standards. Scientists who are part of such accredited programs will be subject to the influences of these external controls.

General Environment

The general environment has an indirect impact on an organization. This environment includes all of the conditions and institutions that have sustained or infrequent impacts on the organization and its functions (Harrison, 1994). Included are the state or conditions of major social institutions (e.g., the economy, political system, educational system, science and technology system, and legal system) as well as the local, national, and international cultures within which an organization operates. The general public, and more specifically the effects of public trust in the research enterprise, are also important components of the general environment. As reflected in Figure 3-2 , the organizations and conditions of the external-task environment (unshaded circles) are embedded within this larger environment (shaded area).

An example of how the broader environment can affect the conduct of research is the recent national debate over embryonic stem cell research; this debate reflects a clash of values that affect the characterization of ethical or unethical research (NAS, 2001; National Bioethics Advisory Commission, 1999). In another instance, the new rules governing the privacy of health records that are part of the Health Insurance Portability and Accountability Act are being challenged by scientists as too restrictive in providing access to identifiable data for research (AAMC, 2001; Annas, 2002). Also, society places a high premium on human rights and the protection of vulnerable persons, values that have been translated into federal regulations for the protection of human research subjects (45 C.F.R. § 46, 1993, and 21 C.F.R. § 50 and 56, 1981).

Other social institutions also have an indirect impact on research environments. Educational systems produce scientists, and these systems affect not only their quantity but also their quality and how well they have been socialized into professional standards of conduct. The technology systems determine the availability of equipment and the methods used to carry out various types of research, factors that may raise questions about the propriety of certain research endeavors. Ethical conflicts are often created when the development of new technologies requires an answer to the question of whether what can be done should be done. Finally, the legal system and the propensity in the United States to resort to litigation may bring about situations in which scientists are caught between the responsible conduct of research and subpoenas for confidential data. These examples are by no means exhaustive, but they reflect the ways in which major social institutions and cultural values can affect research organizations and a scientist's practice of research.

The committee found no comprehensive body of research or writing that can guide the development of hypotheses regarding the relationships between the research environment and the responsible conduct of research. However, viewing the research environment as an open-systems model, which is often used in general organizational and administrative theory, makes it possible to hypothesize how various components affect integrity in research. Inputs of funds and other resources can influence behavior both positively and negatively. The organizational structure and processes that typify the mission and activities of an organization can either promote or detract from the responsible conduct of research. The culture and climate that are unique to an organization both promote and perpetuate certain behaviors. Finally, the external environment, over which individuals and, often, institutions have little control, can affect behavior and alter institutional integrity for better or for worse.

• AAAS (American Association for the Advancement of Science). 2000. The Role and Activities of Scientific Societies in Promoting Research Integrity . A report of a conference, April 10, 2000, Washington, DC. [Online]. Available: http://www.aaas.org/spp/dspp/sfrl/ projects/integrity.htm [Accessed January 7, 2002].
• AAMC (Association of American Medical Colleges). 1997. Developing a Code of Ethics in Research: A Guide for Scientific Societies . Washington, DC: AAMC.
• AAMC. 2001. Letter to Secretary Thompson on Impact of Medical Privacy Rule on Research . [Online]. Available: http://www.aamc.org/advocacy/corres/research/thompson. htm [Accessed February 1, 2002].
• Anderson J, Rungtusanatham M, Schroeder R, Devaraj S. 1995. A path analytic model of a theory of quality management underlying the Deming management method: Preliminary empirical findings. Decision Sciences26:637–658.
• Anderson MS. 1996. Misconduct and departmental context: Evidence from the Acadia Institute's Graduate Education Project. Journal of Information Ethics5(1):15–33.
• Anderson MS, Louis KS. 1994. The graduate student experience and subscription to the norms of science. Research in Higher Education35:273–299.
• Anderson MS, Louis KS, Earle J. 1994. Disciplinary and departmental effects on observations of faculty and graduate student misconduct. Journal of Higher Education65:331– 350.
• Anderson MS, Oju EC, Falkner TMR. 2001. Help from faculty: Findings from the Acadia Institute Graduate Education Study. Science and Engineering Ethics7:487–503. [ PubMed : 11697005 ]
• Andrews KR. 1989. Ethics in practice. Harvard Business Review Sept-Oct:99–104. [ PubMed : 10295476 ]
• Annas GJ. 2002. Medical privacy and medical research—judging the new federal regulations. New England Journal of Medicine346:216–220. [ PubMed : 11796863 ]
• Ashforth BE. 1985. Climate formation: Issues and extensions. Academy of Management Re view10:837–847.
• Bebeau MJ. 2001. Influencing the moral dimensions of professional practice: Implications for teaching and assessing for research integrity. In: Steneck NH, Scheetz MD, eds. [2000]. Investigating Research Integrity: Proceedings of the First ORI Research Conference on Research Integrity . Washington, DC: Office of Research Integrity, U.S. Department of Health and Human Services. Pp.179-188.
• Bebeau MJ, Rest JR, Narvaez D. 1999. Beyond the promise: A perspective on research in moral education. Educational Researcher28(4):18–26.
• Beer M. 1980. Organizational Change and Development—A Systems View . Santa Monica, CA: Goodyear.
• Brookfield SD. 1986. Understanding and Facilitating Adult Learning: A Comprehensive Analysis of Principles and Effective Practices . San Francisco, CA: Jossey-Bass.
• Cross KP. 1981. Adults as Learners: Increasing Participation and Facilitating Learning . San Francisco, CA: Jossey-Bass.
• Daft R. 1992. Organizations: Theory and Design , 4th ed. St. Paul, MN: West Publishing.
• Davidoff F, DeAngelis CD, Drazen JM, Nicholls MG, Hoey J, Hojgaard L, Horton R, Kotzin S, Nylenna M, Overbeke AJPM, Sox HC, Van Der Weyden MB, Wilkes MS. 2001. Sponsorship, authorship and accountability. Canadian Medical Association Journal165:786–788. [ PMC free article : PMC81460 ] [ PubMed : 11584570 ]
• Davis D, O'Brien MAT, Freemantle N, Wolf FM, Mazmanian P, Taylor-Vaisey A. 1999. Impact of formal continuing medical education: Do conferences, workshops, rounds, and other traditional continuing education activities change physician behavior or health care outcomes?Journal of American Medical Association282:867–874. [ PubMed : 10478694 ]
• Deming WE. 1986. Out of Crisis . Cambridge, MA: Center for Advanced Engineering Study, Massachusetts Institute of Technology.
• Donabedian A. 1980. Explorations in Quality Assessment and Monitoring: The Definition of Quality and Approaches to Its Assessment . Vol. 1. Ann Arbor, MI: Health Administration Press.
• Federal Register. 2000. Public Health Service Standards for the Protection of Research . [Online]. Available: http://frwebgate4.access.gpo.gov/cgi-bin/waisgate.cgi?WAISdocID= 47426132664+0+2+0&WAISaction=retrieve [Accessed March 18, 2002].
• Fiol DC, Lyles MA. 1985. Organizational learning. Academy of Management Review10:803– 813.
• Freuberg D. 1986. The Corporate Conscience: Money, Power, and Responsible Business . New York, NY: American Management Association.
• Hackman JR, Wageman R. 1995. Total quality management: Empirical, conceptual, and practical issues. Administrative Science Quarterly40:309–342.
• Hamner WC, Organ DW. 1978. Organizational Behavior: An Applied Psychological Approach . Dallas, TX: Business Publications.
• Harrison MI. 1994. Diagnosing Organizations: Methods, Models, and Processes , 2nd ed. Thousand Oaks, CA: Sage.
• Hegarty WH, Sims HP. 1978. Some determinants of unethical decision behavior: An experiment. Journal of Applied Psychology63:451–457.
• Hegarty WH, Sims HP. 1979. Organizational philosophy, policies, and objectives related to unethical decision behavior: A laboratory experiment. Journal of Applied Psychology64:331–338.
• Heitman E. 2000. Ethical values in the education of biomedical researchers. Hastings Center Report30:S40–S44. [ PubMed : 10971905 ]
• Jendrek MP. 1992. Students' reactions to academic dishonesty. Journal of College Student Development33:260–273.
• Katz D, Kahn R. 1978. The Social Psychology of Organizations , 2nd ed. New York, NY: Wiley.
• Katz R. 1980. Time and work: Toward an integrative perspective. In: Staw BM, Cummings LL, eds. Research in Organizational Behavior, Vol. 2. Greenwich, CT: JAI Press. Pp. 81– 127.
• Knowles MS. 1970. The Modern Practice of Adult Education: Andragagy Versus Pedagogy . New York, NY: New York Association Press.
• Kohlberg L. 1984. The Psychology of Moral Development: The Nature and Validity of Moral Stages . San Francisco: Harper & Row. Essays on Moral Development Vol. 2.
• Kurland N. 1996. Trust, accountability, and sales agents' dueling loyalties. Business Ethics Quarterly6:289–310.
• Louis KS, Anderson MS, Rosenberg L. 1995. Academic misconduct and values: The department's influence. The Review of Higher Education18:393–422.
• McCabe DL. 2000. New research on academic integrity: the success of “modified” honor codes. Synfax Weekly Report [Online]. Available: http://www.collegepubs.com/ref/ SFX000515.shtml [Accessed July 25, 2001].
• McCabe DL, Pavela GM. 1998. The effect of institutional policies and procedures on academic integrity. In: Burnett DD, Rudolph L, Clifford KO, eds. Academic Integrity Mat ters . Washington, DC: National Association of Student Personnel Administrators, Inc. Pp.93–108.
• McCabe DL, Treviño LK. 1993. Academic dishonesty: Honor codes and other contextual influences. Journal of Higher Education64:522–538.
• McCabe DL, Treviño LK, Butterfield KD. 1996. The influence of collegiate and corporate codes of conduct on ethics-related behavior in the workplace. Business Ethics Quarterly6:461–476.
• NAS (National Academy of Sciences). 2001. Biological and Biomedical Applications of Stem Cell Research . Washington, DC: National Academy Press.
• National Bioethics Advisory Commission. 1999. Ethical Issues in Human Stem Cell Research , Vol. I to III. Washington, DC: U.S. Government Printing Office.
• NIST (National Institute of Standards and Technology). 1999. Malcolm Baldrige National Qual ity Award 1998 Education Criteria for Performance Excellence . Washington, DC: U.S. Department of Commerce.
• OGE (U.S. Office of Government Ethics). 2000. Executive Branch Employee Ethics: Survey 2000 . Washington, DC: OGE.
• Perry A, Kane K, Bernesser K, Spicker P. 1990. Type A behavior, competitive achievementstriving, and cheating among college students. Psychological Reports66:459–465. [ PubMed : 2349335 ]
• Peters TJ. 1978. Symbols, patterns, and settings: An optimistic case for getting things done. Organizational Dynamics7(2):3–23.
• Peters TJ, Waterman RH, Jr. 1982. In Search of Excellence: Lessons from America's Best-Run Companies . New York, NY: Harper & Row.
• Pfeffer J. 1981. Management as symbolic action: The creation and maintenance of organizational paradigms. In: Cummings LL, Staw BM eds. Research in Organizational Behavior, Vol. 3. Greenwich, CT: JAI Press. Pp.1–52.
• Posner B, Schmidt W. 1982. What kind of people enter the public and private sectors? An undated comparison of perceptions, stereotypes, and values. Human Resource Manage ment21:35–43.
• Posner B, Schmidt W. 1984. Values and the American manager: An update. California Man agement Review26(3):202–216.
• Rest, J. 1983. Morality. In: Mussen PH (series ed.) and Flavell J, Markman E (vol. eds.), Handbook of Child Psychology, Vol. 3,Cognitive Development, 4th ed. New York, NY: Wiley. Pp.556–629.
• Rest J, Narvaez D, Bebeau MJ, Thoma SJ. 1999. Postconventional Moral Thinking: A Neo Kohlbergian Approach . Hillsdale, NJ: L. Erlbaum Associates.
• Saraph JV, Benson PG, Schroeder RG. 1989. An instrument for measuring the critical factors of quality management. Decision Sciences20:810–829.
• Schein EH. 1968. Organizational socialization and the profession of management. Industrial Management Review9:1-15.
• Schneider B. 1975. Organizational climates: An essay. Personnel Psychology28:447–479.
• Schneider B, Reichers AE. 1983. On the etiology of climates. Personnel Psychology36:19–39.
• Schön DA. 1983. Organizational learning. In: Morgan G, ed. Beyond Method: Strategies for Social Research . Newbury Park, CA: Sage. Pp.114–128.
• Siehl C, Martin J. 1984. The role of symbolic management: How can managers effectively transmit organizational culture? In: Hunt JG, Hosking DM, Schriesheim CA, Stewart R, eds. Leaders and Managers: International Perspectives on Managerial Behavior and Leader ship . Elmsford, NY: Pergamon. Pp.227–239.
• Swazey JP, Anderson MS. 1998. Mentors, advisors, and role models in graduate and professional education. In: Rubin ER, ed. Mission Management: A New Synthesis, Vol. 2. Washington, DC: Association of Academic Health Centers. Pp.165–185.
• Swazey JP, Anderson MS, Louis KS. 1993. Ethical problems in academic research. American Scientist81:542–553.
• Treviño LK. 1990. A cultural perspective on changing and developing organizational ethics. Research in Organizational Change and Development4:195–230.
• Treviño LK, Butterfield KD, McCabe DL. 1996. The ethical context in organizations: Influences on employee attitudes and behaviors. Business Ethics Quarterly8(3):447–476.
• Van Maanen J, Schein EH. 1979. Toward a theory of organizational socialization. In: Staw BM, ed. Research in Organizational Behavior, Vol. 1. Greenwich, CT: JAI Press. Pp. 209– 264.
• Victor B, Cullen JB. 1988. The organizational bases of ethical work climates. Administrative Science Quarterly33:101–125.
• Walker OC, Churchill GA, Ford NM. 1979. Where do we go from here? Selected conceptual and empirical issues concerning the motivation and performance of the industrial sales force. In: Albuam G, Churchill GA, eds. Critical Issues in Sales Management State of the Art and Future Research Needs . Eugene, OR: University of Oregon Press. Pp.10–75.
• Wanous JP. 1980. Organizational Entry: Recruitment, Selection, and Socialization of Newcomers . Reading, MA: Addison-Wesley.
• Wimbush JC, Shepard JM. 1994. Toward an understanding of ethical climate: Its relationship to ethical behavior and supervisory influence. Journal of Business Ethics13:637– 647.

For general references on organizational behavior and processes, see Donabedian (1980), Hamner and Organ (1978), Harrison (1994), Katz (1980), Katz and Kahn (1978), Peters (1978), Peters and Waterman (1982), and Pfeffer (1981). For general references on ethical cultures and climate see Ashforth (1985), Schneider and Reichers (1983), and Victor and Cullen (1988). For general references on moral development, see Kohlberg (1984), Rest (1983), and Rest et al. (1999). For general references on adult learning and educational practices, see Brookfield (1986), Cross (1981), and Knowles (1970). For general references on professional socialization, see Schein (1968), Siehl and Martin (1984), Van Maanen and Schein (1979), and Wanous (1980).

Traditional honor codes generally include a pledge that students sign attesting to the integrity of their work, a strong, often exclusive role for students in the judicial process that addresses dishonesty allegations, and provisions such as unproctored exams. Some also require students to report any cheating observed. Modified honor codes generally include a strong or exclusive role for students in the academic judicial system, but do not usually require unproctored exams or that students sign a pledge. Modified codes do place a strong campus focus on the issue of academic integrity and students are reminded frequently that their institution places a high value on integrity (McCabe, 2000).

A recent review of approaches to the study of morality (Bebeau et al., 1999) has challenged the usefulness of the usual tripartite view that assumes that the elements to be studied and assessed are attitudes, knowledge, and behavior. When researchers have studied the connections among these elements, they usually do not find significant connections and are left with the conclusion that attitudes do not have much to do with knowing and behavior is often devoid of feeling and thinking. A more profitable approach is to assume that many types of cognitions, many types of affects, and many kinds of observable behaviors are involved in morality or integrity. All behavior is the result of cognitive-affective processes. Instead of studying cognitions, affects, and behaviors as separate elements, psychologists suggest that researchers study functional processes that must arise to produce moral behavior (Rest, 1983).

See Chapter 6 for further discussion of the role professional and scientific societies can play in fostering an environment that promotes integrity in research.

• Cite this Page National Research Council (US) and Institute of Medicine (US) Committee on Assessing Integrity in Research Environments. Integrity in Scientific Research: Creating an Environment That Promotes Responsible Conduct. Washington (DC): National Academies Press (US); 2002. 3, The Research Environment and Its Impact on Integrity in Research.
• PDF version of this title (5.4M)

In this Page

Related information.

• PMC PubMed Central citations
• PubMed Links to PubMed

Recent Activity

• The Research Environment and Its Impact on Integrity in Research - Integrity in ... The Research Environment and Its Impact on Integrity in Research - Integrity in Scientific Research

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

Sabine Müller

On creating a good research environment

31 March 2021 | doi:10.5281/zenodo.463628 | 1 Comment

Sabine Müller on the hierarchical system of German academia and why it could be a problem for the wellbeing of young academics and Ph.D. candidates. She compares it to her experiences at Oxford University and sheds light on the differences between the two research cultures.

“Researchers say that their working culture is best when it is collaborative, inclusive, supportive and creative, when researchers are given time to focus on their research priorities, when leadership is transparent and open, and when individuals have a sense of safety and security. But too often research culture is not at its best.” [“What researchers think about the culture they work in”, Wellcome Trust and Shift Learning (London 2020), p. 3 (subsequently referred to as “What researchers think”)] The executive summary of the 2020 Wellcome Trust study on research culture goes on to describe how “many [researchers] are often missing out on critical aspects of good management … [a]nd worse, many have experienced exploitation, discrimination, harassment and bullying.” Notably, members of minority groups more often experience the latter. These results echo those of previous surveys, such as those conducted by Advance HE or the journal Nature in the UK – and which illustrate that these issues are on the radar of public debate. (Woolston, 2019) The situation in Germany is hardly any different, though of course data in Germany is scarce and hardly sufficient to make any reliable statements about early career researchers’ emotional situation. (Most notably, the German Centre for High Education Research and Science Studies (DZHW) runs a National Academics Panel Study since 2017 which promises to give further data on the condition and well-being of doctoral researchers.) Notably, existing studies are frequently a reaction to incidents that reached public attention (Albott, 2019), or rely on surveys by the concerned group such as by the networks of doctoral researchers of non-university research organisations. Accordingly, the existence of guidelines for managing power abuse or mental health are confined to institutions which have struggled with cases. Otherwise the silence on the topic by reputable research organisations such as the DFG or the academies is overwhelming . The reasons for this gap may be manifold. German academia is not immune to the complex range of problems such as non-transparent leadership, a lack of inclusiveness, harassment or mental health issues which resist a positive research culture. Each of these issues by themselves has numerous causes, but all of them are amplified by a system which lays “an excessive focus on measuring performance” as well as institutional structures such as the accumulation of responsibilities and decision making as well as steep hierarchies. (Shore & Wright, 1999) It is the latter aspects which I would like to focus on as crucial elements required in order to foster a cultural change for a positive research environment, and which, compared to the bigger systemic issues could be rather easily fixed.

As a career development adviser in Germany, I was frequently confronted with the following “argument” during discussions with senior researchers about the working conditions of early career researchers: “It was like this when I did my PhD, so why should that not work nowadays?” I always wondered what exactly this was meant to say? Unpacking this claim to myself it seems to implicitly suggest 1) the person, too, did not attain their PhD in good conditions, 2) but the fact that they succeeded makes them think that it was not so bad after all. Is the rationale behind this that a “rough school” toughens people up to prepare them for academic life? In fact, I often also heard that doctoral researchers today are not only too sensitive but too demanding. But is the consequence that only people who are willing to toughen up stay in academia? Besides the questionable psychological rationale, I wonder whether we really think that this is what academia needs: tough personalities? Putting aside the universe of unconscious biases which is touched by such a question, shouldn’t academic work and life not be guided – even more than any other branches of the labour market – by the principle of reason, multiperspectivity, openness, integrity and such, rather than of the dull workings of unconscious bias and self-perpetuation? And should the system not aim to do everything for those qualities to be able to unfold and thrive? Responding to such a statement from my own personal experience, I often felt awkward since I did not share this experience. And this difference in the culture of dealing with issues such as discrimination, mental health, diversity and welfare, as well as power abuse has struck me most notably upon my return to Germany after I had spent eight years at the University of Oxford taking up a position in career development in a research organisation. My experience during my doctoral and postdoctoral research was a very positive one – in many respects, I had the time of my life – and I felt that encouraging people to create an environment for such a good experience would be crucial. In the following piece, I focus on some landmarks of this positive (!) experience in my academic career in order to point to how a fundamental change of culture needs to be human centered, and attend to individual experience. 

I was granted a first memorable insight into a different mind-set before I left for my one year Master’s, which led to my doctoral research at the University of Oxford. I was confronted with the choice between 32 colleges. I was inclined to apply at the college to which my future supervisor was affiliated – a thought which very much agreed with the logic of the German system I was socialized in where doctoral researchers are frequently not only naturally affiliated with the “Lehrstuhl” of their supervisors but also seem to enter into a sort of patronship relation expressed in the still prevalent German term “Doktorvater/-mutter”. However, my supervisor asked me to consider that in case of disagreement or conflict, it would be advantageous to be able to have an independent college adviser to turn to. The sense of responsibility expressed in this thoughtfulness with respect to providing an environment to my advantage profoundly shaped my own actions along the subsequent years as doctoral and postdoctoral researcher, as senior subject tutor and lecturer at Oxford University and beyond. Supervision training might help, but can only partly address the care at work here. The ambition to promote your doctoral researcher, so that s/he can realise her/his potential is connected to a sense of duty to pay attention to the welfare of your supervisee. This attention is promoted and aided by structural aspects. Beside my supervisor – who I am lucky to say was a most conscious and inspiring researcher with whom I met every other week, and who conscientiously read every essay, chapter or anything else I ever submitted – I was then assigned a college adviser. In my case this person was an éminence grise in my field who, as tradition would have it, invited me to a talk at the fireplace and imparted his wisdom to me – and, last but not least, a faculty adviser who was there to offer further opportunities to talk about the programme of my thesis and to whom both my supervisor and I had to submit a progress report by the end of each term. Moreover, the degree at Oxford has a clear milestone system in which supervision and assessment are separated from each other: the vivas for the transfer of status as “Probationer Research Student” to DPhil Candidate after one year into your degree as well as the confirmation of status after two years is taken by two faculty members. The assessment of the submitted final dissertation lies in the hands of an internal and external examiner. This way of organization ensures that the role of the supervisor is focused to act as adviser and to support their supervisee as best as they can. Of course, this means some control for the supervision process: Failure to bring your supervisee to successfully finish their degree will not have consequences for any academic but is not as easily obscured by the possibility to drag the doctoral research on or by marking the thesis accordingly. At the same time, the shared roles opened the opportunity for me, as the supervisee to connect and frankly discuss with other senior academics who took my work seriously. 

Thus, the transparency of a clear milestone system, which details what is expected from the student as well as the separation of the roles of supervision, monitoring and assessment, has the potential to minimise the risk of power abuse and lifts the weight from the relation between supervisors from the start. It affords the supervisee the opportunity to discuss her or his work throughout the process with various researchers, to gain more perspectives and develop an independence of thought and a network from the get-go. Combined with the opportunity to frequently share your intellectual thoughts with established experts in your field and beyond made doctoral and postdoctoral research particularly worthwhile.

I would like to add that as a senior subject tutor for German Studies I experienced the advantages of this disentanglement of examination and supervision for myself: the faculty assigns a committee which designs the end of year exams. Marking and assessment were organised anonymously in an annual rotating system of examiners. Both procedures entail multiple advantages: not only do they limit the power of tutor or supervisor but they also relieve both from that burden of power. Not being the examiner, you can truly fulfil the role as adviser, coach and teacher and accompany your students along their development. Reaching out to your tutor or supervisor is easier, if you do not have to fear any repercussion on your performance. In this context, I also learned to appreciate the carefully built college and university community which provided a network to support students and lecturers alike. It ranges from the so-called common rooms with their mentor for freshers and trained peer advisers to college and university counsellors as well support staff for people of various religious and ethnical background on campus. Coming from a German university, this amount of attention and care which unloaded over my head was at first rather overwhelming and I confess I thought it unnecessary. But over the years, I learned to appreciate this culture which aspired to keep people well and enable them to enjoy their time at the university. Especially later, as a senior subject tutor, when my contract stated in no uncertain terms that tasks comprised the welfare of my students, this community recognised the limits of my competencies and acknowledged the need for welfare offers. 

Another major landmark remains the handling of admission and application procedures. Perhaps it is worth explaining that student admissions at Oxford is a highly professional and formal process which stretches over two weeks in December after the autumn term. Not only are we dealing with standardised applications which aim to highlight the potential of each candidate. Each applicant invited to interviews has the right to get at least two interviews with different academics to assess their performance. In fact, it is a very intricate system with the objective to select students with high potential, no matter their background. In my first year, I was asked to write the protocol for admission interviews and even for that rather small task, I had to complete an online course on legal liabilities, correct interview methods, harassment, discrimination and the mechanism of unconscious bias. As senior subject tutor responsible for admissions in your subject area, I had to take another, more extensive course with on- and offline elements. These courses were a necessary eye-opener to topics which had never been addressed, even in the student council of my German university. It set my expectations of what I consider to be a professional application procedure and to this day I find it hard to accept that none of this type of elementary interview training, which raises awareness of everyone’s unconscious blind spots concerning bias and awareness, is a required standard at German universities or research organisations. It would be easy to implement part of a structured onboarding for each and every academic at the university and at least make the recruiting procedure a bit fairer. 

To sum up, I would like to make clear that I am aware that problems prevail in the UK, as the quoted Wellcome Trust study illustrates. I also want to point out that reason why welfare at Oxford and Cambridge is paid such attention is not entirely altruistic: for a long time, these Universities had to deal with the reproach of higher suicide rates – a critique which cannot be sustained (Hawton et al., 2012).  In addition, it is often pointed out that these institutions are only accessible to elites, which, at an undergraduate level, is very true. At the same time, Oxbridge institutions  understand that in order to attract the best academics they have to cater to people’s wellbeing as human beings in every aspect. So strategic deliberations and monetary concerns are certainly central drivers for the implementation. However, this does not devalue the learnings from such an experience: a collaborative, open, transparent and overall friendly environment relies on the mind-set of the academics who acknowledge the responsibility for their supervisees. This mind-set is supported by structures that foster transparency, independence and exchange by clearly laying out the demands and milestones of a doctoral course (without the need to make people go back to school) by separating the roles of supervision, monitoring and assessment, by carefully building a community with low-threshold support structures catering to various backgrounds as well as training to raise awareness to biases, harassment, stress symptoms etc. None of these suggestions are new but maybe not enough people have experienced how powerful they can be in their small workings and, thus, not enough people can or want to pass on this kind of experience.

Albott, Alison: Germany’s prestigious Max Planck Society investigates new allegations of abuse, in Nature (online) (9 July 2019) https://www.nature.com/articles/d41586-018-05668-y / doi: https://doi.org/10.1038/d41586-018-05668-y

Hawton K., Bergen H et.al : University Students over a 30-years period, in Social Psychiatry and Psychiatric Epidemology 47 (2012), p.43-51, https://doi.org/10.1007/s00127-010-0310-3 (a summary is provided: https://www.psych.ox.ac.uk/publications/168323 ). Further information can be obtained at the Office for National statistics :

Shore, C., & Wright, S. (1999). Audit Culture and Anthropology: Neo-Liberalism in British Higher Education. The Journal of the Royal Anthropological Institute, 5(4), 557-575. doi: 10.2307/2661148 See also:  “A cry for help”, in Nature 575 (14 November 2019), p.257-258: https://www.nature.com/articles/d41586-019-03489-1

Woolston, Cristof: PhDs: torturous truths, in Nature 575 (13 November 2019), p.403-406 https://www.nature.com/articles/d41586-019-03459-7

Thanks for this, Sabine. You very well present the PhD scholar’s perspective. From a professor’s or supervisor’s perspective it is in my view even worse. I remember one of my first PhD examinations in Germany – not as a supervisor. I thought the PhD was really poor and wanted to be nice and make it pass but give it a very bad mark. I talk to a colleagues about this and she adviced me “Are you crazy, you cannot do that!! His supervisor will interpret this as an open war”. I learnt that the assessment of a PhD candidate is also assessment of the supervisor, so if you want to punish a colleague who has not supported you in another situation, you can do it by marking his candidate poorly. The other way around as well, of course: you’ll give high marks to the the PhD candidates of your friends. You can imagine yourself what kind of informal “economy” that goes on among supervisors/professors. I have been part of many PhD examinations, and I tell you that there is no system at all of good candidates getting good marks and bad candidates getting bad marks. What is negotiated at a PhD examination is very much the standing of the professors and their interrelations. A feudal system.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

First we need to confirm, that you're human. × three = three

Author info

Sabine Müller is currently Adviser for Education and Participation in the Digital World at Wikimedia Deutschland. Before, she was a research consultant for humanities and educational research as well as career development at the head office of the Leibniz Association. She holds a DPhil from Oxford University where she also worked as senior subject tutor for German Studies at St John’s and Magdalen College as well as affiliate postdoc on embodied cognition and narration.