quality research articles are

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

How do you determine the quality of a journal article?

Published on October 17, 2014 by Bas Swaen . Revised on March 4, 2019.

In the theoretical framework of your thesis, you support the research that you want to perform by means of a literature review . Here, you are looking for earlier research about your subject. These studies are often published in the form of scientific articles in journals (scientific publications).

Table of contents

Why is good quality important, check the following points.

The better the quality of the articles that you use in the literature review , the stronger your own research will be. When you use articles that are not well respected, you run the risk that the conclusions you draw will be unfounded. Your supervisor will always check the article sources for the conclusions you draw.

We will use an example to explain how you can judge the quality of a scientific article. We will use the following article as our example:

Example article

Perrett, D. I., Burt, D. M., Penton-Voak, I. S., Lee, K. J., Rowland, D. A., & Edwards, R. (1999). Symmetry and Human Facial Attractiveness.  Evolution and Human Behavior ,  20 , 295-307. Retrieved from  http://www.grajfoner.com/Clanki/Perrett%201999%20Symetry%20Attractiveness.pdf

This article is about the possible link between facial symmetry and the attractiveness of a human face.

Prevent plagiarism. Run a free check.

1. where is the article published.

The journal (academic publication) where the article is published says something about the quality of the article. Journals are ranked in the Journal Quality List (JQL). If the journal you used is ranked at the top of your professional field in the JQL, then you can assume that the quality of the article is high.

The article from the example is published in the journal “Evolution and Human Behavior”. The journal is not on the Journal Quality List, but after googling the publication, it seems from multiple sources that it nevertheless is among the top in the field of Psychology (see Journal Ranking at   http://www.ehbonline.org/ ). The quality of the source is thus high enough to use it.

So, if a journal is not listed in the Journal Quality List then it is worthwhile to google it. You will then find out more about the quality of the journal.

2. Who is the author?

The next step is to look at who the author of the article is:

• What do you know about the person who wrote the paper?
• Has the author done much research in this field?
• What do others say about the author?
• What is the author’s background?
• At which university does the author work? Does this university have a good reputation?

The lead author of the article (Perrett) has already done much work within the research field, including prior studies of predictors of attractiveness. Penton-Voak, one of the other authors, also collaborated on these studies. Perrett and Penton-Voak were in 1999 both professors at the University of St Andrews in the United Kingdom. This university is among the top 100 best universities in the world. There is less information available about the other authors. It could be that they were students who helped the professors.

3. What is the date of publication?

In which year is the article published? The more recent the research, the better. If the research is a bit older, then it’s smart to check whether any follow-up research has taken place. Maybe the author continued the research and more useful results have been published.

Tip! If you’re searching for an article in Google Scholar , then click on ‘Since 2014’ in the left hand column. If you can’t find anything (more) there, then select ‘Since 2013’. If you work down the row in this manner, you will find the most recent studies.

The article from the example was published in 1999. This is not extremely old, but there has probably been quite a bit of follow-up research done in the past 15 years. Thus, I quickly found via Google Scholar an article from 2013, in which the influence of symmetry on facial attractiveness in children was researched. The example article from 1999 can probably serve as a good foundation for reading up on the subject, but it is advisable to find out how research into the influence of symmetry on facial attractiveness has further developed.

4. What do other researchers say about the paper?

Find out who the experts are in this field of research. Do they support the research, or are they critical of it?

By searching in Google Scholar, I see that the article has been cited at least 325 times! This says then that the article is mentioned at least in 325 other articles. If I look at the authors of the other articles, I see that these are experts in the research field. The authors who cite the article use the article as support and not to criticize it.

5. Determine the quality

Now look back: how did the article score on the points mentioned above? Based on that, you can determine quality.

The example article scored ‘reasonable’ to ‘good’ on all points. So we can consider the article to be qualitatively good, and therefore it is useful in, for example, a literature review. Because the article is already somewhat dated, however, it is wise to also go in search of more recent research.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Swaen, B. (2019, March 04). How do you determine the quality of a journal article?. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/tips/how-do-you-determine-the-quality-of-a-journal-article/

Is this article helpful?

"I thought AI Proofreading was useless but.."

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

• Search Menu
• Sign in through your institution
• Advance articles
• Author Guidelines
• Submission Site
• Open Access
• Why Publish?
• About Research Evaluation
• Editorial Board
• Advertising and Corporate Services
• Journals Career Network
• Self-Archiving Policy
• Dispatch Dates
• Journals on Oxford Academic
• Books on Oxford Academic

Article Contents

1. introduction, 4. synthesis, 4.1 principles of tdr quality, 5. conclusions, supplementary data, acknowledgements, defining and assessing research quality in a transdisciplinary context.

• Article contents
• Figures & tables
• Supplementary Data

Brian M. Belcher, Katherine E. Rasmussen, Matthew R. Kemshaw, Deborah A. Zornes, Defining and assessing research quality in a transdisciplinary context, Research Evaluation , Volume 25, Issue 1, January 2016, Pages 1–17, https://doi.org/10.1093/reseval/rvv025

• Permissions Icon Permissions

Research increasingly seeks both to generate knowledge and to contribute to real-world solutions, with strong emphasis on context and social engagement. As boundaries between disciplines are crossed, and as research engages more with stakeholders in complex systems, traditional academic definitions and criteria of research quality are no longer sufficient—there is a need for a parallel evolution of principles and criteria to define and evaluate research quality in a transdisciplinary research (TDR) context. We conducted a systematic review to help answer the question: What are appropriate principles and criteria for defining and assessing TDR quality? Articles were selected and reviewed seeking: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, proposed principles of research quality, proposed criteria for research quality assessment, proposed indicators and measures of research quality, and proposed processes for evaluating TDR. We used the information from the review and our own experience in two research organizations that employ TDR approaches to develop a prototype TDR quality assessment framework, organized as an evaluation rubric. We provide an overview of the relevant literature and summarize the main aspects of TDR quality identified there. Four main principles emerge: relevance, including social significance and applicability; credibility, including criteria of integration and reflexivity, added to traditional criteria of scientific rigor; legitimacy, including criteria of inclusion and fair representation of stakeholder interests, and; effectiveness, with criteria that assess actual or potential contributions to problem solving and social change.

Contemporary research in the social and environmental realms places strong emphasis on achieving ‘impact’. Research programs and projects aim to generate new knowledge but also to promote and facilitate the use of that knowledge to enable change, solve problems, and support innovation ( Clark and Dickson 2003 ). Reductionist and purely disciplinary approaches are being augmented or replaced with holistic approaches that recognize the complex nature of problems and that actively engage within complex systems to contribute to change ‘on the ground’ ( Gibbons et al. 1994 ; Nowotny, Scott and Gibbons 2001 , Nowotny, Scott and Gibbons 2003 ; Klein 2006 ; Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Erno-Kjolhede and Hansson 2011 ). Emerging fields such as sustainability science have developed out of a need to address complex and urgent real-world problems ( Komiyama and Takeuchi 2006 ). These approaches are inherently applied and transdisciplinary, with explicit goals to contribute to real-world solutions and strong emphasis on context and social engagement ( Kates 2000 ).

While there is an ongoing conceptual and theoretical debate about the nature of the relationship between science and society (e.g. Hessels 2008 ), we take a more practical starting point based on the authors’ experience in two research organizations. The first author has been involved with the Center for International Forestry Research (CIFOR) for almost 20 years. CIFOR, as part of the Consultative Group on International Agricultural Research (CGIAR), began a major transformation in 2010 that shifted the emphasis from a primary focus on delivering high-quality science to a focus on ‘…producing, assembling and delivering, in collaboration with research and development partners, research outputs that are international public goods which will contribute to the solution of significant development problems that have been identified and prioritized with the collaboration of developing countries.’ ( CGIAR 2011 ). It was always intended that CGIAR research would be relevant to priority development and conservation issues, with emphasis on high-quality scientific outputs. The new approach puts much stronger emphasis on welfare and environmental results; research centers, programs, and individual scientists now assume shared responsibility for achieving development outcomes. This requires new ways of working, with more and different kinds of partnerships and more deliberate and strategic engagement in social systems.

Royal Roads University (RRU), the home institute of all four authors, is a relatively new (created in 1995) public university in Canada. It is deliberately interdisciplinary by design, with just two faculties (Faculty of Social and Applied Science; Faculty of Management) and strong emphasis on problem-oriented research. Faculty and student research is typically ‘applied’ in the Organization for Economic Co-operation and Development (2012) sense of ‘original investigation undertaken in order to acquire new knowledge … directed primarily towards a specific practical aim or objective’.

An increasing amount of the research done within both of these organizations can be classified as transdisciplinary research (TDR). TDR crosses disciplinary and institutional boundaries, is context specific, and problem oriented ( Klein 2006 ; Carew and Wickson 2010 ). It combines and blends methodologies from different theoretical paradigms, includes a diversity of both academic and lay actors, and is conducted with a range of research goals, organizational forms, and outputs ( Klein 2006 ; Boix-Mansilla 2006a ; Erno-Kjolhede and Hansson 2011 ). The problem-oriented nature of TDR and the importance placed on societal relevance and engagement are broadly accepted as defining characteristics of TDR ( Carew and Wickson 2010 ).

The experience developing and using TDR approaches at CIFOR and RRU highlights the need for a parallel evolution of principles and criteria for evaluating research quality in a TDR context. Scientists appreciate and often welcome the need and the opportunity to expand the reach of their research, to contribute more effectively to change processes. At the same time, they feel the pressure of added expectations and are looking for guidance.

In any activity, we need principles, guidelines, criteria, or benchmarks that can be used to design the activity, assess its potential, and evaluate its progress and accomplishments. Effective research quality criteria are necessary to guide the funding, management, ongoing development, and advancement of research methods, projects, and programs. The lack of quality criteria to guide and assess research design and performance is seen as hindering the development of transdisciplinary approaches ( Bergmann et al. 2005 ; Feller 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2008 ; Carew and Wickson 2010 ; Jahn and Keil 2015 ). Appropriate quality evaluation is essential to ensure that research receives support and funding, and to guide and train researchers and managers to realize high-quality research ( Boix-Mansilla 2006a ; Klein 2008 ; Aagaard-Hansen and Svedin 2009 ; Carew and Wickson 2010 ).

Traditional disciplinary research is built on well-established methodological and epistemological principles and practices. Within disciplinary research, quality has been defined narrowly, with the primary criteria being scientific excellence and scientific relevance ( Feller 2006 ; Chataway, Smith and Wield 2007 ; Erno-Kjolhede and Hansson 2011 ). Disciplines have well-established (often implicit) criteria and processes for the evaluation of quality in research design ( Erno-Kjolhede and Hansson 2011 ). TDR that is highly context specific, problem oriented, and includes nonacademic societal actors in the research process is challenging to evaluate ( Wickson, Carew and Russell 2006 ; Aagaard-Hansen and Svedin 2009 ; Andrén 2010 ; Carew and Wickson 2010 ; Huutoniemi 2010 ). There is no one definition or understanding of what constitutes quality, nor a set guide for how to do TDR ( Lincoln 1995 ; Morrow 2005 ; Oberg 2008 ; Andrén 2010 ; Huutoniemi 2010 ). When epistemologies and methods from more than one discipline are used, disciplinary criteria may be insufficient and criteria from more than one discipline may be contradictory; cultural conflicts can arise as a range of actors use different terminology for the same concepts or the same terminology for different concepts ( Chataway, Smith and Wield 2007 ; Oberg 2008 ).

Current research evaluation approaches as applied to individual researchers, programs, and research units are still based primarily on measures of academic outputs (publications and the prestige of the publishing journal), citations, and peer assessment ( Boix-Mansilla 2006a ; Feller 2006 ; Erno-Kjolhede and Hansson 2011 ). While these indicators of research quality remain relevant, additional criteria are needed to address the innovative approaches and the diversity of actors, outputs, outcomes, and long-term social impacts of TDR. It can be difficult to find appropriate outlets for TDR publications simply because the research does not meet the expectations of traditional discipline-oriented journals. Moreover, a wider range of inputs and of outputs means that TDR may result in fewer academic outputs. This has negative implications for transdisciplinary researchers, whose performance appraisals and long-term career progression are largely governed by traditional publication and citation-based metrics of evaluation. Research managers, peer reviewers, academic committees, and granting agencies all struggle with how to evaluate and how to compare TDR projects ( ex ante or ex post ) in the absence of appropriate criteria to address epistemological and methodological variability. The extent of engagement of stakeholders 1 in the research process will vary by project, from information sharing through to active collaboration ( Brandt et al. 2013) , but at any level, the involvement of stakeholders adds complexity to the conceptualization of quality. We need to know what ‘good research’ is in a transdisciplinary context.

As Tijssen ( 2003 : 93) put it: ‘Clearly, in view of its strategic and policy relevance, developing and producing generally acceptable measures of “research excellence” is one of the chief evaluation challenges of the years to come’. Clear criteria are needed for research quality evaluation to foster excellence while supporting innovation: ‘A principal barrier to a broader uptake of TD research is a lack of clarity on what good quality TD research looks like’ ( Carew and Wickson 2010 : 1154). In the absence of alternatives, many evaluators, including funding bodies, rely on conventional, discipline-specific measures of quality which do not address important aspects of TDR.

There is an emerging literature that reviews, synthesizes, or empirically evaluates knowledge and best practice in research evaluation in a TDR context and that proposes criteria and evaluation approaches ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Klein 2008 ; Carew and Wickson 2010 ; ERIC 2010; de Jong et al. 2011 ; Spaapen and Van Drooge 2011 ). Much of it comes from a few fields, including health care, education, and evaluation; little comes from the natural resource management and sustainability science realms, despite these areas needing guidance. National-scale reviews have begun to recognize the need for broader research evaluation criteria but have had difficulty dealing with it and have made little progress in addressing it ( Donovan 2008 ; KNAW 2009 ; REF 2011 ; ARC 2012 ; TEC 2012 ). A summary of the national reviews that we reviewed in the development of this research is provided in Supplementary Appendix 1 . While there are some published evaluation schemes for TDR and interdisciplinary research (IDR), there is ‘substantial variation in the balance different authors achieve between comprehensiveness and over-prescription’ ( Wickson and Carew 2014 : 256) and still a need to develop standardized quality criteria that are ‘uniquely flexible to provide valid, reliable means to evaluate and compare projects, while not stifling the evolution and responsiveness of the approach’ ( Wickson and Carew 2014 : 256).

There is a need and an opportunity to synthesize current ideas about how to define and assess quality in TDR. To address this, we conducted a systematic review of the literature that discusses the definitions of research quality as well as the suggested principles and criteria for assessing TDR quality. The aim is to identify appropriate principles and criteria for defining and measuring research quality in a transdisciplinary context and to organize those principles and criteria as an evaluation framework.

The review question was: What are appropriate principles, criteria, and indicators for defining and assessing research quality in TDR?

This article presents the method used for the systematic review and our synthesis, followed by key findings. Theoretical concepts about why new principles and criteria are needed for TDR, along with associated discussions about evaluation process are presented. A framework, derived from our synthesis of the literature, of principles and criteria for TDR quality evaluation is presented along with guidance on its application. Finally, recommendations for next steps in this research and needs for future research are discussed.

2.1 Systematic review

Systematic review is a rigorous, transparent, and replicable methodology that has become widely used to inform evidence-based policy, management, and decision making ( Pullin and Stewart 2006 ; CEE 2010). Systematic reviews follow a detailed protocol with explicit inclusion and exclusion criteria to ensure a repeatable and comprehensive review of the target literature. Review protocols are shared and often published as peer reviewed articles before undertaking the review to invite critique and suggestions. Systematic reviews are most commonly used to synthesize knowledge on an empirical question by collating data and analyses from a series of comparable studies, though methods used in systematic reviews are continually evolving and are increasingly being developed to explore a wider diversity of questions ( Chandler 2014 ). The current study question is theoretical and methodological, not empirical. Nevertheless, with a diverse and diffuse literature on the quality of TDR, a systematic review approach provides a method for a thorough and rigorous review. The protocol is published and available at http://www.cifor.org/online-library/browse/view-publication/publication/4382.html . A schematic diagram of the systematic review process is presented in Fig. 1 .

Search process.

2.2 Search terms

Search terms were designed to identify publications that discuss the evaluation or assessment of quality or excellence 2 of research 3 that is done in a TDR context. Search terms are listed online in Supplementary Appendices 2 and 3 . The search strategy favored sensitivity over specificity to ensure that we captured the relevant information.

2.3 Databases searched

ISI Web of Knowledge (WoK) and Scopus were searched between 26 June 2013 and 6 August 2013. The combined searches yielded 15,613 unique citations. Additional searches to update the first searchers were carried out in June 2014 and March 2015, for a total of 19,402 titles scanned. Google Scholar (GS) was searched separately by two reviewers during each search period. The first reviewer’s search was done on 2 September 2013 (Search 1) and 3 September 2013 (Search 2), yielding 739 and 745 titles, respectively. The second reviewer’s search was done on 19 November 2013 (Search 1) and 25 November 2013 (Search 2), yielding 769 and 774 titles, respectively. A third search done on 17 March 2015 by one reviewer yielded 98 new titles. Reviewers found high redundancy between the WoK/Scopus searches and the GS searches.

2.4 Targeted journal searches

Highly relevant journals, including Research Evaluation, Evaluation and Program Planning, Scientometrics, Research Policy, Futures, American Journal of Evaluation, Evaluation Review, and Evaluation, were comprehensively searched using broader, more inclusive search strings that would have been unmanageable for the main database search.

2.5 Supplementary searches

References in included articles were reviewed to identify additional relevant literature. td-net’s ‘Tour d’Horizon of Literature’, lists important inter- and transdisciplinary publications collected through an invitation to experts in the field to submit publications ( td-net 2014 ). Six additional articles were identified via supplementary search.

2.6 Limitations of coverage

The review was limited to English-language published articles and material available through internet searches. There was no systematic way to search the gray (unpublished) literature, but relevant material identified through supplementary searches was included.

2.7 Inclusion of articles

This study sought articles that review, critique, discuss, and/or propose principles, criteria, indicators, and/or measures for the evaluation of quality relevant to TDR. As noted, this yielded a large number of titles. We then selected only those articles with an explicit focus on the meaning of IDR and/or TDR quality and how to achieve, measure or evaluate it. Inclusion and exclusion criteria were developed through an iterative process of trial article screening and discussion within the research team. Through this process, inter-reviewer agreement was tested and strengthened. Inclusion criteria are listed in Tables 1 and 2 .

Inclusion criteria for title and abstract screening

Inclusion criteria for abstract and full article screening

Article screening was done in parallel by two reviewers in three rounds: (1) title, (2) abstract, and (3) full article. In cases of uncertainty, papers were included to the next round. Final decisions on inclusion of contested papers were made by consensus among the four team members.

2.8 Critical appraisal

In typical systematic reviews, individual articles are appraised to ensure that they are adequate for answering the research question and to assess the methods of each study for susceptibility to bias that could influence the outcome of the review (Petticrew and Roberts 2006). Most papers included in this review are theoretical and methodological papers, not empirical studies. Most do not have explicit methods that can be appraised with existing quality assessment frameworks. Our critical appraisal considered four criteria adapted from Spencer et al. (2003): (1) relevance to the review question, (2) clarity and logic of how information in the paper was generated, (3) significance of the contribution (are new ideas offered?), and (4) generalizability (is the context specified; do the ideas apply in other contexts?). Disagreements were discussed to reach consensus.

2.9 Data extraction and management

The review sought information on: arguments for or against expanding definitions of research quality, purposes for research quality evaluation, principles of research quality, criteria for research quality assessment, indicators and measures of research quality, and processes for evaluating TDR. Four reviewers independently extracted data from selected articles using the parameters listed in Supplementary Appendix 4 .

2.10 Data synthesis and TDR framework design

Our aim was to synthesize ideas, definitions, and recommendations for TDR quality criteria into a comprehensive and generalizable framework for the evaluation of quality in TDR. Key ideas were extracted from each article and summarized in an Excel database. We classified these ideas into themes and ultimately into overarching principles and associated criteria of TDR quality organized as a rubric ( Wickson and Carew 2014 ). Definitions of each principle and criterion were developed and rubric statements formulated based on the literature and our experience. These criteria (adjusted appropriately to be applied ex ante or ex post ) are intended to be used to assess a TDR project. The reviewer should consider whether the project fully satisfies, partially satisfies, or fails to satisfy each criterion. More information on application is provided in Section 4.3 below.

We tested the framework on a set of completed RRU graduate theses that used transdisciplinary approaches, with an explicit problem orientation and intent to contribute to social or environmental change. Three rounds of testing were done, with revisions after each round to refine and improve the framework.

3.1 Overview of the selected articles

Thirty-eight papers satisfied the inclusion criteria. A wide range of terms are used in the selected papers, including: cross-disciplinary; interdisciplinary; transdisciplinary; methodological pluralism; mode 2; triple helix; and supradisciplinary. Eight included papers specifically focused on sustainability science or TDR in natural resource management, or identified sustainability research as a growing TDR field that needs new forms of evaluation ( Cash et al. 2002 ; Bergmann et al. 2005 ; Chataway, Smith and Wield 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Andrén 2010 ; Carew and Wickson 2010 ; Lang et al. 2012 ; Gaziulusoy and Boyle 2013 ). Carew and Wickson (2010) build on the experience in the TDR realm to propose criteria and indicators of quality for ‘responsible research and innovation’.

The selected articles are written from three main perspectives. One set is primarily interested in advancing TDR approaches. These papers recognize the need for new quality measures to encourage and promote high-quality research and to overcome perceived biases against TDR approaches in research funding and publishing. A second set of papers is written from an evaluation perspective, with a focus on improving evaluation of TDR. The third set is written from the perspective of qualitative research characterized by methodological pluralism, with many characteristics and issues relevant to TDR approaches.

The majority of the articles focus at the project scale, some at the organization level, and some do not specify. Some articles explicitly focus on ex ante evaluation (e.g. proposal evaluation), others on ex post evaluation, and many are not explicit about the project stage they are concerned with. The methods used in the reviewed articles include authors’ reflection and opinion, literature review, expert consultation, document analysis, and case study. Summaries of report characteristics are available online ( Supplementary Appendices 5–8 ). Eight articles provide comprehensive evaluation frameworks and quality criteria specifically for TDR and research-in-context. The rest of the articles discuss aspects of quality related to TDR and recommend quality definitions, criteria, and/or evaluation processes.

3.2 The need for quality criteria and evaluation methods for TDR

Many of the selected articles highlight the lack of widely agreed principles and criteria of TDR quality. They note that, in the absence of TDR quality frameworks, disciplinary criteria are used ( Morrow 2005 ; Boix-Mansilla 2006a , b ; Feller 2006 ; Klein 2006 , 2008 ; Wickson, Carew and Russell 2006 ; Scott 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Oberg 2008 ; Erno-Kjolhede and Hansson 2011 ), and evaluations are often carried out by reviewers who lack cross-disciplinary experience and do not have a shared understanding of quality ( Aagaard-Hansen and Svedin 2009 ). Quality is discussed by many as a relative concept, developed within disciplines, and therefore defined and understood differently in each field ( Morrow 2005 ; Klein 2006 ; Oberg 2008 ; Mitchell and Willets 2009 ; Huutoniemi 2010 ; Hellstrom 2011 ). Jahn and Keil (2015) point out the difficulty of creating a common set of quality criteria for TDR in the absence of a standard agreed-upon definition of TDR. Many of the selected papers argue the need to move beyond narrowly defined ideas of ‘scientific excellence’ to incorporate a broader assessment of quality which includes societal relevance ( Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ). This shift includes greater focus on research organization, research process, and continuous learning, rather than primarily on research outputs ( Hemlin and Rasmussen 2006 ; de Jong et al. 2011 ; Wickson and Carew 2014 ; Jahn and Keil 2015 ). This responds to and reflects societal expectations that research should be accountable and have demonstrated utility ( Cloete 1997 ; Defila and Di Giulio 1999 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Stige 2009 ).

A central aim of TDR is to achieve socially relevant outcomes, and TDR quality criteria should demonstrate accountability to society ( Cloete 1997 ; Hemlin and Rasmussen 2006 ; Chataway, Smith and Wield 2007 ; Ozga 2007 ; Spaapen, Dijstelbloem and Wamelink 2007 ; de Jong et al. 2011 ). Integration and mutual learning are a core element of TDR; it is not enough to transcend boundaries and incorporate societal knowledge but, as Carew and Wickson ( 2010 : 1147) summarize: ‘…the TD researcher needs to put effort into integrating these potentially disparate knowledges with a view to creating useable knowledge. That is, knowledge that can be applied in a given problem context and has some prospect of producing desired change in that context’. The inclusion of societal actors in the research process, the unique and often dispersed organization of research teams, and the deliberate integration of different traditions of knowledge production all fall outside of conventional assessment criteria ( Feller 2006 ).

Not only do the range of criteria need to be updated, expanded, agreed upon, and assumptions made explicit ( Boix-Mansilla 2006a ; Klein 2006 ; Scott 2007 ) but, given the specific problem orientation of TDR, reviewers beyond disciplinary academic peers need to be included in the assessment of quality ( Cloete 1997 ; Scott 2007 ; Spappen et al. 2007 ; Klein 2008 ). Several authors discuss the lack of reviewers with strong cross-disciplinary experience ( Aagaard-Hansen and Svedin 2009 ) and the lack of common criteria, philosophical foundations, and language for use by peer reviewers ( Klein 2008 ; Aagaard-Hansen and Svedin 2009 ). Peer review of TDR could be improved with explicit TDR quality criteria, and appropriate processes in place to ensure clear dialog between reviewers.

Finally, there is the need for increased emphasis on evaluation as part of the research process ( Bergmann et al. 2005 ; Hemlin and Rasmussen 2006 ; Meyrick 2006 ; Chataway, Smith and Wield 2007 ; Stige, Malterud and Midtgarden 2009 ; Hellstrom 2011 ; Lang et al. 2012 ; Wickson and Carew 2014 ). This is particularly true in large, complex, problem-oriented research projects. Ongoing monitoring of the research organization and process contributes to learning and adaptive management while research is underway and so helps improve quality. As stated by Wickson and Carew ( 2014 : 262): ‘We believe that in any process of interpreting, rearranging and/or applying these criteria, open negotiation on their meaning and application would only positively foster transformative learning, which is a valued outcome of good TD processes’.

3.3 TDR quality criteria and assessment approaches

Many of the papers provide quality criteria and/or describe constituent parts of quality. Aagaard-Hansen and Svedin (2009) define three key aspects of quality: societal relevance, impact, and integration. Meyrick (2006) states that quality research is transparent and systematic. Boaz and Ashby (2003) describe quality in four dimensions: methodological quality, quality of reporting, appropriateness of methods, and relevance to policy and practice. Although each article deconstructs quality in different ways and with different foci and perspectives, there is significant overlap and recurring themes in the papers reviewed. There is a broadly shared perspective that TDR quality is a multidimensional concept shaped by the specific context within which research is done ( Spaapen, Dijstelbloem and Wamelink 2007 ; Klein 2008 ), making a universal definition of TDR quality difficult or impossible ( Huutoniemi 2010 ).

Huutoniemi (2010) identifies three main approaches to conceptualizing quality in IDR and TDR: (1) using existing disciplinary standards adapted as necessary for IDR; (2) building on the quality standards of disciplines while fundamentally incorporating ways to deal with epistemological integration, problem focus, context, stakeholders, and process; and (3) radical departure from any disciplinary orientation in favor of external, emergent, context-dependent quality criteria that are defined and enacted collaboratively by a community of users.

The first approach is prominent in current research funding and evaluation protocols. Conservative approaches of this kind are criticized for privileging disciplinary research and for failing to provide guidance and quality control for transdisciplinary projects. The third approach would ‘undermine the prevailing status of disciplinary standards in the pursuit of a non-disciplinary, integrated knowledge system’ ( Huutoniemi 2010 : 313). No predetermined quality criteria are offered, only contextually embedded criteria that need to be developed within a specific research project. To some extent, this is the approach taken by Spaapen, Dijstelbloem and Wamelink (2007) and de Jong et al. (2011) . Such a sui generis approach cannot be used to compare across projects. Most of the reviewed papers take the second approach, and recommend TDR quality criteria that build on a disciplinary base.

Eight articles present comprehensive frameworks for quality evaluation, each with a unique approach, perspective, and goal. Two of these build comprehensive lists of criteria with associated questions to be chosen based on the needs of the particular research project ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ). Wickson and Carew (2014) develop a reflective heuristic tool with questions to guide researchers through ongoing self-evaluation. They also list criteria for external evaluation and to compare between projects. Spaapen, Dijstelbloem and Wamelink (2007) design an approach to evaluate a research project against its own goals and is not meant to compare between projects. Wickson and Carew (2014) developed a comprehensive rubric for the evaluation of Research and Innovation that builds of their extensive previous work in TDR. Finally, Lang et al. (2012) , Mitchell and Willets (2009) , and Jahn and Keil (2015) develop criteria checklists that can be applied across transdisciplinary projects.

Bergmann et al. (2005) and Carew and Wickson (2010) organize their frameworks into managerial elements of the research project, concerning problem context, participation, management, and outcomes. Lang et al. (2012) and Defila and Di Giulio (1999) focus on the chronological stages in the research process and identify criteria at each stage. Mitchell and Willets (2009) , , with a focus on doctoral s tudies, adapt standard dissertation evaluation criteria to accommodate broader, pluralistic, and more complex studies. Spaapen, Dijstelbloem and Wamelink (2007) focus on evaluating ‘research-in-context’. Wickson and Carew (2014) created a rubric based on criteria that span the research process, stages, and all actors included. Jahn and Keil (2015) organized their quality criteria into three categories of quality including: quality of the research problems, quality of the research process, and quality of the research results.

The remaining papers highlight key themes that must be considered in TDR evaluation. Dominant themes include: engagement with problem context, collaboration and inclusion of stakeholders, heightened need for explicit communication and reflection, integration of epistemologies, recognition of diverse outputs, the focus on having an impact, and reflexivity and adaptation throughout the process. The focus on societal problems in context and the increased engagement of stakeholders in the research process introduces higher levels of complexity that cannot be accommodated by disciplinary standards ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Klein 2008 ).

Finally, authors discuss process ( Defila and Di Giulio 1999 ; Bergmann et al. 2005 ; Boix-Mansilla 2006b ; Spaapen, Dijstelbloem and Wamelink 2007 ) and utilitarian values ( Hemlin 2006 ; Ernø-Kjølhede and Hansson 2011 ; Bornmann 2013 ) as essential aspects of quality in TDR. Common themes include: (1) the importance of formative and process-oriented evaluation ( Bergmann et al. 2005 ; Hemlin 2006 ; Stige 2009 ); (2) emphasis on the evaluation process itself (not just criteria or outcomes) and reflexive dialog for learning ( Bergmann et al. 2005 ; Boix-Mansilla 2006b ; Klein 2008 ; Oberg 2008 ; Stige, Malterud and Midtgarden 2009 ; Aagaard-Hansen and Svedin 2009 ; Carew and Wickson 2010 ; Huutoniemi 2010 ); (3) the need for peers who are experienced and knowledgeable about TDR for fair peer review ( Boix-Mansilla 2006a , b ; Klein 2006 ; Hemlin 2006 ; Scott 2007 ; Aagaard-Hansen and Svedin 2009 ); (4) the inclusion of stakeholders in the evaluation process ( Bergmann et al. 2005 ; Scott 2007 ; Andréen 2010 ); and (5) the importance of evaluations that are built in-context ( Defila and Di Giulio 1999 ; Feller 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; de Jong et al. 2011 ).

While each reviewed approach offers helpful insights, none adequately fulfills the need for a broad and adaptable framework for assessing TDR quality. Wickson and Carew ( 2014 : 257) highlight the need for quality criteria that achieve balance between ‘comprehensiveness and over-prescription’: ‘any emerging quality criteria need to be concrete enough to provide real guidance but flexible enough to adapt to the specificities of varying contexts’. Based on our experience, such a framework should be:

Comprehensive: It should accommodate the main aspects of TDR, as identified in the review.

Time/phase adaptable: It should be applicable across the project cycle.

Scalable: It should be useful for projects of different scales.

Versatile: It should be useful to researchers and collaborators as a guide to research design and management, and to internal and external reviews and assessors.

Comparable: It should allow comparison of quality between and across projects/programs.

Reflexive: It should encourage and facilitate self-reflection and adaptation based on ongoing learning.

In this section, we synthesize the key principles and criteria of quality in TDR that were identified in the reviewed literature. Principles are the essential elements of high-quality TDR. Criteria are the conditions that need to be met in order to achieve a principle. We conclude by providing a framework for the evaluation of quality in TDR ( Table 3 ) and guidance for its application.

Transdisciplinary research quality assessment framework

a Research problems are the particular topic, area of concern, question to be addressed, challenge, opportunity, or focus of the research activity. Research problems are related to the societal problem but take on a specific focus, or framing, within a societal problem.

b Problem context refers to the social and environmental setting(s) that gives rise to the research problem, including aspects of: location; culture; scale in time and space; social, political, economic, and ecological/environmental conditions; resources and societal capacity available; uncertainty, complexity, and novelty associated with the societal problem; and the extent of agency that is held by stakeholders ( Carew and Wickson 2010 ).

c Words such as ‘appropriate’, ‘suitable’, and ‘adequate’ are used deliberately to allow for quality criteria to be flexible and specific enough to the needs of individual research projects ( Oberg 2008 ).

d Research process refers to the series of decisions made and actions taken throughout the entire duration of the research project and encompassing all aspects of the research project.

e Reflexivity refers to an iterative process of formative, critical reflection on the important interactions and relationships between a research project’s process, context, and product(s).

f In an ex ante evaluation, ‘evidence of’ would be replaced with ‘potential for’.

There is a strong trend in the reviewed articles to recognize the need for appropriate measures of scientific quality (usually adapted from disciplinary antecedants), but also to consider broader sets of criteria regarding the societal significance and applicability of research, and the need for engagement and representation of stakeholder values and knowledge. Cash et al. (2002) nicely conceptualize three key aspects of effective sustainability research as: salience (or relevance), credibility, and legitimacy. These are presented as necessary attributes for research to successfully produce transferable, useful information that can cross boundaries between disciplines, across scales, and between science and society. Many of the papers also refer to the principle that high-quality TDR should be effective in terms of contributing to the solution of problems. These four principles are discussed in the following sections.

4.1.1 Relevance

Relevance is the importance, significance, and usefulness of the research project's objectives, process, and findings to the problem context and to society. This includes the appropriateness of the timing of the research, the questions being asked, the outputs, and the scale of the research in relation to the societal problem being addressed. Good-quality TDR addresses important social/environmental problems and produces knowledge that is useful for decision making and problem solving ( Cash et al. 2002 ; Klein 2006 ). As Erno-Kjolhede and Hansson ( 2011 : 140) explain, quality ‘is first and foremost about creating results that are applicable and relevant for the users of the research’. Researchers must demonstrate an in-depth knowledge of and ongoing engagement with the problem context in which their research takes place ( Wickson, Carew and Russell 2006 ; Stige, Malterud and Midtgarden 2009 ; Mitchell and Willets 2009 ). From the early steps of problem formulation and research design through to the appropriate and effective communication of research findings, the applicability and relevance of the research to the societal problem must be explicitly stated and incorporated.

4.1.2 Credibility

Credibility refers to whether or not the research findings are robust and the knowledge produced is scientifically trustworthy. This includes clear demonstration that the data are adequate, with well-presented methods and logical interpretations of findings. High-quality research is authoritative, transparent, defensible, believable, and rigorous. This is the traditional purview of science, and traditional disciplinary criteria can be applied in TDR evaluation to an extent. Additional and modified criteria are needed to address the integration of epistemologies and methodologies and the development of novel methods through collaboration, the broad preparation and competencies required to carry out the research, and the need for reflection and adaptation when operating in complex systems. Having researchers actively engaged in the problem context and including extra-scientific actors as part of the research process helps to achieve relevance and legitimacy of the research; it also adds complexity and heightened requirements of transparency, reflection, and reflexivity to ensure objective, credible research is carried out.

Active reflexivity is a criterion of credibility of TDR that may seem to contradict more rigid disciplinary methodological traditions ( Carew and Wickson 2010 ). Practitioners of TDR recognize that credible work in these problem-oriented fields requires active reflexivity, epitomized by ongoing learning, flexibility, and adaptation to ensure the research approach and objectives remain relevant and fit-to-purpose ( Lincoln 1995 ; Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Mitchell and Willets 2009 ; Andreén 2010 ; Carew and Wickson 2010 ; Wickson and Carew 2014 ). Changes made during the research process must be justified and reported transparently and explicitly to maintain credibility.

The need for critical reflection on potential bias and limitations becomes more important to maintain credibility of research-in-context ( Lincoln 1995 ; Bergmann et al. 2005 ; Mitchell and Willets 2009 ; Stige, Malterud and Midtgarden 2009 ). Transdisciplinary researchers must ensure they maintain a high level of objectivity and transparency while actively engaging in the problem context. This point demonstrates the fine balance between different aspects of quality, in this case relevance and credibility, and the need to be aware of tensions and to seek complementarities ( Cash et al. 2002 ).

4.1.3 Legitimacy

Legitimacy refers to whether the research process is perceived as fair and ethical by end-users. In other words, is it acceptable and trustworthy in the eyes of those who will use it? This requires the appropriate inclusion and consideration of diverse values, interests, and the ethical and fair representation of all involved. Legitimacy may be achieved in part through the genuine inclusion of stakeholders in the research process. Whereas credibility refers to technical aspects of sound research, legitimacy deals with sociopolitical aspects of the knowledge production process and products of research. Do stakeholders trust the researchers and the research process, including funding sources and other sources of potential bias? Do they feel represented? Legitimate TDR ‘considers appropriate values, concerns, and perspectives of different actors’ ( Cash et al. 2002 : 2) and incorporates these perspectives into the research process through collaboration and mutual learning ( Bergmann et al. 2005 ; Chataway, Smith and Wield 2007 ; Andrén 2010 ; Huutoneimi 2010 ). A fair and ethical process is important to uphold standards of quality in all research. However, there are additional considerations that are unique to TDR.

Because TDR happens in-context and often in collaboration with societal actors, the disclosure of researcher perspective and a transparent statement of all partnerships, financing, and collaboration is vital to ensure an unbiased research process ( Lincoln 1995 ; Defila and Di Giulio 1999 ; Boaz and Ashby 2003 ; Barker and Pistrang 2005 ; Bergmann et al. 2005 ). The disclosure of perspective has both internal and external aspects, on one hand ensuring the researchers themselves explicitly reflect on and account for their own position, potential sources of bias, and limitations throughout the process, and on the other hand making the process transparent to those external to the research group who can then judge the legitimacy based on their perspective of fairness ( Cash et al. 2002 ).

TDR includes the engagement of societal actors along a continuum of participation from consultation to co-creation of knowledge ( Brandt et al. 2013 ). Regardless of the depth of participation, all processes that engage societal actors must ensure that inclusion/engagement is genuine, roles are explicit, and processes for effective and fair collaboration are present ( Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Hellstrom 2012 ). Important considerations include: the accurate representation of those involved; explicit and agreed-upon roles and contributions of actors; and adequate planning and procedures to ensure all values, perspectives, and contexts are adequately and appropriately incorporated. Mitchell and Willets (2009) consider cultural competence as a key criterion that can support researchers in navigating diverse epistemological perspectives. This is similar to what Morrow terms ‘social validity’, a criterion that asks researchers to be responsive to and critically aware of the diversity of perspectives and cultures influenced by their research. Several authors highlight that in order to develop this critical awareness of the diversity of cultural paradigms that operate within a problem situation, researchers should practice responsive, critical, and/or communal reflection ( Bergmann et al. 2005 ; Wickson, Carew and Russell 2006 ; Mitchell and Willets 2009 ; Carew and Wickson 2010 ). Reflection and adaptation are important quality criteria that cut across multiple principles and facilitate learning throughout the process, which is a key foundation to TD inquiry.

4.1.4 Effectiveness

We define effective research as research that contributes to positive change in the social, economic, and/or environmental problem context. Transdisciplinary inquiry is rooted in the objective of solving real-word problems ( Klein 2008 ; Carew and Wickson 2010 ) and must have the potential to ( ex ante ) or actually ( ex post ) make a difference if it is to be considered of high quality ( Erno-Kjolhede and Hansson 2011 ). Potential research effectiveness can be indicated and assessed at the proposal stage and during the research process through: a clear and stated intention to address and contribute to a societal problem, the establishment of the research process and objectives in relation to the problem context, and the continuous reflection on the usefulness of the research findings and products to the problem ( Bergmann et al. 2005 ; Lahtinen et al. 2005 ; de Jong et al. 2011 ).

Assessing research effectiveness ex post remains a major challenge, especially in complex transdisciplinary approaches. Conventional and widely used measures of ‘scientific impact’ count outputs such as journal articles and other publications and citations of those outputs (e.g. H index; i10 index). While these are useful indicators of scholarly influence, they are insufficient and inappropriate measures of research effectiveness where research aims to contribute to social learning and change. We need to also (or alternatively) focus on other kinds of research and scholarship outputs and outcomes and the social, economic, and environmental impacts that may result.

For many authors, contributing to learning and building of societal capacity are central goals of TDR ( Defila and Di Giulio 1999 ; Spaapen, Dijstelbloem and Wamelink 2007 ; Carew and Wickson 2010 ; Erno-Kjolhede and Hansson 2011 ; Hellstrom 2011 ), and so are considered part of TDR effectiveness. Learning can be characterized as changes in knowledge, attitudes, or skills and can be assessed directly, or through observed behavioral changes and network and relationship development. Some evaluation methodologies (e.g. Outcome Mapping ( Earl, Carden and Smutylo 2001 )) specifically measure these kinds of changes. Other evaluation methodologies consider the role of research within complex systems and assess effectiveness in terms of contributions to changes in policy and practice and resulting social, economic, and environmental benefits ( ODI 2004 , 2012 ; White and Phillips 2012 ; Mayne et al. 2013 ).

4.2 TDR quality criteria

TDR quality criteria and their definitions (explicit or implicit) were extracted from each article and summarized in an Excel database. These criteria were classified into themes corresponding to the four principles identified above, sorted and refined to develop sets of criteria that are comprehensive, mutually exclusive, and representative of the ideas presented in the reviewed articles. Within each principle, the criteria are organized roughly in the sequence of a typical project cycle (e.g. with research design following problem identification and preceding implementation). Definitions of each criterion were developed to reflect the concepts found in the literature, tested and refined iteratively to improve clarity. Rubric statements were formulated based on the literature and our own experience.

The complete set of principles, criteria, and definitions is presented as the TDR Quality Assessment Framework ( Table 3 ).

4.3 Guidance on the application of the framework

4.3.1 timing.

Most criteria can be applied at each stage of the research process, ex ante , mid term, and ex post , using appropriate interpretations at each stage. Ex ante (i.e. proposal) assessment should focus on a project’s explicitly stated intentions and approaches to address the criteria. Mid-term indicators will focus on the research process and whether or not it is being implemented in a way that will satisfy the criteria. Ex post assessment should consider whether the research has been done appropriately for the purpose and that the desired results have been achieved.

4.3.2 New meanings for familiar terms

Many of the terms used in the framework are extensions of disciplinary criteria and share the same or similar names and perhaps similar but nuanced meaning. The principles and criteria used here extend beyond disciplinary antecedents and include new concepts and understandings that encapsulate the unique characteristics and needs of TDR and allow for evaluation and definition of quality in TDR. This is especially true in the criteria related to credibility. These criteria are analogous to traditional disciplinary criteria, but with much stronger emphasis on grounding in both the scientific and the social/environmental contexts. We urge readers to pay close attention to the definitions provided in Table 3 as well as the detailed descriptions of the principles in Section 4.1.

4.3.3 Using the framework

The TDR quality framework ( Table 3 ) is designed to be used to assess TDR research according to a project’s purpose; i.e. the criteria must be interpreted with respect to the context and goals of an individual research activity. The framework ( Table 3 ) lists the main criteria synthesized from the literature and our experience, organized within the principles of relevance, credibility, legitimacy, and effectiveness. The table presents the criteria within each principle, ordered to approximate a typical process of identifying a research problem and designing and implementing research. We recognize that the actual process in any given project will be iterative and will not necessarily follow this sequence, but this provides a logical flow. A concise definition is provided in the second column to explain each criterion. We then provide a rubric statement in the third column, phrased to be applied when the research has been completed. In most cases, the same statement can be used at the proposal stage with a simple tense change or other minor grammatical revision, except for the criteria relating to effectiveness. As discussed above, assessing effectiveness in terms of outcomes and/or impact requires evaluation research. At the proposal stage, it is only possible to assess potential effectiveness.

Many rubrics offer a set of statements for each criterion that represent progressively higher levels of achievement; the evaluator is asked to select the best match. In practice, this often results in vague and relative statements of merit that are difficult to apply. We have opted to present a single rubric statement in absolute terms for each criterion. The assessor can then rank how well a project satisfies each criterion using a simple three-point Likert scale. If a project fully satisfies a criterion—that is, if there is evidence that the criterion has been addressed in a way that is coherent, explicit, sufficient, and convincing—it should be ranked as a 2 for that criterion. A score of 2 means that the evaluator is persuaded that the project addressed that criterion in an intentional, appropriate, explicit, and thorough way. A score of 1 would be given when there is some evidence that the criterion was considered, but it is lacking completion, intention, and/or is not addressed satisfactorily. For example, a score of 1 would be given when a criterion is explicitly discussed but poorly addressed, or when there is some indication that the criterion has been considered and partially addressed but it has not been treated explicitly, thoroughly, or adequately. A score of 0 indicates that there is no evidence that the criterion was addressed or that it was addressed in a way that was misguided or inappropriate.

It is critical that the evaluation be done in context, keeping in mind the purpose, objectives, and resources of the project, as well as other contextual information, such as the intended purpose of grant funding or relevant partnerships. Each project will be unique in its complexities; what is sufficient or adequate in one criterion for one research project may be insufficient or inappropriate for another. Words such as ‘appropriate’, ‘suitable’, and ‘adequate’ are used deliberately to encourage application of criteria to suit the needs of individual research projects ( Oberg 2008 ). Evaluators must consider the objectives of the research project and the problem context within which it is carried out as the benchmark for evaluation. For example, we tested the framework with RRU masters theses. These are typically small projects with limited scope, carried out by a single researcher. Expectations for ‘effective communication’ or ‘competencies’ or ‘effective collaboration’ are much different in these kinds of projects than in a multi-year, multi-partner CIFOR project. All criteria should be evaluated through the lens of the stated research objectives, research goals, and context.

The systematic review identified relevant articles from a diverse literature that have a strong central focus. Collectively, they highlight the complexity of contemporary social and environmental problems and emphasize that addressing such issues requires combinations of new knowledge and innovation, action, and engagement. Traditional disciplinary research has often failed to provide solutions because it cannot adequately cope with complexity. New forms of research are proliferating, crossing disciplinary and academic boundaries, integrating methodologies, and engaging a broader range of research participants, as a way to make research more relevant and effective. Theoretically, such approaches appear to offer great potential to contribute to transformative change. However, because these approaches are new and because they are multidimensional, complex, and often unique, it has been difficult to know what works, how, and why. In the absence of the kinds of methodological and quality standards that guide disciplinary research, there are no generally agreed criteria for evaluating such research.

Criteria are needed to guide and to help ensure that TDR is of high quality, to inform the teaching and learning of new researchers, and to encourage and support the further development of transdisciplinary approaches. The lack of a standard and broadly applicable framework for the evaluation of quality in TDR is perceived to cause an implicit or explicit devaluation of high-quality TDR or may prevent quality TDR from being done. There is a demonstrated need for an operationalized understanding of quality that addresses the characteristics, contributions, and challenges of TDR. The reviewed articles approach the topic from different perspectives and fields of study, using different terminology for similar concepts, or the same terminology for different concepts, and with unique ways of organizing and categorizing the dimensions and quality criteria. We have synthesized and organized these concepts as key TDR principles and criteria in a TDR Quality Framework, presented as an evaluation rubric. We have tested the framework on a set of masters’ theses and found it to be broadly applicable, usable, and useful for analyzing individual projects and for comparing projects within the set. We anticipate that further testing with a wider range of projects will help further refine and improve the definitions and rubric statements. We found that the three-point Likert scale (0–2) offered sufficient variability for our purposes, and rating is less subjective than with relative rubric statements. It may be possible to increase the rating precision with more points on the scale to increase the sensitivity for comparison purposes, for example in a review of proposals for a particular grant application.

Many of the articles we reviewed emphasize the importance of the evaluation process itself. The formative, developmental role of evaluation in TDR is seen as essential to the goals of mutual learning as well as to ensure that research remains responsive and adaptive to the problem context. In order to adequately evaluate quality in TDR, the process, including who carries out the evaluations, when, and in what manner, must be revised to be suitable to the unique characteristics and objectives of TDR. We offer this review and synthesis, along with a proposed TDR quality evaluation framework, as a contribution to an important conversation. We hope that it will be useful to researchers and research managers to help guide research design, implementation and reporting, and to the community of research organizations, funders, and society at large. As underscored in the literature review, there is a need for an adapted research evaluation process that will help advance problem-oriented research in complex systems, ultimately to improve research effectiveness.

This work was supported by funding from the Canada Research Chairs program. Funding support from the Canadian Social Sciences and Humanities Research Council (SSHRC) and technical support from the Evidence Based Forestry Initiative of the Centre for International Forestry Research (CIFOR), funded by UK DfID are also gratefully acknowledged.

Supplementary data is available here

The authors thank Barbara Livoreil and Stephen Dovers for valuable comments and suggestions on the protocol and Gillian Petrokofsky for her review of the protocol and a draft version of the manuscript. Two anonymous reviewers and the editor provided insightful critique and suggestions in two rounds that have helped to substantially improve the article.

Conflict of interest statement . None declared.

1. ‘Stakeholders’ refers to individuals and groups of societal actors who have an interest in the issue or problem that the research seeks to address.

2. The terms ‘quality’ and ‘excellence’ are often used in the literature with similar meaning. Technically, ‘excellence’ is a relative concept, referring to the superiority of a thing compared to other things of its kind. Quality is an attribute or a set of attributes of a thing. We are interested in what these attributes are or should be in high-quality research. Therefore, the term ‘quality’ is used in this discussion.

3. The terms ‘science’ and ‘research’ are not always clearly distinguished in the literature. We take the position that ‘science’ is a more restrictive term that is properly applied to systematic investigations using the scientific method. ‘Research’ is a broader term for systematic investigations using a range of methods, including but not restricted to the scientific method. We use the term ‘research’ in this broad sense.

Aagaard-Hansen J. Svedin U. ( 2009 ) ‘Quality Issues in Cross-disciplinary Research: Towards a Two-pronged Approach to Evaluation’ , Social Epistemology , 23 / 2 : 165 – 76 . DOI: 10.1080/02691720902992323

Google Scholar

Andrén S. ( 2010 ) ‘A Transdisciplinary, Participatory and Action-Oriented Research Approach: Sounds Nice but What do you Mean?’ [unpublished working paper] Human Ecology Division: Lund University, 1–21. < https://lup.lub.lu.se/search/publication/1744256 >

Australian Research Council (ARC) ( 2012 ) ERA 2012 Evaluation Handbook: Excellence in Research for Australia . Australia : ARC . < http://www.arc.gov.au/pdf/era12/ERA%202012%20Evaluation%20Handbook_final%20for%20web_protected.pdf >

Google Preview

Balsiger P. W. ( 2004 ) ‘Supradisciplinary Research Practices: History, Objectives and Rationale’ , Futures , 36 / 4 : 407 – 21 .

Bantilan M. C. et al.  . ( 2004 ) ‘Dealing with Diversity in Scientific Outputs: Implications for International Research Evaluation’ , Research Evaluation , 13 / 2 : 87 – 93 .

Barker C. Pistrang N. ( 2005 ) ‘Quality Criteria under Methodological Pluralism: Implications for Conducting and Evaluating Research’ , American Journal of Community Psychology , 35 / 3-4 : 201 – 12 .

Bergmann M. et al.  . ( 2005 ) Quality Criteria of Transdisciplinary Research: A Guide for the Formative Evaluation of Research Projects . Central report of Evalunet – Evaluation Network for Transdisciplinary Research. Frankfurt am Main, Germany: Institute for Social-Ecological Research. < http://www.isoe.de/ftp/evalunet_guide.pdf >

Boaz A. Ashby D. ( 2003 ) Fit for Purpose? Assessing Research Quality for Evidence Based Policy and Practice .

Boix-Mansilla V. ( 2006a ) ‘Symptoms of Quality: Assessing Expert Interdisciplinary Work at the Frontier: An Empirical Exploration’ , Research Evaluation , 15 / 1 : 17 – 29 .

Boix-Mansilla V. . ( 2006b ) ‘Conference Report: Quality Assessment in Interdisciplinary Research and Education’ , Research Evaluation , 15 / 1 : 69 – 74 .

Bornmann L. ( 2013 ) ‘What is Societal Impact of Research and How can it be Assessed? A Literature Survey’ , Journal of the American Society for Information Science and Technology , 64 / 2 : 217 – 33 .

Brandt P. et al.  . ( 2013 ) ‘A Review of Transdisciplinary Research in Sustainability Science’ , Ecological Economics , 92 : 1 – 15 .

Cash D. Clark W.C. Alcock F. Dickson N. M. Eckley N. Jäger J . ( 2002 ) Salience, Credibility, Legitimacy and Boundaries: Linking Research, Assessment and Decision Making (November 2002). KSG Working Papers Series RWP02-046. Available at SSRN: http://ssrn.com/abstract=372280 .

Carew A. L. Wickson F. ( 2010 ) ‘The TD Wheel: A Heuristic to Shape, Support and Evaluate Transdisciplinary Research’ , Futures , 42 / 10 : 1146 – 55 .

Collaboration for Environmental Evidence (CEE) . ( 2013 ) Guidelines for Systematic Review and Evidence Synthesis in Environmental Management . Version 4.2. Environmental Evidence < www.environmentalevidence.org/Documents/Guidelines/Guidelines4.2.pdf >

Chandler J. ( 2014 ) Methods Research and Review Development Framework: Policy, Structure, and Process . < http://methods.cochrane.org/projects-developments/research >

Chataway J. Smith J. Wield D. ( 2007 ) ‘Shaping Scientific Excellence in Agricultural Research’ , International Journal of Biotechnology 9 / 2 : 172 – 87 .

Clark W. C. Dickson N. ( 2003 ) ‘Sustainability Science: The Emerging Research Program’ , PNAS 100 / 14 : 8059 – 61 .

Consultative Group on International Agricultural Research (CGIAR) ( 2011 ) A Strategy and Results Framework for the CGIAR . < http://library.cgiar.org/bitstream/handle/10947/2608/Strategy_and_Results_Framework.pdf?sequence=4 >

Cloete N. ( 1997 ) ‘Quality: Conceptions, Contestations and Comments’, African Regional Consultation Preparatory to the World Conference on Higher Education , Dakar, Senegal, 1-4 April 1997 .

Defila R. DiGiulio A. ( 1999 ) ‘Evaluating Transdisciplinary Research,’ Panorama: Swiss National Science Foundation Newsletter , 1 : 4 – 27 . < www.ikaoe.unibe.ch/forschung/ip/Specialissue.Pano.1.99.pdf >

Donovan C. ( 2008 ) ‘The Australian Research Quality Framework: A Live Experiment in Capturing the Social, Economic, Environmental, and Cultural Returns of Publicly Funded Research. Reforming the Evaluation of Research’ , New Directions for Evaluation , 118 : 47 – 60 .

Earl S. Carden F. Smutylo T. ( 2001 ) Outcome Mapping. Building Learning and Reflection into Development Programs . Ottawa, ON : International Development Research Center .

Ernø-Kjølhede E. Hansson F. ( 2011 ) ‘Measuring Research Performance during a Changing Relationship between Science and Society’ , Research Evaluation , 20 / 2 : 130 – 42 .

Feller I. ( 2006 ) ‘Assessing Quality: Multiple Actors, Multiple Settings, Multiple Criteria: Issues in Assessing Interdisciplinary Research’ , Research Evaluation 15 / 1 : 5 – 15 .

Gaziulusoy A. İ. Boyle C. ( 2013 ) ‘Proposing a Heuristic Reflective Tool for Reviewing Literature in Transdisciplinary Research for Sustainability’ , Journal of Cleaner Production , 48 : 139 – 47 .

Gibbons M. et al.  . ( 1994 ) The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies . London : Sage Publications .

Hellstrom T. ( 2011 ) ‘Homing in on Excellence: Dimensions of Appraisal in Center of Excellence Program Evaluations’ , Evaluation , 17 / 2 : 117 – 31 .

Hellstrom T. . ( 2012 ) ‘Epistemic Capacity in Research Environments: A Framework for Process Evaluation’ , Prometheus , 30 / 4 : 395 – 409 .

Hemlin S. Rasmussen S. B . ( 2006 ) ‘The Shift in Academic Quality Control’ , Science, Technology & Human Values , 31 / 2 : 173 – 98 .

Hessels L. K. Van Lente H. ( 2008 ) ‘Re-thinking New Knowledge Production: A Literature Review and a Research Agenda’ , Research Policy , 37 / 4 , 740 – 60 .

Huutoniemi K. ( 2010 ) ‘Evaluating Interdisciplinary Research’ , in Frodeman R. Klein J. T. Mitcham C. (eds) The Oxford Handbook of Interdisciplinarity , pp. 309 – 20 . Oxford : Oxford University Press .

de Jong S. P. L. et al.  . ( 2011 ) ‘Evaluation of Research in Context: An Approach and Two Cases’ , Research Evaluation , 20 / 1 : 61 – 72 .

Jahn T. Keil F. ( 2015 ) ‘An Actor-Specific Guideline for Quality Assurance in Transdisciplinary Research’ , Futures , 65 : 195 – 208 .

Kates R. ( 2000 ) ‘Sustainability Science’ , World Academies Conference Transition to Sustainability in the 21st Century 5/18/00 , Tokyo, Japan .

Klein J. T . ( 2006 ) ‘Afterword: The Emergent Literature on Interdisciplinary and Transdisciplinary Research Evaluation’ , Research Evaluation , 15 / 1 : 75 – 80 .

Klein J. T . ( 2008 ) ‘Evaluation of Interdisciplinary and Transdisciplinary Research: A Literature Review’ , American Journal of Preventive Medicine , 35 / 2 Supplment S116–23. DOI: 10.1016/j.amepre.2008.05.010

Royal Netherlands Academy of Arts and Sciences, Association of Universities in the Netherlands, Netherlands Organization for Scientific Research (KNAW) . ( 2009 ) Standard Evaluation Protocol 2009-2015: Protocol for Research Assessment in the Netherlands . Netherlands : KNAW . < www.knaw.nl/sep >

Komiyama H. Takeuchi K. ( 2006 ) ‘Sustainability Science: Building a New Discipline’ , Sustainability Science , 1 : 1 – 6 .

Lahtinen E. et al.  . ( 2005 ) ‘The Development of Quality Criteria For Research: A Finnish approach’ , Health Promotion International , 20 / 3 : 306 – 15 .

Lang D. J. et al.  . ( 2012 ) ‘Transdisciplinary Research in Sustainability Science: Practice , Principles , and Challenges’, Sustainability Science , 7 / S1 : 25 – 43 .

Lincoln Y. S . ( 1995 ) ‘Emerging Criteria for Quality in Qualitative and Interpretive Research’ , Qualitative Inquiry , 1 / 3 : 275 – 89 .

Mayne J. Stern E. ( 2013 ) Impact Evaluation of Natural Resource Management Research Programs: A Broader View . Australian Centre for International Agricultural Research, Canberra .

Meyrick J . ( 2006 ) ‘What is Good Qualitative Research? A First Step Towards a Comprehensive Approach to Judging Rigour/Quality’ , Journal of Health Psychology , 11 / 5 : 799 – 808 .

Mitchell C. A. Willetts J. R. ( 2009 ) ‘Quality Criteria for Inter and Trans - Disciplinary Doctoral Research Outcomes’ , in Prepared for ALTC Fellowship: Zen and the Art of Transdisciplinary Postgraduate Studies ., Sydney : Institute for Sustainable Futures, University of Technology .

Morrow S. L . ( 2005 ) ‘Quality and Trustworthiness in Qualitative Research in Counseling Psychology’ , Journal of Counseling Psychology , 52 / 2 : 250 – 60 .

Nowotny H. Scott P. Gibbons M. ( 2001 ) Re-Thinking Science . Cambridge : Polity .

Nowotny H. Scott P. Gibbons M. . ( 2003 ) ‘‘Mode 2’ Revisited: The New Production of Knowledge’ , Minerva , 41 : 179 – 94 .

Öberg G . ( 2008 ) ‘Facilitating Interdisciplinary Work: Using Quality Assessment to Create Common Ground’ , Higher Education , 57 / 4 : 405 – 15 .

Ozga J . ( 2007 ) ‘Co - production of Quality in the Applied Education Research Scheme’ , Research Papers in Education , 22 / 2 : 169 – 81 .

Ozga J . ( 2008 ) ‘Governing Knowledge: research steering and research quality’ , European Educational Research Journal , 7 / 3 : 261 – 272 .

OECD ( 2012 ) Frascati Manual 6th ed. < http://www.oecd.org/innovation/inno/frascatimanualproposedstandardpracticeforsurveysonresearchandexperimentaldevelopment6thedition >

Overseas Development Institute (ODI) ( 2004 ) ‘Bridging Research and Policy in International Development: An Analytical and Practical Framework’, ODI Briefing Paper. < http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/198.pdf >

Overseas Development Institute (ODI) . ( 2012 ) RAPID Outcome Assessment Guide . < http://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/7815.pdf >

Pullin A. S. Stewart G. B. ( 2006 ) ‘Guidelines for Systematic Review in Conservation and Environmental Management’ , Conservation Biology , 20 / 6 : 1647 – 56 .

Research Excellence Framework (REF) . ( 2011 ) Research Excellence Framework 2014: Assessment Framework and Guidance on Submissions. Reference REF 02.2011. UK: REF. < http://www.ref.ac.uk/pubs/2011-02/ >

Scott A . ( 2007 ) ‘Peer Review and the Relevance of Science’ , Futures , 39 / 7 : 827 – 45 .

Spaapen J. Dijstelbloem H. Wamelink F. ( 2007 ) Evaluating Research in Context: A Method for Comprehensive Assessment . Netherlands: Consultative Committee of Sector Councils for Research and Development. < http://www.qs.univie.ac.at/fileadmin/user_upload/qualitaetssicherung/PDF/Weitere_Aktivit%C3%A4ten/Eric.pdf >

Spaapen J. Van Drooge L. ( 2011 ) ‘Introducing “Productive Interactions” in Social Impact Assessment’ , Research Evaluation , 20 : 211 – 18 .

Stige B. Malterud K. Midtgarden T. ( 2009 ) ‘Toward an Agenda for Evaluation of Qualitative Research’ , Qualitative Health Research , 19 / 10 : 1504 – 16 .

td-net ( 2014 ) td-net. < www.transdisciplinarity.ch/e/Bibliography/new.php >

Tertiary Education Commission (TEC) . ( 2012 ) Performance-based Research Fund: Quality Evaluation Guidelines 2012. New Zealand: TEC. < http://www.tec.govt.nz/Documents/Publications/PBRF-Quality-Evaluation-Guidelines-2012.pdf >

Tijssen R. J. W. ( 2003 ) ‘Quality Assurance: Scoreboards of Research Excellence’ , Research Evaluation , 12 : 91 – 103 .

White H. Phillips D. ( 2012 ) ‘Addressing Attribution of Cause and Effect in Small n Impact Evaluations: Towards an Integrated Framework’. Working Paper 15. New Delhi: International Initiative for Impact Evaluation .

Wickson F. Carew A. ( 2014 ) ‘Quality Criteria and Indicators for Responsible Research and Innovation: Learning from Transdisciplinarity’ , Journal of Responsible Innovation , 1 / 3 : 254 – 73 .

Wickson F. Carew A. Russell A. W. ( 2006 ) ‘Transdisciplinary Research: Characteristics, Quandaries and Quality,’ Futures , 38 / 9 : 1046 – 59

Email alerts

Citing articles via.

• Recommend to your Library

Affiliations

• Online ISSN 1471-5449
• Print ISSN 0958-2029
• Copyright © 2024 Oxford University Press
• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

What is quality research? A guide to identifying the key features and achieving success

Every researcher worth their salt strives for quality. But in research, what does quality mean?

Simply put, quality research is thorough, accurate, original and relevant. And to achieve this, you need to follow specific standards. You need to make sure your findings are reliable and valid. And when you know they're quality assured, you can share them with absolute confidence.

You’ll be able to draw accurate conclusions from your investigations and contribute to the wider body of knowledge in your field.

Importance of quality research

Quality research helps us better understand complex problems. It enables us to make decisions based on facts and evidence. And it empowers us to solve real-world issues. Without quality research, we can't advance knowledge or identify trends and patterns. We also can’t develop new theories and approaches to solving problems.

With rigorous and transparent research methods, you’ll produce reliable findings that other researchers can replicate. This leads to the development of new theories and interventions. On the other hand, low-quality research can hinder progress by producing unreliable findings that can’t be replicated, wasting resources and impeding advancements in the field.

In all cases, quality control is critical. It ensures that decisions are based on evidence rather than gut feeling or bias.

Standards for quality research

Over the years, researchers, scientists and authors have come to a consensus about the standards used to check the quality of research. Determined through empirical observation, theoretical underpinnings and philosophy of science, these include:

1. Having a well-defined research topic and a clear hypothesis

This is essential to verify that the research is focused and the results are relevant and meaningful. The research topic should be well-scoped and the hypothesis should be clearly stated and falsifiable .

For example, in a quantitative study about the effects of social media on behavior, a well-defined research topic could be, "Does the use of TikTok reduce attention span in American adolescents?"

This is good because:

• The research topic focuses on a particular platform of social media (TikTok). And it also focuses on a specific group of people (American adolescents).
• The research question is clear and straightforward, making it easier to design the study and collect relevant data.
• You can test the hypothesis and a research team can evaluate it easily. This can be done through the use of various research methods, such as survey research , experiments or observational studies.
• The hypothesis is focused on a specific outcome (the attention span). Then, this can be measured and compared to control groups or previous research studies.

2. Ensuring transparency

Transparency is crucial when conducting research. You need to be upfront about the methods you used, such as:

• Describing how you recruited the participants.
• How you communicated with them.
• How they were incentivized.

You also need to explain how you analyzed the data, so other researchers can replicate your results if necessary. re-registering your study is a great way to be as transparent in your research as possible. This  involves publicly documenting your study design, methods and analysis plan before conducting the research. This reduces the risk of selective reporting and increases the credibility of your findings.

3. Using appropriate research methods

Depending on the topic, some research methods are better suited than others for collecting data. To use our TikTok example, a quantitative research approach, such as a behavioral test that measures the participants' ability to focus on tasks, might be the most appropriate.

On the other hand, for topics that require a more in-depth understanding of individuals' experiences or perspectives, a qualitative research approach, such as interviews or focus groups, might be more suitable. These methods can provide rich and detailed information that you can’t capture through quantitative data alone.

4. Assessing limitations and the possible impact of systematic bias

When you present your research, it’s important to consider how the limitations of your study could affect the result. This could be systematic bias in the sampling procedure or data analysis, for instance. Let’s say you only study a small sample of participants from one school district. This would limit the generalizability and content validity of your findings.

5. Conducting accurate reporting

This is an essential aspect of any research project. You need to be able to clearly communicate the findings and implications of your study . Also, provide citations for any claims made in your report. When you present your work, it’s vital that you describe the variables involved in your study accurately and how you measured them.

Curious to learn more? Read our Data Quality eBook .

How to identify credible research findings

To determine whether a published study is trustworthy, consider the following:

• Peer review: If a study has been peer-reviewed by recognized experts, rest assured that it’s a reliable source of information. Peer review means that other scholars have read and verified the study before publication.
• Researcher's qualifications: If they're an expert in the field, that’s a good sign that you can trust their findings. However, if they aren't, it doesn’t necessarily mean that the study's information is unreliable. It simply means that you should be extra cautious about accepting its conclusions as fact.
• Study design: The design of a study can make or break its reliability. Consider factors like sample size and methodology.
• Funding source: Studies funded by organizations with a vested interest in a particular outcome may be less credible than those funded by independent sources.
• Statistical significance: You've heard the phrase "numbers don't lie," right? That's what statistical significance is all about. It refers to the likelihood that the results of a study occurred by chance. Results that are statistically significant are more credible.

Achieve quality research with Prolific

Want to ensure your research is high-quality? Prolific can help.

Our platform gives you access to a carefully vetted pool of participants. We make sure they're attentive, honest, and ready to provide rich and detailed answers where needed. This helps to ensure that the data you collect through Prolific is of the highest quality.

With Prolific, you can streamline your research process and feel confident in the results you receive. Our minimum pay threshold and commitment to fair compensation motivate participants to provide valuable responses and give their best effort. This ensures the quality of your research and helps you get the results you need. Sign up as a researcher today to get started!

You might also like

High-quality human data to deliver world-leading research and AIs.

Follow us on

All Rights Reserved Prolific 2024

• - Google Chrome

Intended for healthcare professionals

• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Assessing the quality...

Assessing the quality of research

• Related content
• Peer review

This article has a correction. Please see:

• Errata - September 09, 2004
• Paul Glasziou ([email protected]) , reader 1 ,
• Jan Vandenbroucke , professor of clinical epidemiology 2 ,
• Iain Chalmers , editor, James Lind library 3
• 1 Department of Primary Health Care, University of Oxford, Oxford OX3 7LF
• 2 Leiden University Medical School, Leiden 9600 RC, Netherlands
• 3 James Lind Initiative, Oxford OX2 7LG
• Correspondence to: P Glasziou
• Accepted 20 October 2003

Inflexible use of evidence hierarchies confuses practitioners and irritates researchers. So how can we improve the way we assess research?

The widespread use of hierarchies of evidence that grade research studies according to their quality has helped to raise awareness that some forms of evidence are more trustworthy than others. This is clearly desirable. However, the simplifications involved in creating and applying hierarchies have also led to misconceptions and abuses. In particular, criteria designed to guide inferences about the main effects of treatment have been uncritically applied to questions about aetiology, diagnosis, prognosis, or adverse effects. So should we assess evidence the way Michelin guides assess hotels and restaurants? We believe five issues should be considered in any revision or alternative approach to helping practitioners to find reliable answers to important clinical questions.

Different types of question require different types of evidence

Ever since two American social scientists introduced the concept in the early 1960s, 1 hierarchies have been used almost exclusively to determine the effects of interventions. This initial focus was appropriate but has also engendered confusion. Although interventions are central to clinical decision making, practice relies on answers to a wide variety of types of clinical questions, not just the effect of interventions. 2 Other hierarchies might be necessary to answer questions about aetiology, diagnosis, disease frequency, prognosis, and adverse effects. 3 Thus, although a systematic review of randomised trials would be appropriate for answering questions about the main effects of a treatment, it would be ludicrous to attempt to use it to ascertain the relative accuracy of computerised versus human reading of cervical smears, the natural course of prion diseases in humans, the effect of carriership of a mutation on the risk of venous thrombosis, or the rate of vaginal adenocarcinoma in the daughters of pregnant women given diethylstilboesterol. 4

To answer their everyday questions, practitioners …

Log in using your username and password

BMA Member Log In

If you have a subscription to The BMJ, log in:

• Need to activate
• Log in via institution
• Log in via OpenAthens

Log in through your institution

Subscribe from £184 *.

Subscribe and get access to all BMJ articles, and much more.

* For online subscription

Access this article for 1 day for: £33 / $40 / €36 ( excludes VAT )

You can download a PDF version for your personal record.

Buy this article

Creating a Culture of Quality

• Ashwin Srinivasan
• Bryan Kurey

Financial incentives don’t reduce errors. Employees must be passionate about eliminating mistakes.

In most industries, quality has never mattered more. New technologies have empowered customers to seek out and compare an endless array of products from around the globe. Shoppers can click to find objective data compiled by experts at organizations such as Consumer Reports and J.D. Power and go online to read user-generated reviews at sites such as Amazon; together, these sources provide an early warning system that alerts the public to quality problems. And when customers are unhappy with a product or service, they can use social media to broadcast their displeasure. In surveys, 26% of consumers say they have used social media to air grievances about a company and its products. And this issue isn’t limited to the consumer space—75% of B2B customers say they rely on word of mouth, including social media, when making purchase decisions.

• AS Ashwin Srinivasan is a managing director, and Bryan Kurey is a senior director, at CEB.
• BK Bryan Kurey is the Senior Vice President of Research at SBI Growth Advisory.

Partner Center

This device is too small

If you're on a Galaxy Fold, consider unfolding your phone or viewing it in full screen to best optimize your experience.

• Personal Finance

5 Amazing Costco Buys for Less Than $100

Published on June 22, 2024

By: Natasha Etzel

• You can score some fantastic finds for less than $100 each at Costco.
• Some worthwhile Costco buys for under $100 include an Igloo KoolTunes Bluetooth Speaker Cooler, Waterpik Ultra Plus and Cordless Pearl Water Flosser Combo Pack, and a Midea 11-quart Dual Basket Air Fryer Oven.

Check out our pick for the best cash back credit card of 2024

Investing in a warehouse club membership like Costco can save you money. You can shop members-only deals when buying food, household essentials, toiletries, electronics, clothing, and more. While it costs $60 to $120 a year to be a member, many shoppers find their yearly savings exceed the membership fee. Here are a few worthwhile Costco buys for less than $100.

1. A two-pack of outdoor string lights for $89.99

You'll get two 48-foot LED string lights when you purchase a two-pack. This is an excellent price for heavy-duty indoor and outdoor safe string lights, and this set has great reviews.

2. Bluetooth speaker cooler for $99.99

Whether you're seeking a cooler for game day tailgates, pool days, or vacation, you can buy one at Costco. If you're also a fan of listening to music, consider getting a Bluetooth speaker cooler so you have a cooler and a speaker for your many adventures.

Costco sells the highly rated Igloo KoolTunes Bluetooth Speaker Cooler for $99.99. Igloo sells the same cooler for $149.99 on its own website, so this is a great deal. You'll get a 14-quart cooler with a built-in Bluetooth speaker and a three-foot USB charging cable.

3. Durable, portable chair for $89.99

You never know when you'll need a camping chair. Can you get one for $20? Sure. Will it be the most comfortable chair you can find? Probably not. Costco sells some high-quality portable chairs at affordable price points.

One option is the Mac Sports Heavy Duty Camp Chair, which members can buy for $89.99. This portable chair has two cup holders. Many of the reviews highlight its durability and comfort. You'll be glad you invested in a quality chair like this.

4. Water flosser and cordless water flosser combo for $79.99

A water flosser can step up your oral hygiene routine and improve the health of your teeth, gums, and mouth. Costco sells various dental care products, including water flossers.

You can score the Waterpik Ultra Plus and Cordless Pearl Water Flosser Combo Pack for $79.99. Waterpik sells this device without the cordless Pearl Water Flosser for $89.99, so this is a deal.

5. Dual air basket and toaster oven for $99.99

If you want to buy an air fryer and use toaster ovens, a combination device can help you get what you need while freeing up counter space. Costco members can purchase the Midea 11-quart Dual Basket Air Fryer Oven for $99.99.

The manufacturer sells this device for $159, so you'd get a good deal by purchasing it at Costco. This 11-quart kitchen item features a 6-quart air frying basket and a 5-quart toaster oven, which can simplify your cooking experience.

Get the most from your Costco membership

If you're a Costco member or will join soon, get the most out of your membership perks. Keep alert to new product and service offers and shop the best deals to keep more money in your checking account . In addition, consider the best payment method option before you check out.

Want to save more money at Costco ? To maximize your savings, we suggest using one of the best credit cards for Costco to pay for your Costco hauls. You can earn rewards when you swipe your card. Once you have enough rewards earned, you can redeem them for cash back, allowing you to save even more money when shopping at Costco.

Alert: our top-rated cash back card now has 0% intro APR until 2025

This credit card is not just good – it’s so exceptional that our experts use it personally. It features a lengthy 0% intro APR period, a cash back rate of up to 5%, and all somehow for no annual fee! Click here to read our full review for free and apply in just 2 minutes.

Our Research Expert

Natasha is a freelance writer who specializes in personal finance, credit card, credit card rewards, and travel hacking content.

Share this page

We're firm believers in the Golden Rule, which is why editorial opinions are ours alone and have not been previously reviewed, approved, or endorsed by included advertisers. The Ascent, a Motley Fool service, does not cover all offers on the market. The Ascent has a dedicated team of editors and analysts focused on personal finance, and they follow the same set of publishing standards and editorial integrity while maintaining professional separation from the analysts and editors on other Motley Fool brands.

Related Articles

By: Cole Tretheway | Published on June 7, 2024

By: Lyle Daly | Published on June 5, 2024

By: Christy Bieber | Published on June 5, 2024

By: Lyle Daly | Published on June 4, 2024

The Ascent is a Motley Fool service that rates and reviews essential products for your everyday money matters.

Copyright © 2018 - 2024 The Ascent. All rights reserved.

• Starting a Business
• Growing a Business
• Small Business Guide
• Business News
• Science & Technology
• Money & Finance
• For Subscribers
• Write for Entrepreneur
• Tips White Papers
• Entrepreneur Store
• United States
• Asia Pacific
• Middle East
• South Africa

Copyright © 2024 Entrepreneur Media, LLC All rights reserved. Entrepreneur® and its related marks are registered trademarks of Entrepreneur Media LLC

Here Are the SEO Metrics That Matter in 2024 Explore the crucial SEO metrics for 2024 that every digital marketer needs to focus on for enhanced search engine visibility and performance.

By Nikola Baldikov Edited by Chelsea Brown Jun 28, 2024

Key Takeaways

• Tracking the right metrics is crucial for optimizing your online performance and achieving long-term SEO success.
• This article highlights the specific metrics you need to focus on in 2024 in order to boost your SEO performance.

Opinions expressed by Entrepreneur contributors are their own.

As someone who has been involved in search engine optimization (SEO) for nearly a decade now, I know how important it is to track metrics. While not every metric has the same value, it is essential to keep your finger on the pulse of your online performance.

From my observations and practical experience, there are a few SEO metrics that really matter in 2024. Specifically, these can be divided into on-page and off-page ones. Nevertheless, they will be crucial for your concerted efforts to ensure you skyrocket your SEO performance.

So, without further ado, here are the metrics I measure and keep tabs on and recommend for highly effective SEO endeavors.

Related: Experts Share The 5 Things You Need to Understand About SEO in 2024

Keyword research and analysis

I'd like to kick off by mentioning the importance of keyword research and analysis. This should be the foundation of any SEO strategy. However, choosing the right keywords can feel like looking for a needle in a haystack.

This is where using the right tools comes into play. And once you have these tools at your fingertips, you need to use them wisely. Whether you are creating a sales page or a blog article, you must consider your users' search intent when looking for the right keywords.

A sales page with informational search intent keywords will flop. The same is true for a blog article with keywords that have a commercial search intent. Match the right keywords with your specific content type. For example, a commercial page should have keywords with commercial search intent and a blog article should have keywords with informational search intent.

After you have narrowed down your list of keywords, created your content and published it, you need to track performance. Analyzing keyword data is essential to inform your SEO strategies. Measure aspects such as search engine rankings, keyword rankings, overall organic traffic and other criteria to ensure you are on the right path.

Page speed and mobile-friendliness

The impact of page speed on SEO is huge. Many, if not most, online users feel a sense of frustration when they click on a page and it takes more than three seconds to open. In our world of instant gratification, people want fast-loading websites that are easy to navigate. Wondering how to optimize your page speed performance? A few tips worth considering include:

Compress your images

Avoid hosting videos locally

Use a quality hosting plan

Avoid too many animations

Minimize external resources

Minimize HTTP requests

Optimize Performance With Redis Caching

Enable GZIP compression

Utilize accelerated mobile pages

Reduce the number of redirects

A related page speed factor is ensuring your website is mobile-friendly . Yes, our world is becoming increasingly mobile-first, and if your website doesn't flow and show smoothly on mobile, you've just frustrated your users and lost them to the competition. So, basically, with these strategies, you can keep your website loading speed low and watch user satisfaction increase as you track this metric.

Related: 8 Ways to Make Your Website Faster (and Why It Is Critical to Your Business)

Content quality and relevance

Measuring content quality and relevance can be a tough nut to crack. Despite this, it's still important to do so. There are several metrics in this space that I always keep an eye out for. These include:

Bounce rate

Social shares

Backlinks (for more on this, see my discussion below)

Conversions

Revenue and/or return on investment

Every one of these metrics, when looked at as a whole, will tell you whether your content resonates with your target audience. Another aspect I consider in this regard is not only publishing fresh content but also updating older articles to ensure that my website or my clients' websites perform optimally at any given period of time. Bear in mind that SEO is a long-term game. It's an ongoing process that requires continued and dedicated efforts for overall success.

Link building and authority

And now I come to the crown jewel of SEO: link building. Getting backlinks to your website from other authoritative websites has been described by many in the industry as securing a vote of confidence in the quality of your online presence. But there are good backlinks and bad backlinks. That's why key metrics to monitor when it comes to your backlink profile should include:

High domain authority

Relevance of the linking websites to your content

Anchor text diversity

A low spam score

When it comes to how to build high-quality backlinks , there are many strategies that you can follow. However, I recommend the skyscraper approach, resource link exchanges and a few subcategories of outreach such as guest posting. Ultimately, you want your online presence to dominate. And to achieve this, you need to show that you are credible, authoritative and importantly — trustworthy.

Conversion rates and user experience

Measuring conversion rates involves a bit of an analytical approach. In a nutshell, it's a numbers game. However, measuring the user experience goes beyond numbers and ventures over into the qualitative sphere. Both of these metrics are important for SEO. And while conversion rates can be easily measured by the number of new customers your business has acquired, your users' experience can be measured by using the following approaches:

Customer Satisfaction Score (CSAT)

Net Promoter Score (NPS)

System Usability Scale (SUS)

Task success rate

Task on time

Measured speed

Ultimately, when it comes to boosting your SEO efforts in order to drive conversions through a great user experience, you will want to implement A/B testing and carry out different experiments to ensure that your approach is as refined as possible.

Related: User Experience Is the Most Important Metric You Aren't Measuring

And that's it, folks: the key SEO metrics that matter in 2024. I can't overemphasize the importance of tracking and optimizing your SEO metrics for your entrepreneurial success. After all, if you have an online presence, you need to build, nurture and grow it to reap the benefits.

With this in mind, if you haven't yet started tracking some of these metrics, it's important to do so while coupling the metrics measurements with a combination of the right SEO strategies and approaches. The rewards for your business and online presence will be phenomenal.

Entrepreneur Leadership Network® Contributor

CEO at InBound Blogging

Want to be an Entrepreneur Leadership Network contributor? Apply now to join.

Editor's Pick Red Arrow

• Lock The Average American Can't Afford a House in 99% of the U.S. — Here's a State-By-State Breakdown of the Mortgage Rates That Tip the Scale
• Richard Branson Shares His Extremely Active Morning Routine : 'I've Got to Look After Myself'
• Lock This Flexible, AI-Powered Side Hustle Lets a Dad of Four Make $32 an Hour , Plus Tips: 'You Can Make a Substantial Amount of Money'
• Tennis Champion Coco Gauff Reveals the Daily Habits That Help Her Win On and Off the Court — Plus a 'No Brainer' Business Move
• Lock 3 Essential Skills I Learned By Growing My Business From the Ground Up
• 50 Cent Once Sued Taco Bell for $4 Million. Here's How the Fast-Food Giant Got on the Rapper's Bad Side .

Most Popular Red Arrow

A buddy's franchise is built for success in a recession resistant industry.

Over the past six decades Buddy's has refined a proven operating model and established relational equity with its customers. With best-in-class training, support and service, franchisees have peace of mind knowing that they are being set up for success.

Mark Zuckerberg Sounds Off on Developing AI: 'I Don't Think AI Technology Is a Thing That Should Be Hoarded'

Meta's CEO spoke about AI's future during an interview with YouTuber, Kalloway.

The Key to Preparing Your Business for an Eventual Investment or Sale

How an investment teaser can help diagnose (and fix) flaws in your business.

I Started My Business with $1,000 — It's Now Worth Billions and Serves Over 163 Million People. These 7 Principles Were My Secrets to Success.

How seven simple principles can you help you build the business of your dreams.

Here Are the SEO Metrics That Matter in 2024

Explore the crucial SEO metrics for 2024 that every digital marketer needs to focus on for enhanced search engine visibility and performance.

Walmart Is Making a Major Change to Its Price Displays in Thousands of Stores

The retailer is implementing digital labels in roughly 2,300 locations.

Successfully copied link

Measuring Subjective Well-being Capability: A Multi-Country Empirical Analysis in Europe

• Open access
• Published: 27 June 2024

Cite this article

You have full access to this open access article

• Tomasz Kwarciński   ORCID: orcid.org/0000-0002-9474-4216 1 ,
• Paweł Ulman   ORCID: orcid.org/0000-0002-1911-8821 2 &
• Julia Wdowin 3  

107 Accesses

5 Altmetric

Explore all metrics

The main aim of this paper is to conceptualise and empirically estimate subjective well-being capability. The empirical demonstration of the conceptual framework is applied in a selection of European countries: Poland and its leading emigration destinations the United Kingdom, Germany, the Netherlands, Ireland, France and Italy. The paper advances the measure of subjective well-being capability (SWC) as the integration of the subjective well-being measure with the capability approach in a unified measurement framework. Following the development of a conceptual model, the theoretical framework is operationalized empirically to quantify measures of SWC across the selected countries using a Multiple Indicators and Multiple Causes Model. Data from the European Quality of Life Survey is employed. A comparative analysis compares the SWC measures across countries as well as comparing SWC with conventional well-being measures such as overall happiness and GDP per capita . The results of the study reveal a significant correlation between the SWC based on a general model for all countries, overall happiness, and GDP per capita . However, it also suggests that country-specific SWCs, calculated from tailored models, could substantially deviate from traditional well-being measurements. This attribute suggests that SWC could be a practical tool for assessing individual contexts, as reflected in the tailored models, even though it might not serve as the optimal instrument for country ranking (via the general MIMIC model).

Similar content being viewed by others

Well-being is more than happiness and life satisfaction: a multidimensional analysis of 21 countries

On Well-Being and Public Policy: Are We Capable of Questioning the Hegemony of Happiness?

International determinants of subjective well-being: living in a subjectively material world.

Avoid common mistakes on your manuscript.

Introduction

The 'happiness approach' and the ‘capability approach' are two well-established methodologies for measuring well-being that have been portrayed as alternatives to the standard welfare economic approach. While the study of subjective well-being (SWB), focused on studying quality of life as an end outcome , and expressed in ‘happiness economics’ studies, such as Easterlin’s ( 1974 ) pioneering study exploring psychological happiness and income developments, emphasises an analysis of subjects' feelings of happiness, sadness or life satisfaction, Amartya Sen's ( 1979 , 2005 ) capability approach (CA) draws attention to the objective basis for the realisation of well-being, such as the possibility of living a normal length of life, being able to move around freely or having shelter. The main aim of this paper is to integrate the SWB notion with the CA into a unified measuring framework. This paper contributes to the conceptual and empirical effort towards such integration in particular by (a) exploring the notion of subjective well-being capability (SWC) coined by Martin Binder ( 2014 ) and most recently applied by Yei-Whei Lin et al ( 2023 ) and (b) comparing SWC with conventional well-being measures such as overall happiness and GDP per capita in selected European countries.

Following the SWC conceptualisation, we conduct an empirical analysis that involves a comparative examination of SWC in Poland and its main emigration destinations, such as the United Kingdom, Germany, the Netherlands, Ireland, France, and Italy. Footnote 1 While the criterion of Polish leading emigration destinations is used to identify a group of countries for comparison, and we do not intend to investigate the particular context of migration, the present research can nevertheless offer a valuable starting point to further research on migrants’ choice and migrational motivations. There is a growing body of research that focuses on the happiness and life satisfaction of immigrants, individuals in their country of origin, and host citizens (Bartram, 2011 ; Hendriks & Bartram, 2019 ; Kogan et al., 2018 ; Voicu & Vasile, 2014 ). Developing the concept and measure of SWC, which emphasises the agency aspects of well-being, can hold great value in evaluating people's decisions to emigrate and their relative position in relation to their compatriots and the citizens of the host country. Although for the present article the empirical study is for descriptive purposes, and such an application goes beyond the scope of this analysis, the aim would be to analyse whether differences in SWC can serve as motivation for migration movements.

We propose measuring individual SWC as a latent variable. To calculate SWC, a Multiple Indicators and Multiple Causes Model (MIMIC) is employed. MIMIC is a special case of the general structural equation model (SEM) which consists of a structural model and a measurement model (Jöreskog & Goldberger, 1975 ). The former describes the causal link among the latent variable and its observed (exogenous) determinants while the latter shows how the latent variable is defined by the observed indicators (we understand these observed indicators to be functionings , in the language of the CA). In modelling SWC empirically, we draw on the methodology adopted by Zwierzchowski and Panek ( 2020 ), who utilise the MIMIC model to evaluate the SWB of Polish citizens. Building on this work we, firstly, develop a clear conceptualization that distinguishes SWC from SWB. Secondly, we use a new dataset, namely the European Quality of Life Survey (EQLS), which includes relevant variables regarding overall happiness. Thirdly, our analysis involves a cross-country comparative approach instead of focusing on a single-country analysis. We also highlight the specific aims of the present paper. In our analysis, we aim to explicitly integrate SWB with CA and assess the applicability of a new index of SWC. This issue is somewhat timely as other researchers are also engaging in developing SWC as an approach for evaluating living conditions among different groups of people (Lin et al., 2023 ).

The remainder of this article is structured as follows. Section " Bridging Subjective Well-being and the Capability Approach: A Foundation for Subjective Well-being Capability " describes SWB and the CA as the theoretical foundations for SWC. Section " Conceptual Model of Subjective Well-Being Capability Measurement " examines how individual well-being capability (SWC) may be empirically measured within one framework. The empirical analysis, which comprises data descriptions, the MIMIC model specification and the SWC estimates, is presented in Section " Empirical Analysis ". The results are discussed in Section " Discussion ". Final conclusive remarks close the paper.

Bridging Subjective Well-being and the Capability Approach: A Foundation for Subjective Well-being Capability

A burgeoning set of well-being measures and indices reflect a growing trend and appetite for the broadening of the informational space to assess individual ‘progress’ and well-being (Hoekstra, 2019 ). The two most significant responses to date to the critiques of traditional income-based well-being measures have been the subjective approach to well-being (SWB) and the CA. The SWB literature largely refers to psychological states and how a person experiences particular moments or how she evaluates her life as a whole or over a period of time as measured expressions of well-being. The most common categorical distinctions in SWB are evaluative, eudaimonic and experienced well-being (Dolan et al., 2011 ; Layard & De Neve, 2023 ). The CA, on the other hand, provides a normative framework for assessing individual opportunity to achieve certain objective goods for measuring well-being (Sen, 1979 , 1992 ). Both approaches shift the focus of well-being measurement from means-based (resourcist approach) to ends-based analysis (well-being), however, taken separately are both subject to critiques of offering limited information on individual well-being without due regard for the informational set of the other (Comim, 2005 ). SWB is prone to problems of hedonic adaptation, whereby it is observed that individuals may understate their level of deprivation in a number of areas because of their personal disposition or because of adaptation and acceptance of life’s long-term circumstances. The measures do not account for an individual’s objective circumstances, such as a being in a position of long-term poverty, even if they report being happy or satisfied. Yet, notions of happiness captured by SWB are considered important to an overall assessment of well-being which an objective account, as the CA expresses, can be prone to omitting.

These two approaches are most frequently treated in well-being measurement literature as exclusive to one another. In this regard, scholars commonly take either a CA to well-being measurement or a SWB approach, with limited work on integration of the two. This, however, is changing (Binder, 2014 ; Lin et al., 2023 ). The difficulty comes about from an implicit emphasis on irreconcilable differences in the philosophical premises, such as attention to individual agency and autonomy, intrinsic value of choice, or on the methodological objective or subjective assessment of welfare, on which each approach is grounded. While the SWB approach draws on psychological insights to define the nature of human well-being to be measured and is often grounded in a utilitarian normative framework, the CA can be traced more closely to an Aristotelian concept of eudaimonia (human flourishing and objective notions of the standard of living) to assess individual welfare objectively and in part externally to the person’s own assessment.

A growing number of recent studies have been taken up to draw links between the two approaches and the areas of compatibility between them (Binder, 2014 ; Comim, 2005 ; van Hoorn et al., 2010 ). These few studies have been motivated by the observation that the two approaches are fundamentally linked by several meaningful synergies in the context of individual welfare evaluation (Binder, 2014 ; Comim, 2005 ).

Following these links, there are three ways in the which the literature has bridged these two approaches so far:

The integration of CA into SWB framework scenario: analysis is undertaken of the extent to which capabilities influence or determine SWB (Graham & Nikolova, 2015 ; Muffels & Headey, 2013 ; Veenhoven, 2010 ).

The integration of SWB into the CA framework scenario: SWB or “being happy” is analysed as a relevant dimension or capability for individual welfare among other relevant capabilities (Robeyns, 2017 ; Zwierzchowski & Panek, 2020 ).

The bridging scenario: the question of how either approach can be enriched by the other (Lin et al., 2023 ; Kwarciński and Ulman, 2018 ; Binder, 2014 ; Kotan, 2010 ; Comim, 2005 ).

This paper aims to situate itself in the third strand of literature, and explore and empirically operationalise precisely the concept of SWC. Of particular significance to our present study is the finding that substantial shortcomings of the SWB approach “can be overcome by focusing welfare assessments on ‘subjective well-being capabilities’, that is, focusing on the substantive opportunities of individuals to pursue and achieve happiness” (Binder, 2014 ). The concept of SWC to be developed and operationalised in this paper moves away from depending on psychological profile alone of SWB measures. At the same time, the domains of happiness that these measures focus on (evaluative, eudaimonic and experienced) are analysed from the perspective of the opportunity that an individual has to achieve them. That is, SWC seeks to map the potential individual opportunities to achieve happiness. Conversion factors, a notion adopted from the CA, further allow for an analysis of the factors that determine the person’s individual opportunities to achieve happiness in the SWB domain. In this way, the SWC concept provides a broader informational space for individual well-being analysis.

There are several reasons to be interested in the theoretical notion of SWCs and their measurement. Firstly, in line with Sen’s theoretical stance, the concept of SWC recognises the importance of ‘happiness’ to overall human welfare according to the CA, but acknowledges its partiality and limitation as a sole indicator of individual welfare, (mainly due to the issue of hedonic adaptation). What we would expect to evaluate is a situation where a person may have high SWB (reports a high state of happiness) but low SWC (is objectively materially or otherwise deprived).

Secondly, it brings to light the distinction between Sen’s ( 1993 ) well-being freedom and agency freedom. As autonomous agents, individuals use their agency to determine their overall achieved well-being from the genuine alternative freedoms (capabilities) available to them (their capability set), receiving intrinsic value from the choice itself. Agency plays out in individual welfare measurement in two ways, and is related to the intrinsic value he attaches to choice . It implies that value may be found in the utility or disutility regarding the range of choices available to the individual (i.e. the range in choice of functionings in their capability set).

Additionally, though, there is intrinsic value in the individual simply being able to choose. SWB functionings could exist separately from the concept of SWC, but the concept of freedom or gauge of individual opportunity would be missing from the analysis (Binder, 2014 ). From a Senian perspective, individuals as autonomous agents have their own rankings according to which trade-offs and selections between capabilities that form part of an individual’s capability set are made. SWB functionings may be selected or traded-off according to an individual’s priorities and ranking. With the SWC notion, we suggest that SWB functionings be considered functionings that form part of an individual’s capability set, among other functionings.

Thirdly, the SWC approach allows for the empirical identification of relevant functionings underlying achieved happiness levels, so as not to have to rely on the top-down expert-given opinion of the factors that represent happiness (Lin et al., 2023 ).

Conceptual Model of Subjective Well-Being Capability Measurement

In formulating SWC, we seek to model the opportunities that a person has for happiness, the latter of which (happiness) is directly observable from SWB indicators of happiness. What is not directly observable is the individual opportunity to achieve given happiness levels, which we denote as SWC. From a theoretical perspective, we specify the analytical components of the CA informational space, which includes: resources, conversion factors, functionings, and capabilities. The unobservable element here is SWC. Functionings refer to the various ways individuals experience happiness in their lives. Resources encompass personal assets such as health conditions or educational attainment, while conversion factors include personal, social, and environmental circumstances that influence the way a person can make use of their resources to achieve a desired level of happiness.

The purpose of building a SWC model is to be able to measure SWC – that is, opportunities for happiness in the SWB domain, and study whether there are certain sub-groups of individuals who are more or less disadvantaged in the potential happiness they can achieve, that is, have higher or lower SWC. This would imply that there are certain sub-groups of individuals who have potentially less access to SWB functionings. In the case of our application, individuals’ opportunities are explored at the country-level for a selection of European countries.

For the empirical operationalization of SWC in this study, we adopt a research approach that infers capabilities using latent variable models (Anand et al., 2011 ; Krishnakumar & Ballon, 2008 ; Sarr & Ba, 2017 ; Yeung & Breheny, 2016 ). The methodology has also recently been employed by Zwierzchowski and Panek ( 2020 ), who utilize multiple indicators and multiple causes models (MIMIC) in their study of SWB in Poland. However, in order to address the ambiguity present in their approach – where they interpret the estimated latent variable as both potential SWB (Zwierzchowski & Panek, 2020 , 164) and as a composite indicator of SWB (Zwierzchowski & Panek, 2020 , 165) – we propose the following operationalization of SWC (Fig.  1 ).

Conceptual model of subjective well-being capability measurement. Source: own diagram

In the outlined operationalization, we introduce a clear distinction between SWB as a composite index and potential SWB, which we term SWC. The composite index of SWB would be estimated solely from the (SEM) measurement model (confirmatory factor analysis). This index is also distinct from SWB based on a single ‘overall happiness’ variable. The SWC index, on the other hand, is derived from a structural model. By choosing linear predictors from the structural model instead of factor scores in MIMIC, we follow Krishnakumar and Chávez-Juárez ( 2016 ), who argue that linear predictors perform better when the indicator variables are highly correlated with each other, which is usually the case in empirical analysis, including our study.

In the measurement model we follow the conventional taxonomy on SWB, dividing the concept into three parts: (1) evaluative, (2) eudaimonic and (3) experienced. Evaluative well-being refers to overall life satisfaction or fulfilment; eudaimonic well-being involves seeing one's life as meaningful and taking purposeful actions; while experienced well-being, also known as hedonic well-being, consists of positive and negative effects experienced by individuals (Layard & De Neve, 2023 ; National Research Council, 2013 ). The three-part concept of well-being overlaps with the CA for at least two reasons. First, because the self-reported evaluation of life, the positive or negative feelings experienced, are ways of ‘being’ (and things that can also affect individual ‘doings’)—‘beings’ and ‘doings’ are important ontological categories in Sen’s capability theory, through which welfare opportunity is interpersonally assessed. Second, Sen emphasises the hedonic aspects of well-being in his works, despite rejecting SWB as a primary system for well-being evaluation (Burns, 2022 ; Sen, 1987 , 2008 ).

While capabilities, and opportunities more broadly, are found to be positively correlated with life satisfaction (Ravazzini & Chávez-Juárez, 2018 ; Steckermeier, 2021 ), a possible implication is that some form of individual valuation of opportunity or notions of agency may be implicit in SWB assessments, particularly within evaluative or eudaimonic SWB. However, the SWC measure does not examine the relationship between possessing a specific range of opportunities or having agency, and achieving a given self-assessed level of SWB. Instead, it focuses on the individual’s opportunity to achieve certain SWB functionings. The difference, while conceptually subtle, is central to the present study.

In the structural model we include sets of resources and conversion factors. By resources, we understand any tangible or intangible resources that are at an individual’s disposal and that serve as “inputs” to happiness, that is are related to generating the person’s happiness. These can include such things as education, income or health. The adoption of conversion factors that the CA offers allows for an investigation into whether certain personal, social or environmental characteristics impact upon how and to what extent individuals can differently utilise resources for happiness.

It is worth noting that the distinction between resources and conversion factors in the context of empirical analysis commonly depends on the given functionings or capabilities being analysed, and can be to some degree arbitrary. Conceptually, there is a clear distinction between these two categories. However, in practice, the categories become highly challenging to delineate due to the interplay between resources and conversion factors. In different circumstances, resources can function as conversion factors or influence other resources and conversion factors. For instance, health can serve as a personal asset or as a conversion factor which impacts the extent to which someone can utilise financial resources. This provides a theoretical justification for the high degree of correlation observed between variables describing resources and conversion factors.

To our knowledge there is only one similar theoretical attempt to integrate SWB with a capability-based approach by focusing on SWC (Binder, 2014 ), and one recent empirical application (Lin et al., 2023 ). According to Binder ( 2014 ), policymakers should take into account not only SWB but also happiness-relevant functionings, which are the ways of being and doing which make people happy. He cites employment, having an income, having good health, having friends, etc. as examples of such functionings (Binder, 2014 ). Following Binder’s ( 2014 ) approach, we would first have to identify happiness-relevant functionings by running linear regression of the various functionings on the happiness index. We could then construct a MIMIC model in which we would place within the SWC indicators the most statistically significant happiness-relevant functionings. This approach would also require rethinking which elements make up the set of resources and conversion factors. Selecting relevant resources, and happiness-relevant functionings would be vulnerable to criticism for a paternalistic approach. We adopt a more direct approach, similar to Lin et al. ( 2023 ), where instead of happiness-relevant functionings, we include SWB functionings, such as feeling happy, not feeling of sadness, enjoyment, being full of life, doing worthwhile things, and so on, into the MIMIC model. What we share with Binder is a general view of SWC, as “the substantive opportunities an individual enjoys to pursue and achieve happiness.”

The SWC derived from the MIMIC structural model is an assessment of an individual's capability to achieve SWB, taking into account not only SWB functionings but also available resources and conversion factors in its calculation. This approach aligns with the theoretical foundations that define capabilities as the real freedoms to perform particular functionings, which depend on resources and specific conversion factors. It also adheres to empirical conventions that apply statistical techniques, such as the MIMIC model, to calculate, based on observed functionings, a latent variable interpreted as an unobserved capability.

Empirical Analysis

The study utilises the European Quality of Life Survey (European Foundation for the Improvement of Living and Working Conditions, 2018 ), which allows us to present distinct findings compared to those of Zwierzchowski and Panek ( 2020 ). This dataset offers distinct advantages as it includes the ‘overall happiness’ variable, which is not present in EU-SILC. Incorporating this variable into our analysis serves as a valuable reference point for measuring subjective well-being capability (SWC) outcomes. This article uses data from the last wave of EQLS 2016 for selected EU countries: Poland, United Kingdom, Germany, France, Ireland, Italy and the Netherlands. The assumption of the survey is that the sample size for a given country cannot be less than 1000 units. Thus, the smallest sample size was taken for Poland (1009) and the largest for Italy (2007).

As with any empirical study that involves surveying individuals, missing responses occur, resulting in missing data in the database. Deleting all records in which at least one of the examined variables was missing would lead to a significant reduction in the number of observations in the sample for a given country. It was therefore decided to fill in the mentioned gaps for selected potential variables under analysis. In the first phase, we chose the variables for the study from a large list of variables accessible in the EQLS 2016 dataset. Then, the missing data were filled in using a predictive mean matching algorithm (R package MICE).

The set of variables consists of three subsets: SWB functionings, conversion factors and resources. As mentioned, SWB is composed of three elements: evaluative well-being, eudaimonic well-being and experienced well-being. To measure evaluative well-being we use a life satisfaction ten-point ordinal scale. Eudaimonic well-being is captured by the statement “I generally feel that what I do in life is worthwhile” and measured on a six-point ordinal scale. In the case of experienced well-being, we use three statements to reflect negative affections: “I have felt particularly tense”, “I have felt lonely”, “I have felt downhearted and depressed” and three statements to reflect positive ones: “I have felt cheerful and in good spirits'', “I have felt calm and relaxed”, and “I have felt active and vigorous”. Responses are measured on a six-point ordinal scale.

It is important to note that resources and conversion factors are depicted through variables that serve as stimulants to well-being. This is applicable to variables where the order of states they represent can be distinctly aligned within the context of well-being. For example, the binary variable ‘absence of unemployment’ is coded so that the value zero represents a person who is unemployed, and one indicates a person who is not experiencing unemployment. Detailed information on all variables can be found in the appendix, specifically in Tables 5 and 6 .

Furthermore, we employ the variables ‘overall happiness’ and ‘GDP per capita ’ for comparison with the SWC, which is calculated from the estimated model. The ‘overall happiness’ variable is distinct from the other related ‘happiness’ variables in the model and is determined based on responses to the question: “Taking all things together on a scale of 1 to 10, how happy would you say you are?” Additionally, the GDP per capita in 2016 is measured using purchasing power parity (current international $) and comes from the World Bank dataset.

Estimation of the MIMIC Models and SWC

Using the prepared data for seven countries, we estimate the Multiple Indicator and Multiple Causes Model (MIMIC). MIMIC is a special case of Structural Equation Models (SEM) Footnote 2 . A feature of this type of model is that it contains two parts: measurement and structural. The measurement part defines the latent (unobservable) dependent variable, while the structural part contains exogenous observable variables, i.e. factors that potentially vary the level of the above-mentioned unobservable dependent variable measured by the measurement part of the model.

Formally, MIMIC can be written as:

\(y\) – vector of observable endogenous variables,

\(\Lambda\) – vector of factor loadings of endogenous variables,

\(f\) – a latent endogenous variable (composite indicator of SWC),

\(B\) – vector of coefficients of latent variable to observable exogenous variables x ,

\(x\) – vector of observable exogenous structural variables,

\(\varepsilon,\psi\)    – vectors of error terms.

The estimation of model parameters can be performed by different methods. This paper uses maximum likelihood estimation (MLE) implemented in the IBM AMOS package. The MIMIC models are estimated using the weighting system contained in the EQLS dataset.

An important step in the construction of a MIMIC model is to define its measurement part. This part of the model describes the relationship between the unobservable (latent) variable and the observable variables used to measure it. Task-wise, the measurement model should be defined before the structural part of the MIMIC model is introduced. The SWB functionings depicted in Fig.  1 and Table 5 inform the configuration of the measurement model. This model's framework is shaped by theoretical underpinnings that posit SWB as comprising evaluative well-being, eudaimonic well-being, and experienced well-being. The latter encompasses both positive and negative life experiences. Figure  2 presents an example of a MIMIC model in the context of Poland. The results of the estimation of the parameters of the measurement model for all countries studied are presented in Tables 2 and 4 . They confirm the significant importance of the individual variables (functionings) in determining the latent variable: SWC.

Once the form of the measurement model is established, the structural part of the MIMIC model is constructed by selecting the exogenous variables, which are the resources and conversion factors (see Table 6 ). In the course of estimating the full MIMIC model, some of the exogenous variables prove to be statistically insignificant, with the set of variables varying from country to country. The outcomes of the structural and measurement model estimations are also detailed in terms of standardised assessments in Tables 3 and 4 .

The model quality metrics employed (R-square and RMSEA) suggest an adequate fit of the model to the empirical data when using the individual-level dataset. In the analysis of metric invariance for measurement model, comparing the comprehensive model (MIMIC GEN) to individual country models reveals that for Germany, Ireland, Italy, the Netherlands, and the United Kingdom, the disparity in chi-square values (between the country-specific model and the general model) suggests a rejection of the metric invariance hypothesis. Nevertheless, the incremental fit indices (Normed fit index—NFI, Incremental fit index—IFI, Relative fit index—RFI, Tucker-Lewis index—TLI) reveal only a nominal variation between these models. Moreover, the fit metrics for models without constraints and those with subsequent restrictions differ marginally, which could imply that metric invariance is maintained despite the hypothesis being rejected. The rejection of the metric invariance hypothesis may be attributable to the relatively large sample sizes, which could lead to the nullification of the hypothesis even when the discrepancies between the models under comparison are minor.

To assess the stability of outcomes, we estimated the results under the different assumption of no correlations among variables in the measurement model. The deviations from values obtained with the initial methodology were minimal. However, it is important to note that the model quality indicators showed a slight decrease in performance compared to the original models.

The MIMIC model enables the estimation of individual SWC values. These values are derived from the structural model estimated for each country individually as well as for all countries collectively (referred to as MIMIC GEN). They are then normalised using the formula:

\({f}_{ij}-\) estimated individual SWC value for i household and j country (or generally for all countries).

Two approaches are used: within-state normalisation and across-states normalisation (the results of the SWC estimation are combined for all states and normalised based on these). The first approach follows the relative way of identifying poverty, where the national median income is determined and the equivalent household income is related to the corresponding share of this median within a country. In contrast, the second approach – it seems – more broadly allows for comparisons of welfare levels between countries.

Comparative analysis of the SWC and overall happiness over age in different countries is made by locally estimated scatterplot smoothing (LOESS) in R-package ggplot2.

Example Diagram of the MIMIC Model: A Case Study for Poland. Source: own calculations

Descriptive Statistics

Using data from the EQLS, our analysis covers SWC results for seven European countries. When examining SWB indicators, the descriptive statistics reveal that SWB remains relatively consistent among the seven countries, with minor variations. For instance, the range for life satisfaction sits between 6.56 (Italy) and 7.74 (Netherlands), suggesting that there is not a significant variation in life satisfaction among the countries. In general, Ireland appears to have higher levels of positive attributes such as life satisfaction and happiness, whilst maintaining lower levels of negative indicators such as nervousness and sadness. Conversely, Italy exhibits the opposite pattern with increased negative attributes and lower positive ones. Variability in responses differs considerably across countries, indicating differing levels of consensus among the populace concerning SWB. For example, Italy often shows high variability, indicating that responses in this country may be quite diverse. On the other hand, the Netherlands often shows lower variability, implying more consistency in responses.

When considering the determinants of SWC, such as resources and conversion factors, a noticeable disparity between countries becomes apparent. The descriptive statistics indicate distinct patterns based on average income. There are two main groups of countries: (1) the affluent group consisting of the Netherlands, Germany, France, and the United Kingdom, with an average equalised monthly income in purchasing power parity exceeding 1500 Euro, and (2) the less affluent group comprising Italy and Poland, with an average income in purchasing power parity per person less than 1300 Euro. Ireland falls in between the two groups, with an income approaching 1400 Euro. We find a partial confirmation of this division of countries by analysing the percentage of the poor, which is the highest in Poland and Italy. Ireland, in this case, is included in the group of wealthy countries with one of the lowest shares of the poor. This situation can be explained by a relatively young population with a large number of people in the household, which reduces the equivalised income and shapes the distribution of this income in such a way that 60% of its median (as the poverty line) generates the lowest poverty rate. Poland belongs to the group of countries with a low level of health quality, along with Italy and Germany, but only Italians have the lowest level of medical needs satisfied, with approximately 60%, while in other countries the percentage is at least around 80.

Regarding gender, samples in all countries are generally balanced, with slightly fewer men than women, typically around 48% men. Ireland also has the youngest average age of respondents compared to other countries (46.2 years), where the average age is around 50 years, excluding Poland with an average age of 47 years. This relatively young Irish population is potentially associated with the level of other characteristics, e.g. household size (average highest), percentage retired (by far the lowest), percentage of students (the highest together with Poland) or perceived health status (most often rated as very good). Among the variables that make up the conversion factors, one that stands out is the risk of crime experience. The highest risk of crime is experienced in the Netherlands, and France, the lowest in Italy. Table 7 presents descriptive statistics for SWB indicators and SWC determinants, along with overall happiness and GDP per capita for each country.

Comparative Analysis of SWC

Comparing the countries studied on the basis of separately estimated MIMIC models, we observe correlations between three factors and SWC across all countries. In each case, the lower the level of material deprivation, the lower the perception of crime in the neighbourhood, and the better the self-perceived health the higher the estimated SWC (Table 1 ). Self-perceived health consistently exhibits the largest effect size relative to other factors across all countries (Table 3 ). In the case of Poland, our analysis partially confirms the findings of Zwierzchowski and Panek ( 2020 , 167). Similar to their research, the most significant factors influencing SWC in Poland are self-perceived health and absence of material deprivation (Table 3 ). Furthermore, there are statistically significant factors that are unique to each of the countries analysed. Household equivalent income emerges as a significant factor in Ireland, where higher income is linked to higher SWC. Additionally, not being a student is associated with higher SWC in Poland (Table 1 ).

The affluent group of countries, comprising the Netherlands, Germany, France, and the United Kingdom, share three primary factors that significantly influence SWC: larger household size, unemployment status, and the absence of unmet medical needs. Interestingly, since the average household size in these countries is relatively low, increasing family structures or cohabitation affects SWC. The United Kingdom also indicates an association between lower levels of education and SWC. Retirement status significantly and positively influences SWC in France and the United Kingdom, which might imply a certain level of satisfaction or quality of life linked with retirement in these countries.

Italy and Poland, despite being grouped as less affluent countries, exhibit dissimilar influencing factors. In Italy, living above the monetary poverty line, not perceiving pollution, larger household size, unemployment status, and absence of unmet medical needs are pivotal. It is worth noting that the last three factors mentioned make Italy similar to its more affluent counterparts. Poland, in contrast, experiences the influence of higher education, female gender, unemployment status, low economic activity, and non-student status on SWC. This juxtaposition between Italy and Poland highlights diverse socio-economic conditions and their distinct impacts on SWC.

In Ireland, which falls between more and less affluent countries, retirement status is positively associated with SWC, similarly to France and the United Kingdom. Additionally, the perception of pollution influences SWC in Ireland, similarly to Italy but in the opposite direction. In Ireland, a lack of perception of pollution is linked to an increase in SWC, whereas in Italy, people have a perception of pollution, which negatively affects SWC. Moreover, the results for Ireland indicates that older age is negatively linked to SWC, contrasting with findings from Germany, where older individuals experience higher SWC. This stands in opposition to the typical association of higher SWC in older, retired populations.

In the general model (MIMIC GEN) estimated for all countries combined, almost all variables are statistically significant, with the exception of gender and being a student (Table 1 ). The factors that exhibit the largest effect sizes, compared to others, include self-perceived health, absence of material deprivation, and perception of crime in the neighbourhood (Table 3 ). SWC cross-sectional analyses are conducted using the MIMIC GEN model, as it allows us to assess the variability between countries consistently.

In most cross-sections, the same general pattern is found in all countries. Analysing the SWC from the perspective of respondents' education level, health status, poverty status, and gender we notice similar dependencies in all the countries. In general, the higher the education level of the respondent, the higher the SWC (Fig.  3 ), the worse the self-perceived health status, the lower the SWC (Fig.  4 ), and belonging to the highest level of poverty group is associated with lower SWC (Fig.  5 ). What is more, in all countries, men's SWC appears to be higher than women's, although in the case of Poland and Italy both genders have significantly lower SWC compared to the other countries surveyed (Fig.  6 ).

Only when respondents are stratified by being retired are there significant differences between countries with respect to SWC. There is also an apparent division into two groups of countries, those where retirees have higher or equal SWC relative to non-retirees (Netherlands, United Kingdom, Ireland), and those where SWC is lower among retirees (Germany, France, and notably in Poland, Italy) (Fig.  7 ).

The analysis of SWC variability, as discussed with reference to Figs.  3 – 7 , is further supported by the Welch test. The Welch test, a corrected version of the One-Way ANOVA, identified significant differences (at a significance level of 0.05) across all countries based on respondents' education level, self-perceived health status, poverty, and gender. Regarding gender, a notable exception is Poland, where a difference between genders is not statistically significant. Moreover, significant differences due to retirement status are observed exclusively in Germany, France, Italy, and Poland.

SWC and education. Source: own calculations

SWC and health. Source: own calculations

SWC and poverty. Source: own calculations

SWC and gender. Source: own calculations

SWC and retirement. Source: own calculations

SWC and conventional well-being measures

The relationship between SWC and the conventional well-being measures (overall happiness and GDP per capita ) can provide valuable insights into the reliability and applicability of SWC. When examining the mean SWC based on MIMIC GEN, it becomes apparent that there is a strong and significant correlation with both the mean overall happiness and GDP per capita across countries (r = 0.80 and r = 0.81, respectively). However, when considering the mean SWC estimated separately for each country, the correlation with mean overall happiness is notably weaker and insignificant (r = 0.39). Similarly, the correlation between the mean SWC estimated separately for each country and GDP per capita is low and insignificant (r = -0.05).

Considering SWC based on MIMIC GEN and across-states normalised (green line in Fig.  8 ), only Italy has a similarly low average SWC compared to Poland. Moreover, although the average overall happiness index for Poland is higher than in Italy, the average SWC in both countries is at similar levels. It is also worth noting that the average overall happiness and average SWC in most countries, with the exception of Poland and Italy, are very close (Table 8 ). Considering SWC based on MIMIC models calculated separately with within-state normalization for the countries (illustrated by the blue line in Fig.  8 ), it can be observed that Ireland has the lowest value of SWC and exhibits the greatest disparity between the calculated SWC, SWC based on MIMIC GEN, and overall happiness.

Tukey's Range Test (Tukey's HSD) confirmed that for SWC derived from separately calculated MIMIC models, there are no significant differences between Germany and France, Ireland and Italy, Ireland and Poland, as well as Italy and Poland. Regarding Happiness, no significant differences were found between Germany and France, Germany and Poland, France and Poland, Ireland and the Netherlands, Ireland and the United Kingdom, as well as between the United Kingdom and the Netherlands. For SWC based on the MIMIC GEN model estimated for all countries combined, significant differences are absent between Germany and France, France and the Netherlands, France and the United Kingdom, Ireland and the Netherlands, Italy and Poland, as well as between the Netherlands and the United Kingdom.

Comparing SWC with overall happiness over the lifetimes of people in each country yields some interesting results (Fig.  9 ). For most countries, both SWC and overall happiness decrease as respondents’ age increases (Germany, France, Italy, Poland). More importantly, however, is the fact that only in selected countries and only in certain periods of the respondent's life is the SWC index statistically different from the overall happiness index. This is particularly evident in Italy until the age of 60. This suggests that, when taking into account the resources at their disposal and the influence of conversion factors, individuals below 60 years of age have more opportunities to experience happiness than they tend to report in terms of the overall happiness level.

SWC and overall happiness. Note: Differences between overall happiness within-state normalised (dotted line) and SWC estimated separately for each country, within-state normalised (solid line) and SWC estimated for all countries combined (MIMIC GEN), across-states normalised (dashed line). Source: own calculation 

SWC and overall happiness across life span. Source: own calculations

Integrating the SWB and the CA into a unified framework allows us to conceptualise and measure the real opportunities for achieving SWC among citizens of selected European countries. While these countries' populations exhibit numerous similarities in terms of SWC, such as important factors influencing SWC and similar patterns regarding the relationships between education levels, health status, poverty, gender, place of residence and SWC, which can be attributed to shared socio-economic conditions, there are specific aspects that warrant further exploration.

Firstly, it is noteworthy that looking at the MIMIC GEN model Poland and Italy exhibit the lowest levels of SWC among the investigated countries. Both are found in the less affluent country group. Furthermore, there are distinctive patterns in the relationship between retirement status and SWC in these two countries compared to the others. Italy also experiences a notable discrepancy between overall happiness and SWC (MIMIC GEN) throughout individuals' lifespans. Moreover, Ireland demonstrates a significant gap between SWC based on MIMIC GEN and SWC based on the tailored model, highlighting the need for further examination. Additionally, the strong correlation between SWC based on MIMIC GEN and conventional measures of well-being emphasises the issue of the practical application and utility of such sophisticated models. What follows is a discussion regarding the above four findings.

Poland and Italy display overall lower levels of SWC (MIMIC GEN) for both women and men than the remaining countries in this study (Fig.  6 ). Both of these countries are situated in the less affluent country group in terms of income PPP, and additionally, of the countries analysed, have the highest percentage of individuals living below 60% of the country median equivalent income. Both countries also display similarly low levels of tertiary education relative to other countries. The two countries furthermore have relatively low levels of health quality. This finding aligns with our expectation that opportunities for well-being, including the aspect of individual agency not captured in conventional happiness measures like overall happiness, may be reduced in less affluent countries, and can be linked to the proportion of highly educated individuals. The SWC measure, which captures individuals’ opportunities, can serve to reflect the role of individual agency for achieving well-being, and while further research would be needed, it could be conjectured that there is some context or policy-specific political infrastructure in Poland and Italy which may reduce individuals’ agency, and subsequently SWC.

We also observe that the biggest differences between measured SWC (MIMIC GEN) and overall happiness are noted in Italy and Poland (Fig.  8 ). It can be noted that this is again the case for two countries categorised as ‘less affluent’ and with the highest percentage of poor individuals. However, among these countries the relationship between SWC and overall happiness is inverse, that is, in Italy the estimated SWC is higher than the overall happiness measure, while in Poland the opposite relationship can be observed. We have two suggestions regarding this observation, which would warrant further study.

Firstly, as we have noted, SWC is to do with individual opportunity. In this way, it can bypass the problem of hedonic adaptation to deprived or difficult conditions. SWC as a measure can generate a different estimate to that of overall happiness if it controls for the problem of hedonic adaptation, which we suggest could be the case for Italy (France and Ireland), where SWC estimates are higher than overall happiness estimates. Opportunity-based measurement is in theory less vulnerable to hedonic adaptation than a measure of SWB because it leans on objective criteria (resources and conversion factors). In Poland, we observe a similar difference between the SWC and overall happiness estimate, but the inverse relationship, where SWC measures are higher than happiness measures. This suggests to us that there may be country-specific cultural and political contexts, namely specific policies, that can generate a difference in individuals’ happiness and opportunities. A specific policy could generate higher SWC than overall happiness in a given less affluent country, and higher happiness than SWC in a different less affluent country. For example, a particular retirement-related policy, such as the retirement age requirement, retirement replacement rate, quality of public services could generate the sense of more agency and opportunity in one country, but it is plausible that the same retirement policy, or indeed, a different one could generate the sense of less opportunity in another depending on the cultural context and how work is viewed and valued culturally and materially in the given country.

It may also be that certain highly valued functionings may produce a necessary trade-off with SWB functionings, with some functionings moving in the same direction, while others in conflicting directions. For example, the functioning of employment may move in the same positive direction as the functioning of ‘feeling happy’ or ‘feeling calm.’ In other instances, functionings may be rivalrous. If we consider the functioning of employment, for example, depending on the nature of the work, the functioning may be associated with negative SWB functionings, such as ‘feeling nervous’ or ‘feeling sad’. The concept of SWC takes on particular significance for this situation: it is the individual’s hierarchy of preferences and values that will determine the balance of functionings, including negative SWB functionings, that are pursued by the individual in achieving overall welfare. In particular, SWB functionings may not align with eudaimonic (worthwhile) functionings. In the situation of a trade-off between these categories and categories of other valuable functionings, the individual, exercising his agency, chooses the relevant balance in pursuit of overall welfare. The SWB account, on the other, makes no normative reference to agency.

We suggest that a similar logic regarding the differences in the country-specific policy contexts can be applied to understanding differences in SWC in specific domains such as retired/non-retired individuals as in Fig.  7 (MIMIC GEN), as well as the striking result of a notable discrepancy between overall happiness and SWC estimates throughout individuals’ lifespans in Italy (Fig.  9 , Italy, MIMIC GEN). In the case of the latter (Fig.  9 , Italy) what is observed is that opportunities for SWB functionings are higher than the happiness measure until about 60, after which point happiness measures are generally estimated as higher than SWC. Further investigation would be required to uncover what it is within the given policy or political context that might be influencing lower opportunities for well-being achievement for, in the cases of Poland and Italy (and Germany and France to a lesser extent), retired individuals. The case of Italy (Fig.  9 ) gives some plausible indication that policies dividing retired and non-retired individuals in Italy may be influencing the level of personal agency available to individuals at these different life stages.

Ireland presents an intriguing case study that reveals interesting insights. This country exhibits striking differences between SWC based on the tailored model and SWC based on the MIMIC GEN model (Fig.  8 ). From a statistical perspective, this peculiarity can be understood as the specific factors in Ireland lose their significance when included in a broader sample of all seven countries. However, Ireland's SWC, based on the tailored model, is significantly different from its average level of overall happiness, which sets it apart from other countries, too. We cannot rule out the possibility that there is a problem of hedonic adaptation, but we also hypothesise that the unique social conditions in this country, especially its young population and the associated high percentage of students and low percentage of retirees, conceal valuable information about social relationships. These factors, in turn, have an impact on SWC that is not apparent in an overall happiness variable. As shown by Berlingieri et al. ( 2023 ), loneliness is most prevalent in Ireland among European countries, with 20% of respondents reporting feeling lonely. These feelings are negatively correlated with age and positively correlated with experiencing major life events. For instance, individuals who finish their studies often feel more lonely due to their social circle suddenly shrinking. We observe that in our SWC model for Ireland, the age variable has a negative impact on SWC, while retirement status and economic activity have a positive impact, which lends credibility to our hypotheses. In general, Ireland's example convinces us that the set of resource variables should be expanded to include variables related to interpersonal relationships as social capital.

Not only does the lack of correlation between overall happiness and SWC based on the tailored model require attention, furthermore the alignment between SWC based on MIMIC GEN, overall happiness, and GDP per capita indicates a high correlation among these variables. These findings provide valuable guidance for the application of SWC. Instead of comparing SWC between countries using a common model (such as MIMIC GEN), it is preferable to utilise the simpler variable of 'overall happiness' or GDP per capita . However, for a deeper understanding of the specific factors influencing SWC in a given country, it is more suitable to employ SWC specifically designed for that country.

As we have endeavoured to demonstrate, the SWC aims to capture, and draws insight from the CA conceptual analysis. It brings to the fore the central CA concept of individual human agency . The SWC measure for a given country introduces the notion of agency in achieving well-being, which conventional overall happiness measures do not account for. Due to these specific features, SWC could hold great value in analysing migration movements. We can hypothesise, for example, that the opportunity to achieve happiness is one of the significant factors enabling migrants' assimilation in the host country.

Summary and Conclusion

The article introduces, measures, and discusses subjective well-being capability (SWC) as a consolidated measurement framework for subjective well-being (SWB) and the capability approach (CA). As a practical example, we estimated SWC for citizens of Poland and other selected European countries, identified based on their status as primary destinations for Polish emigrants. These estimates were made using the general model (MIMIC GEN) and specific models tailored to each country's unique context. Furthermore, we attempt to assess the usefulness of the SWC measure by comparing it with traditional well-being metrics like overall happiness (a measure of subjective well-being) and GDP per capita (a measure of objective well-being).

Our research yields several key findings. Firstly, there exists a significant correlation between SWC based on the MIMIC GEN model, overall happiness, and GDP per capita . When applying the same model to citizens from different countries, we note that higher average wealth and greater average happiness correspond with similarly elevated SWC values. This may be linked to similar underlying resources and conversion factors that are associated with SWB across all countries.

Secondly, country-specific SWC based on tailored models may significantly diverge from conventional well-being measures. This is particularly noticeable in the case of Ireland, where relatively high average overall happiness coexists with a low SWC. We have identified three potential reasons for this phenomenon. One might be tied to Ireland’s unique socio-economic factors, such as having the youngest population and the fewest retirees. Another possibility could be the unaccounted impact of the quality of social relationships on SWC, which is not explicitly represented in the model. Finally, an adaptation problem may exist, wherein some individuals report feeling happy even if objective conditions might not support this sentiment.

Thirdly, it is not only possible for the SWC, in terms of either the general or tailored MIMIC model, to be significantly lower than overall happiness, but the reverse can also occur. Italy and Poland provide clear examples of such a contrast. On one hand, we might hypothesise that large disparities between these two measures relate to the countries' relatively low wealth status, as both are classified as less affluent. On the other hand, we can speculate that a higher level of overall happiness and a lower level of SWC (as in Poland) may reveal an adaptation problem. Conversely, a higher level of SWC and a lower happiness level (as in Italy) might be associated with a trade-off faced by individuals striving to realise various available functionings, not exclusively related to happiness.

All of these observations lead us to the final conclusion that SWC encompasses additional information, compared to both subjective and objective well-being measures alone. This information pertains to the agency aspects of human welfare. This characteristic makes SWC useful for evaluating specific personal contexts (as demonstrated by the tailored models), though it may not necessarily serve as an ideal tool for country ranking (MIMIC GEN). Consequently, we posit that due to its agency aspect, the SWC could be a beneficial measure for critically evaluating ‘opportunities for happiness’, for example, in the social environment of immigrants, their countries of origin and their host societies.

While our study yields intriguing findings, we must acknowledge some limitations of our approach, related to both data quality and model specification. Concerning the data, our samples were drawn from a survey conducted in 2016, and, as of now, no more recent EQLS database has been made available. The dataset has a high percentage of missing data for some variables, which either precludes the use of these variables or necessitates their imputation. In terms of model specification, it's important to note that the distinction between resources and conversion factors is not clear-cut and context-dependent, making it challenging to create a more precise structural model. Furthermore, our model lacks variables directly associated with social capital. The use of SWC as a single index necessitates the coding of all variables as stimulants, which sometimes requires arbitrary decisions. Despite these drawbacks, we believe that further research on a unifying framework for SWB and CA is worth pursuing.

It is also worth noting that we do not intend to analyse the Polish diasporas in different countries.

MIMIC was formulated by Hauser and Goldberger ( 1971 ) and was later popularised by Jöreskog and Goldberger ( 1975 ). On the grounds of measurement and analysis of the level of well-being (capabilities) this model was used by Krishnakumar and Ballon ( 2008 ), Krishnakumar and Chávez-Juárez ( 2016 ), Zwierzchowski and Panek ( 2020 ), Lin et al. ( 2023 ).

Anand, P., Krishnakumar, J., & Tran, N. B. (2011). Measuring welfare: Latent variable models for happiness and capabilities in the presence of unobservable heterogeneity. Journal of Public Economics, 95 (3–4), 205–215.

Article   Google Scholar  

Bartram, D. (2011). Economic migration and happiness: Comparing immigrants’ and natives’ happiness gains from income. Social Indicators Research, 103 , 57–76.

Berlingieri, F., Colagrossi, M., & Mauri, C. (2023). Loneliness and social connectedness: Insights from a new EU-wide survey (Report No. JRC133351). European Commission.

Binder, M. (2014). Subjective well-being capabilities: Bridging the gap between the capability approach and subjective well-being research. Journal of Happiness Studies, 15 (5), 1197–1217.

Burns, T. (2022). Amartya Sen and the capabilities versus happiness debate: An Aristotelian perspective. In F. Irtelli & F. Gabrielli (Eds.), Happiness and wellness: Biopsychosocial and anthropological perspectives . IntechOpen . https://doi.org/10.5772/intechopen.108512

Comim, F. (2005). Capabilities and happiness: Potential synergies. Review of Social Economy, 63 (2), 161–176.

Dolan, P., Layard, R., & Metcalfe, R. (2011). Measuring subjective well-being for public policy . London School of Economics and Political Science, LSE Library. https://eprints.lse.ac.uk/35420/

Easterlin, R. A. (1974). Does economic growth improve the human lot? Some empirical evidence. In P. A. David & M. W. Reder (Eds.), Nations and households in economic growth: Essays in honor of Moses Abramovitz (pp. 89–125). Academic Press.

European Foundation for the Improvement of Living and Working Conditions. (2018). European Quality of Life Survey Integrated Data File, 2003-2016. [data collection]. 3rd Edition. UK Data Service. SN: 7348, 10.5255/UKDA-SN-7348-3

Graham, C., & Nikolova, M. (2015). Bentham or Aristotle in the development process? An empirical investigation of capabilities and subjective well-being. World Development, 68 , 163–179.

Hauser, R. M., & Goldberger, A. S. (1971). The treatment of unobservable variables in path analysis. Sociological Methodology, 3 , 81–117.

Hendriks, M., & Bartram, D. (2019). Bringing happiness into the study of migration and its consequences: What, why, and how? Journal of Immigrant & Refugee Studies, 17 (3), 279–298.

Hoekstra, R. (2019). Replacing GDP by 2030: Towards a common language for the well-being and sustainability community . Cambridge University Press. https://doi.org/10.1017/9781108608558

Jöreskog, K. G., & Goldberger, A. S. (1975). Estimation of a Model with Multiple Indicators and Multiple Causes of a Single Latent Variable. Journal of the American Statistical Association, 70 (351), 631–639.

Kogan, I., Shen, J., & Siegert, M. (2018). What makes a satisfied immigrant? Host-country characteristics and immigrants’ life satisfaction in eighteen European countries. Journal of Happiness Studies, 19 , 1783–1809.

Kotan, M. (2010). Freedom or happiness? Agency and subjective well-being in the capability approach. The Journal of Socio-Economics, 39 (3), 369–375.

Krishnakumar, J., & Ballon, P. (2008). Estimating Basic Capabilities: A Structural Equations Model Applied to Bolivia. World Development, 36 (6), 992–1010.

Krishnakumar, J., & Chávez-Juárez, F. (2016). Estimating Capabilities with Structural Equation Models: How Well are We Doing in a ‘Real’ World? Social Indicators Research, 129 , 717–737.

Kwarciński, T., & Ulman, P. (2018). A hybrid version of well-being: An attempt at operationalisation. Journal of Public Governance, 4 (46), 30–49.

Google Scholar  

Layard, R., & De Neve, J. E. (2023). Wellbeing . Cambridge University Press.

Book   Google Scholar  

Lin, Y. W., Lin, C. H., & Chen, C. N. (2023). Opportunities for Happiness and Its Determinants Among Children in China: A Study of Three Waves of the China Family Panel Studies Survey. Child Indicators Research, 16 (2), 551–579.

Muffels, R., & Headey, B. (2013). Capabilities and choices: Do they make Sen’se for understanding objective and subjective well-being? An empirical test of Sen’s capability framework on German and British panel data. Social Indicators Research, 110 (3), 1159–1185.

National Research Council. (2013). Subjective Well-Being: Measuring Happiness, Suffering, and Other Dimensions of Experience. The National Academies Press. https://doi.org/10.17226/18548 .

Ravazzini, L., & Chávez-Juárez, F. (2018). Which Inequality Makes People Dissatisfied with Their Lives? Evidence of the Link Between Life Satisfaction and Inequalities. Social Indicators Research, 137 , 1119–1143.

Robeyns, I. (2017). Wellbeing, Freedom . Open Book Publishers.

Sarr, F., & Ba, M. (2017). The capability approach and evaluation of the well-being in Senegal: An operationalization with the structural equations models. Modern Economy, 8 (1), 90–110.

Sen, A. (1979). Equality of What? In S. McMurrin (Ed.), The tanner lectures on human values (pp. 197–220). Cambridge University Press.

Sen, A. (1987). On Ethics and Economics . Oxford University Press.

Sen, A. (1992). Inequality reexamined . Clarendon Press.

Sen, A. (1993). Capability and well-being. In M. Nussbaum & A. Sen (Eds.), The Quality of Life (pp. 30–53). Oxford University Press.

Chapter   Google Scholar  

Sen, A. (2005). Commodities and Capabilities . Oxford University Press.

Sen, A. (2008). The economics of happiness and capability. In L. Bruni, F. Comim, & M. Pugno (Eds.), Capabilities and happiness (pp. 16–27). Oxford University Press.

Steckermeier, L. C. (2021). The Value of Autonomy for the Good Life. An Empirical Investigation of Autonomy and Life Satisfaction in Europe. Social Indicators Research, 154 , 693–723.

van Hoorn, A. A. J., Mabsout, R., & Sent, E. M. (2010). Happiness and capability: Introduction to the Symposium. Journal of Socio-Economics, 39 (3), 339–343.

Veenhoven, R. (2010). Capability and happiness: Conceptual difference and reality links. The Journal of Socio-Economics, 39 (3), 344–350.

Voicu, B., & Vasile, M. (2014). Do ‘cultures of life satisfaction’ travel? Current Sociology, 62 (1), 81–99.

Yeung, P., & Breheny, M. (2016). Using the capability approach to understand the determinants of subjective well-being among community-dwelling older people in New Zealand. Age and Ageing, 45 (2), 292–298.

Zwierzchowski, J., & Panek, T. (2020). Measurement of Subjective Well-being under Capability Approach in Poland. Polish Sociological Review, 210 (2), 157–178.

Download references

The publication was financed from the subsidy granted to the Krakow University of Economics—Project nr 084/EIT/2022/POT.

Author information

Authors and affiliations.

Department of Philosophy, Krakow University of Economics, Kraków, Poland

Tomasz Kwarciński

Department of Statistics, Krakow University of Economics, Kraków, Poland

Paweł Ulman

Bennett Institute for Public Policy, University of Cambridge, Cambridge, United Kingdom

Julia Wdowin

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Tomasz Kwarciński .

Ethics declarations

Conflicts of interest/competing interests.

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table 1 , Table 2 , Table 3 , Table 4 , Table 5 , Table 6 , Table 7 , Table 8 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Kwarciński, T., Ulman, P. & Wdowin, J. Measuring Subjective Well-being Capability: A Multi-Country Empirical Analysis in Europe. Applied Research Quality Life (2024). https://doi.org/10.1007/s11482-024-10334-9

Download citation

Received : 06 December 2023

Accepted : 04 June 2024

Published : 27 June 2024

DOI : https://doi.org/10.1007/s11482-024-10334-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Subjective well-being
• Capabilities
• Find a journal
• Publish with us
• Track your research

• Do Not Sell My Personal Info

•  ⋅ 

Google Reveals Its Methods For Measuring Search Quality

Google measures search results using surveys, experts, and user behavior. Improving quality leads to trickier searches and ongoing challenges.

• Google uses multiple methods to measure search quality, including human evaluators and behavioral analysis.
• Users tend to make more complex queries as search quality improves.
• Not everything important in search quality is measurable, Google says.

How does Google know if its search results are improving?

As Google rolls out algorithm updates and claims to reduce “unhelpful” content, many wonder about the true impact of these changes.

In an episode of Google’s Search Off The Record podcast , Google Search Directer, Product Management, Elizabeth Tucker discusses how Google measures search quality.

This article explores Tucker’s key revelations, the implications for marketers, and how you can adapt to stay ahead.

Multifaceted Approach To Measurement

Tucker, who transitioned to product management after 15 years as a data scientist at Google, says it’s difficult to determine whether search quality is improving.

“It’s really hard,” she admitted, describing a comprehensive strategy that includes user surveys, human evaluators, and behavioral analysis.

Tucker explained

“We use a lot of metrics where we sample queries and have human evaluators go through and evaluate the results for things like relevance.”

She also noted that Google analyzes user behavior patterns to infer whether people successfully find the information they seek.

The Moving Target Of User Behavior

Tucker revealed that users make more complex queries as search quality improves.

This creates a constantly shifting landscape for Google’s teams to navigate.

Tucker observed:

“The better we’re able to do this, the more interesting and difficult searches people will do.”

Counterintuitive Metrics

Tucker shared that in the short term, poor search performance might lead to increased search activity as users struggle to find information.

However, this trend reverses long-term, with sustained poor performance resulting in decreased usage.

Tucker cautioned:

“A measurement that can be good in the long term can be misleading in the short term.”

Quantifying Search Quality

To tackle the challenge of quantifying search quality, Google relies on an expansive (and expanding) set of metrics that gauge factors like relevance, accuracy, trustworthiness, and “freshness.”

But numbers don’t always tell the full story, Tucker cautioned:

“I think one important thing that we all have to acknowledge is that not everything important is measurable, and not everything that is measurable is important.”

For relatively straightforward queries, like a search for “Facebook,” delivering relevant results is a comparatively simple task for modern search engines.

However, more niche or complex searches demand rigorous analysis and attention, especially concerning critical health information.

The Human Element

Google aims to surface the most helpful information for searchers’ needs, which are as diverse as they are difficult to pin down at the scales Google operates at.

Tucker says:

“Understanding if we’re getting it right, where we’re getting it right, where needs focus out of those billions of queries – man, is that a hard problem.”

As developments in AI and machine learning push the boundaries of what’s possible in search, Tucker sees the “human element” as a key piece of the puzzle.

From the search quality raters who assess real-world results to the engineers and product managers, Google’s approach to quantifying search improvements blends big data with human insight.

Looking Ahead

As long as the web continues to evolve, Google’s work to refine its search quality measurements will be ongoing, Tucker says:

“Technology is constantly changing, websites are constantly changing. If we just stood still, search would get worse.”

What Does This Mean?

Google’s insights can help align your strategies with Google’s evolving standards.

Key takeaways include:

• Quality over quantity : Given Google’s focus on relevance and helpfulness, prioritize creating high-quality, user-centric content rather than aiming for sheer volume.
• Embrace complexity : Develop content that addresses more nuanced and specific user needs.
• Think long-term : Remember that short-term metrics can be misleading. Focus on sustained performance and user satisfaction rather than quick wins.
• Holistic approach : Like Google, adopt a multifaceted approach to measuring your content’s success, combining quantitative metrics with qualitative assessments.
• Stay adaptable : Given the constant changes in technology and user behavior, remain flexible and ready to adjust your strategies as needed.
• Human-centric : While leveraging AI and data analytics, don’t underestimate the importance of human insight in understanding and meeting user needs.

As Tucker’s insights show, this user-first approach is at the heart of Google’s efforts to improve search quality – and it should be at the center of every marketer’s strategy as well.

Listen to the discussion on measuring search quality in the video below, starting at the 17:39 mark:

Featured Image: Screenshot from YouTube.com/GoogleSearchCentral, June 2024

Matt G. Southern, Senior News Writer, has been with Search Engine Journal since 2013. With a bachelor’s degree in communications, ...

Subscribe To Our Newsletter.

Conquer your day with daily search marketing news.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.368; 2020

Clinical Updates

Quality improvement into practice, adam backhouse.

1 North London Partners in Health and Care, Islington CCG, London N1 1TH, UK

Fatai Ogunlayi

2 Institute of Applied Health Research, Public Health, University of Birmingham, B15 2TT, UK

What you need to know

• Thinking of quality improvement (QI) as a principle-based approach to change provides greater clarity about ( a ) the contribution QI offers to staff and patients, ( b ) how to differentiate it from other approaches, ( c ) the benefits of using QI together with other change approaches
• QI is not a silver bullet for all changes required in healthcare: it has great potential to be used together with other change approaches, either concurrently (using audit to inform iterative tests of change) or consecutively (using QI to adapt published research to local context)
• As QI becomes established, opportunities for these collaborations will grow, to the benefit of patients.

The benefits to front line clinicians of participating in quality improvement (QI) activity are promoted in many health systems. QI can represent a valuable opportunity for individuals to be involved in leading and delivering change, from improving individual patient care to transforming services across complex health and care systems. 1

However, it is not clear that this promotion of QI has created greater understanding of QI or widespread adoption. QI largely remains an activity undertaken by experts and early adopters, often in isolation from their peers. 2 There is a danger of a widening gap between this group and the majority of healthcare professionals.

This article will make it easier for those new to QI to understand what it is, where it fits with other approaches to improving care (such as audit or research), when best to use a QI approach, making it easier to understand the relevance and usefulness of QI in delivering better outcomes for patients.

How this article was made

AB and FO are both specialist quality improvement practitioners and have developed their expertise working in QI roles for a variety of UK healthcare organisations. The analysis presented here arose from AB and FO’s observations of the challenges faced when introducing QI, with healthcare providers often unable to distinguish between QI and other change approaches, making it difficult to understand what QI can do for them.

How is quality improvement defined?

There are many definitions of QI ( box 1 ). The BMJ ’s Quality Improvement series uses the Academy of Medical Royal Colleges definition. 6 Rather than viewing QI as a single method or set of tools, it can be more helpful to think of QI as based on a set of principles common to many of these definitions: a systematic continuous approach that aims to solve problems in healthcare, improve service provision, and ultimately provide better outcomes for patients.

Definitions of quality improvement

• Improvement in patient outcomes, system performance, and professional development that results from a combined, multidisciplinary approach in how change is delivered. 3
• The delivery of healthcare with improved outcomes and lower cost through continuous redesigning of work processes and systems. 4
• Using a systematic change method and strategies to improve patient experience and outcome. 5
• To make a difference to patients by improving safety, effectiveness, and experience of care by using understanding of our complex healthcare environment, applying a systematic approach, and designing, testing, and implementing changes using real time measurement for improvement. 6

In this article we discuss QI as an approach to improving healthcare that follows the principles outlined in box 2 ; this may be a useful reference to consider how particular methods or tools could be used as part of a QI approach.

Principles of QI

• Primary intent— To bring about measurable improvement to a specific aspect of healthcare delivery, often with evidence or theory of what might work but requiring local iterative testing to find the best solution. 7
• Employing an iterative process of testing change ideas— Adopting a theory of change which emphasises a continuous process of planning and testing changes, studying and learning from comparing the results to a predicted outcome, and adapting hypotheses in response to results of previous tests. 8 9
• Consistent use of an agreed methodology— Many different QI methodologies are available; commonly cited methodologies include the Model for Improvement, Lean, Six Sigma, and Experience-based Co-design. 4 Systematic review shows that the choice of tools or methodologies has little impact on the success of QI provided that the chosen methodology is followed consistently. 10 Though there is no formal agreement on what constitutes a QI tool, it would include activities such as process mapping that can be used within a range of QI methodological approaches. NHS Scotland’s Quality Improvement Hub has a glossary of commonly used tools in QI. 11
• Empowerment of front line staff and service users— QI work should engage staff and patients by providing them with the opportunity and skills to contribute to improvement work. Recognition of this need often manifests in drives from senior leadership or management to build QI capability in healthcare organisations, but it also requires that frontline staff and service users feel able to make use of these skills and take ownership of improvement work. 12
• Using data to drive improvement— To drive decision making by measuring the impact of tests of change over time and understanding variation in processes and outcomes. Measurement for improvement typically prioritises this narrative approach over concerns around exactness and completeness of data. 13 14
• Scale-up and spread, with adaptation to context— As interventions tested using a QI approach are scaled up and the degree of belief in their efficacy increases, it is desirable that they spread outward and be adopted by others. Key to successful diffusion of improvement is the adaption of interventions to new environments, patient and staff groups, available resources, and even personal preferences of healthcare providers in surrounding areas, again using an iterative testing approach. 15 16

What other approaches to improving healthcare are there?

Taking considered action to change healthcare for the better is not new, but QI as a distinct approach to improving healthcare is a relatively recent development. There are many well established approaches to evaluating and making changes to healthcare services in use, and QI will only be adopted more widely if it offers a new perspective or an advantage over other approaches in certain situations.

A non-systematic literature scan identified the following other approaches for making change in healthcare: research, clinical audit, service evaluation, and clinical transformation. We also identified innovation as an important catalyst for change, but we did not consider it an approach to evaluating and changing healthcare services so much as a catch-all term for describing the development and introduction of new ideas into the system. A summary of the different approaches and their definition is shown in box 3 . Many have elements in common with QI, but there are important difference in both intent and application. To be useful to clinicians and managers, QI must find a role within healthcare that complements research, audit, service evaluation, and clinical transformation while retaining the core principles that differentiate it from these approaches.

Alternatives to QI

• Research— The attempt to derive generalisable new knowledge by addressing clearly defined questions with systematic and rigorous methods. 17
• Clinical audit— A way to find out if healthcare is being provided in line with standards and to let care providers and patients know where their service is doing well, and where there could be improvements. 18
• Service evaluation— A process of investigating the effectiveness or efficiency of a service with the purpose of generating information for local decision making about the service. 19
• Clinical transformation— An umbrella term for more radical approaches to change; a deliberate, planned process to make dramatic and irreversible changes to how care is delivered. 20
• Innovation— To develop and deliver new or improved health policies, systems, products and technologies, and services and delivery methods that improve people’s health. Health innovation responds to unmet needs by employing new ways of thinking and working. 21

Why do we need to make this distinction for QI to succeed?

Improvement in healthcare is 20% technical and 80% human. 22 Essential to that 80% is clear communication, clarity of approach, and a common language. Without this shared understanding of QI as a distinct approach to change, QI work risks straying from the core principles outlined above, making it less likely to succeed. If practitioners cannot communicate clearly with their colleagues about the key principles and differences of a QI approach, there will be mismatched expectations about what QI is and how it is used, lowering the chance that QI work will be effective in improving outcomes for patients. 23

There is also a risk that the language of QI is adopted to describe change efforts regardless of their fidelity to a QI approach, either due to a lack of understanding of QI or a lack of intention to carry it out consistently. 9 Poor fidelity to the core principles of QI reduces its effectiveness and makes its desired outcome less likely, leading to wasted effort by participants and decreasing its credibility. 2 8 24 This in turn further widens the gap between advocates of QI and those inclined to scepticism, and may lead to missed opportunities to use QI more widely, consequently leading to variation in the quality of patient care.

Without articulating the differences between QI and other approaches, there is a risk of not being able to identify where a QI approach can best add value. Conversely, we might be tempted to see QI as a “silver bullet” for every healthcare challenge when a different approach may be more effective. In reality it is not clear that QI will be fit for purpose in tackling all of the wicked problems of healthcare delivery and we must be able to identify the right tool for the job in each situation. 25 Finally, while different approaches will be better suited to different types of challenge, not having a clear understanding of how approaches differ and complement each other may mean missed opportunities for multi-pronged approaches to improving care.

What is the relationship between QI and other approaches such as audit?

Academic journals, healthcare providers, and “arms-length bodies” have made various attempts to distinguish between the different approaches to improving healthcare. 19 26 27 28 However, most comparisons do not include QI or compare QI to only one or two of the other approaches. 7 29 30 31 To make it easier for people to use QI approaches effectively and appropriately, we summarise the similarities, differences, and crossover between QI and other approaches to tackling healthcare challenges ( fig 1 ).

How quality improvement interacts with other approaches to improving healthcare

QI and research

Research aims to generate new generalisable knowledge, while QI typically involves a combination of generating new knowledge or implementing existing knowledge within a specific setting. 32 Unlike research, including pragmatic research designed to test effectiveness of interventions in real life, QI does not aim to provide generalisable knowledge. In common with QI, research requires a consistent methodology. This method is typically used, however, to prove or disprove a fixed hypothesis rather than the adaptive hypotheses developed through the iterative testing of ideas typical of QI. Both research and QI are interested in the environment where work is conducted, though with different intentions: research aims to eliminate or at least reduce the impact of many variables to create generalisable knowledge, whereas QI seeks to understand what works best in a given context. The rigour of data collection and analysis required for research is much higher; in QI a criterion of “good enough” is often applied.

Relationship with QI

Though the goal of clinical research is to develop new knowledge that will lead to changes in practice, much has been written on the lag time between publication of research evidence and system-wide adoption, leading to delays in patients benefitting from new treatments or interventions. 33 QI offers a way to iteratively test the conditions required to adapt published research findings to the local context of individual healthcare providers, generating new knowledge in the process. Areas with little existing knowledge requiring further research may be identified during improvement activities, which in turn can form research questions for further study. QI and research also intersect in the field of improvement science, the academic study of QI methods which seeks to ensure QI is carried out as effectively as possible. 34

Scenario: QI for translational research

Newly published research shows that a particular physiotherapy intervention is more clinically effective when delivered in short, twice-daily bursts rather than longer, less frequent sessions. A team of hospital physiotherapists wish to implement the change but are unclear how they will manage the shift in workload and how they should introduce this potentially disruptive change to staff and to patients.

• Before continuing reading think about your own practice— How would you approach this situation, and how would you use the QI principles described in this article?

Adopting a QI approach, the team realise that, although the change they want to make is already determined, the way in which it is introduced and adapted to their wards is for them to decide. They take time to explain the benefits of the change to colleagues and their current patients, and ask patients how they would best like to receive their extra physiotherapy sessions.

The change is planned and tested for two weeks with one physiotherapist working with a small number of patients. Data are collected each day, including reasons why sessions were missed or refused. The team review the data each day and make iterative changes to the physiotherapist’s schedule, and to the times of day the sessions are offered to patients. Once an improvement is seen, this new way of working is scaled up to all of the patients on the ward.

The findings of the work are fed into a service evaluation of physiotherapy provision across the hospital, which uses the findings of the QI work to make recommendations about how physiotherapy provision should be structured in the future. People feel more positive about the change because they know colleagues who have already made it work in practice.

QI and clinical audit

Clinical audit is closely related to QI: it is often used with the intention of iteratively improving the standard of healthcare, albeit in relation to a pre-determined standard of best practice. 35 When used iteratively, interspersed with improvement action, the clinical audit cycle adheres to many of the principles of QI. However, in practice clinical audit is often used by healthcare organisations as an assurance function, making it less likely to be carried out with a focus on empowering staff and service users to make changes to practice. 36 Furthermore, academic reviews of audit programmes have shown audit to be an ineffective approach to improving quality due to a focus on data collection and analysis without a well developed approach to the action section of the audit cycle. 37 Clinical audits, such as the National Clinical Audit Programme in the UK (NCAPOP), often focus on the management of specific clinical conditions. QI can focus on any part of service delivery and can take a more cross-cutting view which may identify issues and solutions that benefit multiple patient groups and pathways. 30

Audit is often the first step in a QI process and is used to identify improvement opportunities, particularly where compliance with known standards for high quality patient care needs to be improved. Audit can be used to establish a baseline and to analyse the impact of tests of change against the baseline. Also, once an improvement project is under way, audit may form part of rapid cycle evaluation, during the iterative testing phase, to understand the impact of the idea being tested. Regular clinical audit may be a useful assurance tool to help track whether improvements have been sustained over time.

Scenario: Audit and QI

A foundation year 2 (FY2) doctor is asked to complete an audit of a pre-surgical pathway by looking retrospectively through patient documentation. She concludes that adherence to best practice is mixed and recommends: “Remind the team of the importance of being thorough in this respect and re-audit in 6 months.” The results are presented at an audit meeting, but a re-audit a year later by a new FY2 doctor shows similar results.

• Before continuing reading think about your own practice— How would you approach this situation, and how would you use the QI principles described in this paper?

Contrast the above with a team-led, rapid cycle audit in which everyone contributes to collecting and reviewing data from the previous week, discussed at a regular team meeting. Though surgical patients are often transient, their experience of care and ideas for improvement are captured during discharge conversations. The team identify and test several iterative changes to care processes. They document and test these changes between audits, leading to sustainable change. Some of the surgeons involved work across multiple hospitals, and spread some of the improvements, with the audit tool, as they go.

QI and service evaluation

In practice, service evaluation is not subject to the same rigorous definition or governance as research or clinical audit, meaning that there are inconsistencies in the methodology for carrying it out. While the primary intent for QI is to make change that will drive improvement, the primary intent for evaluation is to assess the performance of current patient care. 38 Service evaluation may be carried out proactively to assess a service against its stated aims or to review the quality of patient care, or may be commissioned in response to serious patient harm or red flags about service performance. The purpose of service evaluation is to help local decision makers determine whether a service is fit for purpose and, if necessary, identify areas for improvement.

Service evaluation may be used to initiate QI activity by identifying opportunities for change that would benefit from a QI approach. It may also evaluate the impact of changes made using QI, either during the work or after completion to assess sustainability of improvements made. Though likely planned as separate activities, service evaluation and QI may overlap and inform each other as they both develop. Service evaluation may also make a judgment about a service’s readiness for change and identify any barriers to, or prerequisites for, carrying out QI.

QI and clinical transformation

Clinical transformation involves radical, dramatic, and irreversible change—the sort of change that cannot be achieved through continuous improvement alone. As with service evaluation, there is no consensus on what clinical transformation entails, and it may be best thought of as an umbrella term for the large scale reform or redesign of clinical services and the non-clinical services that support them. 20 39 While it is possible to carry out transformation activity that uses elements of QI approach, such as effective engagement of the staff and patients involved, QI which rests on iterative test of change cannot have a transformational approach—that is, one-off, irreversible change.

There is opportunity to use QI to identify and test ideas before full scale clinical transformation is implemented. This has the benefit of engaging staff and patients in the clinical transformation process and increasing the degree of belief that clinical transformation will be effective or beneficial. Transformation activity, once completed, could be followed up with QI activity to drive continuous improvement of the new process or allow adaption of new ways of working. As interventions made using QI are scaled up and spread, the line between QI and transformation may seem to blur. The shift from QI to transformation occurs when the intention of the work shifts away from continuous testing and adaptation into the wholesale implementation of an agreed solution.

Scenario: QI and clinical transformation

An NHS trust’s human resources (HR) team is struggling to manage its junior doctor placements, rotas, and on-call duties, which is causing tension and has led to concern about medical cover and patient safety out of hours. A neighbouring trust has launched a smartphone app that supports clinicians and HR colleagues to manage these processes with the great success.

This problem feels ripe for a transformation approach—to launch the app across the trust, confident that it will solve the trust’s problems.

• Before continuing reading think about your own organisation— What do you think will happen, and how would you use the QI principles described in this article for this situation?

Outcome without QI

Unfortunately, the HR team haven’t taken the time to understand the underlying problems with their current system, which revolve around poor communication and clarity from the HR team, based on not knowing who to contact and being unable to answer questions. HR assume that because the app has been a success elsewhere, it will work here as well.

People get excited about the new app and the benefits it will bring, but no consideration is given to the processes and relationships that need to be in place to make it work. The app is launched with a high profile campaign and adoption is high, but the same issues continue. The HR team are confused as to why things didn’t work.

Outcome with QI

Although the app has worked elsewhere, rolling it out without adapting it to local context is a risk – one which application of QI principles can mitigate.

HR pilot the app in a volunteer specialty after spending time speaking to clinicians to better understand their needs. They carry out several tests of change, ironing out issues with the process as they go, using issues logged and clinician feedback as a source of data. When they are confident the app works for them, they expand out to a directorate, a division, and finally the transformational step of an organisation-wide rollout can be taken.

Education into practice

Next time when faced with what looks like a quality improvement (QI) opportunity, consider asking:

• How do you know that QI is the best approach to this situation? What else might be appropriate?
• Have you considered how to ensure you implement QI according to the principles described above?
• Is there opportunity to use other approaches in tandem with QI for a more effective result?

How patients were involved in the creation of this article

This article was conceived and developed in response to conversations with clinicians and patients working together on co-produced quality improvement and research projects in a large UK hospital. The first iteration of the article was reviewed by an expert patient, and, in response to their feedback, we have sought to make clearer the link between understanding the issues raised and better patient care.

This article is part of the Quality Improvement series ( https://www.bmj.com/quality-improvement ) produced by The BMJ in partnership with and funded by The Health Foundation.

Contributors: This work was initially conceived by AB. AB and FO were responsible for the research and drafting of the article. AB is the guarantor of the article.

Competing interests: We have read and understood BMJ policy on declaration of interests and have no relevant interests to declare.

Provenance and peer review: This article is part of a series commissioned by The BMJ based on ideas generated by a joint editorial group with members from the Health Foundation and The BMJ , including a patient/carer. The BMJ retained full editorial control over external peer review, editing, and publication. Open access fees and The BMJ ’s quality improvement editor post are funded by the Health Foundation.

Ephemeral Streams Likely to Have Significant Effect on U.S. Water Quality

A new study co-authored by Yale researchers quantifies the consequences of a U.S. Supreme Court decision that weakened the Clean Water Act.

  Listen to Article

Ephemeral streams, or those streams that flow only briefly after precipitation events, are a substantial pathway for water transfer with significant implications for water quality, a first-of-its kind study co-authored by Yale researchers has found.

These streams — which transport water pollutants, sediments, and nutrients from land surfaces to rivers, lakes, reservoirs, and ultimately the oceans — influence a substantial amount of water output of the nation’s rivers, the researchers found. Following a 2023 U.S. Supreme Court decision, however, they are no longer regulated by the Clean Water Act (CWA).

Peter A. Raymond Senior Associate Dean of Research & Director of Doctoral Studies; Oastler Professor of Biogeochemistry

“Our findings show that ephemeral streams are likely a substantial pathway through which pollution may influence downstream water quality, a finding that can inform evaluation of the consequences of limiting U.S. federal jurisdiction over ephemeral streams under the CWA,” the researchers from Yale and the University of Massachusetts Amherst said in the study published in Science .

In Sackett v. EPA, the U.S. Supreme Court narrowly defined “waters of the United States” (WOTUS) within the scope of the CWA, as encompassing “only those relatively permanent, standing, or continuously flowing bodies of water” — effectively removing ephemeral streams from U.S. federal jurisdiction.

For the new study, led by Craig Brinkerhoff, an incoming Yale postdoctoral fellow, and co-authored by Peter Raymond , Oastler Professor of Biogeochemistry at Yale School of the Environment (YSE), Matthew Kotchen , professor of economics at YSE, and Doug Kysar , Joseph M. Field ’55 Professor of Law at Yale Law School, the researchers modeled ephemeral stream contributions to the U.S. network of more than 20 million rivers, lakes, reservoirs, canals, and ditches.

The chemistry of water is dependent on how you manage the entire watershed, not just pieces of it. These streams are a critical source of water and pollutants and have to be regulated.”

In the nation’s largest rivers, such as the Mississippi and Columbia, more than 50% of the water originates from ephemeral streams at average annual discharge, they found. In some waterways, such as the Rio Grande, more than 90% of water comes from ephemeral streams. While the size of the river basin influences results, ephemeral streams on average account for 59% of drainage networks by length. These streams, the researchers say, pick up nitrogen, pesticides, and other pollutants that are likely relayed to the rivers at similar magnitudes as their water input.

“When the Supreme Court narrowed the scope of the federal Clean Water Act, it did so by referring to abstract dictionary definitions rather than science. This research underscores the impact of that approach since, by our estimate, over half of annual discharge from U.S. drainage networks will no longer be protected by the Act,” Kysar said.

By documenting the significance of ephemeral stream flow to downstream water quality, the results provide a basis for Congress to amend the CWA to expressly include ephemeral streams as an exercise of its power over interstate commerce, Kysar said, adding that findings also point to the need of enhanced regulation by state and local governments

“The chemistry of water is dependent on how you manage the entire watershed, not just pieces of it,” said Raymond. “These streams are a critical source of water and pollutants and have to be regulated.”

Craig Brinkerhoff, who led the research while completing his doctoral degree at the University of Massachusetts Amherst, said it shows the vast impacts of waterways that were once considered to influence only their immediate areas.

“Our study provides more concrete evidence that all of these things are connected,” he said.

• Peter A. Raymond
• Matthew Kotchen
• Douglas Kysar
• Environmental Policy Analysis
• Water Resource Science and Management

Media Contact

Fran Silverman

Associate Director of Communications

Research in the News

2024 Environmental Performance Index: A Surprise Top Ranking, Global Biodiversity Commitment Tested

Climate Change Threatens Resilience of Sri Lankan Rainforests

An Inside Look at Beech Leaf Disease

Connect with us.

• Request Information
• Register for Events

This is an alert popup's content... Read More

Topeka Shiner (Photo: Iowa Soybean Association / Joclyn Bushman)

Edge-of-field practice creates habitat and improves water quality

June 27, 2024 | Kriss Nelson

Linda Evans is proud to be the second generation of her family to own a piece of her family’s farm. As a self-proclaimed environmentalist, she is conservation-focused and welcomed the Iowa Soybean Association’s (ISA) invitation for an oxbow restoration project.

“I saw this project as a part of land stewardship,” says Evans. “Whether it’s planting cover crops, installing waterways, restoring oxbows or other conservation practices, most farmers want to be good stewards of the land.”

Three oxbows were restored on Evans’ land near Paton in Greene County. Recently, ISA hosted an Innovation to Profit meeting centered on Evans’ oxbows.

The event held on June 13 also highlighted other edge-of-field practices, water quality and research opportunities.

Evans worked closely with Brandon Iddings, ISA’s field service program manager and Darrick Weissenfluh, private lands biologist with the U.S. Fish and Wildlife Service to restore her oxbows.

“These are the first oxbows that have been restored on East Buttrick Creek and we have plans to do more,” says Iddings. “At one time, they were connected to the nearby river. After the river shifted 50 years ago or so, the oxbows became disconnected and filled with dirt.”

Unique to Evans’ oxbows is the installation of control structures, which Iddings says can be used to help manipulate how much water is being held.

Now the restoration is complete, and the banks have been seeded with native grasses and plants creating a space for a wildlife habitat, including the Topeka shiner.

Iddings says oxbows are crucial habitats for the Topeka shiner, an endangered species in Iowa.

After the flood water recede, oxbows allow small minnows like the Topeka shiner to lay eggs. With the smaller pool sizes, predator fish can’t get as big; smaller fish have a better chance at survival.

According to the United States Fish and Wildlife Service (USFWS) Species Status Assessment, the primary conservation action in Iowa for the recovery of the Topeka shiner is the restoration of their habitat through oxbow restoration. The discovery of a significant population of Topeka shiner was confirmed during the field day.

Evans entrusts the operation of her farmland to her nephew, Jonathan Marshall.

Evans initially worried that the oxbow restoration would take away pastureland from Marshall’s cow herd, but that turned out not to be the case.

“That land was low, swampy ground,” says Marshall. “The grass that grows there isn’t attractive to cows. By restoring the oxbow, it now allows them to have a place to drink and cool off.”

The task of excavating and restoring the oxbow was a low-cost and easy endeavor for Evans.

“It was a simple process and I appreciate the help from Brandon,” she says. “It’s a small step to doing better. We are working to make our land a better place and our food and water safer.”

According to the Journal of the American Water Resources Association, oxbows have shown the ability to treat an average of 62% of nitrates from tile-fed multipurpose oxbows.

To learn more about oxbow restoration and funding opportunities, contact Iddings at [email protected] or call 515-729-0039 .

Related News