research methodology question paper model

• Privacy Policy

Home » Research Methodology – Types, Examples and writing Guide

Research Methodology – Types, Examples and writing Guide

Table of Contents

Research Methodology

Definition:

Research Methodology refers to the systematic and scientific approach used to conduct research, investigate problems, and gather data and information for a specific purpose. It involves the techniques and procedures used to identify, collect , analyze , and interpret data to answer research questions or solve research problems . Moreover, They are philosophical and theoretical frameworks that guide the research process.

Structure of Research Methodology

Research methodology formats can vary depending on the specific requirements of the research project, but the following is a basic example of a structure for a research methodology section:

I. Introduction

• Provide an overview of the research problem and the need for a research methodology section
• Outline the main research questions and objectives

II. Research Design

• Explain the research design chosen and why it is appropriate for the research question(s) and objectives
• Discuss any alternative research designs considered and why they were not chosen
• Describe the research setting and participants (if applicable)

III. Data Collection Methods

• Describe the methods used to collect data (e.g., surveys, interviews, observations)
• Explain how the data collection methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or instruments used for data collection

IV. Data Analysis Methods

• Describe the methods used to analyze the data (e.g., statistical analysis, content analysis )
• Explain how the data analysis methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or software used for data analysis

V. Ethical Considerations

• Discuss any ethical issues that may arise from the research and how they were addressed
• Explain how informed consent was obtained (if applicable)
• Detail any measures taken to ensure confidentiality and anonymity

VI. Limitations

• Identify any potential limitations of the research methodology and how they may impact the results and conclusions

VII. Conclusion

• Summarize the key aspects of the research methodology section
• Explain how the research methodology addresses the research question(s) and objectives

Research Methodology Types

Types of Research Methodology are as follows:

Quantitative Research Methodology

This is a research methodology that involves the collection and analysis of numerical data using statistical methods. This type of research is often used to study cause-and-effect relationships and to make predictions.

Qualitative Research Methodology

This is a research methodology that involves the collection and analysis of non-numerical data such as words, images, and observations. This type of research is often used to explore complex phenomena, to gain an in-depth understanding of a particular topic, and to generate hypotheses.

Mixed-Methods Research Methodology

This is a research methodology that combines elements of both quantitative and qualitative research. This approach can be particularly useful for studies that aim to explore complex phenomena and to provide a more comprehensive understanding of a particular topic.

Case Study Research Methodology

This is a research methodology that involves in-depth examination of a single case or a small number of cases. Case studies are often used in psychology, sociology, and anthropology to gain a detailed understanding of a particular individual or group.

Action Research Methodology

This is a research methodology that involves a collaborative process between researchers and practitioners to identify and solve real-world problems. Action research is often used in education, healthcare, and social work.

Experimental Research Methodology

This is a research methodology that involves the manipulation of one or more independent variables to observe their effects on a dependent variable. Experimental research is often used to study cause-and-effect relationships and to make predictions.

Survey Research Methodology

This is a research methodology that involves the collection of data from a sample of individuals using questionnaires or interviews. Survey research is often used to study attitudes, opinions, and behaviors.

Grounded Theory Research Methodology

This is a research methodology that involves the development of theories based on the data collected during the research process. Grounded theory is often used in sociology and anthropology to generate theories about social phenomena.

Research Methodology Example

An Example of Research Methodology could be the following:

Research Methodology for Investigating the Effectiveness of Cognitive Behavioral Therapy in Reducing Symptoms of Depression in Adults

Introduction:

The aim of this research is to investigate the effectiveness of cognitive-behavioral therapy (CBT) in reducing symptoms of depression in adults. To achieve this objective, a randomized controlled trial (RCT) will be conducted using a mixed-methods approach.

Research Design:

The study will follow a pre-test and post-test design with two groups: an experimental group receiving CBT and a control group receiving no intervention. The study will also include a qualitative component, in which semi-structured interviews will be conducted with a subset of participants to explore their experiences of receiving CBT.

Participants:

Participants will be recruited from community mental health clinics in the local area. The sample will consist of 100 adults aged 18-65 years old who meet the diagnostic criteria for major depressive disorder. Participants will be randomly assigned to either the experimental group or the control group.

Intervention :

The experimental group will receive 12 weekly sessions of CBT, each lasting 60 minutes. The intervention will be delivered by licensed mental health professionals who have been trained in CBT. The control group will receive no intervention during the study period.

Data Collection:

Quantitative data will be collected through the use of standardized measures such as the Beck Depression Inventory-II (BDI-II) and the Generalized Anxiety Disorder-7 (GAD-7). Data will be collected at baseline, immediately after the intervention, and at a 3-month follow-up. Qualitative data will be collected through semi-structured interviews with a subset of participants from the experimental group. The interviews will be conducted at the end of the intervention period, and will explore participants’ experiences of receiving CBT.

Data Analysis:

Quantitative data will be analyzed using descriptive statistics, t-tests, and mixed-model analyses of variance (ANOVA) to assess the effectiveness of the intervention. Qualitative data will be analyzed using thematic analysis to identify common themes and patterns in participants’ experiences of receiving CBT.

Ethical Considerations:

This study will comply with ethical guidelines for research involving human subjects. Participants will provide informed consent before participating in the study, and their privacy and confidentiality will be protected throughout the study. Any adverse events or reactions will be reported and managed appropriately.

Data Management:

All data collected will be kept confidential and stored securely using password-protected databases. Identifying information will be removed from qualitative data transcripts to ensure participants’ anonymity.

Limitations:

One potential limitation of this study is that it only focuses on one type of psychotherapy, CBT, and may not generalize to other types of therapy or interventions. Another limitation is that the study will only include participants from community mental health clinics, which may not be representative of the general population.

Conclusion:

This research aims to investigate the effectiveness of CBT in reducing symptoms of depression in adults. By using a randomized controlled trial and a mixed-methods approach, the study will provide valuable insights into the mechanisms underlying the relationship between CBT and depression. The results of this study will have important implications for the development of effective treatments for depression in clinical settings.

How to Write Research Methodology

Writing a research methodology involves explaining the methods and techniques you used to conduct research, collect data, and analyze results. It’s an essential section of any research paper or thesis, as it helps readers understand the validity and reliability of your findings. Here are the steps to write a research methodology:

• Start by explaining your research question: Begin the methodology section by restating your research question and explaining why it’s important. This helps readers understand the purpose of your research and the rationale behind your methods.
• Describe your research design: Explain the overall approach you used to conduct research. This could be a qualitative or quantitative research design, experimental or non-experimental, case study or survey, etc. Discuss the advantages and limitations of the chosen design.
• Discuss your sample: Describe the participants or subjects you included in your study. Include details such as their demographics, sampling method, sample size, and any exclusion criteria used.
• Describe your data collection methods : Explain how you collected data from your participants. This could include surveys, interviews, observations, questionnaires, or experiments. Include details on how you obtained informed consent, how you administered the tools, and how you minimized the risk of bias.
• Explain your data analysis techniques: Describe the methods you used to analyze the data you collected. This could include statistical analysis, content analysis, thematic analysis, or discourse analysis. Explain how you dealt with missing data, outliers, and any other issues that arose during the analysis.
• Discuss the validity and reliability of your research : Explain how you ensured the validity and reliability of your study. This could include measures such as triangulation, member checking, peer review, or inter-coder reliability.
• Acknowledge any limitations of your research: Discuss any limitations of your study, including any potential threats to validity or generalizability. This helps readers understand the scope of your findings and how they might apply to other contexts.
• Provide a summary: End the methodology section by summarizing the methods and techniques you used to conduct your research. This provides a clear overview of your research methodology and helps readers understand the process you followed to arrive at your findings.

When to Write Research Methodology

Research methodology is typically written after the research proposal has been approved and before the actual research is conducted. It should be written prior to data collection and analysis, as it provides a clear roadmap for the research project.

The research methodology is an important section of any research paper or thesis, as it describes the methods and procedures that will be used to conduct the research. It should include details about the research design, data collection methods, data analysis techniques, and any ethical considerations.

The methodology should be written in a clear and concise manner, and it should be based on established research practices and standards. It is important to provide enough detail so that the reader can understand how the research was conducted and evaluate the validity of the results.

Applications of Research Methodology

Here are some of the applications of research methodology:

• To identify the research problem: Research methodology is used to identify the research problem, which is the first step in conducting any research.
• To design the research: Research methodology helps in designing the research by selecting the appropriate research method, research design, and sampling technique.
• To collect data: Research methodology provides a systematic approach to collect data from primary and secondary sources.
• To analyze data: Research methodology helps in analyzing the collected data using various statistical and non-statistical techniques.
• To test hypotheses: Research methodology provides a framework for testing hypotheses and drawing conclusions based on the analysis of data.
• To generalize findings: Research methodology helps in generalizing the findings of the research to the target population.
• To develop theories : Research methodology is used to develop new theories and modify existing theories based on the findings of the research.
• To evaluate programs and policies : Research methodology is used to evaluate the effectiveness of programs and policies by collecting data and analyzing it.
• To improve decision-making: Research methodology helps in making informed decisions by providing reliable and valid data.

Purpose of Research Methodology

Research methodology serves several important purposes, including:

• To guide the research process: Research methodology provides a systematic framework for conducting research. It helps researchers to plan their research, define their research questions, and select appropriate methods and techniques for collecting and analyzing data.
• To ensure research quality: Research methodology helps researchers to ensure that their research is rigorous, reliable, and valid. It provides guidelines for minimizing bias and error in data collection and analysis, and for ensuring that research findings are accurate and trustworthy.
• To replicate research: Research methodology provides a clear and detailed account of the research process, making it possible for other researchers to replicate the study and verify its findings.
• To advance knowledge: Research methodology enables researchers to generate new knowledge and to contribute to the body of knowledge in their field. It provides a means for testing hypotheses, exploring new ideas, and discovering new insights.
• To inform decision-making: Research methodology provides evidence-based information that can inform policy and decision-making in a variety of fields, including medicine, public health, education, and business.

Advantages of Research Methodology

Research methodology has several advantages that make it a valuable tool for conducting research in various fields. Here are some of the key advantages of research methodology:

• Systematic and structured approach : Research methodology provides a systematic and structured approach to conducting research, which ensures that the research is conducted in a rigorous and comprehensive manner.
• Objectivity : Research methodology aims to ensure objectivity in the research process, which means that the research findings are based on evidence and not influenced by personal bias or subjective opinions.
• Replicability : Research methodology ensures that research can be replicated by other researchers, which is essential for validating research findings and ensuring their accuracy.
• Reliability : Research methodology aims to ensure that the research findings are reliable, which means that they are consistent and can be depended upon.
• Validity : Research methodology ensures that the research findings are valid, which means that they accurately reflect the research question or hypothesis being tested.
• Efficiency : Research methodology provides a structured and efficient way of conducting research, which helps to save time and resources.
• Flexibility : Research methodology allows researchers to choose the most appropriate research methods and techniques based on the research question, data availability, and other relevant factors.
• Scope for innovation: Research methodology provides scope for innovation and creativity in designing research studies and developing new research techniques.

Research Methodology Vs Research Methods

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

Dissertation – Format, Example and Template

Research Questions – Types, Examples and Writing...

Research Paper Conclusion – Writing Guide and...

Data Analysis – Process, Methods and Types

Research Objectives – Types, Examples and...

Dissertation vs Thesis – Key Differences

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

Here's What You Need to Understand About Research Methodology

Table of Contents

Research methodology involves a systematic and well-structured approach to conducting scholarly or scientific inquiries. Knowing the significance of research methodology and its different components is crucial as it serves as the basis for any study.

Typically, your research topic will start as a broad idea you want to investigate more thoroughly. Once you’ve identified a research problem and created research questions , you must choose the appropriate methodology and frameworks to address those questions effectively.

What is the definition of a research methodology?

Research methodology is the process or the way you intend to execute your study. The methodology section of a research paper outlines how you plan to conduct your study. It covers various steps such as collecting data, statistical analysis, observing participants, and other procedures involved in the research process

The methods section should give a description of the process that will convert your idea into a study. Additionally, the outcomes of your process must provide valid and reliable results resonant with the aims and objectives of your research. This thumb rule holds complete validity, no matter whether your paper has inclinations for qualitative or quantitative usage.

Studying research methods used in related studies can provide helpful insights and direction for your own research. Now easily discover papers related to your topic on SciSpace and utilize our AI research assistant, Copilot , to quickly review the methodologies applied in different papers.

The need for a good research methodology

While deciding on your approach towards your research, the reason or factors you weighed in choosing a particular problem and formulating a research topic need to be validated and explained. A research methodology helps you do exactly that. Moreover, a good research methodology lets you build your argument to validate your research work performed through various data collection methods, analytical methods, and other essential points.

Just imagine it as a strategy documented to provide an overview of what you intend to do.

While undertaking any research writing or performing the research itself, you may get drifted in not something of much importance. In such a case, a research methodology helps you to get back to your outlined work methodology.

A research methodology helps in keeping you accountable for your work. Additionally, it can help you evaluate whether your work is in sync with your original aims and objectives or not. Besides, a good research methodology enables you to navigate your research process smoothly and swiftly while providing effective planning to achieve your desired results.

What is the basic structure of a research methodology?

Usually, you must ensure to include the following stated aspects while deciding over the basic structure of your research methodology:

1. Your research procedure

Explain what research methods you’re going to use. Whether you intend to proceed with quantitative or qualitative, or a composite of both approaches, you need to state that explicitly. The option among the three depends on your research’s aim, objectives, and scope.

2. Provide the rationality behind your chosen approach

Based on logic and reason, let your readers know why you have chosen said research methodologies. Additionally, you have to build strong arguments supporting why your chosen research method is the best way to achieve the desired outcome.

3. Explain your mechanism

The mechanism encompasses the research methods or instruments you will use to develop your research methodology. It usually refers to your data collection methods. You can use interviews, surveys, physical questionnaires, etc., of the many available mechanisms as research methodology instruments. The data collection method is determined by the type of research and whether the data is quantitative data(includes numerical data) or qualitative data (perception, morale, etc.) Moreover, you need to put logical reasoning behind choosing a particular instrument.

4. Significance of outcomes

The results will be available once you have finished experimenting. However, you should also explain how you plan to use the data to interpret the findings. This section also aids in understanding the problem from within, breaking it down into pieces, and viewing the research problem from various perspectives.

5. Reader’s advice

Anything that you feel must be explained to spread more awareness among readers and focus groups must be included and described in detail. You should not just specify your research methodology on the assumption that a reader is aware of the topic.  

All the relevant information that explains and simplifies your research paper must be included in the methodology section. If you are conducting your research in a non-traditional manner, give a logical justification and list its benefits.

6. Explain your sample space

Include information about the sample and sample space in the methodology section. The term "sample" refers to a smaller set of data that a researcher selects or chooses from a larger group of people or focus groups using a predetermined selection method. Let your readers know how you are going to distinguish between relevant and non-relevant samples. How you figured out those exact numbers to back your research methodology, i.e. the sample spacing of instruments, must be discussed thoroughly.

For example, if you are going to conduct a survey or interview, then by what procedure will you select the interviewees (or sample size in case of surveys), and how exactly will the interview or survey be conducted.

7. Challenges and limitations

This part, which is frequently assumed to be unnecessary, is actually very important. The challenges and limitations that your chosen strategy inherently possesses must be specified while you are conducting different types of research.

The importance of a good research methodology

You must have observed that all research papers, dissertations, or theses carry a chapter entirely dedicated to research methodology. This section helps maintain your credibility as a better interpreter of results rather than a manipulator.

A good research methodology always explains the procedure, data collection methods and techniques, aim, and scope of the research. In a research study, it leads to a well-organized, rationality-based approach, while the paper lacking it is often observed as messy or disorganized.

You should pay special attention to validating your chosen way towards the research methodology. This becomes extremely important in case you select an unconventional or a distinct method of execution.

Curating and developing a strong, effective research methodology can assist you in addressing a variety of situations, such as:

• When someone tries to duplicate or expand upon your research after few years.
• If a contradiction or conflict of facts occurs at a later time. This gives you the security you need to deal with these contradictions while still being able to defend your approach.
• Gaining a tactical approach in getting your research completed in time. Just ensure you are using the right approach while drafting your research methodology, and it can help you achieve your desired outcomes. Additionally, it provides a better explanation and understanding of the research question itself.
• Documenting the results so that the final outcome of the research stays as you intended it to be while starting.

Instruments you could use while writing a good research methodology

As a researcher, you must choose which tools or data collection methods that fit best in terms of the relevance of your research. This decision has to be wise.

There exists many research equipments or tools that you can use to carry out your research process. These are classified as:

a. Interviews (One-on-One or a Group)

An interview aimed to get your desired research outcomes can be undertaken in many different ways. For example, you can design your interview as structured, semi-structured, or unstructured. What sets them apart is the degree of formality in the questions. On the other hand, in a group interview, your aim should be to collect more opinions and group perceptions from the focus groups on a certain topic rather than looking out for some formal answers.

In surveys, you are in better control if you specifically draft the questions you seek the response for. For example, you may choose to include free-style questions that can be answered descriptively, or you may provide a multiple-choice type response for questions. Besides, you can also opt to choose both ways, deciding what suits your research process and purpose better.

c. Sample Groups

Similar to the group interviews, here, you can select a group of individuals and assign them a topic to discuss or freely express their opinions over that. You can simultaneously note down the answers and later draft them appropriately, deciding on the relevance of every response.

d. Observations

If your research domain is humanities or sociology, observations are the best-proven method to draw your research methodology. Of course, you can always include studying the spontaneous response of the participants towards a situation or conducting the same but in a more structured manner. A structured observation means putting the participants in a situation at a previously decided time and then studying their responses.

Of all the tools described above, it is you who should wisely choose the instruments and decide what’s the best fit for your research. You must not restrict yourself from multiple methods or a combination of a few instruments if appropriate in drafting a good research methodology.

Types of research methodology

A research methodology exists in various forms. Depending upon their approach, whether centered around words, numbers, or both, methodologies are distinguished as qualitative, quantitative, or an amalgamation of both.

1. Qualitative research methodology

When a research methodology primarily focuses on words and textual data, then it is generally referred to as qualitative research methodology. This type is usually preferred among researchers when the aim and scope of the research are mainly theoretical and explanatory.

The instruments used are observations, interviews, and sample groups. You can use this methodology if you are trying to study human behavior or response in some situations. Generally, qualitative research methodology is widely used in sociology, psychology, and other related domains.

2. Quantitative research methodology

If your research is majorly centered on data, figures, and stats, then analyzing these numerical data is often referred to as quantitative research methodology. You can use quantitative research methodology if your research requires you to validate or justify the obtained results.

In quantitative methods, surveys, tests, experiments, and evaluations of current databases can be advantageously used as instruments If your research involves testing some hypothesis, then use this methodology.

3. Amalgam methodology

As the name suggests, the amalgam methodology uses both quantitative and qualitative approaches. This methodology is used when a part of the research requires you to verify the facts and figures, whereas the other part demands you to discover the theoretical and explanatory nature of the research question.

The instruments for the amalgam methodology require you to conduct interviews and surveys, including tests and experiments. The outcome of this methodology can be insightful and valuable as it provides precise test results in line with theoretical explanations and reasoning.

The amalgam method, makes your work both factual and rational at the same time.

Final words: How to decide which is the best research methodology?

If you have kept your sincerity and awareness intact with the aims and scope of research well enough, you must have got an idea of which research methodology suits your work best.

Before deciding which research methodology answers your research question, you must invest significant time in reading and doing your homework for that. Taking references that yield relevant results should be your first approach to establishing a research methodology.

Moreover, you should never refrain from exploring other options. Before setting your work in stone, you must try all the available options as it explains why the choice of research methodology that you finally make is more appropriate than the other available options.

You should always go for a quantitative research methodology if your research requires gathering large amounts of data, figures, and statistics. This research methodology will provide you with results if your research paper involves the validation of some hypothesis.

Whereas, if  you are looking for more explanations, reasons, opinions, and public perceptions around a theory, you must use qualitative research methodology.The choice of an appropriate research methodology ultimately depends on what you want to achieve through your research.

Frequently Asked Questions (FAQs) about Research Methodology

1. how to write a research methodology.

You can always provide a separate section for research methodology where you should specify details about the methods and instruments used during the research, discussions on result analysis, including insights into the background information, and conveying the research limitations.

2. What are the types of research methodology?

There generally exists four types of research methodology i.e.

• Observation
• Experimental
• Derivational

3. What is the true meaning of research methodology?

The set of techniques or procedures followed to discover and analyze the information gathered to validate or justify a research outcome is generally called Research Methodology.

4. Where lies the importance of research methodology?

Your research methodology directly reflects the validity of your research outcomes and how well-informed your research work is. Moreover, it can help future researchers cite or refer to your research if they plan to use a similar research methodology.

You might also like

Consensus GPT vs. SciSpace GPT: Choose the Best GPT for Research

Literature Review and Theoretical Framework: Understanding the Differences

Using AI for research: A beginner’s guide

We are the No 1 PhD Consultancy in South India

20+ students register for phd every month, coursework – research methodology -sample question paper with answers.

Multiple Choice Questions

• Conference proceedings are considered as.documents. a. Conventional b. Primary c. Secondary d. Tertiary Answer : b. Primary
• Informationis….. a. RawData b. Processed Data c. Inputdata d. Organized data Answer : b. Processed Data
• Information acquired by experience or experimentation is called as: a. Empirical b. Scientific c. Facts d. Scientific Evidence Answer : b. Scientific
• Abstract elements representing classes of phenomena within the field of study are called : a.Concepts b.Theories c.Variables d.Hypothesis Answer: a. Concepts
• All living things are made up of cells Blue whale is a living being, Thereforeblue whale is made up of cells’ The reasoning used here is a. Inductive b. Deductive c. Hypothetic deductive d. Both a and b Answer : b. Deductive
• Questionnaire is a: a. Research method b. Measurement technique c. Tool for data collection d. Data analysis technique Answer : b. Measurement Technique
• Mean, Median and Mode are a. Measures of deviation b. Ways of sampling c. Measure of control tendency d. None of the above Answer : c. Measure of control tendency
• The reasoning that uses general principle to predict specific results is calledas- a. Inductive b. Deductive c. Both a and b d. Hypothetic o-deductive Answer : b. Deductive
• A research paper is a brief report of research work based on a. Primary Data only b. Secondary Data only c. Both a and b d. None of the above Answer : c. Both a and b
• Research is a. Searching again and again b. Finding solutions to any problem c. Working in a scientific way to d. None -of the above Answer : c. Working in a scientific way to
• Multiple-choice questions are an example of a. OrdinalMeasure b. Nominal Measure c. RatioMeasure d. None of the above Answer : b. Nominal Measure
• Which of the variables cannot be expressed in quantitative terms a. Socio economic status b. Marital status c. Numerical aptitude d. Professional attitude Answer : d. Professional attitude
• The essential qualities of a researcher are : a. Spirit of free enquiry b. Reliance on observation c. Reliance on evidences d. All of the above Answer : d. All the above
• A research process starts with- a. Hypothesis b. Experiment to test hypothesis c. Observation d. None of the above Answer : a. Hypothesis
• Who was the proponent of deductive method- a. FrancisBacon b. Christian Huygenes c. Aristotle d. Isaac Newton Answer : b. Christian Huygenes
• The non-random sampling type that involves selecting a convenience sample from a population with a specific set of characteristics for your research study is called a. Convenience sampling b. Quota sampling c. Purposive sampling d. None of the above Answer : a. Convenience Sampling
• Which of the following is NOT an example of a non-random sampling technique? a. Purposive b. Quota c. Convenience d. Cluster Answer : c. Convenience
• The purpose of drawing sample from a population is known as a. Sampling b. Census c. Survey research d. None of the above Answer : a. Sampling
• Sampling in qualitative research is similar to which type of sampling in quantitative research a. Simple random sampling b. Systematic sampling c. Quotasampling d. Purposive sampling Answer : d. Purposive sampling
• A set of rules that govern overall data communications system is popularly known as……….. a. Protocol b. Agreement c. Pact d. Memorandum Answer : a. Protocol

Essay Questions

•  Basic Research: In this type of research, data is collected to enhance knowledge. The purpose is non-commercial research that is generally not used to invent anything.
•  Applied research: The focus of this research is to analyze and solve real-life problems. It prefers to help solve a practical problem with scientific methods.
•  Problem-Oriented research: It focuses on understanding the nature of the problem to find a relevant solution. The problem could be in various forms; this research analyses the situation.
•  Problem-solving research: Companies usually conduct this type of research to understand and resolve their problems. The research is to find a solution to an existing problem.
•  Qualitative research is a process of inquiry that helps to create an in-depth understanding of problems and issues. It has open ended questions
• State the purpose clearly
• Define the concepts used
• Describe the research procedure in sufficient detail that allows another researcher to make further advancement on the topic
• Design the procedure carefully to achieve desired results
• Data analysis should reveal adequate significance
• Appropriate analysis methods should be used.
• Carefully check the validity and reliability of the data.
• Conclusions should be confined to justify the research data and limit for the which data provides and adequate basis
• Systematic research: Conduct research in structured format with specified steps, rules while keeping in perspective the creative thinking.
•  Research is guided by logical reasoning and process of deduction and induction, which serves as a great value in carrying out research.
•  It is empirical: research is related to one or more than one aspects in real situation that deals with concrete data
•  It is replicable: the characteristics allow researchers to replicate study and building a sound basis for decisions.
• Observing Behaviors of Participants:
• Questionnaire Method
• Interview Method
• Schedules Method
• Information from Correspondents
• Identify the problem
• Review the Literature
• Clarify the Problem
• Clearly Define Terms and Concepts
• Define the Population
• Develop the Instrumentation Plan
• Collect Data
• Analyze the Data

For brochure with pricing details, please mention your contact details below and we will get back to you. Alternatively, you can also call us at  +91-8130872449 . If that number is busy, you can reach us at  91-9160743777  [Hyderabad] or  +91-7411845787  [Bangalore]

Our Branches

My Indian Academy, Vaswani Chambers, 1st floor 264-265, Dr. Annie Besant Rd, Worli, Mumbai, Maharashtra 400051. +91-8976286890

My Indian Academy, 1st floor, D-1/38, near Ramphal Chowk Road, Block D, Sector 7 Dwarka, New Delhi, Delhi 110075. +91-7678657219

My Indian Academy, Fairfield by Marriot, Sankalpa Apartment 2 Block-B Street Number 189, CB Block(Newtown), Action Area I, Newtown, New Town, West Bengal 700156. +91-7679482290

My Indian Academy, 91 Springboard, Gopala Krishna Complex, 45/3, Residency Road, MG Road, Shanthala Nagar, Ashok Nagar, Bangalore – 560025. +91-8548062559

My Indian Academy, HatchStation, 1-8-303, 48/15, PG Road, Begumpet, Hyderabad, Telangana 500003 Phone : +91-9177449218

430+ Research Methodology (RM) Solved MCQs

65 Research Methodology Question Paper PDF Download Free

In case you are preparing for your final year dissertation, you might be interested in downloading a research methodology question paper PDF.

This article will discuss where to find such a question bank. We have given the direct links to download the question papers .

The links in this article will guide you through the steps needed to download and use the question bank. Moreover, it will help you understand what exactly goes into creating a good research methodology question paper. Here are some tips that will help you succeed in this course.

Interesting

• How to Complete Syllabus Fast Easily
• Top 20 Ways To Earn Money As A Student In India
• 2022s Best Diet for Students Preparing for Competitive Exams
• Best 11 ways How to Improve Handwriting Easily
• Effective Guide to Prepare for Exams in a Week

Research Methodology Question Paper PDF

The Research methodology MCQ questions are designed to test the knowledge of the students in their field. They are applicable to competitive and academic MBA, regular, and distance-learning courses. It is important to choose the research question carefully, as it determines the purpose of your study.

The Research Methodology Question Paper questions may be related to a gap in knowledge, analyzing academic assumptions, monitoring a practice development, testing theories, or comparing methods. Ensure that the research question is appropriate for the discipline and institution and is based on a problem-solving framework.

Research Methodology Question Bank pdf

The M.Com Accountancy (IDOL) Research Methodology question paper is available in pdf format for download. It is a good practice to download previous year’s question papers from the Internet as this will help you to learn the techniques used in solving them.

Also, you can look up the solutions of the sample papers to know how to do better in your own research. However, be sure to select the right question paper for your course.

The first step in any research is the formulation of a hypothesis. A hypothesis is a statement regarding a particular population’s size or behaviour. This is followed by research designs. Samples are the group of individuals chosen for a study.

The data collected is analyzed using various techniques. They may be observational or experimental. Regardless of the technique used, the study must answer two research questions. The data collection is based on several methods, including observational research, experimentation, simulation, and derived methodology.

Research Methodology Question Bank with Answers pdf

The first thing you need to do is figure out the research methodology. The research methodology includes the methods and procedures for conducting research. These Research Methodology Question Paper may include physical surveys, questionnaires, interviews, or participant observations. Once you have decided what method to use, you can begin preparing for the exam. Here are some things you need to know about this type of exam.

This article will help you decide what method you want to use. You can also use the research methodology question bank to study various subject aspects.

The objective of the Research Methodology Question Papers is to discover a frequency or an association. It involves collecting data and conducting research using a systematic, logical approach. It involves data analysis, interpretation, and formation of principles.

With the help of Research Methodology Question Papers, you will also need to figure out if the research problem is true or false and whether it’s valid. The process includes steps and criteria for valid research. You will need to know how to gather data, evaluate results, and evaluate findings correctly.

Assignment Questions

• Define and explain the term “Research”. Explain its importance and objectives.
• Define the main issues which should receive the attention of the researcher in formulating the research problem. Give suitable examples.
• Explain in brief the stages in the data processing.
• Briefly explain the significance of data processing. What is the problem associated with data processing?

Firstly, you need to determine whether you’re conducting an ex post facto study. This type of research design is used to investigate the effects of a situation after it happens.

In this case, the researcher can use data not relevant to the problem understudies, such as cash register receipts or customer grievance data. Alternatively, you can use a sample from a national survey.

This type of research is very useful for various situations, such as market research, because it allows you to compare different factors, such as cost, quality, and consistency.

Research Methodology Question Paper in English

Research Methodology Question Paper in Hindi

Ultimately, it would be best if you defined your research question. You need to know your objectives and goals. Once you’ve defined your research question, you need to define your variables. T

hese will result in your final result. By defining your variables and identifying the variables in a study, you will know what to focus on for the final analysis. It’s a must-have tool for any researcher who needs to conduct research. You may take the help of Research Methodology Question Papers.

Experimentation is another method you’ll need to know. In this method, you can establish a causal relationship among variables. The method is also known as experimental research. You can also perform qualitative research, which is mostly exploratory and relies on subjective data. By collecting data, you can develop testable hypotheses and make predictions about the outcome. These results then prove the hypotheses. This method can be used for any research, whether you’re experimenting with science or social work.

A problem is usually defined. Once you’ve defined the problem and chosen the methodology, you can begin collecting data. Research Methodology Question Paper will help you develop the problem, gather data, and test hypotheses. You’ll also need to write a study plan. After that, you’ll need to write the hypothesis, which outlines the methodology you’ll use for your research. If you’re unsure, refer to a research methodology question bank with answers pdf.

PhD Entrance Exam Question Paper For Research Methodology

The PhD entrance exam for research methodology focuses on the principles of a research project. It includes questions on the following: the characteristics of a good researcher, the types of research, case study research, and statistical applications. It also covers the types of research reports, their structure, and their reasoning. The exam will also cover current affairs. Here are some tips to help you ace the research methodology section of the entrance exam with the help of Research Methodology Question Papers.

Answer any TWO of the following

(2 x20 = 40 Marks)

• Write an essay on the preparation of  a manuscript for publications
• Describe various methods of collection of insects in the field.
• Describe any two types of Taxonomic keys with examples.
• Give an account of the types of preservation of insects.

Attempt Research Methodology Question Paper:  Attempt a PHD entrance exam in research methodology to familiarize yourself with the topics. By doing this, you will have an idea of the types of questions you will be asked during the examination.

This is also a good time to get advice from senior students. Ask them for help and clarification on any question that they feel is tricky. In this way, you can maximize your chances of acing the PhD entrance exam.

PhD Research Methodology Old Question Papers

Are you planning to write your PhD in Research Methodology? Are you wondering how to prepare for the PhD research methodology question paper? You can learn how to prepare from the below tips.

The research methodology syllabus consists of several topics that cover the qualities of a good researcher. Other topics in this section are the various types of research and their merits. The topics also cover logic and reasoning, the structure of a research report, and its components.

Class Test Questions

• What are ethical issues concerning the research participants of the research activity? Explain.
• What are the characteristics of research? Explain how quantitative researches differ from qualitative research.
• Explain the significance of primary data. What are the limitations of primary data? Explain in brief the stages in the data processing.
• Discuss the various methods of research? Explain the various factors to be considered in making the decision on the sampling method.

1. PhD Research Methodology Old Question Paper 2020

In the PhD research methodology question paper 2020, students must explain the main reasons behind their chosen data collection and analysis method.

They must also highlight the practical limitations of the method used. This information should be limited to the reasons for the choice of the methodology, which should not include irrelevant details.

For instance, basic procedures should only be explained when the research subject is unfamiliar with them. This will help them score a high mark in the research methodology section.

2. PhD Research Methodology Old Question Paper 2019

A key feature of a PHD research methodology question paper is that it should be based on an academic question that is of interest to researchers and practitioners in the subject. It should be derived from the literature, current situation, or practice of the subject.

Research Methodology Question Papersuld has a clearly stated purpose and can be anything from filling a knowledge gap to analyzing academic assumptions and monitoring developments in practice. It can also be about testing theories within a certain population. The question should also be appropriate for the discipline, institution, and community.

SECTION – A  

(10 x 2 = 20 Marks)

Answer ALL questions

• What are the advantages of Websites in literature collections?
• What is Curating?
• Distinguish conceptual from empirical research.
• What are Tautonyms?
• Define international code of Zoological Nomenclature.
• Define the term – Null hypothesis.
• Distinguish bar diagram from a histogram.
• What is a holotype?
• Mention any two objectives of the Research.
• Differentiate acclimation from acclimatisation

UGC NET Research Methodology Old Question Paper

In the recent UGC NET exam, questions were asked about the research methods. Research Methodology Question Papers covered critical features of research methods, the definition of the research question, hypotheses, data collection, analysis, and reporting procedures.

Students should familiarize themselves with these topics and the questions that will be asked during the examination. If you are preparing for the UGC NET exam, you should prepare by referring to UGC NET mock tests.

In addition to studying the UGC NET syllabus, it is also important to learn the different referencing styles. The UGC NET Research methodology question paper includes study notes on the various accepted referencing styles. The study notes also feature a list of helpful ICT tools to aid you in your preparation. These include tools such as Mendeley and IBM SPSS. Also, the notes include information about the application of ICT to research.

SECTION – B

(4X 10  = 40 Marks)

Answer any four of the following

• Explain briefly the types of sampling.
• What is Berlese Funnel?  Explain its use in the collection of arthropods.
• Write an account of types in insect taxonomy.
• Summarize statistical techniques used in experiments.
• Explain the impact of physicochemical parameters in stress experiments.
• Bring out the significance of current contents and review volumes in research.

MBA Research Methodology Old Question Paper

The MBA research methodology question paper is for the first year of the course and has a simple pattern. Each question carries the same number of marks and includes long answer types. The questions in the question paper cover different aspects of the research process, such as the types of experimental designs, the scales used for measuring variables, and the different factor analyses.

The research methodology question paper also includes the format and layout of a research report. To ensure a good score, candidates are advised to read previous year’s papers to familiarize themselves with the format and content of the exam.

Students should understand the importance of selecting the appropriate statistical technique for the analysis. They must be able to distinguish between non-parametric statistical techniques and parametric statistics. They must also be able to define the research report, thesis, and the significance of the research work.

The question paper also requires the students to understand the format and pattern of the Business Research methodology question paper. The syllabus of the course is available for download in pdf format.

Research Methodology Old Question Paper MCQs

Students preparing for MBA exams should familiarize themselves with Research methodology MCQ questions. Research Methodology Question Papers may be asked in the regular or distance mode. They can be asked about academic, competitive, and managerial research topics. Identifying the problem is the first step in designing a research study.

This process will help you develop a hypothesis and plan of investigation. Then you will have to reduce the problem to a more manageable one.

Moreover, you should understand that Secondary data does not necessarily have to be relevant to the problem under study. For example, you can use cash register receipts or customer grievances data.

Secondary data may also be obtained from national surveys, such as the Statistical Abstracts of India. The purpose of secondary data is to inform decision-makers and make them aware of the importance of their findings. The next step is to analyze the data.

• अनुसंधान पद्धति प्रश्न पत्र 2014
• अनुसंधान पद्धति प्रश्न पत्र 2015
• अनुसंधान पद्धति प्रश्न पत्र 2016
• अनुसंधान पद्धति प्रश्न पत्र 2017
• अनुसंधान पद्धति प्रश्न पत्र 2018
• अनुसं धा न पद्धति प्रश्न पत्र 2019
• अनुसंधान पद्धति प्रश्न पत्र 2020

How to Answer Research Methodology MCQs

This article will teach you how to answer the research methodology MCQs on the UGC NET and MBA research methodology question papers. Hopefully, this will make the task a little easier. But, first, you should be familiar with how the research methodology question paper is formatted.

Below are some questions to keep in mind. When possible, make sure to use an example question paper to illustrate the different methods. If possible, you should also record your research methods. This will ensure that you capture all the small details.

1. “One of the methods of logical reasoning process” is called

   a) Induction   b) Deduction   c) Research d) Experiment

2. “A systematic step-by-step Procedure following the logical process of reasoning” called

   a) Experiment b) Observation c) Deduction d) Scientific method

3. An essential Criterion of Scientific study is called

   a) Belief   b) Value    c) Objectivity d) Subjective

4. “Reasoning from general to particular “is called

   a) Induction   b) deduction c) Observation d) experience

5“Deduction and induction are a part of the system of reasoning” – stated by

a) Caroline   b) P.V.Young  c) Dewey John d) Emory

6 ……….. is a “systematically conceptual structure of interrelated elements in some schematic form”

a) Concept   b) Variable c) Model   d) Facts

7 The method by which a sample is chosen

a) Unit   b) design   c) Random   d) Census

8 Research conducted to find a solution for an immediate problem is ………….

a) Fundamental Research              b) Analytical Research

c) Survey.                                          d) Action Research

9 Fundamental Research is otherwise called

a) Action Research   b) Survey   c) Pilot study   d) Pure Research

10 A research which follows the case study method is called

a) Clinical or diagnostic    b) Causal

c) Analytical                        d) Qualitative

11 Research conducted in a classroom atmosphere is called

a) Field study                        b) Survey

c) Laboratory Research      d) Empirical Research

12. Research through experiment and observation is called

   a) Clinical Research             b) Experimental Research

   c) Laboratory Research       d) Empirical Research

13 ………….. is a way to systematically solve the research problem

About The Author

Question paper guru.

A step-by-step guide to causal study design using real-world data

• Open access
• Published: 19 June 2024

Cite this article

You have full access to this open access article

• Sarah Ruth Hoffman 1 ,
• Nilesh Gangan 1 ,
• Xiaoxue Chen 2 ,
• Joseph L. Smith 1 ,
• Arlene Tave 1 ,
• Yiling Yang 1 ,
• Christopher L. Crowe 1 ,
• Susan dosReis 3 &
• Michael Grabner 1  

166 Accesses

Explore all metrics

Due to the need for generalizable and rapidly delivered evidence to inform healthcare decision-making, real-world data have grown increasingly important to answer causal questions. However, causal inference using observational data poses numerous challenges, and relevant methodological literature is vast. We endeavored to identify underlying unifying themes of causal inference using real-world healthcare data and connect them into a single schema to aid in observational study design, and to demonstrate this schema using a previously published research example. A multidisciplinary team (epidemiology, biostatistics, health economics) reviewed the literature related to causal inference and observational data to identify key concepts. A visual guide to causal study design was developed to concisely and clearly illustrate how the concepts are conceptually related to one another. A case study was selected to demonstrate an application of the guide. An eight-step guide to causal study design was created, integrating essential concepts from the literature, anchored into conceptual groupings according to natural steps in the study design process. The steps include defining the causal research question and the estimand; creating a directed acyclic graph; identifying biases and design and analytic techniques to mitigate their effect, and techniques to examine the robustness of findings. The cardiovascular case study demonstrates the applicability of the steps to developing a research plan. This paper used an existing study to demonstrate the relevance of the guide. We encourage researchers to incorporate this guide at the study design stage in order to elevate the quality of future real-world evidence.

Similar content being viewed by others

Examples of Applying Causal-Inference Roadmap to Real-World Studies

Selection Mechanisms and Their Consequences: Understanding and Addressing Selection Bias

Assessing causality in epidemiology: revisiting Bradford Hill to incorporate developments in causal thinking

Avoid common mistakes on your manuscript.

1 Introduction

Approximately 50 new drugs are approved each year in the United States (Mullard 2022 ). For all new drugs, randomized controlled trials (RCTs) are the gold-standard by which potential effectiveness (“efficacy”) and safety are established. However, RCTs cannot guarantee how a drug will perform in a less controlled context. For this reason, regulators frequently require observational, post-approval studies using “real-world” data, sometimes even as a condition of drug approval. The “real-world” data requested by regulators is often derived from insurance claims databases and/or healthcare records. Importantly, these data are recorded during routine clinical care without concern for potential use in research. Yet, in recent years, there has been increasing use of such data for causal inference and regulatory decision making, presenting a variety of methodologic challenges for researchers and stakeholders to consider (Arlett et al. 2022 ; Berger et al. 2017 ; Concato and ElZarrad 2022 ; Cox et al. 2009 ; European Medicines Agency 2023 ; Franklin and Schneeweiss 2017 ; Girman et al. 2014 ; Hernán and Robins 2016 ; International Society for Pharmacoeconomics and Outcomes Research (ISPOR) 2022 ; International Society for Pharmacoepidemiology (ISPE) 2020 ; Stuart et al. 2013 ; U.S. Food and Drug Administration 2018 ; Velentgas et al. 2013 ).

Current guidance for causal inference using observational healthcare data articulates the need for careful study design (Berger et al. 2017 ; Cox et al. 2009 ; European Medicines Agency 2023 ; Girman et al. 2014 ; Hernán and Robins 2016 ; Stuart et al. 2013 ; Velentgas et al. 2013 ). In 2009, Cox et al. described common sources of bias in observational data and recommended specific strategies to mitigate these biases (Cox et al. 2009 ). In 2013, Stuart et al. emphasized counterfactual theory and trial emulation, offered several approaches to address unmeasured confounding, and provided guidance on the use of propensity scores to balance confounding covariates (Stuart et al. 2013 ). In 2013, the Agency for Healthcare Research and Quality (AHRQ) released an extensive, 200-page guide to developing a protocol for comparative effectiveness research using observational data (Velentgas et al. 2013 ). The guide emphasized development of the research question, with additional chapters on study design, comparator selection, sensitivity analyses, and directed acyclic graphs (Velentgas et al. 2013 ). In 2014, Girman et al. provided a clear set of steps for assessing study feasibility including examination of the appropriateness of the data for the research question (i.e., ‘fit-for-purpose’), empirical equipoise, and interpretability, stating that comparative effectiveness research using observational data “should be designed with the goal of drawing a causal inference” (Girman et al. 2014 ). In 2017 , Berger et al. described aspects of “study hygiene,” focusing on procedural practices to enhance confidence in, and credibility of, real-world data studies (Berger et al. 2017 ). Currently, the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) maintains a guide on methodological standards in pharmacoepidemiology which discusses causal inference using observational data and includes an overview of study designs, a chapter on methods to address bias and confounding, and guidance on writing statistical analysis plans (European Medicines Agency 2023 ). In addition to these resources, the “target trial framework” provides a structured approach to planning studies for causal inferences from observational databases (Hernán and Robins 2016 ; Wang et al. 2023b ). This framework, published in 2016, encourages researchers to first imagine a clinical trial for the study question of interest and then to subsequently design the observational study to reflect the hypothetical trial (Hernán and Robins 2016 ).

While the literature addresses critical issues collectively, there remains a need for a framework that puts key components, including the target trial approach, into a simple, overarching schema (Loveless 2022 ) so they can be more easily remembered, and communicated to all stakeholders including (new) researchers, peer-reviewers, and other users of the research findings (e.g., practicing providers, professional clinical societies, regulators). For this reason, we created a step-by-step guide for causal inference using administrative health data, which aims to integrate these various best practices at a high level and complements existing, more specific guidance, including those from the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the International Society for Pharmacoepidemiology (ISPE) (Berger et al. 2017 ; Cox et al. 2009 ; Girman et al. 2014 ). We demonstrate the application of this schema using a previously published paper in cardiovascular research.

This work involved a formative phase and an implementation phase to evaluate the utility of the causal guide. In the formative phase, a multidisciplinary team with research expertise in epidemiology, biostatistics, and health economics reviewed selected literature (peer-reviewed publications, including those mentioned in the introduction, as well as graduate-level textbooks) related to causal inference and observational healthcare data from the pharmacoepidemiologic and pharmacoeconomic perspectives. The potential outcomes framework served as the foundation for our conception of causal inference (Rubin 2005 ). Information was grouped into the following four concepts: (1) Defining the Research Question; (2) Defining the Estimand; (3) Identifying and Mitigating Biases; (4) Sensitivity Analysis. A step-by-step guide to causal study design was developed to distill the essential elements of each concept, organizing them into a single schema so that the concepts are clearly related to one another. References for each step of the schema are included in the Supplemental Table.

In the implementation phase we tested the application of the causal guide to previously published work (Dondo et al. 2017 ). The previously published work utilized data from the Myocardial Ischaemia National Audit Project (MINAP), the United Kingdom’s national heart attack register. The goal of the study was to assess the effect of β-blockers on all-cause mortality among patients hospitalized for acute myocardial infarction without heart failure or left ventricular systolic dysfunction. We selected this paper for the case study because of its clear descriptions of the research goal and methods, and the explicit and methodical consideration of potential biases and use of sensitivity analyses to examine the robustness of the main findings.

3.1 Overview of the eight steps

The step-by-step guide to causal inference comprises eight distinct steps (Fig.  1 ) across the four concepts. As scientific inquiry and study design are iterative processes, the various steps may be completed in a different order than shown, and steps may be revisited.

A step-by-step guide for causal study design

Abbreviations: GEE: generalized estimating equations; IPC/TW: inverse probability of censoring/treatment weighting; ITR: individual treatment response; MSM: marginal structural model; TE: treatment effect

Please refer to the Supplemental Table for references providing more in-depth information.

1 Ensure that the exposure and outcome are well-defined based on literature and expert opinion.

2 More specifically, measures of association are not affected by issues such as confounding and selection bias because they do not intend to isolate and quantify a single causal pathway. However, information bias (e.g., variable misclassification) can negatively affect association estimates, and association estimates remain subject to random variability (and are hence reported with confidence intervals).

3 This list is not exhaustive; it focuses on frequently encountered biases.

4 To assess bias in a nonrandomized study following the target trial framework, use of the ROBINS-I tool is recommended ( https://www.bmj.com/content/355/bmj.i4919 ).

5 Only a selection of the most popular approaches is presented here. Other methods exist; e.g., g-computation and g-estimation for both time-invariant and time-varying analysis; instrumental variables; and doubly-robust estimation methods. There are also program evaluation methods (e.g., difference-in-differences, regression discontinuities) that can be applied to pharmacoepidemiologic questions. Conventional outcome regression analysis is not recommended for causal estimation due to issues determining covariate balance, correct model specification, and interpretability of effect estimates.

6 Online tools include, among others, an E-value calculator for unmeasured confounding ( https://www.evalue-calculator.com /) and the P95 outcome misclassification estimator ( http://apps.p-95.com/ISPE /).

3.2 Defining the Research question (step 1)

The process of designing a study begins with defining the research question. Research questions typically center on whether a causal relationship exists between an exposure and an outcome. This contrasts with associative questions, which, by their nature, do not require causal study design elements because they do not attempt to isolate a causal pathway from a single exposure to an outcome under study. It is important to note that the phrasing of the question itself should clarify whether an association or a causal relationship is of interest. The study question “Does statin use reduce the risk of future cardiovascular events?” is explicitly causal and requires that the study design addresses biases such as confounding. In contrast, the study question “Is statin use associated with a reduced risk of future cardiovascular events?” can be answered without control of confounding since the word “association” implies correlation. Too often, however, researchers use the word “association” to describe their findings when their methods were created to address explicitly causal questions (Hernán 2018 ). For example, a study that uses propensity score-based methods to balance risk factors between treatment groups is explicitly attempting to isolate a causal pathway by removing confounding factors. This is different from a study that intends only to measure an association. In fact, some journals may require that the word “association” be used when causal language would be more appropriate; however, this is beginning to change (Flanagin et al. 2024 ).

3.3 Defining the estimand (steps 2, 3, 4)

The estimand is the causal effect of research interest and is described in terms of required design elements: the target population for the counterfactual contrast, the kind of effect, and the effect/outcome measure.

In Step 2, the study team determines the target population of interest, which depends on the research question of interest. For example, we may want to estimate the effect of the treatment in the entire study population, i.e., the hypothetical contrast between all study patients taking the drug of interest versus all study patients taking the comparator (the average treatment effect; ATE). Other effects can be examined, including the average treatment effect in the treated or untreated (ATT or ATU).When covariate distributions are the same across the treated and untreated populations and there is no effect modification by covariates, these effects are generally the same (Wang et al. 2017 ). In RCTs, this occurs naturally due to randomization, but in non-randomized data, careful study design and statistical methods must be used to mitigate confounding bias.

In Step 3, the study team decides whether to measure the intention-to-treat (ITT), per-protocol, or as-treated effect. The ITT approach is also known as “first-treatment-carried-forward” in the observational literature (Lund et al. 2015 ). In trials, the ITT measures the effect of treatment assignment rather than the treatment itself, and in observational data the ITT can be conceptualized as measuring the effect of treatment as started . To compute the ITT effect from observational data, patients are placed into the exposure group corresponding to the treatment that they initiate, and treatment switching or discontinuation are purposely ignored in the analysis. Alternatively, a per-protocol effect can be measured from observational data by classifying patients according to the treatment that they initiated but censoring them when they stop, switch, or otherwise change treatment (Danaei et al. 2013 ; Yang et al. 2014 ). Finally, “as-treated” effects are estimated from observational data by classifying patients according to their actual treatment exposure during follow-up, for example by using multiple time windows to measure exposure changes (Danaei et al. 2013 ; Yang et al. 2014 ).

Step 4 is the final step in specifying the estimand in which the research team determines the effect measure of interest. Answering this question has two parts. First, the team must consider how the outcome of interest will be measured. Risks, rates, hazards, odds, and costs are common ways of measuring outcomes, but each measure may be best suited to a particular scenario. For example, risks assume patients across comparison groups have equal follow-up time, while rates allow for variable follow-up time (Rothman et al. 2008 ). Costs may be of interest in studies focused on economic outcomes, including as inputs to cost-effectiveness analyses. After deciding how the outcome will be measured, it is necessary to consider whether the resulting quantity will be compared across groups using a ratio or a difference. Ratios convey the effect of exposure in a way that is easy to understand, but they do not provide an estimate of how many patients will be affected. On the other hand, differences provide a clearer estimate of the potential public health impact of exposure; for example, by allowing the calculation of the number of patients that must be treated to cause or prevent one instance of the outcome of interest (Tripepi et al. 2007 ).

3.4 Identifying and mitigating biases (steps 5, 6, 7)

Observational, real-world studies can be subject to multiple potential sources of bias, which can be grouped into confounding, selection, measurement, and time-related biases (Prada-Ramallal et al. 2019 ).

In Step 5, as a practical first approach in developing strategies to address threats to causal inference, researchers should create a visual mapping of factors that may be related to the exposure, outcome, or both (also called a directed acyclic graph or DAG) (Pearl 1995 ). While creating a high-quality DAG can be challenging, guidance is increasingly available to facilitate the process (Ferguson et al. 2020 ; Gatto et al. 2022 ; Hernán and Robins 2020 ; Rodrigues et al. 2022 ; Sauer 2013 ). The types of inter-variable relationships depicted by DAGs include confounders, colliders, and mediators. Confounders are variables that affect both exposure and outcome, and it is necessary to control for them in order to isolate the causal pathway of interest. Colliders represent variables affected by two other variables, such as exposure and outcome (Griffith et al. 2020 ). Colliders should not be conditioned on since by doing so, the association between exposure and outcome will become distorted. Mediators are variables that are affected by the exposure and go on to affect the outcome. As such, mediators are on the causal pathway between exposure and outcome and should also not be conditioned on, otherwise a path between exposure and outcome will be closed and the total effect of the exposure on the outcome cannot be estimated. Mediation analysis is a separate type of analysis aiming to distinguish between direct and indirect (mediated) effects between exposure and outcome and may be applied in certain cases (Richiardi et al. 2013 ). Overall, the process of creating a DAG can create valuable insights about the nature of the hypothesized underlying data generating process and the biases that are likely to be encountered (Digitale et al. 2022 ). Finally, an extension to DAGs which incorporates counterfactual theory is available in the form of Single World Intervention Graphs (SWIGs) as described in a 2013 primer (Richardson and Robins 2013 ).

In Step 6, researchers comprehensively assess the possibility of different types of bias in their study, above and beyond what the creation of the DAG reveals. Many potential biases have been identified and summarized in the literature (Berger et al. 2017 ; Cox et al. 2009 ; European Medicines Agency 2023 ; Girman et al. 2014 ; Stuart et al. 2013 ; Velentgas et al. 2013 ). Every study can be subject to one or more biases, each of which can be addressed using one or more methods. The study team should thoroughly and explicitly identify all possible biases with consideration for the specifics of the available data and the nuances of the population and health care system(s) from which the data arise. Once the potential biases are identified and listed, the team can consider potential solutions using a variety of study design and analytic techniques.

In Step 7, the study team considers solutions to the biases identified in Step 6. “Target trial” thinking serves as the basis for many of these solutions by requiring researchers to consider how observational studies can be designed to ensure comparison groups are similar and produce valid inferences by emulating RCTs (Labrecque and Swanson 2017 ; Wang et al. 2023b ). Designing studies to include only new users of a drug and an active comparator group is one way of increasing the similarity of patients across both groups, particularly in terms of treatment history. Careful consideration must be paid to the specification of the time periods and their relationship to inclusion/exclusion criteria (Suissa and Dell’Aniello 2020 ). For instance, if a drug is used intermittently, a longer wash-out period is needed to ensure adequate capture of prior use in order to avoid bias (Riis et al. 2015 ). The study team should consider how to approach confounding adjustment, and whether both time-invariant and time-varying confounding may be present. Many potential biases exist, and many methods have been developed to address them in order to improve causal estimation from observational data. Many of these methods, such as propensity score estimation, can be enhanced by machine learning (Athey and Imbens 2019 ; Belthangady et al. 2021 ; Mai et al. 2022 ; Onasanya et al. 2024 ; Schuler and Rose 2017 ; Westreich et al. 2010 ). Machine learning has many potential applications in the causal inference discipline, and like other tools, must be used with careful planning and intentionality. To aid in the assessment of potential biases, especially time-related ones, and the development of a plan to address them, the study design should be visualized (Gatto et al. 2022 ; Schneeweiss et al. 2019 ). Additionally, we note the opportunity for collaboration across research disciplines (e.g., the application of difference-in-difference methods (Zhou et al. 2016 ) to the estimation of comparative drug effectiveness and safety).

3.5 Quality Control & sensitivity analyses (step 8)

Causal study design concludes with Step 8, which includes planning quality control and sensitivity analyses to improve the internal validity of the study. Quality control begins with reviewing study output for prima facie validity. Patient characteristics (e.g., distributions of age, sex, region) should align with expected values from the researchers’ intuition and the literature, and researchers should assess reasons for any discrepancies. Sensitivity analyses should be conducted to determine the robustness of study findings. Researchers can test the stability of study estimates using a different estimand or type of model than was used in the primary analysis. Sensitivity analysis estimates that are similar to those of the primary analysis might confirm that the primary analysis estimates are appropriate. The research team may be interested in how changes to study inclusion/exclusion criteria may affect study findings or wish to address uncertainties related to measuring the exposure or outcome in the administrative data by modifying the algorithms used to identify exposure or outcome (e.g., requiring hospitalization with a diagnosis code in a principal position rather than counting any claim with the diagnosis code in any position). As feasible, existing validation studies for the exposure and outcome should be referenced, or new validation efforts undertaken. The results of such validation studies can inform study estimates via quantitative bias analyses (Lanes and Beachler 2023 ). The study team may also consider biases arising from unmeasured confounding and plan quantitative bias analyses to explore how unmeasured confounding may impact estimates. Quantitative bias analysis can assess the directionality, magnitude, and uncertainty of errors arising from a variety of limitations (Brenner and Gefeller 1993 ; Lash et al. 2009 , 2014 ; Leahy et al. 2022 ).

3.6 Illustration using a previously published research study

In order to demonstrate how the guide can be used to plan a research study utilizing causal methods, we turn to a previously published study (Dondo et al. 2017 ) that assessed the causal relationship between the use of 𝛽-blockers and mortality after acute myocardial infarction in patients without heart failure or left ventricular systolic dysfunction. The investigators sought to answer a causal research question (Step 1), and so we proceed to Step 2. Use (or no use) of 𝛽-blockers was determined after discharge without taking into consideration discontinuation or future treatment changes (i.e., intention-to-treat). Considering treatment for whom (Step 3), both ATE and ATT were evaluated. Since survival was the primary outcome, an absolute difference in survival time was chosen as the effect measure (Step 4). While there was no explicit directed acyclic graph provided, the investigators specified a list of confounders.

Robust methodologies were established by consideration of possible sources of biases and addressing them using viable solutions (Steps 6 and 7). Table  1 offers a list of the identified potential biases and their corresponding solutions as implemented. For example, to minimize potential biases including prevalent-user bias and selection bias, the sample was restricted to patients with no previous use of 𝛽-blockers, no contraindication for 𝛽-blockers, and no prescription of loop diuretics. To improve balance across the comparator groups in terms of baseline confounders, i.e., those that could influence both exposure (𝛽-blocker use) and outcome (mortality), propensity score-based inverse probability of treatment weighting (IPTW) was employed. However, we noted that the baseline look-back period to assess measured covariates was not explicitly listed in the paper.

Quality control and sensitivity analysis (Step 8) is described extensively. The overlap of propensity score distributions between comparator groups was tested and confounder balance was assessed. Since observations in the tail-end of the propensity score distribution may violate the positivity assumption (Crump et al. 2009 ), a sensitivity analysis was conducted including only cases within 0.1 to 0.9 of the propensity score distribution. While not mentioned by the authors, the PS tails can be influenced by unmeasured confounders (Sturmer et al. 2021 ), and the findings were robust with and without trimming. An assessment of extreme IPTW weights, while not included, would further help increase confidence in the robustness of the analysis. An instrumental variable approach was employed to assess potential selection bias due to unmeasured confounding, using hospital rates of guideline-indicated prescribing as the instrument. Additionally, potential bias caused by missing data was attenuated through the use of multiple imputation, and separate models were built for complete cases only and imputed/complete cases.

4 Discussion

We have described a conceptual schema for designing observational real-world studies to estimate causal effects. The application of this schema to a previously published study illuminates the methodologic structure of the study, revealing how each structural element is related to a potential bias which it is meant to address. Real-world evidence is increasingly accepted by healthcare stakeholders, including the FDA (Concato and Corrigan-Curay 2022 ; Concato and ElZarrad 2022 ), and its use for comparative effectiveness and safety assessments requires appropriate causal study design; our guide is meant to facilitate this design process and complement existing, more specific, guidance.

Existing guidance for causal inference using observational data includes components that can be clearly mapped onto the schema that we have developed. For example, in 2009 Cox et al. described common sources of bias in observational data and recommended specific strategies to mitigate these biases, corresponding to steps 6–8 of our step-by-step guide (Cox et al. 2009 ). In 2013, the AHRQ emphasized development of the research question, corresponding to steps 1–4 of our guide, with additional chapters on study design, comparator selection, sensitivity analyses, and directed acyclic graphs which correspond to steps 7 and 5, respectively (Velentgas et al. 2013 ). Much of Girman et al.’s manuscript (Girman et al. 2014 ) corresponds with steps 1–4 of our guide, and the matter of equipoise and interpretability specifically correspond to steps 3 and 7–8. The current ENCePP guide on methodological standards in pharmacoepidemiology contains a section on formulating a meaningful research question, corresponding to step 1, and describes strategies to mitigate specific sources of bias, corresponding to steps 6–8 (European Medicines Agency 2023 ). Recent works by the FDA Sentinel Innovation Center (Desai et al. 2024 ) and the Joint Initiative for Causal Inference (Dang et al. 2023 ) provide more advanced exposition of many of the steps in our guide. The target trial framework contains guidance on developing seven components of the study protocol, including eligibility criteria, treatment strategies, assignment procedures, follow-up period, outcome, causal contrast of interest, and analysis plan (Hernán and Robins 2016 ). Our work places the target trial framework into a larger context illustrating its relationship with other important study planning considerations, including the creation of a directed acyclic graph and incorporation of prespecified sensitivity and quantitative bias analyses.

Ultimately, the feasibility of estimating causal effects relies on the capabilities of the available data. Real-world data sources are complex, and the investigator must carefully consider whether the data on hand are sufficient to answer the research question. For example, a study that relies solely on claims data for outcome ascertainment may suffer from outcome misclassification bias (Lanes and Beachler 2023 ). This bias can be addressed through medical record validation for a random subset of patients, followed by quantitative bias analysis (Lanes and Beachler 2023 ). If instead, the investigator wishes to apply a previously published, claims-based algorithm validated in a different database, they must carefully consider the transportability of that algorithm to their own study population. In this way, causal inference from real-world data requires the ability to think creatively and resourcefully about how various data sources and elements can be leveraged, with consideration for the strengths and limitations of each source. The heart of causal inference is in the pairing of humility and creativity: the humility to acknowledge what the data cannot do, and the creativity to address those limitations as best as one can at the time.

4.1 Limitations

As with any attempt to synthesize a broad array of information into a single, simplified schema, there are several limitations to our work. Space and useability constraints necessitated simplification of the complex source material and selections among many available methodologies, and information about the relative importance of each step is not currently included. Additionally, it is important to consider the context of our work. This step-by-step guide emphasizes analytic techniques (e.g., propensity scores) that are used most frequently within our own research environment and may not include less familiar study designs and analytic techniques. However, one strength of the guide is that additional designs and techniques or concepts can easily be incorporated into the existing schema. The benefit of a schema is that new information can be added and is more readily accessed due to its association with previously sorted information (Loveless 2022 ). It is also important to note that causal inference was approached as a broad overarching concept defined by the totality of the research, from start to finish, rather than focusing on a particular analytic technique, however we view this as a strength rather than a limitation.

Finally, the focus of this guide was on the methodologic aspects of study planning. As a result, we did not include steps for drafting or registering the study protocol in a public database or for communicating results. We strongly encourage researchers to register their study protocols and communicate their findings with transparency. A protocol template endorsed by ISPOR and ISPE for studies using real-world data to evaluate treatment effects is available (Wang et al. 2023a ). Additionally, the steps described above are intended to illustrate an order of thinking in the study planning process, and these steps are often iterative. The guide is not intended to reflect the order of study execution; specifically, quality control procedures and sensitivity analyses should also be formulated up-front at the protocol stage.

5 Conclusion

We outlined steps and described key conceptual issues of importance in designing real-world studies to answer causal questions, and created a visually appealing, user-friendly resource to help researchers clearly define and navigate these issues. We hope this guide serves to enhance the quality, and thus the impact, of real-world evidence.

Data availability

No datasets were generated or analysed during the current study.

Arlett, P., Kjaer, J., Broich, K., Cooke, E.: Real-world evidence in EU Medicines Regulation: Enabling Use and establishing value. Clin. Pharmacol. Ther. 111 (1), 21–23 (2022)

Article   PubMed   Google Scholar  

Athey, S., Imbens, G.W.: Machine Learning Methods That Economists Should Know About. Annual Review of Economics 11(Volume 11, 2019): 685–725. (2019)

Belthangady, C., Stedden, W., Norgeot, B.: Minimizing bias in massive multi-arm observational studies with BCAUS: Balancing covariates automatically using supervision. BMC Med. Res. Methodol. 21 (1), 190 (2021)

Article   PubMed   PubMed Central   Google Scholar  

Berger, M.L., Sox, H., Willke, R.J., Brixner, D.L., Eichler, H.G., Goettsch, W., Madigan, D., Makady, A., Schneeweiss, S., Tarricone, R., Wang, S.V., Watkins, J.: and C. Daniel Mullins. 2017. Good practices for real-world data studies of treatment and/or comparative effectiveness: Recommendations from the joint ISPOR-ISPE Special Task Force on real-world evidence in health care decision making. Pharmacoepidemiol Drug Saf. 26 (9): 1033–1039

Brenner, H., Gefeller, O.: Use of the positive predictive value to correct for disease misclassification in epidemiologic studies. Am. J. Epidemiol. 138 (11), 1007–1015 (1993)

Article   CAS   PubMed   Google Scholar  

Concato, J., Corrigan-Curay, J.: Real-world evidence - where are we now? N Engl. J. Med. 386 (18), 1680–1682 (2022)

Concato, J., ElZarrad, M.: FDA Issues Draft Guidances on Real-World Evidence, Prepares to Publish More in Future [accessed on 2022]. (2022). https://www.fda.gov/drugs/news-events-human-drugs/fda-issues-draft-guidances-real-world-evidence-prepares-publish-more-future

Cox, E., Martin, B.C., Van Staa, T., Garbe, E., Siebert, U., Johnson, M.L.: Good research practices for comparative effectiveness research: Approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: The International Society for Pharmacoeconomics and Outcomes Research Good Research Practices for Retrospective Database Analysis Task Force Report–Part II. Value Health. 12 (8), 1053–1061 (2009)

Crump, R.K., Hotz, V.J., Imbens, G.W., Mitnik, O.A.: Dealing with limited overlap in estimation of average treatment effects. Biometrika. 96 (1), 187–199 (2009)

Article   Google Scholar  

Danaei, G., Rodriguez, L.A., Cantero, O.F., Logan, R., Hernan, M.A.: Observational data for comparative effectiveness research: An emulation of randomised trials of statins and primary prevention of coronary heart disease. Stat. Methods Med. Res. 22 (1), 70–96 (2013)

Dang, L.E., Gruber, S., Lee, H., Dahabreh, I.J., Stuart, E.A., Williamson, B.D., Wyss, R., Diaz, I., Ghosh, D., Kiciman, E., Alemayehu, D., Hoffman, K.L., Vossen, C.Y., Huml, R.A., Ravn, H., Kvist, K., Pratley, R., Shih, M.C., Pennello, G., Martin, D., Waddy, S.P., Barr, C.E., Akacha, M., Buse, J.B., van der Laan, M., Petersen, M.: A causal roadmap for generating high-quality real-world evidence. J. Clin. Transl Sci. 7 (1), e212 (2023)

Desai, R.J., Wang, S.V., Sreedhara, S.K., Zabotka, L., Khosrow-Khavar, F., Nelson, J.C., Shi, X., Toh, S., Wyss, R., Patorno, E., Dutcher, S., Li, J., Lee, H., Ball, R., Dal Pan, G., Segal, J.B., Suissa, S., Rothman, K.J., Greenland, S., Hernan, M.A., Heagerty, P.J., Schneeweiss, S.: Process guide for inferential studies using healthcare data from routine clinical practice to evaluate causal effects of drugs (PRINCIPLED): Considerations from the FDA Sentinel Innovation Center. BMJ. 384 , e076460 (2024)

Digitale, J.C., Martin, J.N., Glymour, M.M.: Tutorial on directed acyclic graphs. J. Clin. Epidemiol. 142 , 264–267 (2022)

Dondo, T.B., Hall, M., West, R.M., Jernberg, T., Lindahl, B., Bueno, H., Danchin, N., Deanfield, J.E., Hemingway, H., Fox, K.A.A., Timmis, A.D., Gale, C.P.: beta-blockers and Mortality after Acute myocardial infarction in patients without heart failure or ventricular dysfunction. J. Am. Coll. Cardiol. 69 (22), 2710–2720 (2017)

Article   CAS   PubMed   PubMed Central   Google Scholar  

European Medicines Agency: ENCePP Guide on Methodological Standards in Pharmacoepidemiology [accessed on 2023]. (2023). https://www.encepp.eu/standards_and_guidances/methodologicalGuide.shtml

Ferguson, K.D., McCann, M., Katikireddi, S.V., Thomson, H., Green, M.J., Smith, D.J., Lewsey, J.D.: Evidence synthesis for constructing directed acyclic graphs (ESC-DAGs): A novel and systematic method for building directed acyclic graphs. Int. J. Epidemiol. 49 (1), 322–329 (2020)

Flanagin, A., Lewis, R.J., Muth, C.C., Curfman, G.: What does the proposed causal inference Framework for Observational studies Mean for JAMA and the JAMA Network Journals? JAMA (2024)

U.S. Food and Drug Administration: Framework for FDA’s Real-World Evidence Program [accessed on 2018]. (2018). https://www.fda.gov/media/120060/download

Franklin, J.M., Schneeweiss, S.: When and how can Real World Data analyses substitute for randomized controlled trials? Clin. Pharmacol. Ther. 102 (6), 924–933 (2017)

Gatto, N.M., Wang, S.V., Murk, W., Mattox, P., Brookhart, M.A., Bate, A., Schneeweiss, S., Rassen, J.A.: Visualizations throughout pharmacoepidemiology study planning, implementation, and reporting. Pharmacoepidemiol Drug Saf. 31 (11), 1140–1152 (2022)

Girman, C.J., Faries, D., Ryan, P., Rotelli, M., Belger, M., Binkowitz, B., O’Neill, R.: and C. E. R. S. W. G. Drug Information Association. 2014. Pre-study feasibility and identifying sensitivity analyses for protocol pre-specification in comparative effectiveness research. J. Comp. Eff. Res. 3 (3): 259–270

Griffith, G.J., Morris, T.T., Tudball, M.J., Herbert, A., Mancano, G., Pike, L., Sharp, G.C., Sterne, J., Palmer, T.M., Davey Smith, G., Tilling, K., Zuccolo, L., Davies, N.M., Hemani, G.: Collider bias undermines our understanding of COVID-19 disease risk and severity. Nat. Commun. 11 (1), 5749 (2020)

Hernán, M.A.: The C-Word: Scientific euphemisms do not improve causal inference from Observational Data. Am. J. Public Health. 108 (5), 616–619 (2018)

Hernán, M.A., Robins, J.M.: Using Big Data to emulate a target Trial when a Randomized Trial is not available. Am. J. Epidemiol. 183 (8), 758–764 (2016)

Hernán, M., Robins, J.: Causal Inference: What if. Chapman & Hall/CRC, Boca Raton (2020)

Google Scholar  

International Society for Pharmacoeconomics and Outcomes Research (ISPOR): Strategic Initiatives: Real-World Evidence [accessed on 2022]. (2022). https://www.ispor.org/strategic-initiatives/real-world-evidence

International Society for Pharmacoepidemiology (ISPE): Position on Real-World Evidence [accessed on 2020]. (2020). https://pharmacoepi.org/pub/?id=136DECF1-C559-BA4F-92C4-CF6E3ED16BB6

Labrecque, J.A., Swanson, S.A.: Target trial emulation: Teaching epidemiology and beyond. Eur. J. Epidemiol. 32 (6), 473–475 (2017)

Lanes, S., Beachler, D.C.: Validation to correct for outcome misclassification bias. Pharmacoepidemiol Drug Saf. (2023)

Lash, T.L., Fox, M.P., Fink, A.K.: Applying Quantitative bias Analysis to Epidemiologic data. Springer (2009)

Lash, T.L., Fox, M.P., MacLehose, R.F., Maldonado, G., McCandless, L.C., Greenland, S.: Good practices for quantitative bias analysis. Int. J. Epidemiol. 43 (6), 1969–1985 (2014)

Leahy, T.P., Kent, S., Sammon, C., Groenwold, R.H., Grieve, R., Ramagopalan, S., Gomes, M.: Unmeasured confounding in nonrandomized studies: Quantitative bias analysis in health technology assessment. J. Comp. Eff. Res. 11 (12), 851–859 (2022)

Loveless, B.: A Complete Guide to Schema Theory and its Role in Education [accessed on 2022]. (2022). https://www.educationcorner.com/schema-theory/

Lund, J.L., Richardson, D.B., Sturmer, T.: The active comparator, new user study design in pharmacoepidemiology: Historical foundations and contemporary application. Curr. Epidemiol. Rep. 2 (4), 221–228 (2015)

Mai, X., Teng, C., Gao, Y., Governor, S., He, X., Kalloo, G., Hoffman, S., Mbiydzenyuy, D., Beachler, D.: A pragmatic comparison of logistic regression versus machine learning methods for propensity score estimation. Supplement: Abstracts of the 38th International Conference on Pharmacoepidemiology: Advancing Pharmacoepidemiology and Real-World Evidence for the Global Community, August 26–28, 2022, Copenhagen, Denmark. Pharmacoepidemiology and Drug Safety 31(S2). (2022)

Mullard, A.: 2021 FDA approvals. Nat. Rev. Drug Discov. 21 (2), 83–88 (2022)

Onasanya, O., Hoffman, S., Harris, K., Dixon, R., Grabner, M.: Current applications of machine learning for causal inference in healthcare research using observational data. International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Atlanta, GA. (2024)

Pearl, J.: Causal diagrams for empirical research. Biometrika. 82 (4), 669–688 (1995)

Prada-Ramallal, G., Takkouche, B., Figueiras, A.: Bias in pharmacoepidemiologic studies using secondary health care databases: A scoping review. BMC Med. Res. Methodol. 19 (1), 53 (2019)

Richardson, T.S., Robins, J.M.: Single World Intervention Graphs: A Primer [accessed on 2013]. (2013). https://www.stats.ox.ac.uk/~evans/uai13/Richardson.pdf

Richiardi, L., Bellocco, R., Zugna, D.: Mediation analysis in epidemiology: Methods, interpretation and bias. Int. J. Epidemiol. 42 (5), 1511–1519 (2013)

Riis, A.H., Johansen, M.B., Jacobsen, J.B., Brookhart, M.A., Sturmer, T., Stovring, H.: Short look-back periods in pharmacoepidemiologic studies of new users of antibiotics and asthma medications introduce severe misclassification. Pharmacoepidemiol Drug Saf. 24 (5), 478–485 (2015)

Rodrigues, D., Kreif, N., Lawrence-Jones, A., Barahona, M., Mayer, E.: Reflection on modern methods: Constructing directed acyclic graphs (DAGs) with domain experts for health services research. Int. J. Epidemiol. 51 (4), 1339–1348 (2022)

Rothman, K.J., Greenland, S., Lash, T.L.: Modern Epidemiology. Wolters Kluwer Health/Lippincott Williams & Wilkins, Philadelphia (2008)

Rubin, D.B.: Causal inference using potential outcomes. J. Am. Stat. Assoc. 100 (469), 322–331 (2005)

Article   CAS   Google Scholar  

Sauer, B.V.: TJ. Use of Directed Acyclic Graphs. In Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide , edited by P. Velentgas, N. Dreyer, and P. Nourjah: Agency for Healthcare Research and Quality (US) (2013)

Schneeweiss, S., Rassen, J.A., Brown, J.S., Rothman, K.J., Happe, L., Arlett, P., Dal Pan, G., Goettsch, W., Murk, W., Wang, S.V.: Graphical depiction of longitudinal study designs in Health Care databases. Ann. Intern. Med. 170 (6), 398–406 (2019)

Schuler, M.S., Rose, S.: Targeted maximum likelihood estimation for causal inference in Observational studies. Am. J. Epidemiol. 185 (1), 65–73 (2017)

Stuart, E.A., DuGoff, E., Abrams, M., Salkever, D., Steinwachs, D.: Estimating causal effects in observational studies using Electronic Health data: Challenges and (some) solutions. EGEMS (Wash DC) 1 (3). (2013)

Sturmer, T., Webster-Clark, M., Lund, J.L., Wyss, R., Ellis, A.R., Lunt, M., Rothman, K.J., Glynn, R.J.: Propensity score weighting and trimming strategies for reducing Variance and Bias of Treatment Effect estimates: A Simulation Study. Am. J. Epidemiol. 190 (8), 1659–1670 (2021)

Suissa, S., Dell’Aniello, S.: Time-related biases in pharmacoepidemiology. Pharmacoepidemiol Drug Saf. 29 (9), 1101–1110 (2020)

Tripepi, G., Jager, K.J., Dekker, F.W., Wanner, C., Zoccali, C.: Measures of effect: Relative risks, odds ratios, risk difference, and ‘number needed to treat’. Kidney Int. 72 (7), 789–791 (2007)

Velentgas, P., Dreyer, N., Nourjah, P., Smith, S., Torchia, M.: Developing a Protocol for Observational Comparative Effectiveness Research: A User’s Guide. Agency for Healthcare Research and Quality (AHRQ) Publication 12(13). (2013)

Wang, A., Nianogo, R.A., Arah, O.A.: G-computation of average treatment effects on the treated and the untreated. BMC Med. Res. Methodol. 17 (1), 3 (2017)

Wang, S.V., Pottegard, A., Crown, W., Arlett, P., Ashcroft, D.M., Benchimol, E.I., Berger, M.L., Crane, G., Goettsch, W., Hua, W., Kabadi, S., Kern, D.M., Kurz, X., Langan, S., Nonaka, T., Orsini, L., Perez-Gutthann, S., Pinheiro, S., Pratt, N., Schneeweiss, S., Toussi, M., Williams, R.J.: HARmonized Protocol Template to enhance reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force. Pharmacoepidemiol Drug Saf. 32 (1), 44–55 (2023a)

Wang, S.V., Schneeweiss, S., Initiative, R.-D., Franklin, J.M., Desai, R.J., Feldman, W., Garry, E.M., Glynn, R.J., Lin, K.J., Paik, J., Patorno, E., Suissa, S., D’Andrea, E., Jawaid, D., Lee, H., Pawar, A., Sreedhara, S.K., Tesfaye, H., Bessette, L.G., Zabotka, L., Lee, S.B., Gautam, N., York, C., Zakoul, H., Concato, J., Martin, D., Paraoan, D.: and K. Quinto. Emulation of Randomized Clinical Trials With Nonrandomized Database Analyses: Results of 32 Clinical Trials. JAMA 329(16): 1376-85. (2023b)

Westreich, D., Lessler, J., Funk, M.J.: Propensity score estimation: Neural networks, support vector machines, decision trees (CART), and meta-classifiers as alternatives to logistic regression. J. Clin. Epidemiol. 63 (8), 826–833 (2010)

Yang, S., Eaton, C.B., Lu, J., Lapane, K.L.: Application of marginal structural models in pharmacoepidemiologic studies: A systematic review. Pharmacoepidemiol Drug Saf. 23 (6), 560–571 (2014)

Zhou, H., Taber, C., Arcona, S., Li, Y.: Difference-in-differences method in comparative Effectiveness Research: Utility with unbalanced groups. Appl. Health Econ. Health Policy. 14 (4), 419–429 (2016)

Download references

The authors received no financial support for this research.

Author information

Authors and affiliations.

Carelon Research, Wilmington, DE, USA

Sarah Ruth Hoffman, Nilesh Gangan, Joseph L. Smith, Arlene Tave, Yiling Yang, Christopher L. Crowe & Michael Grabner

Elevance Health, Indianapolis, IN, USA

Xiaoxue Chen

University of Maryland School of Pharmacy, Baltimore, MD, USA

Susan dosReis

You can also search for this author in PubMed   Google Scholar

Contributions

SH, NG, JS, AT, CC, MG are employees of Carelon Research, a wholly owned subsidiary of Elevance Health, which conducts health outcomes research with both internal and external funding, including a variety of private and public entities. XC was an employee of Elevance Health at the time of study conduct. YY was an employee of Carelon Research at the time of study conduct. SH, MG, and JLS are shareholders of Elevance Health. SdR receives funding from GlaxoSmithKline for a project unrelated to the content of this manuscript and conducts research that is funded by state and federal agencies.

Corresponding author

Correspondence to Sarah Ruth Hoffman .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary material 2, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Hoffman, S.R., Gangan, N., Chen, X. et al. A step-by-step guide to causal study design using real-world data. Health Serv Outcomes Res Method (2024). https://doi.org/10.1007/s10742-024-00333-6

Download citation

Received : 07 December 2023

Revised : 31 May 2024

Accepted : 10 June 2024

Published : 19 June 2024

DOI : https://doi.org/10.1007/s10742-024-00333-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Causal inference
• Real-world data
• Confounding
• Non-randomized data
• Bias in pharmacoepidemiology
• Find a journal
• Publish with us
• Track your research

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Questionnaire Design | Methods, Question Types & Examples

Questionnaire Design | Methods, Question Types & Examples

Published on July 15, 2021 by Pritha Bhandari . Revised on June 22, 2023.

A questionnaire is a list of questions or items used to gather data from respondents about their attitudes, experiences, or opinions. Questionnaires can be used to collect quantitative and/or qualitative information.

Questionnaires are commonly used in market research as well as in the social and health sciences. For example, a company may ask for feedback about a recent customer service experience, or psychology researchers may investigate health risk perceptions using questionnaires.

Table of contents

Questionnaires vs. surveys, questionnaire methods, open-ended vs. closed-ended questions, question wording, question order, step-by-step guide to design, other interesting articles, frequently asked questions about questionnaire design.

A survey is a research method where you collect and analyze data from a group of people. A questionnaire is a specific tool or instrument for collecting the data.

Designing a questionnaire means creating valid and reliable questions that address your research objectives , placing them in a useful order, and selecting an appropriate method for administration.

But designing a questionnaire is only one component of survey research. Survey research also involves defining the population you’re interested in, choosing an appropriate sampling method , administering questionnaires, data cleansing and analysis, and interpretation.

Sampling is important in survey research because you’ll often aim to generalize your results to the population. Gather data from a sample that represents the range of views in the population for externally valid results. There will always be some differences between the population and the sample, but minimizing these will help you avoid several types of research bias , including sampling bias , ascertainment bias , and undercoverage bias .

Here's why students love Scribbr's proofreading services

Discover proofreading & editing

Questionnaires can be self-administered or researcher-administered . Self-administered questionnaires are more common because they are easy to implement and inexpensive, but researcher-administered questionnaires allow deeper insights.

Self-administered questionnaires

Self-administered questionnaires can be delivered online or in paper-and-pen formats, in person or through mail. All questions are standardized so that all respondents receive the same questions with identical wording.

Self-administered questionnaires can be:

• cost-effective
• easy to administer for small and large groups
• anonymous and suitable for sensitive topics

But they may also be:

• unsuitable for people with limited literacy or verbal skills
• susceptible to a nonresponse bias (most people invited may not complete the questionnaire)
• biased towards people who volunteer because impersonal survey requests often go ignored.

Researcher-administered questionnaires

Researcher-administered questionnaires are interviews that take place by phone, in-person, or online between researchers and respondents.

Researcher-administered questionnaires can:

• help you ensure the respondents are representative of your target audience
• allow clarifications of ambiguous or unclear questions and answers
• have high response rates because it’s harder to refuse an interview when personal attention is given to respondents

But researcher-administered questionnaires can be limiting in terms of resources. They are:

• costly and time-consuming to perform
• more difficult to analyze if you have qualitative responses
• likely to contain experimenter bias or demand characteristics
• likely to encourage social desirability bias in responses because of a lack of anonymity

Your questionnaire can include open-ended or closed-ended questions or a combination of both.

Using closed-ended questions limits your responses, while open-ended questions enable a broad range of answers. You’ll need to balance these considerations with your available time and resources.

Closed-ended questions

Closed-ended, or restricted-choice, questions offer respondents a fixed set of choices to select from. Closed-ended questions are best for collecting data on categorical or quantitative variables.

Categorical variables can be nominal or ordinal. Quantitative variables can be interval or ratio. Understanding the type of variable and level of measurement means you can perform appropriate statistical analyses for generalizable results.

Examples of closed-ended questions for different variables

Nominal variables include categories that can’t be ranked, such as race or ethnicity. This includes binary or dichotomous categories.

It’s best to include categories that cover all possible answers and are mutually exclusive. There should be no overlap between response items.

In binary or dichotomous questions, you’ll give respondents only two options to choose from.

White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander

Ordinal variables include categories that can be ranked. Consider how wide or narrow a range you’ll include in your response items, and their relevance to your respondents.

Likert scale questions collect ordinal data using rating scales with 5 or 7 points.

When you have four or more Likert-type questions, you can treat the composite data as quantitative data on an interval scale . Intelligence tests, psychological scales, and personality inventories use multiple Likert-type questions to collect interval data.

With interval or ratio scales , you can apply strong statistical hypothesis tests to address your research aims.

Pros and cons of closed-ended questions

Well-designed closed-ended questions are easy to understand and can be answered quickly. However, you might still miss important answers that are relevant to respondents. An incomplete set of response items may force some respondents to pick the closest alternative to their true answer. These types of questions may also miss out on valuable detail.

To solve these problems, you can make questions partially closed-ended, and include an open-ended option where respondents can fill in their own answer.

Open-ended questions

Open-ended, or long-form, questions allow respondents to give answers in their own words. Because there are no restrictions on their choices, respondents can answer in ways that researchers may not have otherwise considered. For example, respondents may want to answer “multiracial” for the question on race rather than selecting from a restricted list.

• How do you feel about open science?
• How would you describe your personality?
• In your opinion, what is the biggest obstacle for productivity in remote work?

Open-ended questions have a few downsides.

They require more time and effort from respondents, which may deter them from completing the questionnaire.

For researchers, understanding and summarizing responses to these questions can take a lot of time and resources. You’ll need to develop a systematic coding scheme to categorize answers, and you may also need to involve other researchers in data analysis for high reliability .

Question wording can influence your respondents’ answers, especially if the language is unclear, ambiguous, or biased. Good questions need to be understood by all respondents in the same way ( reliable ) and measure exactly what you’re interested in ( valid ).

Use clear language

You should design questions with your target audience in mind. Consider their familiarity with your questionnaire topics and language and tailor your questions to them.

For readability and clarity, avoid jargon or overly complex language. Don’t use double negatives because they can be harder to understand.

Use balanced framing

Respondents often answer in different ways depending on the question framing. Positive frames are interpreted as more neutral than negative frames and may encourage more socially desirable answers.

Use a mix of both positive and negative frames to avoid research bias , and ensure that your question wording is balanced wherever possible.

Unbalanced questions focus on only one side of an argument. Respondents may be less likely to oppose the question if it is framed in a particular direction. It’s best practice to provide a counter argument within the question as well.

Avoid leading questions

Leading questions guide respondents towards answering in specific ways, even if that’s not how they truly feel, by explicitly or implicitly providing them with extra information.

It’s best to keep your questions short and specific to your topic of interest.

• The average daily work commute in the US takes 54.2 minutes and costs $29 per day. Since 2020, working from home has saved many employees time and money. Do you favor flexible work-from-home policies even after it’s safe to return to offices?
• Experts agree that a well-balanced diet provides sufficient vitamins and minerals, and multivitamins and supplements are not necessary or effective. Do you agree or disagree that multivitamins are helpful for balanced nutrition?

Keep your questions focused

Ask about only one idea at a time and avoid double-barreled questions. Double-barreled questions ask about more than one item at a time, which can confuse respondents.

This question could be difficult to answer for respondents who feel strongly about the right to clean drinking water but not high-speed internet. They might only answer about the topic they feel passionate about or provide a neutral answer instead – but neither of these options capture their true answers.

Instead, you should ask two separate questions to gauge respondents’ opinions.

Strongly Agree Agree Undecided Disagree Strongly Disagree

Do you agree or disagree that the government should be responsible for providing high-speed internet to everyone?

You can organize the questions logically, with a clear progression from simple to complex. Alternatively, you can randomize the question order between respondents.

Logical flow

Using a logical flow to your question order means starting with simple questions, such as behavioral or opinion questions, and ending with more complex, sensitive, or controversial questions.

The question order that you use can significantly affect the responses by priming them in specific directions. Question order effects, or context effects, occur when earlier questions influence the responses to later questions, reducing the validity of your questionnaire.

While demographic questions are usually unaffected by order effects, questions about opinions and attitudes are more susceptible to them.

• How knowledgeable are you about Joe Biden’s executive orders in his first 100 days?
• Are you satisfied or dissatisfied with the way Joe Biden is managing the economy?
• Do you approve or disapprove of the way Joe Biden is handling his job as president?

It’s important to minimize order effects because they can be a source of systematic error or bias in your study.

Randomization

Randomization involves presenting individual respondents with the same questionnaire but with different question orders.

When you use randomization, order effects will be minimized in your dataset. But a randomized order may also make it harder for respondents to process your questionnaire. Some questions may need more cognitive effort, while others are easier to answer, so a random order could require more time or mental capacity for respondents to switch between questions.

Step 1: Define your goals and objectives

The first step of designing a questionnaire is determining your aims.

• What topics or experiences are you studying?
• What specifically do you want to find out?
• Is a self-report questionnaire an appropriate tool for investigating this topic?

Once you’ve specified your research aims, you can operationalize your variables of interest into questionnaire items. Operationalizing concepts means turning them from abstract ideas into concrete measurements. Every question needs to address a defined need and have a clear purpose.

Step 2: Use questions that are suitable for your sample

Create appropriate questions by taking the perspective of your respondents. Consider their language proficiency and available time and energy when designing your questionnaire.

• Are the respondents familiar with the language and terms used in your questions?
• Would any of the questions insult, confuse, or embarrass them?
• Do the response items for any closed-ended questions capture all possible answers?
• Are the response items mutually exclusive?
• Do the respondents have time to respond to open-ended questions?

Consider all possible options for responses to closed-ended questions. From a respondent’s perspective, a lack of response options reflecting their point of view or true answer may make them feel alienated or excluded. In turn, they’ll become disengaged or inattentive to the rest of the questionnaire.

Step 3: Decide on your questionnaire length and question order

Once you have your questions, make sure that the length and order of your questions are appropriate for your sample.

If respondents are not being incentivized or compensated, keep your questionnaire short and easy to answer. Otherwise, your sample may be biased with only highly motivated respondents completing the questionnaire.

Decide on your question order based on your aims and resources. Use a logical flow if your respondents have limited time or if you cannot randomize questions. Randomizing questions helps you avoid bias, but it can take more complex statistical analysis to interpret your data.

Step 4: Pretest your questionnaire

When you have a complete list of questions, you’ll need to pretest it to make sure what you’re asking is always clear and unambiguous. Pretesting helps you catch any errors or points of confusion before performing your study.

Ask friends, classmates, or members of your target audience to complete your questionnaire using the same method you’ll use for your research. Find out if any questions were particularly difficult to answer or if the directions were unclear or inconsistent, and make changes as necessary.

If you have the resources, running a pilot study will help you test the validity and reliability of your questionnaire. A pilot study is a practice run of the full study, and it includes sampling, data collection , and analysis. You can find out whether your procedures are unfeasible or susceptible to bias and make changes in time, but you can’t test a hypothesis with this type of study because it’s usually statistically underpowered .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

A questionnaire is a data collection tool or instrument, while a survey is an overarching research method that involves collecting and analyzing data from people using questionnaires.

Closed-ended, or restricted-choice, questions offer respondents a fixed set of choices to select from. These questions are easier to answer quickly.

Open-ended or long-form questions allow respondents to answer in their own words. Because there are no restrictions on their choices, respondents can answer in ways that researchers may not have otherwise considered.

A Likert scale is a rating scale that quantitatively assesses opinions, attitudes, or behaviors. It is made up of 4 or more questions that measure a single attitude or trait when response scores are combined.

To use a Likert scale in a survey , you present participants with Likert-type questions or statements, and a continuum of items, usually with 5 or 7 possible responses, to capture their degree of agreement.

You can organize the questions logically, with a clear progression from simple to complex, or randomly between respondents. A logical flow helps respondents process the questionnaire easier and quicker, but it may lead to bias. Randomization can minimize the bias from order effects.

Questionnaires can be self-administered or researcher-administered.

Researcher-administered questionnaires are interviews that take place by phone, in-person, or online between researchers and respondents. You can gain deeper insights by clarifying questions for respondents or asking follow-up questions.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bhandari, P. (2023, June 22). Questionnaire Design | Methods, Question Types & Examples. Scribbr. Retrieved June 24, 2024, from https://www.scribbr.com/methodology/questionnaire/

Is this article helpful?

Pritha Bhandari

Other students also liked, survey research | definition, examples & methods, what is a likert scale | guide & examples, reliability vs. validity in research | difference, types and examples, what is your plagiarism score.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 19 June 2024

Detecting hallucinations in large language models using semantic entropy

• Sebastian Farquhar   ORCID: orcid.org/0000-0002-9185-6415 1   na1 ,
• Jannik Kossen 1   na1 ,
• Lorenz Kuhn 1   na1 &
• Yarin Gal   ORCID: orcid.org/0000-0002-2733-2078 1  

Nature volume  630 ,  pages 625–630 ( 2024 ) Cite this article

34k Accesses

1426 Altmetric

Metrics details

• Computer science
• Information technology

Large language model (LLM) systems, such as ChatGPT 1 or Gemini 2 , can show impressive reasoning and question-answering capabilities but often ‘hallucinate’ false outputs and unsubstantiated answers 3 , 4 . Answering unreliably or without the necessary information prevents adoption in diverse fields, with problems including fabrication of legal precedents 5 or untrue facts in news articles 6 and even posing a risk to human life in medical domains such as radiology 7 . Encouraging truthfulness through supervision or reinforcement has been only partially successful 8 . Researchers need a general method for detecting hallucinations in LLMs that works even with new and unseen questions to which humans might not know the answer. Here we develop new methods grounded in statistics, proposing entropy-based uncertainty estimators for LLMs to detect a subset of hallucinations—confabulations—which are arbitrary and incorrect generations. Our method addresses the fact that one idea can be expressed in many ways by computing uncertainty at the level of meaning rather than specific sequences of words. Our method works across datasets and tasks without a priori knowledge of the task, requires no task-specific data and robustly generalizes to new tasks not seen before. By detecting when a prompt is likely to produce a confabulation, our method helps users understand when they must take extra care with LLMs and opens up new possibilities for using LLMs that are otherwise prevented by their unreliability.

Similar content being viewed by others

Testing theory of mind in large language models and humans

Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT

ThoughtSource: A central hub for large language model reasoning data

‘Hallucinations’ are a critical problem 9 for natural language generation systems using large language models (LLMs), such as ChatGPT 1 or Gemini 2 , because users cannot trust that any given output is correct.

Hallucinations are often defined as LLMs generating “content that is nonsensical or unfaithful to the provided source content” 9 , 10 , 11 but they have come to include a vast array of failures of faithfulness and factuality. We focus on a subset of hallucinations which we call ‘confabulations’ 12 for which LLMs fluently make claims that are both wrong and arbitrary—by which we mean that the answer is sensitive to irrelevant details such as random seed. For example, when asked a medical question “What is the target of Sotorasib?” an LLM confabulates by sometimes answering KRASG12 ‘C’ (correct) and other times KRASG12 ‘D’ (incorrect) despite identical instructions. We distinguish this from cases in which a similar ‘symptom’ is caused by the following different mechanisms: when LLMs are consistently wrong as a result of being trained on erroneous data such as common misconceptions 13 ; when the LLM ‘lies’ in pursuit of a reward 14 ; or systematic failures of reasoning or generalization. We believe that combining these distinct mechanisms in the broad category hallucination is unhelpful. Our method makes progress on a portion of the problem of providing scalable oversight 15 by detecting confabulations that people might otherwise find plausible. However, it does not guarantee factuality because it does not help when LLM outputs are systematically bad. Nevertheless, we significantly improve question-answering accuracy for state-of-the-art LLMs, revealing that confabulations are a great source of error at present.

We show how to detect confabulations by developing a quantitative measure of when an input is likely to cause an LLM to generate arbitrary and ungrounded answers. Detecting confabulations allows systems built on LLMs to avoid answering questions likely to cause confabulations, to make users aware of the unreliability of answers to a question or to supplement the LLM with more grounded search or retrieval. This is essential for the critical emerging field of free-form generation in which naive approaches, suited to closed vocabulary and multiple choice, fail. Past work on uncertainty for LLMs has focused on simpler settings, such as classifiers 16 , 17 and regressors 18 , 19 , whereas the most exciting applications of LLMs relate to free-form generations.

The term hallucination in the context of machine learning originally comes from filling in ungrounded details, either as a deliberate strategy 20 or as a reliability problem 4 . The appropriateness of the metaphor has been questioned as promoting undue anthropomorphism 21 . Although we agree that metaphor must be used carefully with LLMs 22 , the widespread adoption of the term hallucination reflects the fact that it points to an important phenomenon. This work represents a step towards making that phenomenon more precise.

To detect confabulations, we use probabilistic tools to define and then measure the ‘semantic’ entropy of the generations of an LLM—an entropy that is computed over meanings of sentences. High entropy corresponds to high uncertainty 23 , 24 , 25 —so semantic entropy is one way to estimate semantic uncertainties. Semantic uncertainty, the broader category of measures we introduce, could be operationalized with other measures of uncertainty, such as mutual information, instead. Entropy in free-form generation is normally hard to measure because answers might mean the same thing (be semantically equivalent) despite being expressed differently (being syntactically or lexically distinct). This causes naive estimates of entropy or other lexical variation scores 26 to be misleadingly high when the same correct answer might be written in many ways without changing its meaning.

By contrast, our semantic entropy moves towards estimating the entropy of the distribution of meanings of free-form answers to questions, insofar as that is possible, rather than the distribution over the ‘tokens’ (words or word-pieces) which LLMs natively represent. This can be seen as a kind of semantic consistency check 27 for random seed variation. An overview of our approach is provided in Fig. 1 and a worked example in Supplementary Table 1 .

a , Naive entropy-based uncertainty measures variation in the exact answers, treating ‘Paris’, ‘It’s Paris’ and ‘France’s capital Paris’ as different. But this is unsuitable for language tasks for which sometimes different answers mean the same things. Our semantic entropy clusters answers which share meanings before computing the entropy. A low semantic entropy shows that the LLM is confident about the meaning. b , Semantic entropy can also detect confabulations in longer passages. We automatically decompose a long generated answer into factoids. For each factoid, an LLM generates questions to which that factoid might have been the answer. The original LLM then samples  M possible answers to these questions. Finally, we compute the semantic entropy over the answers to each specific question, including the original factoid. Confabulations are indicated by high average semantic entropy for questions associated with that factoid. Here, semantic entropy classifies Fact 1 as probably not a confabulation because generations often mean the same thing, despite very different wordings, which a naive entropy would have missed.

Intuitively, our method works by sampling several possible answers to each question and clustering them algorithmically into answers that have similar meanings, which we determine on the basis of whether answers in the same cluster entail each other bidirectionally 28 . That is, if sentence A entails that sentence B is true and vice versa, then we consider them to be in the same semantic cluster. We measure entailment using both general-purpose LLMs and natural language inference (NLI) tools developed specifically for detecting entailment for which we show direct evaluations in Supplementary Tables 2 and 3 and Supplementary Fig. 1 . Textual entailment has previously been shown to correlate with faithfulness 10 in the context of factual consistency 29 as well as being used to measure factuality in abstractive summarization 30 , especially when applied at the right granularity 31 .

Semantic entropy detects confabulations in free-form text generation across a range of language models and domains, without previous domain knowledge. Our evaluations cover question answering in trivia knowledge (TriviaQA 32 ), general knowledge (SQuAD 1.1; ref. 33 ), life sciences (BioASQ 34 ) and open-domain natural questions (NQ-Open 35 ) derived from actual queries to Google Search 36 . In addition, semantic entropy detects confabulations in mathematical word problems (SVAMP 37 ) and in a biography-generation dataset, FactualBio, accompanying this paper.

Our results for TriviaQA, SQuAD, BioASQ, NQ-Open and SVAMP are all evaluated context-free and involve sentence-length answers (96 ± 70 characters, mean ± s.d.) and use LLaMA 2 Chat (7B, 13B and 70B parameters) 38 , Falcon Instruct (7B and 40B) 39 and Mistral Instruct (7B) 40 . In the Supplementary Information , we further consider short-phrase-length answers. Results for FactualBio (442 ± 122 characters) use GPT-4 (ref. 1 ). At the time of writing, GPT-4 (ref. 1 ) did not expose output probabilities 41 or hidden states, although it does now. As a result, we propose a discrete approximation of our estimator for semantic entropy which allows us to run experiments without access to output probabilities, which we use for all GPT-4 results in this paper and which performs similarly well.

Our confabulation detection with semantic entropy is more robust to user inputs from previously unseen domains than methods which aim to ‘learn’ how to detect confabulations from a set of example demonstrations. Our method is unsupervised, meaning that we do not need labelled examples of confabulations. By contrast, supervised methods detect confabulations by learning patterns behind examples of confabulations, assuming that future questions preserve these patterns. But this assumption is often untrue in new situations or with confabulations that human overseers are unable to identify (compare Fig. 17 of ref. 24 ). As a strong supervised baseline, we compare to an embedding regression method inspired by ref. 24 which trains a logistic regression classifier to predict whether the model correctly answered a question on the basis of the final ‘embedding’ (hidden state) of the LLM. We also use the P (True) method 24 which looks at the probability with which an LLM predicts that the next token is ‘True’ when few-shot prompted to compare a main answer with ‘brainstormed’ alternatives.

Confabulations contribute substantially to incorrect answers given by language models. We show that semantic entropy can be used to predict many incorrect model answers and to improve question-answering accuracy by refusing to answer those questions the model is uncertain about. Corresponding to these two uses, we evaluate two main metrics. First, the widely used area under the receiver operating characteristic (AUROC) curve for the binary event that a given answer is incorrect. This measure captures both precision and recall and ranges from 0 to 1, with 1 representing a perfect classifier and 0.5 representing an un-informative classifier. We also show a new measure, the area under the ‘rejection accuracy’ curve (AURAC). This studies the case in which the confabulation detection score is used to refuse to answer the questions judged most likely to cause confabulations. Rejection accuracy is the accuracy of the answers of the model on the remaining questions and the area under this curve is a summary statistic over many thresholds (representative threshold accuracies are provided in Supplementary Material ). The AURAC captures the accuracy improvement which users would experience if semantic entropy was used to filter out questions causing the highest entropy.

Detecting confabulations in QA and math

In Fig. 2 , we show that both semantic entropy and its discrete approximation outperform our best baselines for sentence-length generations. These results are averaged across datasets and provide the actual scores on the held-out evaluation dataset. We report the raw average score across held-out evaluation datasets without standard error because the distributional characteristics are more a property of the models and datasets selected than the method. Consistency of relative results across different datasets is a stronger indicator of variation in this case.

Semantic entropy outperforms leading baselines and naive entropy. AUROC (scored on the y -axes) measures how well methods predict LLM mistakes, which correlate with confabulations. AURAC (likewise scored on the y -axes) measures the performance improvement of a system that refuses to answer questions which are judged likely to cause confabulations. Results are an average over five datasets, with individual metrics provided in the Supplementary Information .

Semantic entropy greatly outperforms the naive estimation of uncertainty using entropy: computing the entropy of the length-normalized joint probability of the token sequences. Naive entropy estimation ignores the fact that token probabilities also express the uncertainty of the model over phrasings that do not change the meaning of an output.

Our methods also outperform the supervised embedding regression method both in- and out-of-distribution. In pale-yellow bars we show that embedding regression performance deteriorates when its training data do not match the deployment distribution—which mirrors the common real-world case in which there is a distribution shift between training and deployment 42 —the plotted value is the average metric for embedding regression trained on one of the four ‘off-distribution’ datasets for that evaluation. This is critical because reliable uncertainty is most important when the data distribution shifts. Semantic entropy also outperforms P (True) which is supervised ‘in-context’; that is, it is adapted to the deployment task with a few training examples provided in the LLM prompt itself. The discrete variant of semantic entropy performs similarly to our standard estimator, despite not requiring exact output probabilities.

Averaged across the 30 combinations of tasks and models we study, semantic entropy achieves the best AUROC value of 0.790 whereas naive entropy (0.691), P (True) (0.698) and the embedding regression baseline (0.687) lag behind it. Semantic entropy performs well consistently, with stable performance (between 0.78 and 0.81 AUROC) across the different model families (LLaMA, Falcon and Mistral) and scales (from 7B to 70B parameters) which we study (we report summary statistics for each dataset and model as before). Although semantic entropy outperforms the baselines across all model sizes, P (True) seems to improve with model size, suggesting that it might become more competitive for very capable honest models in settings that the model understands well (which are, however, not the most important cases to have good uncertainty). We use ten generations to compute entropy, selected using analysis in Supplementary Fig. 2 . Further results for short-phrase generations are described in Supplementary Figs. 7 – 10 .

The results in Fig. 2 offer a lower bound on the effectiveness of semantic entropy at detecting confabulations. These evaluations determine whether semantic entropy and baseline methods can detect when the answers of the model are incorrect (which we validate against human correctness evaluations in Supplementary Table 4 ). In addition to errors from confabulations (arbitrary incorrectness), this also includes other types of mistakes for which semantic entropy is not suited, such as consistent errors learned from the training data. The fact that methods such as embedding regression are able to spot other kinds of errors, not just confabulations, but still are outperformed by semantic entropy, suggests that confabulations are a principal category of errors for actual generations.

Examples of questions and answers from TriviaQA, SQuAD and BioASQ, for LLaMA 2 Chat 70B, are shown in Table 1 . These illustrate how only semantic entropy detects when the meaning is constant but the form varies (the first row of the table) whereas semantic entropy and naive entropy both correctly predict the presence of confabulations when the form and meaning vary together (second row) and predict the absence of confabulations when the form and meaning are both constant across several resampled generations (third row). In the final row, we give an example in which semantic entropy is erroneously high as a result of overly sensitive semantic clustering relative to the reference answer. Our clustering method distinguishes the answers which provide a precise date from those which only provide a year. For some contexts that would have been correct but in this context the distinction between the specific day and the year is probably irrelevant. This highlights the importance of context and judgement in clustering, especially in subtle cases, as well as the shortcomings of evaluating against fixed reference answers which do not capture the open-ended flexibility of conversational deployments of LLMs.

Detecting confabulations in biographies

Semantic entropy is most natural for sentences that express a single proposition but the idea of semantic equivalence is trickier to apply to longer passages which express many propositions which might only agree partially 43 . Nevertheless, we can use semantic entropy to detect confabulations in longer generations, such as entire paragraphs of text. To show this, we develop a dataset of biographical generations from GPT-4 (v.0613) for 21 individuals notable enough to have their own Wikipedia page but without extensive online biographies. From each biography generated by GPT-4, we automatically extract propositional factual claims about the individual (150 factual claims in total), which we manually label as true or false.

Applying semantic entropy to this problem is challenging. Naively, one might simply regenerate each sentence (conditioned on the text so far) and then compute semantic entropy over these regenerations. However, the resampled sentences often target different aspects of the biography: for example, one time describing family and the next time profession. This is analogous to the original problem semantic entropy was designed to resolve: the model is uncertain about the right ordering of facts, not about the facts themselves. To address this, we break down the entire paragraph into factual claims and reconstruct questions which might have been answered by those claims. Only then do we apply semantic entropy (Fig. 1 ) by generating three new answers to each question (selected with analysis in Supplementary Figs. 3 and 4 ) and computing the semantic entropy over those generations plus the original factual claim. We aggregate these by averaging the semantic entropy over all the questions to get an uncertainty score for each proposition, which we use to detect confabulations. Unaggregated results are shown in Supplementary Figs. 5 and 6 .

As GPT-4 did not allow access to the probability of the generation at the time of writing, we use a discrete variant of semantic entropy which makes the further approximation that we can infer a discrete empirical distribution over semantic meaning clusters from only the generations ( Methods ). This allows us to compute semantic entropy using only the black-box outputs of an LLM. However, we were unable to compute the naive entropy baseline, the standard semantic entropy estimator or the embedding regression baseline for GPT-4 without output probabilities and embeddings.

In Fig. 3 we show that the discrete variant of semantic entropy effectively detects confabulations on this dataset. Its AUROC and AURAC are higher than either a simple ‘self-check’ baseline—which just asks the LLM whether the factoid is likely to be true—or a variant of P (True) which has been adapted to work for the paragraph-length setting. Discrete semantic entropy has better rejection accuracy performance until 20% of the questions have been rejected at which point P (True) has a narrow edge. This indicates that the questions predicted to cause confabulations are indeed more likely to be wrong.

The discrete variant of our semantic entropy estimator outperforms baselines both when measured by AUROC and AURAC metrics (scored on the y -axis). The AUROC and AURAC are substantially higher than for both baselines. At above 80% of questions being answered, semantic entropy has the highest accuracy. Only when the top 20% of answers judged most likely to be confabulations are rejected does the answer accuracy on the remainder for the P (True) baseline exceed semantic entropy.

Our probabilistic approach, accounting for semantic equivalence, detects an important class of hallucinations: those that are caused by a lack of LLM knowledge. These are a substantial portion of the failures at present and will continue even as models grow in capabilities because situations and cases that humans cannot reliably supervise will persist. Confabulations are a particularly noteworthy failure mode for question answering but appear in other domains too. Semantic entropy needs no previous domain knowledge and we expect that algorithmic adaptations to other problems will allow similar advances in, for example, abstractive summarization. In addition, extensions to alternative input variations such as rephrasing or counterfactual scenarios would allow a similar method to act as a form of cross-examination 44 for scalable oversight through debate 45 .

The success of semantic entropy at detecting errors suggests that LLMs are even better at “knowing what they don’t know” than was argued by ref. 24 —they just don’t know they know what they don’t know. Our method explicitly does not directly address situations in which LLMs are confidently wrong because they have been trained with objectives that systematically produce dangerous behaviour, cause systematic reasoning errors or are systematically misleading the user. We believe that these represent different underlying mechanisms—despite similar ‘symptoms’—and need to be handled separately.

One exciting aspect of our approach is the way it makes use of classical probabilistic machine learning methods and adapts them to the unique properties of modern LLMs and free-form language generation. We hope to inspire a fruitful exchange of well-studied methods and emerging new problems by highlighting the importance of meaning when addressing language-based machine learning problems.

Semantic entropy as a strategy for overcoming confabulation builds on probabilistic tools for uncertainty estimation. It can be applied directly to any LLM or similar foundation model without requiring any modifications to the architecture. Our ‘discrete’ variant of semantic uncertainty can be applied even when the predicted probabilities for the generations are not available, for example, because access to the internals of the model is limited.

In this section we introduce background on probabilistic methods and uncertainty in machine learning, discuss how it applies to language models and then discuss our contribution, semantic entropy, in detail.

Uncertainty and machine learning

We aim to detect confabulations in LLMs, using the principle that the model will be uncertain about generations for which its output is going to be arbitrary.

One measure of uncertainty is the predictive entropy of the output distribution, which measures the information one has about the output given the input 25 . The predictive entropy (PE) for an input sentence x is the conditional entropy ( H ) of the output random variable Y with realization y given x ,

A low predictive entropy indicates an output distribution which is heavily concentrated whereas a high predictive entropy indicates that many possible outputs are similarly likely.

Aleatoric and epistemic uncertainty

We do not distinguish between aleatoric and epistemic uncertainty in our analysis. Researchers sometimes separate aleatoric uncertainty (uncertainty in the underlying data distribution) from epistemic uncertainty (caused by having only limited information) 46 . Further advances in uncertainty estimation which separate these kinds of uncertainty would enhance the potential for our semantic uncertainty approach by allowing extensions beyond entropy.

Joint probabilities of sequences of tokens

Generative LLMs produce strings of text by selecting tokens in sequence. Each token is a wordpiece that often represents three or four characters (though especially common sequences and important words such as numbers typically get their own token). To compute entropies, we need access to the probabilities the LLM assigns to the generated sequence of tokens. The probability of the entire sequence, s , conditioned on the context, x , is the product of the conditional probabilities of new tokens given past tokens, whose resulting log-probability is \(\log P({\bf{s}}| {\boldsymbol{x}})={\sum }_{i}\log P({s}_{i}| {{\bf{s}}}_{ < i},{\boldsymbol{x}})\) , where s i is the i th output token and s < i denotes the set of previous tokens.

Length normalization

When comparing the log-probabilities of generated sequences, we use ‘length normalization’, that is, we use an arithmetic mean log-probability, \(\frac{1}{N}{\sum }_{i}^{N}\log P({s}_{i}| {{\bf{s}}}_{ < i},{\boldsymbol{x}})\) , instead of the sum. In expectation, longer sequences have lower joint likelihoods because of the conditional independence of the token probabilities 47 . The joint likelihood of a sequence of length N shrinks exponentially in N . Its negative log-probability therefore grows linearly in N , so longer sentences tend to contribute more to entropy. We therefore interpret length-normalizing the log-probabilities when estimating the entropy as asserting that the expected uncertainty of generations is independent of sentence length. Length normalization has some empirical success 48 , including in our own preliminary experiments, but little theoretical justification in the literature.

Principles of semantic uncertainty

If we naively calculate the predictive entropy directly from the probabilities of the generated sequence of tokens, we conflate the uncertainty of the model over the meaning of its answer with the uncertainty over the exact tokens used to express that meaning. For example, even if the model is confident in the meaning of a generation, there are still usually many different ways for phrasing that generation without changing its meaning. For the purposes of detecting confabulations, the uncertainty of the LLM over meanings is more important than the uncertainty over the exact tokens used to express those meanings.

Our semantic uncertainty method therefore seeks to estimate only the uncertainty the LLM has over the meaning of its generation, not the choice of words. To do this, we introduce an algorithm that clusters model generations by meaning and subsequently calculates semantic uncertainty. At a high level this involves three steps:

Generation: sample output sequences of tokens from the predictive distribution of a LLM given a context x .

Clustering: cluster sequences by their meaning using our clustering algorithm based on bidirectional entailment.

Entropy estimation: estimate semantic entropy by summing probabilities of sequences that share a meaning following equation ( 2 ) and compute their entropy.

Generating a set of answers from the model

Given some context x as input to the LLM, we sample M sequences, { s (1) , …,  s ( M ) } and record their token probabilities, { P ( s (1) ∣ x ), …,  P ( s ( M ) ∣ x )}. We sample all our generations from a single model, varying only the random seed used for sampling from the token probabilities. We do not observe the method to be particularly sensitive to details of the sampling scheme. In our implementation, we sample at temperature 1 using nucleus sampling ( P  = 0.9) (ref. 49 ) and top- K sampling ( K  = 50) (ref. 50 ). We also sample a single generation at low temperature (0.1) as an estimate of the ‘best generation’ of the model to the context, which we use to assess the accuracy of the model. (A lower sampling temperature increases the probability of sampling the most likely tokens).

Clustering by semantic equivalence

To estimate semantic entropy we need to cluster generated outputs from the model into groups of outputs that mean the same thing as each other.

This can be described using ‘semantic equivalence’ which is the relation that holds between two sentences when they mean the same thing. We can formalize semantic equivalence mathematically. Let the space of tokens in a language be \({\mathcal{T}}\) . The space of all possible sequences of tokens of length N is then \({{\mathcal{S}}}_{N}\equiv {{\mathcal{T}}}^{N}\) . Note that N can be made arbitrarily large to accommodate whatever size of sentence one can imagine and one of the tokens can be a ‘padding’ token which occurs with certainty for each token after the end-of-sequence token. For some sentence \({\bf{s}}\in {{\mathcal{S}}}_{N}\) , composed of a sequence of tokens, \({s}_{i}\in {\mathcal{T}}\) , there is an associated meaning. Theories of meaning are contested 51 . However, for specific models and deployment contexts many considerations can be set aside. Care should be taken comparing very different models and contexts.

Let us introduce a semantic equivalence relation, E (  ⋅  ,  ⋅  ), which holds for any two sentences that mean the same thing—we will operationalize this presently. Recall that an equivalence relation is any reflexive, symmetric and transitive relation and that any equivalence relation on a set corresponds to a set of equivalence classes. Each semantic equivalence class captures outputs that can be considered to express the same meaning. That is, for the space of semantic equivalence classes \({\mathcal{C}}\) the sentences in the set \(c\in {\mathcal{C}}\) can be regarded in many settings as expressing a similar meaning such that \(\forall {\bf{s}},{{\bf{s}}}^{{\prime} }\in c:E({\bf{s}},{{\bf{s}}}^{{\prime} })\) . So we can build up these classes of semantically equivalent sentences by checking if new sentences share a meaning with any sentences we have already clustered and, if so, adding them into that class.

We operationalize E (  ⋅  ,  ⋅  ) using the idea of bidirectional entailment, which has a long history in linguistics 52 and natural language processing 28 , 53 , 54 . A sequence, s , means the same thing as a second sequence, s ′, only if the sequences entail (that is, logically imply) each other. For example, ‘The capital of France is Paris’ entails ‘Paris is the capital of France’ and vice versa because they mean the same thing. (See later for a discussion of soft equivalence and cases in which bidirectional entailment does not guarantee equivalent meanings).

Importantly, we require that the sequences mean the same thing with respect to the context—key meaning is sometimes contained in the context. For example, ‘Paris’ does not entail ‘The capital of France is Paris’ because ‘Paris’ is not a declarative sentence without context. But in the context of the question ‘What is the capital of France?’, the one-word answer does entail the longer answer.

Detecting entailment has been the object of study of a great deal of research in NLI 55 . We rely on language models to predict entailment, such as DeBERTa-Large-MNLI 56 , which has been trained to predict entailment, or general-purpose LLMs such as GPT-3.5 (ref. 57 ), which can predict entailment given suitable prompts.

We then cluster sentences according to whether they bidirectionally entail each other using the algorithm presented in Extended Data Fig. 1 . Note that, to check if a sequence should be added to an existing cluster, it is sufficient to check if the sequence bidirectionally entails any of the existing sequences in that cluster (we arbitrarily pick the first one), given the transitivity of semantic equivalence. If a sequence does not share meaning with any existing cluster, we assign it its own cluster.

Computing the semantic entropy

Having determined the classes of generated sequences that mean the same thing, we can estimate the likelihood that a sequence generated by the LLM belongs to a given class by computing the sum of the probabilities of all the possible sequences of tokens which can be considered to express the same meaning as

Formally, this treats the output as a random variable whose event-space is the space of all possible meaning-classes, C , a sub- σ -algebra of the standard event-space S . We can then estimate the semantic entropy (SE) as the entropy over the meaning-distribution,

There is a complication which prevents direct computation: we do not have access to every possible meaning-class c . Instead, we can only sample c from the sequence-generating distribution induced by the model. To handle this, we estimate the expectation in equation ( 3 ) using a Rao–Blackwellized Monte Carlo integration over the semantic equivalence classes C ,

where \(P({C}_{i}| {\boldsymbol{x}})=\frac{P({c}_{i}| {\boldsymbol{x}})}{{\sum }_{c}P(c| {\boldsymbol{x}})}\) estimates a categorical distribution over the cluster meanings, that is, ∑ i P ( C i ∣ x ) = 1. Without this normalization step cluster ‘probabilities’ could exceed one because of length normalization, resulting in degeneracies. Equation ( 5 ) is the estimator giving our main method that we refer to as semantic entropy throughout the text.

For scenarios in which the sequence probabilities are not available, we propose a variant of semantic entropy which we call ‘discrete’ semantic entropy. Discrete semantic entropy approximates P ( C i ∣ x ) directly from the number of generations in each cluster, disregarding the token probabilities. That is, we approximate P ( C i ∣ x ) as \({\sum }_{1}^{M}\frac{{I}_{c={C}_{i}}}{M}\) , the proportion of all the sampled answers which belong to that cluster. Effectively, this just assumes that each output that was actually generated was equally probable—estimating the underlying distribution as the categorical empirical distribution. In the limit of M the estimator converges to equation ( 5 ) by the law of large numbers. We find that discrete semantic entropy results in similar performance empirically.

We provide a worked example of the computation of semantic entropy in Supplementary Note  1 .

Semantic entropy is designed to detect confabulations, that is, model outputs with arbitrary meaning. In our experiments, we use semantic uncertainty to predict model accuracy, demonstrating that confabulations make up a notable fraction of model mistakes. We further show that semantic uncertainty can be used to improve model accuracy by refusing to answer questions when semantic uncertainty is high. Last, semantic uncertainty can be used to give users a way to know when model generations are probably unreliable.

We use the datasets BioASQ 34 , SQuAD 33 , TriviaQA 32 , SVAMP 37 and NQ-Open 35 . BioASQ is a life-sciences question-answering dataset based on the annual challenge of the same name. The specific dataset we use is based on the QA dataset from Task B of the 2023 BioASQ challenge (11B). SQuAD is a reading comprehension dataset whose context passages are drawn from Wikipedia and for which the answers to questions can be found in these passages. We use SQuAD 1.1 which excludes the unanswerable questions added in v.2.0 that are deliberately constructed to induce mistakes so they do not in practice cause confabulations to occur. TriviaQA is a trivia question-answering dataset. SVAMP is a word-problem maths dataset containing elementary-school mathematical reasoning tasks. NQ-Open is a dataset of realistic questions aggregated from Google Search which have been chosen to be answerable without reference to a source text. For each dataset, we use 400 train examples and 400 test examples randomly sampled from the original larger dataset. Note that only some of the methods require training, for example semantic entropy does not use the training data. If the datasets themselves are already split into train and test (or validation) samples, we sample our examples from within the corresponding split.

All these datasets are free-form, rather than multiple choice, because this better captures the opportunities created by LLMs to produce free-form sentences as answers. We refer to this default scenario as our ‘sentence-length’ experiments. In Supplementary Note  7 , we also present results for confabulation detection in a ‘short-phrase’ scenario, in which we constrain model answers on these datasets to be as concise as possible.

To make the problems more difficult and induce confabulations, we do not provide the context passages for any of the datasets. When the context passages are provided, the accuracy rate is too high for these datasets for the latest generations of models to meaningfully study confabulations.

For sentence-length generations we use: Falcon 39 Instruct (7B and 40B), LLaMA 2 Chat 38 (7B, 13B and 70B) and Mistral 40 Instruct (7B).

In addition to reporting results for semantic entropy, discrete semantic entropy and naive entropy, we consider two strong baselines.

Embedding regression is a supervised baseline inspired by the P (IK) method 24 . In that paper, the authors fine-tune their proprietary LLM on a dataset of questions to predict whether the model would have been correct. This requires access to a dataset of ground-truth answers to the questions. Rather than fine-tuning the entire LLM in this way, we simply take the final hidden units and train a logistic regression classifier to make the same prediction. By contrast to their method, this is much simpler because it does not require fine-tuning the entire language model, as well as being more reproducible because the solution to the logistic regression optimization problem is not as seed-dependent as the fine-tuning procedure. As expected, this supervised approach performs well in-distribution but fails when the distribution of questions is different from that on which the classifier is trained.

The second baseline we consider is the P (True) method 24 , in which the model first samples M answers (identically to our semantic entropy approach) and then is prompted with the list of all answers generated followed by the highest probability answer and a question whether this answer is “(a) True” or “(b) False”. The confidence score is then taken to be the probability with which the LLM responds with ‘a’ to the multiple-choice question. The performance of this method is boosted with a few-shot prompt, in which up to 20 examples from the training set are randomly chosen, filled in as above, but then provided with the actual ground truth of whether the proposed answer was true or false. In this way, the method can be considered as supervised ‘in-context’ because it makes use of some ground-truth training labels but can be used without retraining the model. Because of context-size constraints, this method cannot fit a full 20 few-shot examples in the context when input questions are long or large numbers of generations are used. As a result, we sometimes have to reduce the number of few-shot examples to suit the context size and we note this in the  Supplementary Material .

Entailment estimator

Any NLI classification system could be used for our bidirectional entailment clustering algorithm. We consider two different kinds of entailment detector.

One option is to use an instruction-tuned LLM such as LLaMA 2, GPT-3.5 (Turbo 1106) or GPT-4 to predict entailment between generations. We use the following prompt:

We are evaluating answers to the question {question} Here are two possible answers: Possible Answer 1: {text1} Possible Answer 2: {text2} Does Possible Answer 1 semantically entail Possible Answer 2? Respond with entailment, contradiction, or neutral.

Alternatively, we consider using a language model trained for entailment prediction, specifically the DeBERTa-large model 56 fine-tuned on the NLI dataset MNLI 58 . This builds on past work towards paraphrase identification based on embedding similarity 59 , 60 and BERT-style models 61 , 62 . We template more simply, checking if DeBERTa predicts entailment between the concatenation of the question and one answer and the concatenation of the question and another answer. Note that DeBERTa-large is a relatively lightweight model with only 1.5B parameters which is much less powerful than most of the LLMs under study.

In Supplementary Note 2 , we carefully evaluate the benefits and drawbacks of these methods for entailment prediction. We settle on using GPT-3.5 with the above prompt, as its entailment predictions agree well with human raters and lead to good confabulation detection performance.

In Supplementary Note  3 , we provide a discussion of the computational cost and choosing the number of generations for reliable clustering.

Prompting templates

We use a simple generation template for all sentence-length answer datasets:

Answer the following question in a single brief but complete sentence. Question: {question} Answer:

Metrics and accuracy measurements

We use three main metrics to evaluate our method: AUROC, rejection accuracy and AURAC. Each of these is grounded in an automated factuality estimation measurement relative to the reference answers provided by the datasets that we use.

AUROC, rejection accuracy and AURAC

First, we use the AUROC curve, which measures the reliability of a classifier accounting for both precision and recall. The AUROC can be interpreted as the probability that a randomly chosen correct answer has been assigned a higher confidence score than a randomly chosen incorrect answer. For a perfect classifier, this is 1.

Second, we compute the ‘rejection accuracy at X %’, which is the question-answering accuracy of the model on the most-confident X % of the inputs as identified by the respective uncertainty method. If an uncertainty method works well, predictions on the confident subset should be more accurate than predictions on the excluded subset and the rejection accuracy should increase as we reject more inputs.

To summarize this statistic we compute the AURAC—the total area enclosed by the accuracies at all cut-off percentages X %. This should increase towards 1 as given uncertainty method becomes more accurate and better at detecting likely-inaccurate responses but it is more sensitive to the overall accuracy of the model than the AUROC metric.

In Supplementary Note  5 , we provide the unaggregated rejection accuracies for sentence-length generations.

Assessing accuracy

For the short-phrase-length generation setting presented in Supplementary Note  7 , we simply assess the accuracy of the generations by checking if the F1 score of the commonly used SQuAD metric exceeds 0.5. There are limitations to such simple scoring rules 63 but this method is widely used in practice and its error is comparatively small on these standard datasets.

For our default scenario, the longer sentence-length generations, this measure fails, as the overlap between the short reference answer and our long model answer is invariably too small. For sentence-length generations, we therefore automatically determine whether an answer to the question is correct or incorrect by using GPT-4 to compare the given answer to the reference answer. We use the template:

We are assessing the quality of answers to the following question: {question} The expected answer is: {reference answer} The proposed answer is: {predicted answer} Within the context of the question, does the proposed answer mean the same as the expected answer? Respond only with yes or no.

We make a small modification for datasets with several reference answers: line two becomes “The following are expected answers to this question:” and the final line asks “does the proposed answer mean the same as any of the expected answers?”.

In Supplementary Note 6 , we check the quality of our automated ground-truth evaluations against human judgement by hand. We find that GPT-4 gives the best results for determining model accuracy and thus use it in all our sentence-length experiments.

In this section we describe the application of semantic entropy to confabulation detection in longer model generations, specifically paragraph-length biographies.

We introduce a biography-generation dataset—FactualBio—available alongside this paper. FactualBio is a collection of biographies of individuals who are notable enough to have Wikipedia pages but not notable enough to have large amounts of detailed coverage, generated by GPT-4 (v.0613). To generate the dataset, we randomly sampled 21 individuals from the WikiBio dataset 64 . For each biography, we generated a list of factual claims contained in each biography using GPT-4, with 150 total factual claims (the total number is only coincidentally a round number). For each of these factual claims, we manually determined whether the claim was correct or incorrect. Out of 150 claims, 45 were incorrect. As before, we apply confabulation detection to detect incorrect model predictions, even though there may be model errors which are not confabulations.

Prompting and generation

Given a paragraph-length piece of LLM-generated text, we apply the following sequence of steps:

Automatically decompose the paragraph into specific factual claims using an LLM (not necessarily the same as the original).

For each factual claim, use an LLM to automatically construct Q questions which might have produced that claim.

For each question, prompt the original LLM to generate M answers.

For each question, compute the semantic entropy of the answers, including the original factual claim.

Average the semantic entropies over the questions to arrive at a score for the original factual claim.

We pursue this slightly indirect way of generating answers because we find that simply resampling each sentence creates variation unrelated to the uncertainty of the model about the factual claim, such as differences in paragraph structure.

We decompose the paragraph into factual claims using the following prompt:

Please list the specific factual propositions included in the answer above. Be complete and do not leave any factual claims out. Provide each claim as a separate sentence in a separate bullet point.

We found that we agreed with the decompositions in all cases in the dataset.

We then generate six questions for each of the facts from the decomposition. We generate these questions by prompting the model twice with the following:

Following this text: {text so far} You see the sentence: {proposition} Generate a list of three questions, that might have generated the sentence in the context of the preceding original text, as well as their answers. Please do not use specific facts that appear in the follow-up sentence when formulating the question. Make the questions and answers diverse. Avoid yes-no questions. The answers should not be a full sentence and as short as possible, e.g. only a name, place, or thing. Use the format “1. {question} – {answer}”.

These questions are not necessarily well-targeted and the difficulty of this step is the main source of errors in the procedure. We generate three questions with each prompt, as this encourages diversity of the questions, each question targeting a different aspect of the fact. However, we observed that the generated questions will sometimes miss obvious aspects of the fact. Executing the above prompt twice (for a total of six questions) can improve coverage. We also ask for brief answers because the current version of GPT-4 tends to give long, convoluted and highly hedged answers unless explicitly told not to.

Then, for each question, we generate three new answers using the following prompt:

We are writing an answer to the question “{user question}”. So far we have written: {text so far} The next sentence should be the answer to the following question: {question} Please answer this question. Do not answer in a full sentence. Answer with as few words as possible, e.g. only a name, place, or thing.

We then compute the semantic entropy over these answers plus the original factual claim. Including the original fact ensures that the estimator remains grounded in the original claim and helps detect situations in which the question has been interpreted completely differently from the original context. We make a small modification to handle the fact that GPT-4 generations often include refusals to answer questions. These refusals were not something we commonly observe in our experiments with LLaMA 2, Falcon or Mistral models. If more than half of the answers include one of the strings ‘not available’, ‘not provided’, ‘unknown’ or ‘unclear’ then we treat the semantic uncertainty as maximal.

We then average the semantic entropies for each question corresponding to the factual claim to get an entropy for this factual claim.

Despite the extra assumptions and complexity, we find that this method greatly outperforms the baselines.

To compute semantic entailment between the original claim and regenerated answers, we rely on the DeBERTa entailment prediction model as we find empirically that DeBERTa predictions result in higher train-set AUROC than other methods. Because DeBERTa has slightly lower recall than GPT-3.5/4, we use a modified set-up for which we say the answers mean the same as each other if at least one of them entails the other and neither is seen to contradict the other—a kind of ‘non-defeating’ bidirectional entailment check rather than true bidirectional entailment. The good performance of DeBERTa in this scenario is not surprising as both factual claims and regenerated answers are relatively short. We refer to Supplementary Notes 2 and 3 for ablations and experiments regarding our choice of entailment estimator for paragraph-length generations.

We implement two baselines. First, we implement a variant of the P (True) method, which is adapted to the new setting. For each factoid, we generate a question with answers in the same way as for semantic entropy. We then use the following prompt:

Question: {question} Here are some brainstormed ideas: {list of regenerated answers} Possible answer: {original answer} Is the possible answer true? Respond with “yes” or “no”.

As we cannot access the probabilities GPT-4 assigns to predicting ‘yes’ and ‘no’ as the next token, we approximate this using Monte Carlo samples. Concretely, we execute the above prompt ten times (at temperature 1) and then take the fraction of answers which was ‘yes’ as our unbiased Monte Carlo estimate of the token probability GPT-4 assigns to ‘yes’.

As a second, simpler, baseline we check if the model thinks the answer is true. We simply ask:

Following this text: {text so far} You see this statement: {proposition} Is it likely that the statement is true? Respond with ‘yes’ or ‘no’.

It is interesting that this method ought to perform very well if we think that the model has good ‘self-knowledge’ (that is, if “models mostly know what they don’t know” 24 ) but in fact semantic entropy is much better at detecting confabulations.

Data availability

The data used for the short-phrase and sentence-length generations are publicly available and the released code details how to access it. We release a public version of the FactualBio dataset as part of the code base for reproducing the paragraph-length experiments.

Code availability

We release all code used to produce the main experiments. The code for short-phrase and sentence-length experiments can be found at github.com/jlko/semantic_uncertainty and https://doi.org/10.5281/zenodo.10964366 (ref. 65 ). The code for paragraph-length experiments can be found at github.com/jlko/long_hallucinations and https://doi.org/10.5281/zenodo.10964366 (ref. 65 ).

GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023).

Gemini: a family of highly capable multimodal models. Preprint at https://arxiv.org/abs/2312.11805 (2023).

Xiao, Y. & Wang, W. Y. On hallucination and predictive uncertainty in conditional language generation. In Proc. 16th Conference of the European Chapter of the Association for Computational Linguistics 2734–2744 (Association for Computational Linguistics, 2021).

Rohrbach, A., Hendricks, L. A., Burns, K., Darrell, T. & Saenko, K. Object hallucination in image captioning. In Proc. 2018 Conference on Empirical Methods in Natural Language Processing (eds Riloff, E., Chiang, D., Hockenmaier, J. & Tsujii, J.) 4035–4045 (Association for Computational Linguistics, 2018).

Weiser, B. Lawyer who used ChatGPT faces penalty for made up citations. The New York Times (8 Jun 2023).

Opdahl, A. L. et al. Trustworthy journalism through AI. Data Knowl. Eng . 146 , 102182 (2023).

Shen, Y. et al. ChatGPT and other large language models are double-edged swords. Radiology 307 , e230163 (2023).

Article   PubMed   Google Scholar  

Schulman, J. Reinforcement learning from human feedback: progress and challenges. Presented at the Berkeley EECS Colloquium. YouTube www.youtube.com/watch?v=hhiLw5Q_UFg (2023).

Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. 55 , 248 (2023).

Maynez, J., Narayan, S., Bohnet, B. & McDonald, R. On faithfulness and factuality in abstractive summarization. In Proc. 58th Annual Meeting of the Association for Computational Linguistics (eds Jurafsky, D., Chai, J., Schluter, N. & Tetreault, J.) 1906–1919 (Association for Computational Linguistics, 2020).

Filippova, K. Controlled hallucinations: learning to generate faithfully from noisy data. In Findings of the Association for Computational Linguistics: EMNLP 2020 (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 864–870 (Association for Computational Linguistics, 2020).

Berrios, G. Confabulations: a conceptual history. J. Hist. Neurosci. 7 , 225–241 (1998).

Article   CAS   PubMed   Google Scholar  

Lin, S., Hilton, J. & Evans, O. Teaching models to express their uncertainty in words. Transact. Mach. Learn. Res. (2022).

Evans, O. et al. Truthful AI: developing and governing AI that does not lie. Preprint at https://arxiv.org/abs/2110.06674 (2021).

Amodei, D. et al. Concrete problems in AI safety. Preprint at https://arxiv.org/abs/1606.06565 (2016).

Jiang, Z., Araki, J., Ding, H. & Neubig, G. How can we know when language models know? On the calibration of language models for question answering. Transact. Assoc. Comput. Linguist. 9 , 962–977 (2021).

Article   Google Scholar  

Desai, S. & Durrett, G. Calibration of pre-trained transformers. In Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (eds Webber, B., Cohn, T., He, Y. & Liu, Y.) 295–302 (Association for Computational Linguistics, 2020).

Glushkova, T., Zerva, C., Rei, R. & Martins, A. F. Uncertainty-aware machine translation evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2021 (eds Moens, M-F., Huang, X., Specia, L. & Yih, S.) 3920–3938 (Association for Computational Linguistics, 2021).

Wang, Y., Beck, D., Baldwin, T. & Verspoor, K. Uncertainty estimation and reduction of pre-trained models for text regression. Transact. Assoc. Comput. Linguist. 10 , 680–696 (2022).

Baker, S. & Kanade, T. Hallucinating faces. In Proc. Fourth IEEE International Conference on Automatic Face and Gesture Recognition . 83–88 (IEEE, Catalogue no PR00580, 2002).

Eliot, L. AI ethics lucidly questioning this whole hallucinating AI popularized trend that has got to stop. Forbes Magazine (24 August 2022).

Shanahan, M. Talking about large language models. Commun. Assoc. Comp. Machinery 67 , 68–79 (2024).

MacKay, D. J. C. Information-based objective functions for active data selection. Neural Comput. 4 , 590–604 (1992).

Kadavath, S. et al. Language models (mostly) know what they know. Preprint at https://arxiv.org/abs/2207.05221 (2022).

Lindley, D. V. On a measure of the information provided by an experiment. Ann. Math. Stat. 27 , 986–1005 (1956).

Article   MathSciNet   Google Scholar  

Xiao, T. Z., Gomez, A. N. & Gal, Y. Wat zei je? Detecting out-of-distribution translations with variational transformers. In Workshop on Bayesian Deep Learning at the Conference on Neural Information Processing Systems (NeurIPS, Vancouver, 2019).

Christiano, P., Cotra, A. & Xu, M. Eliciting Latent Knowledge (Alignment Research Center, 2021); https://docs.google.com/document/d/1WwsnJQstPq91_Yh-Ch2XRL8H_EpsnjrC1dwZXR37PC8/edit .

Negri, M., Bentivogli, L., Mehdad, Y., Giampiccolo, D. & Marchetti, A. Divide and conquer: crowdsourcing the creation of cross-lingual textual entailment corpora. In Proc. 2011 Conference on Empirical Methods in Natural Language Processing 670–679 (Association for Computational Linguistics, 2011).

Honovich, O. et al. TRUE: Re-evaluating factual consistency evaluation. In Proc. Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering 161–175 (Association for Computational Linguistics, 2022).

Falke, T., Ribeiro, L. F. R., Utama, P. A., Dagan, I. & Gurevych, I. Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In Proc. 57th Annual Meeting of the Association for Computational Linguistics 2214–2220 (Association for Computational Linguistics, 2019).

Laban, P., Schnabel, T., Bennett, P. N. & Hearst, M. A. SummaC: re-visiting NLI-based models for inconsistency detection in summarization. Trans. Assoc. Comput. Linguist. 10 , 163–177 (2022).

Joshi, M., Choi, E., Weld, D. S. & Zettlemoyer, L. TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proc. 55th Annual Meeting of the Association for Computational Linguistics 1601–1611 (Association for Computational Linguistics. 2017).

Rajpurkar, P., Zhang, J., Lopyrev, K. & Liang, P. SQuAD: 100,000+ questions for machine compression of text. In Proc. 2016 Conference on Empirical Methods in Natural Language Processing (eds Su, J., Duh, K. & Carreras, X.) 2383–2392 (Association for Computational Linguistics, 2016).

Tsatsaronis, G. et al. An overview of the BIOASQ large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics 16 , 138 (2015).

Article   PubMed   PubMed Central   Google Scholar  

Lee, K., Chang, M.-W. & Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Proc. 57th Annual Meeting of the Association for Computational Linguistics 6086–6096 (Association for Computational Linguistics, 2019).

Kwiatkowski, T. et al. Natural questions: a benchmark for question answering research. Transact. Assoc. Comput. Linguist. 7 , 452–466 (2019).

Patel, A., Bhattamishra, S. & Goyal, N. Are NLP models really able to solve simple math word problems? In Proc. 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Toutanova, K. et al.) 2080–2094 (Assoc. Comp. Linguistics, 2021).

Touvron, H. et al. Llama 2: open foundation and fine-tuned chat models. Preprint at https://arxiv.org/abs/2307.09288 (2023).

Penedo, G. et al. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. In Proc. 36th Conference on Neural Information Processing Systems (eds Oh, A. et al.) 79155–79172 (Curran Associates, 2023)

Jiang, A. Q. et al. Mistral 7B. Preprint at https://arxiv.org/abs/2310.06825 (2023).

Manakul, P., Liusie, A. & Gales, M. J. F. SelfCheckGPT: Zero-Resource Black-Box hallucination detection for generative large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023 (eds Bouamor, H., Pino, J. & Bali, K.) 9004–9017 (Assoc. Comp. Linguistics, 2023).

Mukhoti, J., Kirsch, A., van Amersfoort, J., Torr, P. H. & Gal, Y. Deep deterministic uncertainty: a new simple baseline. In IEEE/CVF Conference on Computer Vision and Pattern Recognition 24384–24394 (Computer Vision Foundation, 2023).

Schuster, T., Chen, S., Buthpitiya, S., Fabrikant, A. & Metzler, D. Stretching sentence-pair NLI models to reason over long documents and clusters. In Findings of the Association for Computational Linguistics: EMNLP 2022 (eds Goldberg, Y. et al.) 394–412 (Association for Computational Linguistics, 2022).

Barnes, B. & Christiano, P. Progress on AI Safety via Debate. AI Alignment Forum www.alignmentforum.org/posts/Br4xDbYu4Frwrb64a/writeup-progress-on-ai-safety-via-debate-1 (2020).

Irving, G., Christiano, P. & Amodei, D. AI safety via debate. Preprint at https://arxiv.org/abs/1805.00899 (2018).

Der Kiureghian, A. & Ditlevsen, O. Aleatory or epistemic? Does it matter? Struct. Saf. 31 , 105–112 (2009).

Malinin, A. & Gales, M. Uncertainty estimation in autoregressive structured prediction. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=jN5y-zb5Q7m (2021).

Murray, K. & Chiang, D. Correcting length bias in neural machine translation. In Proc. Third Conference on Machine Translation (eds Bojar, O. et al.) 212–223 (Assoc. Comp. Linguistics, 2018).

Holtzman, A., Buys, J., Du, L., Forbes, M. & Choi, Y. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=rygGQyrFvH (2020).

Fan, A., Lewis, M. & Dauphin, Y. Hierarchical neural story generation. In Proc. 56th Annual Meeting of the Association for Computational Linguistics (eds Gurevych, I. & Miyao, Y.) 889–898 (Association for Computational Linguistics, 2018).

Speaks, J. in The Stanford Encyclopedia of Philosophy (ed. Zalta, E. N.) (Metaphysics Research Lab, Stanford Univ., 2021).

Culicover, P. W. Paraphrase generation and information retrieval from stored text. Mech. Transl. Comput. Linguist. 11 , 78–88 (1968).

Google Scholar  

Padó, S., Cer, D., Galley, M., Jurafsky, D. & Manning, C. D. Measuring machine translation quality as semantic equivalence: a metric based on entailment features. Mach. Transl. 23 , 181–193 (2009).

Androutsopoulos, I. & Malakasiotis, P. A survey of paraphrasing and textual entailment methods. J. Artif. Intell. Res. 38 , 135–187 (2010).

MacCartney, B. Natural Language Inference (Stanford Univ., 2009).

He, P., Liu, X., Gao, J. & Chen, W. Deberta: decoding-enhanced BERT with disentangled attention. In International Conference on Learning Representations https://openreview.net/forum?id=XPZIaotutsD (2021).

Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33 , 1877–1901 (2020).

Williams, A., Nangia, N. & Bowman, S. R. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (eds Walker, M. et al.) 1112–1122 (Assoc. Comp. Linguistics, 2018).

Yu, L., Hermann, K. M., Blunsom, P. & Pulman, S. Deep learning for answer sentence selection. Preprint at https://arxiv.org/abs/1412.1632 (2014).

Socher, R., Huang, E., Pennin, J., Manning, C. D. & Ng, A. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of the 24th Conference on Neural Information Processing Systems (eds Shawe-Taylor, J. et al.) (2011)

He, R., Ravula, A., Kanagal, B. & Ainslie, J. Realformer: Transformer likes residual attention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 (eds Zhong, C., et al.) 929–943 (Assoc. Comp. Linguistics, 2021).

Tay, Y. et al. Charformer: fast character transformers via gradient-based subword tokenization. In Proceedings of the International Conference on Learning Representations https://openreview.net/forum?id=JtBRnrlOEFN (2022).

Kane, H., Kocyigit, Y., Abdalla, A., Ajanoh, P. & Coulibali, M. Towards neural similarity evaluators. In Workshop on Document Intelligence at the 32nd conference on Neural Information Processing (2019).

Lebret, R., Grangier, D. & Auli, M. Neural text generation from structured data with application to the biography domain. In Proc. 2016 Conference on Empirical Methods in Natural Language Processing (eds Su, J. et al.) 1203–1213 (Association for Computational Linguistics, 2016).

Kossen, J., jlko/semantic_uncertainty: Initial release v.1.0.0. Zenodo https://doi.org/10.5281/zenodo.10964366 (2024).

Download references

Acknowledgements

We thank G. Irving, K. Perlin, J. Richens, L. Rimell and M. Turpin for their comments or discussion related to this work. We thank K. Handa for his help with the human evaluation of our automated accuracy assessment. We thank F. Bickford Smith and L. Melo for their code review. Y.G. is supported by a Turing AI Fellowship funded by the UK government’s Office for AI, through UK Research and Innovation (grant reference EP/V030302/1), and delivered by the Alan Turing Institute.

Author information

These authors contributed equally: Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn

Authors and Affiliations

OATML, Department of Computer Science, University of Oxford, Oxford, UK

Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn & Yarin Gal

You can also search for this author in PubMed   Google Scholar

Contributions

S.F. led the work from conception to completion and proposed using bidirectional entailment to cluster generations as a way of computing entropy in LLMs. He wrote the main text, most of the Methods and Supplementary Information and prepared most of the figures. J.K. improved the mathematical formalization of semantic entropy; led the extension of semantic entropy to sentence- and paragraph-length generations; wrote the code for, and carried out, all the experiments and evaluations; wrote much of the Methods and Supplementary Information and prepared drafts of many figures; and gave critical feedback on the main text. L.K. developed the initial mathematical formalization of semantic entropy; wrote code for, and carried out, the initial experiments around semantic entropy and its variants which demonstrated the promise of the idea and helped narrow down possible research avenues to explore; and gave critical feedback on the main text. Y.G. ideated the project, proposing the idea to differentiate semantic and syntactic diversity as a tool for detecting hallucinations, provided high-level guidance on the research and gave critical feedback on the main text; he runs the research laboratory in which the work was carried out.

Corresponding author

Correspondence to Sebastian Farquhar .

Ethics declarations

Competing interests.

S.F. is currently employed by Google DeepMind and L.K. by OpenAI. For both, this paper was written under their University of Oxford affiliation. The remaining authors declare no competing interests.

Peer review

Peer review information.

Nature thanks Mirella Lapata and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Extended data figures and tables

Extended data fig. 1 algorithm outline for bidirectional entailment clustering..

Given a set of outputs in response to a context, the bidirectional entailment answer returns a set of sets of outputs which have been classified as sharing a meaning.

Supplementary information

Supplementary information.

Supplementary Notes 1–7, Figs. 1–10, Tables 1–4 and references. Includes, worked example for semantic entropy calculation, discussion of limitations and computational cost of entailment clustering, ablation of entailment prediction and clustering methods, discussion of automated accuracy assessment, unaggregated results for sentence-length generations and further results for short-phrase generations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Farquhar, S., Kossen, J., Kuhn, L. et al. Detecting hallucinations in large language models using semantic entropy. Nature 630 , 625–630 (2024). https://doi.org/10.1038/s41586-024-07421-0

Download citation

Received : 17 July 2023

Accepted : 12 April 2024

Published : 19 June 2024

Issue Date : 20 June 2024

DOI : https://doi.org/10.1038/s41586-024-07421-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Advancing Cough Research: Methodological Insights into Cough Challenge in Guinea Pig Models Using Double Chamber vs Whole-Body Plethysmography

19 Pages Posted: 21 Jun 2024

Jana Plevkova

affiliation not provided to SSRN

Janka Jakusova

Mariana brozmanova, zuzana biringerova, tomas buday.

Objective: This study compares two methods of citric acid-induced cough in guinea pigs in whole-body plethysmography (WBP) and double chamber plethysmography (DCP) to evaluate their efficacy.Methods: Sixteen specific pathogen-free (SPF) and sixteen conventionally-bred (CON) animals were exposed to 0.4M citric acid aerosol. They underwent cough provocation using both DCP and WBP methods. The number of coughs and latency to the first cough were recorded and analysed using statistical methods to determine significant differences between the two techniques.Results: WBP resulted in significantly higher cough counts (WBP vs. DCP: 13±8.85 vs 2±3 for SPF; 14±8 vs 5±5 for CON; p

Keywords: cough provocation, guinea pig model, double chamber plethysmography, whole-body plethysmography

Suggested Citation: Suggested Citation

affiliation not provided to SSRN ( email )

No Address Available

Tomas Buday (Contact Author)

Do you have a job opening that you would like to promote on ssrn, paper statistics, related ejournals, physiology ejournal.

Subscribe to this fee journal for more curated articles on this topic

• Open access
• Published: 18 June 2024

Impact of a multi-disciplinary team-based care model for patients living with diabetes on health outcomes: a mixed-methods study

• Jacquelyn Jacobs 1 ,
• Alyn Dougherty 1 ,
• Banita McCarn 1 ,
• Nazia S. Saiyed 1 ,
• Stacy Ignoffo 1 ,
• Christina Wagener 2 ,
• Cindy San Miguel 1 &
• Linda Martinez 1  

BMC Health Services Research volume  24 , Article number:  746 ( 2024 ) Cite this article

182 Accesses

Metrics details

Individuals facing socioeconomic hardship experience higher than average rates of chronic disease, such as diabetes, with less access to evidence-based treatment. One solution to address these inequities is a team-based care (TBC) model, defined as one in which at least two providers work collaboratively with a patient and their caregiver(s) to make healthcare decisions. This paper seeks to describe the implementation of a TBC model within a safety-net healthcare setting and determine the extent to which it can be an effective, patient-centered approach to treating individuals with diabetes.

Semi-structured interviews were conducted with staff ( n  = 15) and patients ( n  = 18). Clinical data were extracted from the electronic medical record of patients ( n  = 1,599) seen at a safety-net health system in Chicago, Illinois, United States. The mixed methods study was guided by implementation science and participatory research principles. Staff interviews were 60 min and covered patient care activities, work flow, perceived patient experience, and facilitators/barriers to care coordination. Patient interviews were 60 min and covered satisfaction, attitudes about diabetes management, quality of life, and technology. Patient interviews were co-analyzed by research staff and members of a patient advisory committee. Clinical data were collected at an index visit, two years prior and at one-year follow up ( n  = 1,599).

Four themes emerged from the interviews: (1) patients perceived the TBC model to be patient centered and of high quality; (2) technology can be an innovative tool, but barriers exist; (3) diabetes management is a complex process; and (4) staff communication enhances care coordination, but misinterpreting roles reduces care coordination. From pre-enrollment to the follow-up period, we found a statistically significant increase in missed visits, decrease in hemoglobin A1c (HbA1c), decrease in body mass index, and decrease in the percent of patients with high blood pressure. We found that each medical visit during the follow-up period was associated with an HbA1c decrease of 0.26 points.

Conclusions

A TBC model is a patient-centered approach to providing care to patients with complex health needs, such as diabetes, patients were satisfied with the care they were receiving, and the model was associated with an improvement in clinical outcomes.

Peer Review reports

In 2021, the United States (U.S.) Centers for Disease Control and Prevention (CDC) reported 38.1 million U.S. adults aged 18 years or older—or 14.7% of all U.S. adults—were living with diabetes [ 1 ]. It is estimated that an additional 8.5 million U.S. adults have undiagnosed diabetes, demonstrating the burden of disease that requires coordinated care and quality treatment [ 2 ]. In addition to high prevalence, the total cost of diagnosed diabetes continues to rise. Between 2012 and 2017, the total cost increased by 26% [ 3 ]. Like other chronic diseases, the burden of diabetes management lies heavily upon the patient and requires strict medication adherence, regular monitoring of glucose levels, intentional eating habits and other behavioral interventions [ 4 ].

Individuals facing socioeconomic hardship experience higher than average rates of chronic disease, with less access to evidence-based treatment. Research has shown that low-income patients have higher chronic disease prevalence rates, incidence rates, and per patient disease-related costs compared to the national estimates for all adults [ 5 , 6 , 7 ]. Data from the 2021 Illinois Diabetes Burden report showed a significantly higher diabetes prevalence for those with an annual household income less than $15,000 [ 8 ].

One solution to address the needs of patients with complex health conditions is a team-based care (TBC) model. TBC is defined by the Institute of Medicine as “the provision of health services to individuals, families, and/or their communities by at least two health providers who work collaboratively with patients and their caregivers—to the extent preferred by each patient to accomplish shared goals within and across settings to achieve coordinated, high-quality care” [ 9 ]. Existing literature suggests that TBC can be effective in managing diabetes, specifically in lowering HbA1c, blood pressure, and cholesterol [ 10 ]. The Community Preventive Services Task Force also found TBC to be patient-centered and flexible in a variety of settings. However, less is known about its effectiveness as a model of care within healthcare settings primarily serving racial and ethnic minorities and/or low-income patients [ 11 , 12 , 13 , 14 , 15 , 16 ]. The present study seeks to fill a gap in the current literature around the implementation of a TBC model with a multi-disciplinary team in a safety-net health system that serves predominately low-income patients of color. The purpose of this paper is to describe the implementation of a TBC model for patients living with diabetes who receive care at a safety-net health system, describe patient satisfaction of the model, and measure changes to utilization of health care services and diabetes-related health outcomes.

In Chicago, Illinois, 12.4% of adults are living with diabetes [ 17 ]. Like other parts of the U.S., people of color (POC) experience disproportionately higher rates compared to their white counterparts. Nearly 17% of non-Hispanic Black individuals in Chicago are living with diabetes compared to 12.3% of Hispanic individuals, and just 8.9% of white individuals [ 17 ]. Rates of diabetes are also unequally distributed across geographic regions. Those on the West and Southwest sides of the city, communities which are predominately comprised of POC, have rates as high as 33.2%. Yet the highest neighborhood-level prevalence rate on the North side, predominantly comprised of white individuals, is 18.6% [ 17 ].

Sinai Chicago is the largest private safety-net health system in Illinois. In the U.S., a safety-net health system is a designation for systems that are obligated to provide care for all individuals, regardless of their insurance status or ability to pay. This often results in high proportions of uninsured or underinsured and low-income patients. Sinai Chicago serves the historically disinvested communities on the West and Southwest sides of Chicago, which are comprised of predominately un- or under-insured POC [ 18 ]. Many of Sinai’s patients also experience a disproportionate burden of chronic diseases such as diabetes, cancer, and heart disease. In many communities served by Sinai Chicago, an estimated 30–50% of residents have a family history of diabetes [ 19 ].

Team-based care model

For decades, Sinai primary care providers, endocrinologists, nurses, and dieticians have provided clinical care and individualized diabetes self-management education to patients with complex clinical and social needs. The patient would see separate providers, on different days, who may review the medical record, but not engage with other providers in a manner to collectively address patient goals. However, this model lacked the integration that is required for adequate and holistic management of uncontrolled diabetes often seen in this population. In response, Sinai launched the Center for Diabetes and Endocrinology (the Center) in September 2020 with the goal of establishing a destination of choice for patients with diabetes in Sinai’s primary service area. The Center treats patients with prediabetes, diabetes (types 1 and 2 and gestational), and other endocrine disorders in a multidisciplinary setting that offers education, nutrition, and prevention. This robust, patient-centered program provides streamlined services to address the full spectrum of patients’ medical (physical, pharmaceutical, and behavioral) and non-medical (nutrition, social, and emotional) needs in one clinical setting. The Center’s clinical team addresses physical needs from mild to severe cases and is supported by pharmacy and a variety of wraparound services (social work, community health workers). Throughout the patient experience, technology is integrated to ensure ongoing monitoring of patient adherence to treatment plans, rapid response to changes in disease status, and educational outreach. For example, patients received access to telehealth appointments to encourage appointment adherence. Similarly, patients may be offered remote glucose monitoring (RGM) which allows for blood glucose data to be integrated into the medical record and shared between the patient and provider for proactive follow up. Medical assistants, pharmacists and diabetes educators review blood glucose data and contact patients who show clinically concerning changes in glucose levels.

Study design and frameworks

We conducted a convergent mixed methods process and outcome evaluation guided by implementation science and participatory research principles [ 20 ]. The objectives of this study were to: (1) describe the implementation of a TBC model for patients living with diabetes who receive care at a safety-net health system; (2) describe patient satisfaction and the extent to which the Center engages patients and responds to their needs; and (3) measure changes to utilization of health care services and diabetes-related health outcomes.

We employed the Consolidated Framework for Implementation Research (CFIR) to guide our understanding of the various components of the Center’s TBC model [ 21 ]. CFIR is one of the most popular frameworks for implementation research and can be used to identify the factors that may influence effectiveness of the implementation process [ 22 ]. This framework is organized into five domains: the intervention, the inner setting, the outer setting, the individuals involved in the intervention, and the process of implementation. Within the context of this study, the outer setting refers to Sinai Chicago’s patient population, the inner setting refers to the Center, the individuals refer to both the providers implementing the TBC model and the patients receiving care, the process refers to the implementation of the intervention (TBC), and the intervention refers to the TBC model. The research team identified implementation constructs for each domain to guide the overall evaluation (Fig.  1 ).

Conceptual framework for evaluating the Sinai center for diabetes and endocrinology

The evaluation also employed a community-based participatory (CBP) approach by including community residents with lived experience in our data collection, analysis, and interpretation processes [ 23 ]. At the project onset, the research team, with extensive experience in community-engaged research, convened a Patient Advisory Committee (PAC) [ 24 ]. The PAC comprised seven individuals 18 years of age or older who are living with diabetes. The PAC met almost every other month for 18 months. PAC meetings were organized and led by the research team. Topics included: development of the patient interview guide, human subjects research training, qualitative data analysis training, coding and theme development, sense-making, and developing dissemination products. We followed CBP research principles by building on strengths within the community, recognizing and highlighting the lived experience of individuals living with diabetes, facilitating partnership between community residents and researchers, and encouraging co-learning and empowerment [ 25 ]. The Mount Sinai Hospital (MSH) Institutional Review Board approved this project (protocol #21–37).

Data collection and analysis

The study team conducted semi-structured interviews with Center staff and patients, and extracted clinical data from the electronic medical record (EMR). Data from our qualitative and quantitative methods were collected and analyzed separately, and findings were merged to achieve study objects.

Semi-structured interviews

Sixty-minute semi-structured interviews were conducted with 15 Center staff, who were recruited through email. Any staff employed by the Center were eligible to participate. Interview guides were created to align with the relevant implementation domain and constructs from Fig.  1 . For example, interviews with staff included questions related to: the inner setting (communication between staff, involvement and accountability of staff, and leadership engagement), the intervention (daily activities, the degree to which the Center is appropriate for the target population), the individuals (perceived patient experiences), and the process (work flows and protocols, facilitators and barriers to coordination across the Center). The full staff interview guide can be found in the supplemental materials.

Sixty-minute semi-structured interviews were also conducted with 18 patients. Patients were recruited via Sinai’s social media platforms, flyers, and referrals from Center staff. The research team trained two community health workers (CHW) to conduct all patient interviews. The CHWs who served as interviewers are from the target communities, but not affiliated with the Center. Interviews were conducted in English or Spanish. Patients were eligible to participate if they were 18 years of age or older, an active patient of the Center (had at least one appointment during the study period), currently living with pre-diabetes, diabetes or other endocrine disorder and competent to provide consent. The interview guide was developed to align with the relevant implementation domain and constructs from Fig.  1 and in collaboration with the PAC to ensure that members of the community informed the research process. The interview guide included questions related to: process (satisfaction with their care at the Center), intervention (knowledge, attitudes and beliefs about diabetes management; quality of life; perceptions of the care team; experience with education received from staff; perceptions of how the Center influences health), individuals (attitudes and experiences with patient-centered technology, application of diabetes education). The full patient interview guide can be found in the supplemental materials.

All interviews were conducted virtually using a HIPAA-compliant version of Webex, and audio recorded with permission. Recordings were transcribed verbatim by a professional transcription company. Following the qualitative analysis training for PAC members, the research team led two virtual coding sessions with the PAC to develop initial codes for a subset of transcripts. The research team incorporated the feedback from the PAC into the final codebook.

Transcribed interviews were coded using QSR NVivo. Two researchers coded the English interviews and two bilingual (English and Spanish) researchers coded the Spanish interviews following the usual standards of qualitative research analysis [ 26 , 27 ]. The researchers followed a grounded theory approach to analyze interview data [ 28 ]. Each reviewed the transcriptions to determine themes and sub-themes, reviewed each other’s themes and discussed discrepancies to arrive at consensus. The findings were then narrowed based on the most significant themes. After conducting the initial analysis, the research team presented themes and sub-themes back to the PAC in sense-making sessions to understand results, elicit contextual feedback and ensure that interpretations accurately reflect the insights of patients.

Patient clinical data

Outcomes of the TBC model and patient’s health outcomes were assessed via EMR data. Each patient’s index visit was defined as their first visit after the formation of the Center. Up to three years of data were collected for each patient: two years prior to the index visit, and one year of follow-up after the index visit.

Outcome variables

Clinical outcomes of interest were HbA1c, body mass index (BMI), and elevated blood pressure. HbA1c and BMI were treated as continuous variables. Blood pressure was dichotomized into elevated blood pressure (systolic blood pressure [SBP] ≥ 130 mmHg or diastolic blood pressure [DBP] ≥ 80 mmHg) and not elevated blood pressure (SBP < 130 mmHg and DBP < 80 mmHg) [ 29 ]. Healthcare utilization outcomes of interest were the number and type of visits completed (e.g., endocrinology-related medical visits, community health worker visits, other visits such as retinal eye exam and diabetes educator visits), the number of missed visits, emergency department visits, and inpatient hospitalizations. The type of visit was defined by categorizing the text typed into the “event” and “location” fields of each visit record in the dataset. Additionally, a binary healthcare utilization variable was created using emergency department visits and inpatient hospitalizations (zero emergency department visits or inpatient hospitalizations vs. at least one emergency department or inpatient hospitalization).

Explanatory variables and covariates

Comorbidities were categorized into three groups: concordant microvascular (including renal disease, neuropathy, and eye disease), concordant macrovascular (coronary disease, cerebrovascular disease, hypertension, and heart failure), and discordant (arthritis, cancer, and chronic lower respiratory disease) [ 30 ]. Sociodemographic variables included age, race/ethnicity, language, and insurance type. Age was treated as a continuous variable; race/ethnicity was categorized into non-Hispanic Black, non-Hispanic White, Hispanic/Latino, and Other/Unknown; language was categorized into English, Spanish and Other/Unknown, and insurance type was dichotomized into private insurance and other coverage (Medicaid, Medicare, charity, other/unknown).

Patient health outcomes at the index visit were compared to those one-year post-enrollment. We considered the data closest to the date of the index visit (within 60 days) to be the index measurement. The last measurement collected before the end of the 12-month follow-up period was considered the post-enrollment measurement. The association between index and post-enrollment blood pressure outcomes were tested for significance using a χ 2 square test; all other health outcomes were tested using paired t-tests.

The annual rate of missed visits, emergency department visits, and inpatient hospitalizations during the pre-enrollment period was compared to the follow-up period. The annual pre-enrollment rate was calculated as the average rate over the two-year pre-enrollment data collection period. Paired t-tests were used to compare pre- and post-enrollment rates.

Finally, linear and logistic regression models were fit to determine which variables, if any, were significantly associated with patient health- and health care utilization outcomes. Regression models were developed using a step-wise reduction approach. A p -value < 0.05 was considered statistically significant. Quantitative analyses were conducted using Stata version 15.1 (StataCorp LP, College Station, TX).

Data from semi-structured interviews with staff and patients and electronic medical record data on health care service utilization were triangulated to provide insights into the implementation of the TBC model and utilization of health care services at the Center. Fifteen Center Staff and 18 patients were interviewed. Participating staff roles included: CHW, certified diabetes care and education specialist (CDCES), pharmacist, endocrinologist, registered nurse, nurse practitioner, medical assistant, and administrator. The majority of patients interviewed were female (56%), Hispanic or Latinx (69%), over 50 years of age (75%), had less than a high school diploma (56%), and living with type 2 diabetes (50%). Clinical data was analyzed for 1,599 Center patients. Two-thirds of those patients were female, the mean age was 51.5 years, and the majority (69%) were English-speaking. Among known race/ethnicity, 54% were Hispanic/Latinx and 40% were Non-Hispanic Black. Nearly half (45%) of patients were on Medicaid, 44% of patients had one or more concordant microvascular comorbidity, 55% had concordant macrovascular comorbidity, and 26% had discordant comorbidity (Table 1 ). The following describes results from the triangulation of qualitative and quantitative analyses. First, we will describe the implementation of the TBC model in practice, including routine care metrics and patient satisfaction, and then we will describe how health care service utilization and health outcomes changed over time Table.

Implementation of a TBC model and patient satisfaction

Based on the analyses of interview data collected from patients and staff, three themes emerged related to TBC implementation and patient satisfaction: (1) patients perceived the TBC model to be patient centered and of high quality; (2) technology has the potential to be an innovative tool; and (3) staff communication can enhance care coordination. Additional illustrative quotes are provided in Table 2 .

Patients perceived the TBC model to be patient centered and of high quality

Patients interviewed described high levels of satisfaction with the TBC model employed by the Center. Patients consistently mentioned feeling important and cared for from their very first appointment at the Center and thereafter. As one patient shared:

“I mean when my first visit in – I saw three people talking to me about the things that they can do for me. And they make me feel like I’m really important… I mean, they made me feel part of the team. They made me feel like I’m really important to them. And that’s – I mean, I have never felt this before with nobody. ” 58-year-old Hispanic patient

Patients reported that having access to a variety of services in one place felt coordinated, and supported patients’ diverse needs. There was a wide range of the number of visits patients had with each provider type, with a mean of 2.81 visits, 0.61 visits, and 3.88 visits with an endocrinologist, community health worker (CHW), or other provider (e.g., pharmacist, diabetes education or dietitian), respectively. During interviews, several patients mentioned Center staff by name and felt that coordination between staff ensured patients had an advocate with insurance companies, access to medication or medical equipment, support to improve diet and nutrition, and were provided a bilingual staff as needed. Specifically, Spanish-speaking patients were grateful to have access to bilingual staff, including a CHW, who could provide clarity on providers’ treatment instructions, help patients understand and navigate health insurance and the cost of healthcare for uninsured individuals, and offer diabetes health education in Spanish.

“La ayuda que siempre me ha brindado [la promotora], ella siempre está atenta, pendiente de ayudarlo a uno, ella me ha ayudado muchísimo, lo que yo no entiendo, lo que yo no puedo hacer, ella hasta me ha ayudado a ir donde uno va para los pagos, se me olvido el nombre.” / “The help that [the CHW] has always given me, she is always attentive, always trying to help me, she has helped me a lot, what I do not understand, what I cannot do, she has even helped me to go where I go for payments.” 50-year-old Hispanic patient

Patients believed the multi-disciplinary TBC model was comprehensive, thorough, and centered around their needs.

Technology has the potential to be an innovative tool

The Center considers the use of technology to be an innovative tool to support patient’s diabetes self-management. For example, continuous glucose monitoring (CGM) is an automated process of tracking blood sugar levels throughout the day using a CGM device. Remote glucose monitoring (RGM) refers to the process of transmitting that information directly from a personal smartphone device to a healthcare provider using a remote data transmitter. Using a mobile application, patients can: document their glucose levels; sync data from a meter, insulin pump, pen or continuous glucose monitoring device; track their weight; log food and meals; access educational resources; track steps; and track blood pressure. Remotely uploading data provides patients with a direct line to their provider for regular feedback on progress and answers to questions between clinic visits. As one staff member describes:

If patients are able to be set up with a continuous glucose monitor they really like it, because it's giving them a lot more data about their blood sugars. Tools like that could be life changing for a patient. Diabetes Center Staff , Registered Nurse

Patients reported feeling more connected to their care and considered this to be a patient-centered tool.

I know what my sugars are now. And that I didn’t know for a long time. They got me a machine where I know my sugar at all times. 50-year-old Black patient

Yet patients also expressed barriers to utilizing technology. Some patients expressed a lack of knowledge about CGM/RGM, or frustrations when it broke. There was a desire to use technology, however patients need high-touch and regular support on how to use the devices, and use the data from the devices. As one patient describes,

“Lo que pasa es que yo no sé si perdí una cita o no me la dieron y ya dejé de ponerme este aparatito porque es para 10 días nada más, ya después ya no me sirve.” / “What happened is that I don't know if I missed an appointment or they didn't give it to me and I stopped using this little device because it is only for 10 days, after that it stopped working for me.” 58-year-old Hispanic patient

Communication enhances care coordination

Center staff reported that information sharing across roles was underpinned by regular and varied modes of communication. Staff indicated that coordination across staff may happen through email, EMR documentation, team meetings, or curbside consultation, where information is shared informally and verbally.

All through EHR [ Electronic Health Record ], because they all document as they go. So when the MA [Medical Assistant] sees them and takes the vitals, she documents everything on EHR and then the endocrinologist does the same thing and then it just goes down. And then as they're rotating in and out, they talk to each other to let each other know what's going on. Diabetes Center Staff , Receptionist

Yet, care coordination was threatened by the lack of knowledge each staff person had for other roles. For example, one staff member had a dual role as a CHW and a retinal specialist. Several staff were aware of his work as a retinal specialist, but were not aware he was a CHW and what services he could provide. This ultimately impacted their ability to deliver the full range of services available within the Center. Further, staff reported different levels of team engagement to discuss cases. When asked if the healthcare team ever meet to make decisions about a patient’s care, an endocrinology stated “No, not mutually. At least not with my patients. I don’t know what anyone else does.” However, when this was asked of a nurse, she replied “Either before or while the patients there. Yeah, if needed we would all coordinate about what could help the patient [with]… the nurse, the doctor, and the pharmacy team.”

Urgent care and clinical outcomes

A fourth theme that emerged from the data is that despite the TBC model and wraparound support, diabetes management is complicated and an ongoing process. Patients discussed the complexities of managing diabetes despite the TBC model and in some cases, poor control was attributed to their lack of self-management. In other cases, patients expressed awareness that despite their adherence to treatment regimens recommended by their clinician, they still did not feel they were able to maintain control. Patients reflected on the challenges of managing stress, maintaining healthy eating habits, and getting enough rest. As one patient shared:

Well, my latest appointment with the nurse, I didn’t do my best for this time around, getting checked. My A1c did go up just a little bit only because it’s been a stressful time for me. My grandma passed away recently, so that was hard for me, and things just got difficult. I started not paying attention to what I was eating. 30-year-old Hispanic patient

Patients reflected on their experience living with diabetes and acknowledged the resources that it takes to properly manage their disease. They identified the need to have a good support system, connecting with others who have “healthy” habits, and even managing challenges.

The challenge in diabetes management is strengthened by the analysis of clinical data. Index data was collected from August 2020 to June 2021, pre-enrollment data went back as far as August 2018, and follow-up data was collected through June 2022. In the two years prior to the Center, 9% of patients had at least one emergency department visit at Mt. Sinai Hospital (mean of 0.07 visits per patient) and 5% were hospitalized at least once at Mt. Sinai Hospital (mean of 0.04 hospitalizations per patient). These percentages were not statistically significantly different at the follow-up period (4% and 2%, respectively). The mean number of missed visits significantly increased from 4.71 at pre-enrollment to 8.54 in the follow-up period ( p  < 0.001). There was a statistically significant decrease in HbA1c from the index to the follow-up period (9.28% to 8.08%, p  < 0.001). Small but statistically significant changes were also seen in BMI (33.16 kg/m 2 to 32.79 kg/m 2 , p  = 0.01) and the percent of patients with elevated blood pressure (64.39% to 60.43%, p  < 0.001) (Tables 3 and 4 ).

The regression analysis focused on four outcome variables: change in HbA1c from index to post-enrollment, the number of endocrinology-related medical visits, number of missed visits, and percent of patients with at least one instance of healthcare utilization. After controlling for sex, we found that each provider visit during the follow-up period was associated with an HbA1c decrease of 0.26 points from index to post-enrollment ( p  < 0.001). Urgent health care utilization during the follow-up period was associated with an additional 3.1 missed visits during the follow-up period compared to patients with no urgent health care utilization, after controlling for concordant microvascular comorbidities, discordant comorbidities, and HbA1c at the index visit ( p  < 0.001). In a logistic regression model, we found a 26% increase in the odds of a hospitalization or emergency department visit for every 1.0% increase in HbA1c at the index visit compared to their pre-enrollment HbA1c, after controlling for concordant microvascular comorbidities and age ( p  < 0.001) (see Table  4 ).

Diabetes is the seventh leading cause of death in the United States, and is associated with heart disease, vision loss, kidney disease, and even death [ 31 ]. Evidence-based interventions and tailored approaches to implementation are critical to appropriately address the high rates of diabetes and prediabetes, and equitably provide care to patients. The present study offers important findings from a process and outcome evaluation of a multi-disciplinary TBC model for patients living with diabetes. This is particularly important for low-income and racially diverse patients with co-occurring comorbidities who, like other patients served by health systems in the U.S. safety net, often have more complex needs that require the additional social support and wraparound services provided at the Center. For instance, these patients are more likely to have competing priorities or child-care or transportation needs that prevent them from attending multiple appointments on different days [ 32 ].

The triangulation of key findings from staff and patient interviews, and clinical data describe the implementation of the TBC model at the Center and illuminate the components of the model that satisfy patient needs and preferences, facilitate coordination of care, and even components that require adjustments in protocol or approach. Patients expressed high levels of satisfaction with the quality of care they received, noting attentive staff and noticeable coordination among clinical staff regarding their care. This is aligned with other research of TBC models, in which slight increases in patient satisfaction were documented [ 33 ].

Some research suggests that patients who regularly use CGM/RGM technology have better clinical outcomes than patients who are not actively engaged [ 34 ]. Our findings support the perceived benefits of CGM/RGM, but also identified barriers during real-world implementation that must be addressed such as insurance coverage, knowledge around utilization, resources for troubleshooting, and access to smart phones for ideal integration. We found that with high-touch support from staff, these barriers can be overcome.

We hypothesized that by increasing the availability of telehealth appointments, prompted by the emergence of the COVID-19 pandemic, along with the TBC model, we would see a decrease in missed appointments. Counter to this hypothesis, missed appointments increased from pre to post TBC implementation. Existing research of TBC models have not explored the impact of this model on missed visits. The Center and its TBC model were launched at the onset of the COVID-19 pandemic which may have necessitated patients’ prioritization of other more urgent issues such as childcare, food accessibility and other essential daily needs. It’s also possible that during the transition, there was confusion about how to schedule appointments with new providers (i.e., pharmacists, CHWs) or where to go to meet them. However, we also hypothesized that uncontrolled A1c at the index visit would lead to an increased likelihood in attended endocrinology-related visits, which was confirmed. This may be because patients had newfound access to additional resources within the clinic (i.e., appointments with a diabetes educator, pharmacist, etc.).

The TBC model evaluated in the present study included several non-clinical staff members such as CDCES and CHWs. Multi-disciplinary teams that include both clinical and non-clinical staff have been shown to improve patient outcomes and improve social determinants of health [ 35 ]. Interventions with CHWs as non-clinical staff, specifically, have been recommended based on improved outcomes in diabetes and cardiovascular management, among other areas [ 36 ]. CHWs integrated within traditional clinical care teams can address the complex health-related social needs within the individual and community context of the patients, which we found can serve as barriers to proper diabetes self-management, and are not typically within the scope of clinical providers. As of July 2022, 29 US states reported that they allow payments through Medicaid for CHW-related services, but further work is needed to effectively integrate, supervise, and sustain CHW efforts within the healthcare system and community settings at large [ 37 ].

Implementation science frameworks, such as CFIR, provide a guide for how to measure, and even improve, the integration of promising interventions into routine practice [ 38 ]. The complexities of systemic racism, organizational infrastructure, and unique patient needs require public health professionals and clinicians to think differently about impactful interventions for patients and the strategies used to implement these interventions. Our study used the CFIR framework to systematically assess and understand multilevel factors that may influence the implementation of, and outcomes related to a TBC model. To strengthen our methodology, we paired the implementation science approach with participatory methods. Studies that include stakeholders in the research process, often have high-quality processes and outcomes, developed capacity and competency of stakeholders, and higher likelihood of sustainability [ 39 , 40 ].

While our study provides evidence for how TBC can impact patient outcomes, we did not examine TBC sustainability or maintenance, particularly as it relates to cost effectiveness. The US traditionally uses a fee-for-service payment model, but this model been criticized for incentivizing providers to perform more procedures rather than focus on outcomes. Other payment models have been tested such as accountable care organizations, bundled payments, and patient-centered medical homes. A systematic review from 2023 found that TBC for blood pressure control was cost effective and had a significant impact on the quality adjusted life years of patients. However, this review also found that compared to other high-income countries, the US intervention costs were higher and varied by composition of the care team and racial/ethnic makeup of the patient population. Future research should examine the degree to which a TBC model for low-income communities of color living with diabetes has the potential for long-term cost savings.

There are several strengths to this study. First, our partnership with the PAC ensured that our data collection tools, recruitment approaches, analyses, and interpretation were guided by lived experience. Their meaningful participation improved the quality of the study. Second, because the Center was opened within an existing patient population, this presented a unique opportunity to capture data on the same patients at three different time points: before the opening of the Center, at an index visit, and at a one-year follow up. Third, our triangulation of qualitative and quantitative data allows us to more fully understand the impact of the TBC model on Sinai’s patient population.

There are also important limitations that must be noted. First, emergency department and hospitalization data was only available for Mt. Sinai Hospital. If patients visited an emergency department or were admitted at another hospital, this data was not included in our analysis. Second, nearly half of race/ethnicity data was missing from our EMR data. Third, it was not possible to easily identify unique patient meetings with a pharmacist, diabetes educator, or retinal specialist. These meetings were only documented as a qualitative note in the medical record and could not be easily extracted. It was also not possible to isolate endocrine-related visits across the same study population prior to the implementation of the TBC model. Fourth, this study began just six months after the U.S. declared COVID-19 to be a public health emergency, as a result, patient volume was initially lower than expected and took several months to increase. Finally, because this was an observational study, we are not able to conclude these findings are solely attributed to the effects of the TBC model. They may be due to unmeasured factors such as implementation of telemedicine, the ongoing effects of the COVID-19 pandemic, changes in staff, etc.

In summary, a multi-disciplinary TBC model can be an appropriate and effective model to improve health outcomes for patients living with diabetes in an urban setting with complex needs. Patients perceived this to be a patient-centered model, and improved patient outcomes were documented compared to prior to the implementation of this model. Specifically, we found a reduction in HbA1, weight, BMI, and proportion of patients with elevated blood pressure. Improvements can be made by ensuring clear understanding of staff roles. These findings suggest that despite integrating clinical and social service providers in one space, a TBC model is not a panacea for addressing complex chronic disease and social needs for patients from low-income communities. Policy changes such as expanded health insurance, adjustments to the healthcare payment system, and additional funding for national and local efforts to address social determinants of health are required for broad change. As of the writing of this manuscript, the TBC model of care is still being offered at Sinai Chicago, and research continues using advanced study designs to measure causal effects.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

United States

Centers for Disease Control and Prevention

• Team-based care

People of color

Consolidated Framework for Implementation Research

Community-based participatory

Mount Sinai Hospital

Patient Advisory Committee

Electronic medical record

Community health worker

Body mass index

Systolic blood pressure

Diastolic blood pressure

Diabetes care and education specialist

Continuous glucose monitoring

Remote glucose monitoring

Hemoglobin A1c

Centers for Disease Control and Prevention. National diabetes statistics report website. https://www.cdc.gov/diabetes/data/statistics-report/index.html . Accessed 20 Dec 2023.

Centers for Disease Control and Prevention. Prediabetes – your chance to prevent type 2 diabetes. 2022.  https://www.cdc.gov/diabetes/basics/prediabetes.html . Accessed 6 Feb 2023.

Riddle MC, Herman WH. The cost of diabetes care—an elephant in the room. Diabetes Care. 2018;41(5):929–32. https://doi.org/10.2337/dci18-0012 .

Article   PubMed   Google Scholar  

American Diabetes Association. Health insurance aid. 2023. https://diabetes.org/tools-support/health-insurance . Accessed 8 Feb 2023.

Buescher PA, Whitmire JT, Pullen-Smith B. Medical care costs for diabetes associated with health disparities among adult Medicaid enrollees in North Carolina. N C Med J. 2010;71(4):319–24.

PubMed   Google Scholar  

Decker SL, Kostova D, Kenney GM, Long SK. Health status, risk factors, and medical conditions among persons enrolled in Medicaid vs uninsured low-income adults potentially eligible for Medicaid under the Affordable Care Act. JAMA. 2013;309(24):2579–86. https://doi.org/10.1001/jama.2013.7106 .

Article   CAS   PubMed   Google Scholar  

Chapel JM, Ritchey MD, Zhang D, Wang G. Prevalence and medical costs of chronic diseases among adult medicaid beneficiaries. Am J Prev Med. 2017;53(6s2):S143-s154. https://doi.org/10.1016/j.amepre.2017.07.019 .

Article   PubMed   PubMed Central   Google Scholar  

Illinois Department of Public Health. Diabetes in Illinois. 2021. https://dph.illinois.gov/topics-services/diseases-and-conditions/diabetes.html . Accessed 8 Feb 2023.

Mitchell P, M Wynia, R Golden, B McNellis, S Okun, CE Webb, V Rohrbach, I Von Kohorn. Core principles & values of effective team-based health care. Washington, DC: Discussion Paper, Institute of Medicine; 2012. www.iom.edu/tbc .

Book   Google Scholar  

Team-based care to improve type 2 diabetes management: recommendation of the community preventive services task force. Am J Prev Med. 2019;57(1):e27-e29. https://doi.org/10.1016/j.amepre.2019.02.005 .

Levengood TW, Peng Y, Xiong KZ, et al. Team-based care to improve diabetes management: a community guide meta-analysis. Am J Prev Med. 2019;57(1):e17–26. https://doi.org/10.1016/j.amepre.2019.02.005 .

Tricco AC, Ivers NM, Grimshaw JM, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet. 2012;379(9833):2252–61. https://doi.org/10.1016/S0140-6736(12)60480-2 .

Gary TL, Bone LR, Hill MN, et al. Randomized controlled trial of the effects of nurse case manager and community health worker interventions on risk factors for diabetes-related complications in urban African Americans. Prev Med. 2003;37(1):23–32. https://doi.org/10.1016/s0091-7435(03)00040-9 .

Pape GA, Hunt JS, Butler KL, et al. Team-based care approach to cholesterol management in diabetes mellitus: two-year cluster randomized controlled trial. Arch Intern Med. 2011;171(16):1480–6. https://doi.org/10.1001/archinternmed.2011.417 .

Al Mazroui NR, Kamal MM, Ghabash NM, Yacout TA, Kole PL, McElnay JC. Influence of pharmaceutical care on health outcomes in patients with Type 2 diabetes mellitus. Br J Clin Pharmacol. 2009;67(5):547–57. https://doi.org/10.1111/j.1365-2125.2009.03391.x .

Article   CAS   PubMed   PubMed Central   Google Scholar  

Proia KK, Thota AB, Njie GJ, et al. Team-based care and improved blood pressure control: a community guide systematic review. Am J Prev Med. 2014;47(1):86–99. https://doi.org/10.1016/j.amepre.2014.03.004 .

Chicago Health Atlas. Adult diabetes rate. 2023. https://chicagohealthatlas.org . Accessed 8 Feb 2023.

Sinai Health System. Mount Sinai hospital community health needs assessment. 2022. https://www.sinaichicago.org/wp-content/uploads/2022/10/Sinai-Chicago-2022-CHNA_Mount-Sinai-Hospital.pdf . Accessed 8 Feb 2023.

Sinai Urban Health Institute. Sinai community health survey 2.0 health snapshots. 2017. https://www.sinaisurvey.org/health-snapshots . Accessed 8 Feb 2023.

Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48(6 Pt 2):2134–56. https://doi.org/10.1111/1475-6773.12117 .

Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation Sci. 2009;4:50. https://doi.org/10.1186/1748-5908-4-50 .

Article   Google Scholar  

Skolarus TA, Lehmann T, Tabak RG, Harris J, Lecy J, Sales AE. Assessing citation networks for dissemination and implementation research frameworks. Implementation Sci. 2017;12(1):97.

Springer MV, Skolarus LE. Community-based participatory research. Stroke. 2019;50(3):e48–50. https://doi.org/10.1161/STROKEAHA.118.024241 .

Monnard K, Benjamins MR, Hirschtick JL, Castro M, Roesch PT. Co-creation of knowledge: a community-based approach to multilevel dissemination of health information. Health Promot Pract. 2021;22(2):215–23. https://doi.org/10.1177/1524839919865228 . Epub 2019 Aug 30 PMID: 31470741.

Israel BA, Schulz AJ, Parker EA, Becker AB. Review of community-based research: assessing partnership approaches to improve public health. Annu Rev Public Health. 1998;19:173–202. https://doi.org/10.1146/annurev.publhealth.19.1.173 .

Silverman D. A guide to the principles of qualitative research. London: Sage; 2011.

Google Scholar  

Kvale S, Brinkman S. Interview quality. Interviews: Learning the craft of qualitative research interviewing. 2009. p. 161–75.

Chapman AL, Hadfield M, Chapman CJ. Qualitative research in healthcare: an introduction to grounded theory using thematic analysis. J R Coll Physicians Edinb. 2015;45(3):201–5. https://doi.org/10.4997/JRCPE.2015.305 . PMID: 26517098.

Whelton PK, Carey RM, Mancia G, Kreutz R, Bundy JD, Williams B. Harmonization of the American College of Cardiology/American Heart Association and European Society of Cardiology/European Society of Hypertension Blood Pressure/Hypertension Guidelines: Comparisons, Reflections, and Recommendations. Circulation. 2022;146(11):868–77. https://doi.org/10.1161/CIRCULATIONAHA.121.054602 .

Kerr EA, Heisler M, Krein SL, et al. Beyond comorbidity counts: how do comorbidity type and severity influence diabetes patients’ treatment priorities and self-management? J Gen Intern Med. 2007;22(12):1635–40. https://doi.org/10.1007/s11606-007-0313-2 .

Centers for Disease Control and Prevention. Diabetes fast facts. 2022. https://www.cdc.gov/diabetes/basics/quick-facts.html . Accessed 27 Feb 2023.

Alkomos MF, Mendez D, Mazzei-Pifano D, Shafeek F, Rodriguez C, Ali F, Banks C, Melki G, Michael P. Patients’ reasons for missing scheduled clinic appointments and their solutions at a major urban-based academic medical center. J Community Hosp Intern Med Perspect. 2020;10(5):426–30. https://doi.org/10.1080/20009666.2020.1796903 . PMID:33235676;PMCID:PMC7671744.

Litaker D, Mion L, Planavsky L, Kippes C, Mehta N, Frolkis J. Physician−nurse practitioner teams in chronic disease management: the impact on costs, clinical effectiveness, and patients’ perception of care. J Interprof Care. 2003;17(3):223–37. https://doi.org/10.1080/1356182031000122852 .

Jethwani K, Ling E, Mohammed M, Myint-U K, Pelletier A, Kvedar JC. Diabetes connect: an evaluation of patient adoption and engagement in a web-based remote glucose monitoring program. J Diabetes Sci Technol. 2012;6(6):1328–36. https://doi.org/10.1177/193229681200600611 . Published 2012 Nov 1.

Epstein NE. Multidisciplinary in-hospital teams improve patient outcomes: A review. Surg Neurol Int. 2014;5(Suppl 7):S295–303. https://doi.org/10.4103/2152-7806.139612 . Published 2014 Aug 28.

Guide to Community Preventive Services. Community health workers. 2021. https://www.thecommunityguide.org/pages/community-health-workers.html . Accessed 28 Feb 2023.

Haldar S, Hinton E. State Policies for Expanding Medicaid Coverage of Community Health Worker (CHW) services. 2023. https://www.kff.org/medicaid/issue-brief/state-policies-for-expanding-medicaid-coverage-of-community-health-worker-chw-services/ . Accessed 24 Feb 2023.

Glasgow RE, Emmons KM. How can we increase translation of research into practice? Types of evidence needed. Annu Rev Public Health. 2007;28:413–33. https://doi.org/10.1146/annurev.publhealth.28.021406.144145 .

Jagosh J, Macaulay AC, Pluye P, et al. Uncovering the benefits of participatory research: implications of a realist review for health research and practice. Milbank Q. 2012;90(2):311–46. https://doi.org/10.1111/j.1468-0009.2012.00665.x .

Ramanadhan S, Davis MM, Armstrong R, et al. Participatory implementation science to increase the impact of evidence-based cancer prevention and control. Cancer Causes Control. 2018;29(3):363–9. https://doi.org/10.1007/s10552-018-1008-1 .

Download references

Acknowledgements

The research team would like to thank the Models of Care Patient Advisory Committee, Sinai Center for Diabetes and Endocrinology staff, and patients who provided their time and deep insight to enrich this study.

Support for this publication was provided by a grant from the Robert Wood Johnson Foundation (#78233).

Author information

Authors and affiliations.

Sinai Urban Health Institute, 1500 South Fairfield Avenue, Chicago, IL, 60608, USA

Jacquelyn Jacobs, Alyn Dougherty, Banita McCarn, Nazia S. Saiyed, Stacy Ignoffo, Cindy San Miguel & Linda Martinez

Sinai Chicago, 1500 South Fairfield Avenue, Chicago, IL, 60608, USA

Christina Wagener

You can also search for this author in PubMed   Google Scholar

Contributions

J.J.: Conceptualization, Methodology, Formal Analysis, Interpretation, Writing—Original Draft, Supervision, Funding acquisition; A.D.: Data Curation, Formal Analysis, Writing- Original Draft; B.M.: Data Curation, Methodology, Formal Analysis; N.S.: Methodology, Formal analysis, Writing – Review & Editing; S.I.: Funding acquisition, Supervision, Writing – Review & Editing; C.W.: Funding acquisition, Supervision, Writing – Review & Editing; C.S.M.: Formal Analysis, Writing – Review & Editing; L.M.: Formal Analysis, Writing – Review & Editing.

Corresponding author

Correspondence to Jacquelyn Jacobs .

Ethics declarations

Ethics approval and consent to participate.

Mount Sinai Hospital Institutional Review Board approved the study protocol (#21–37). Research staff obtained consent from all patient and staff participants prior to qualitative data collection. They were informed that participation was voluntary and they were able to withdraw at any time. Due to the retrospective nature of the quantitative analysis of clinical data, the study team received a Health Insurance Portability and Accountability Act (HIPAA) waiver of informed consent, under 45cfr46.116(d) of the United States Department of Health and Human Services regulations for protection of human subjects in research.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1. , rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Jacobs, J., Dougherty, A., McCarn, B. et al. Impact of a multi-disciplinary team-based care model for patients living with diabetes on health outcomes: a mixed-methods study. BMC Health Serv Res 24 , 746 (2024). https://doi.org/10.1186/s12913-024-11062-4

Download citation

Received : 17 January 2024

Accepted : 30 April 2024

Published : 18 June 2024

DOI : https://doi.org/10.1186/s12913-024-11062-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Multi-disciplinary team
• Implementation science
• Participatory research
• Endocrinology

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Research Methodology and IPR VTU Question Papers 1st sem CWM 2020 scheme

MTech 20RMI17 Question Paper

Waste Water?Management Health & Safety Engineering

Branch : CWM

1st Semester

Post Graduate Engineering Course

VTU Research Methodology and IPR FEB 2021 Question Paper

Last updated: monday, february 13, 2023.

• VTU Time Table New
• VTU Model Question Papers New
• VTU Exam Form
• VTU Syllabus Updated
• VTU Results
• VTU Question Bank

RNAfold: RNA tertiary structure prediction using variational autoencoder.

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• For correspondence: [email protected]
• Info/History
• Preview PDF

Understanding the RNA tertiary structure is crucial for advancing therapeutic development and vaccine design. Previous methods, such as dynamic simulations, have been used to study RNA structure. However, these methods often struggle to capture the complex non-linear relationships in RNA sequences. Other approaches focused on simple models, resulting in lower performance. Indeed, one of the main challenges in predicting RNA tertiary structure is the lack of data, and using complicated algorithms is limited. In this paper, we introduce the Variational Autoencoder (VAE). In this paper, we introduce the Variational Autoencoder (VAE) for the problem of RNA folding and demonstrate that the Variational Autoencoder could yield promising results. Our model achieves a Root Mean Square Error (RMSE) of approximately 3.3 Angstrom in predicting nucleotide positions, significantly outperforming existing approaches.

Competing Interest Statement

The authors have declared no competing interest.

https://github.com/quantori/RNAfold

View the discussion thread.

Thank you for your interest in spreading the word about bioRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One

Subject Area

• Bioinformatics
• Animal Behavior and Cognition (5394)
• Biochemistry (12174)
• Bioengineering (9121)
• Bioinformatics (30078)
• Biophysics (15444)
• Cancer Biology (12558)
• Cell Biology (18014)
• Clinical Trials (138)
• Developmental Biology (9730)
• Ecology (14574)
• Epidemiology (2067)
• Evolutionary Biology (18738)
• Genetics (12529)
• Genomics (17187)
• Immunology (12286)
• Microbiology (28966)
• Molecular Biology (12006)
• Neuroscience (63062)
• Paleontology (462)
• Pathology (1931)
• Pharmacology and Toxicology (3361)
• Physiology (5173)
• Plant Biology (10777)
• Scientific Communication and Education (1706)
• Synthetic Biology (2996)
• Systems Biology (7528)
• Zoology (1690)