problem solving techniques delphi

• Memberships

Delphi Technique explained

Delphi Technique: this article describes the theory and concept of the Delphi Technique , developed by RAND Corporation in a practical way. Next to what this research technique is, this article also highlights the approach, the definition and elemenst, how to apply and a practical Delphi Technique example. After reading you will understand the basics of this powerful decision making and problem solving tool. Enjoy reading!

What is Delphi Technique?

The Delphi Technique, or Delphi studies, started out as a technique to predict the future. It was developed by the RAND Corporation in 1950, as for the US air force.

The RAND Corporation is an American think-tank that was founded in 1946 by the US air force. A few years later, it became an independent non-profit. Originally, the Delphi Technique was aimed at predicting the impact of technology on warfare.

For this decision making method, a group of experts (expert consensus) are asked to anonymously answer a survey and provide feedback on each other’s answers. This process repeats itself. The aim is to come up with concrete solutions.

Approach of the Delphi Process

The Delphi Technique uses a carefully designed list of questions that is developed and refined in several rounds. Sometimes, interviews are used.

The experts in question answer the questionnaire anonymously and are then confronted with each other’s knowledge and views. They do not know each other and do not know who else is in the panel of experts. These expert opinions are very valuable in the process.

There is a feedback mechanism; obtained information is recorded and reported to participants regularly, as part of a new attempt at information processing. The feedback is not just about the actual future expectations that are being expressed, but also the arguments and considerations that underpin these expectations. In this way, you can work towards a well-founded joint position.

By confronting the participants with each other’s ideas and insights in each new round, you can work towards a consensus. In this way, the Delphi Method asks experts for their recommendations.

The number of participants is not relevant. However, there must be the assurance that different interests and problem definitions are addressed, so that a problem or topic is viewed from different perspectives and discussed, and all participants respect each other’s contributions.

Figure 1 – The Delphi Technique Process

Delphi Technique definition and elements

The Delphi Technique is part of the research technique that is also known as ‘interactive survey’, which involves dialogue with the public (experimental application). It is based on an articulate research field, which in principle is capable of introducing the necessary expertise. It aims for a solution that consists of knowledge from the research field, rather than knowledge about it. Important elements within the Delphi method are:

• It uses the opinions of a limited group of experts or people who have knowledge of the subject from their experience.
• The subjects that an organisation wants to understand through the Delphi method are obtained by subjective means; the opinions of the group of experts are therefore very important.
• The group of experts works towards a consensus, through step-by-step feedback of information, by repeating the question and providing feedback on each other’s answers.

Application of delphi technique

The possible applications of Delphi are endless. It is not only applicable for making forecasts or future forecasts, but it can also be useful for determining complicated policy decisions. In principle, the Delphi Technique can be properly applied for the following situations:

• For an organisation in which the formal and informal communication lines have become blocked or are lacking. The anonymous method prevents personal contact, which means that stuck relationships do not get damaged any further and there is more room for new solutions and more respect for each other’s opinions.
• In the case of problems that can be solved mainly when multiple experts are confronted with each other’s varying visions. Everyone’s individual subjective opinion contributes, so that their shared thinking will lead to an effective approach.
• In the case of solutions that are too subject to individual opinions and perceptions of those involved, it may help to work with an anonymous group who independently express their views on the problem.
• In the absence of objective data, such as customer reviews and other audience reviews. When the opinion of all these groups is important in a specific problem solving, the Delphi Method works very well.

The great advantage of the Delphi Technique is the anonymity. It is not shared who the participants are or who provided the information. This prevents distortion that can sometimes occur when participants conform to the opinion of an expert who has very high prestige.

What is a Delphi technique example?

Suppose there is a problem in a municipality regarding informing residents in different districts. If the Delphi Technique were to be used for this problem, this municipality would initially appoint a project leader and gather a group of experts involved in this problem statement. With the Delphi Method, the experts in the field of communication to citizens are systematically used to address this problem.

In the first round of the method, participants will fill in a questionnaire, with concrete questions about the problem statement. The project leader then compiles a summary of all completed questionnaires, which is then returned to all participants. By sharing all the views and relevant information, participants will be better able to reflect on their own views.

In the second round, participants re-enter the questionnaire. After each round, experts’ answers will become more similar without affecting each other. Of course, their own opinions and ideas are subjective and based on everyone’s individual experience or knowledge.

The experts are people working in research fields that directly relate to communication between residents and municipalities.

For example, consider a representative from every neighbourhood, a communications officer from the municipality, a communications agency employee and a local newspaper representative.

The query and feedback process is repeated until a consensus is reached between the participants.

Finally, by combining the expertise of the experts, recommendations can be formulated based on the more complete information, resulting in high-quality and useful recommendations.

Brainstorming and Delphi Technique

The Delphi Technique is often compared to brainstorming .

These is one essential difference between the Delphi Technique and the brainstorm technique : with the Delphi Technique, there is no direct group interaction. The participants do not engage in conversation with each other, cannot discuss with each other and do not build on each other’s ideas, as is done in brainstorming.

It’s Your Turn

What do you think? Is the Delphi Technique applicable in today’s modern business and team challenges? Do you recognize the practical explanation of the Delphi oracle or do you have more suggestions? What are your success factors for a good decision making process?

Share your experience and knowledge in the comments box below.

More information

• Dalkey, N. C., Brown, B. B., & Cochran, S. (1969). The Delphi method: An experimental study of group opinion (Vol. 3) . Santa Monica, CA: Rand Corporation.
• Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications (Vol. 29) . Reading, MA: Addison-Wesley.
• Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International journal of forecasting, 15(4), 353-375.

How to cite this article: Mulder, P. (2017). Delphi Technique . Retrieved [insert date] from Toolshero: https://www.toolshero.com/decision-making/delphi-technique/

Original publication date: 04/04/2018 | Last update: 08/31/2023

Add a link to this page on your website: <a href=”https://www.toolshero.com/decision-making/delphi-technique/”>Toolshero: Delphi Technique</a>

Did you find this article interesting?

Your rating is more than welcome or share this article via Social media!

Average rating 4.6 / 5. Vote count: 5

No votes so far! Be the first to rate this post.

We are sorry that this post was not useful for you!

Let us improve this post!

Tell us how we can improve this post?

Patty Mulder

Patty Mulder is an Dutch expert on Management Skills, Personal Effectiveness and Business Communication. She is also a Content writer, Business Coach and Company Trainer and lives in the Netherlands (Europe). Note: all her articles are written in Dutch and we translated her articles to English!

Related ARTICLES

Simplex Problem Solving Process by Marino Basadur

Six Thinking Hats Technique explained

Brainstorming method

Value Stream Mapping (VSM)

Consensus Oriented Decision Making Model by Tim Hartnett

Vroom Yetton Jago Decision Model

Also interesting.

Decision Tree Analysis: the Theory and an Example

What If Analysis: the Basics and an Example

Borda Count Method: Basics and an Example

One response to “delphi technique explained”.

The Delphi technique seems like one of the most efficient and non-organized ways of coming up with a direct solution. Some of the reasons why it wouldn’t be such a good idea to use the Delphi technique would be the ego of all the smart people. Basically, since all the geniuses think on their own and have different morals or ethics they will be subjected to think that only their opinion matters. Hence, they will be influenced by themselves to disagree with every opinion that isn’t theirs. This would most likely result to no solid solution to the problem which either some or all the smart people to come to a disagreement thus disapproving the Delphi technique. Not only would the experts believe that they are only right base don ego but by the fact that the opinions are anonymous so of course they would be biased towards their own word since they don’t know if the other person is an expert too. But on the other hand, if that doesn’t happen it would be very useful to use the Delphi technique. The experts can learn from each other mistakes and when agreeing on only one solution then they can use that exact solution on the specific problem going on in their lives. Not only that but there would be a guarantee of bias towards another’s opinion since all of them would be anonymous to each other. Ergo, the technique would be a guarantee way to solve an issue.

Leave a Reply Cancel reply

You must be logged in to post a comment.

BOOST YOUR SKILLS

Toolshero supports people worldwide ( 10+ million visitors from 100+ countries ) to empower themselves through an easily accessible and high-quality learning platform for personal and professional development.

By making access to scientific knowledge simple and affordable, self-development becomes attainable for everyone, including you! Join our learning platform and boost your skills with Toolshero.

POPULAR TOPICS

• Change Management
• Marketing Theories
• Problem Solving Theories
• Psychology Theories

ABOUT TOOLSHERO

• Free Toolshero e-book
• Memberships & Pricing

• Media Center

The Delphi Method

The basic idea, theory, meet practice.

TDL is an applied research consultancy. In our work, we leverage the insights of diverse fields—from psychology and economics to machine learning and behavioral data science—to sculpt targeted solutions to nuanced problems.

The Delphi method is a technique used in group decision-making and some forms of qualitative research. It involves gathering a panel of experts, having them complete a survey or questionnaire individually, and sharing these anonymised answers within the panel to allow for feedback and debate. Each expert is presented with the questions again, and the process is repeated. It is expected that all opinions will eventually converge around a general consensus.

The Delphi method has been used extensively in forecasting, especially in business and technology. It’s also a commonly used tool in public policy: when policymakers use panels of experts to inform decisions around issues like healthcare, education and climate change, they frequently make use of the Delphi method.

Where all think alike, no one thinks very much. – Walter Lippman

Delphi Method  – A research/decision-making technique in which individual experts give their opinion on a particular subject, then respond to the opinions of others.  Afterward, they evaluate their decision again before presenting it for the final time. In most cases, an alignment of opinions arises surrounding a general consensus.

Wisdom of Crowds  – The theory that decisions made by groups are usually of better quality than those made by individuals.

Consensus  – A general agreement reached by members of a group.

Although the name was inspired by the ‘Oracle of Delphi’, a high priestess of Ancient Greece, The Delphi method has its roots in military warfare. The method was developed in the United States at the onset of the Cold War, as a way to predict the role that technology would play in future combative events. 1  So, unlike the vast majority of theories and methods we encounter in behavioral science, the Delphi method didn’t really come about by way of academic research.

In 1944, General Henry Arnold commissioned a report for the United States Air Corps on technological capabilities that might be deployed by the military in the future. After much trial-and-error of conventional approaches to forecasting, including quantitative models and trend extrapolation, it became apparent that a novel technique was required when forecasting situations that involve yet-to-be-determined parameters. As a result, the American public policy think-tank RAND Corporation, led by Norman Dalkey and Olaf Helmer, developed the Delphi method. In its early applications, the technique was used to investigate the probability and potential effects of future attacks on the United States. Experts made estimates, discussed them, and then estimated again, in a process that would be known as ‘Estimate-Talk-Estimate’. The idea was that opinions would eventually start to converge around the same repeated estimations.

Since then, the Delphi method has been deployed in a wide range of settings, including business, government, medicine, and science. 2  While there is great variation in how the technique is used, the general structure of Estimate-Talk-Estimate defines the Delphi method. In policy-making, the Policy Delphi 3  is used to generate the most divergent political views on how a major policy issue should be addressed. It’s also been influential in the development of direct democracy and stakeholder engagement, as policymakers increasingly try to involve a wide range of experts in their decision-making.

That said, the Delphi method was not the first technique to advocate for the leveraging of groups in decision-making. Scientists and mathematicians had long observed the benefits of using groups instead of individuals to make decisions. In 1907, Francis Galton (a cousin of Charles Darwin) observed how the average of all the entries in a county fair ‘guess the weight of the ox’ competition proved incredibly accurate – more so than the guesses of most individuals, even farmers and so-called cattle experts. 4  The concept was developed and is known today  as the  Wisdom of Crowds.

The organisation behind the Delphi method. RAND is a public policy think tank headquartered in Santa Monica, California. Norman Dalkey and Olaf Helmer led the Delphi method project within RAND, initially with the aim of fostering collaboration amongst military experts.

James Surowiecki

American journalist and former staff writer at  The   New Yorker . Surowiecki is best known for his 2004 book entitled  The wisdom of crowds: why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations . He argues that in most circumstances, large groups show more intelligence and generate better decisions than individuals.

Consequences

Once declassified by the US government, the Delphi method rose in prominence and was adopted by several fields. By 1980, a range of papers and books on the subject had been published, especially in the social sciences and public policy.

In recent years, the Delphi method has developed from its initial application as a forecasting tool, to wider use in qualitative research and stakeholder consultation. Scaled-down formats appear in business settings today; for example, during face-to-face meetings during which participants give initial opinions, offer feedback to the group, and then re-confirm their final viewpoints. This variation is sometimes used as an alternative to brainstorming in strategy meetings.

Other variations include the Policy Delphi 3 , which seeks to consult the most extreme opinions around an important public issue, giving governments an idea of some arguments they need to consider when drafting new policy. An example of this is the ‘Citizen Assembly’ consultation tool that’s been successfully deployed in countries like Canada, Ireland and the Netherlands, amongst others. 6

Finally, the Argument Delphi 7  is more process-driven, used to foster continued discussion and evaluate relevant arguments surrounding a topic, instead of aiming for consensus. Through its many variations and use in more and more fields, the Delphi method remains more influential now than ever.

Controversies

The main argument against the Delphi method is that it strips group-decision making of the group context. Real-time discussion and debate doesn’t happen because experts give their opinions and receive feedback without interacting with the other members of the panel. The influence of contextual factors that come with being in a group – for example, social pressure -is not as prominent  in the Delphi method as in conventional group decision-making (committees or meetings). Experts don’t get the opportunity to debate with one another because the Delphi method calls for ‘controlled feedback’, which is generally not interactive. 8

Another problem arises when participants either misunderstand the questions being asked, or the items used in a questionnaire invoke a variety of interpretations. Open-ended questions are a particular challenge here, because they make it difficult for facilitators to identify patterns in opinions. Challenges also arise when the ‘experts’ involved possess varying levels of expertise. Unfortunately, the Delphi method does not account for psychological forces such as the imposter syndrome, or the role of cognitive  biases .

Case Studies

United states air corps: cold war scenario forecasting.

As we’ve mentioned, the Delphi method was initially developed for the US Air Corps in an effort to predict the likelihood of Soviet attacks during the Cold War. Dalkey, Helmer and their team at RAND Corporation (known as Project RAND at the time) sought to use expert advice to predict how the Soviets would select and attack US military targets. They were specifically looking for estimations of the number of atomic missiles that the Soviets would be expected to use in their attacks, and the impact these would have on US resources. In essence, they were trying to predict how Soviet military strategists would act towards the US in the future.

To do this, the team at RAND Corporation  involved a range of experts and collected their estimations through what became the Delphi method. Their goal was to  “collect the most reliable consensus of opinion of a group of experts by a series of intensive questionnaires interspersed with controlled opinion feedback.’ ’ 2

Healthcare – Clinical Guidelines

The Delphi method has proven to be particularly useful in the development of clinical guidelines for certain diseases and illnesses. What’s especially beneficial is that it allows for a broad range of expert opinions to be considered; in fact, most clinical research has evolved to extend this ‘expert’ title to patients, caregivers, (e.g. parents) and patient advocacy groups. As Natalie Street from the Centers for Disease Control and Prevention observes,  “Clinicians are experts, but the patients with the disease are also experts. They’re the ones living with it on a daily basis.”   9  The anonymity of the Delphi method allows patients to share their voice, without fear of being challenged, or at worst, ridiculed, by medical experts. Patients and caregivers  can offer their opinions in a controlled environment, and the chance of the  Reactive Devaluation Bias  occurring is much lower. Finally, the very fact that patients are involved in the consultation phase greatly increases the likelihood that they will follow the guidelines themselves (see:  Ikea effect . )

Related TDL resources

Group Decision Making: How to Be Effective and Efficient

Yasmine Kalkstein  discusses how groups can avoid groupthink and make meetings more effective and efficient.

COVID-19 and the Science of Risk Perception

How the ‘Wisdom of Crowds’ can improve fact-checking around COVID-19 risk.

• RAND Corp. (2021). Delphi Method. Retrieved 11 February 2021, from  https://www.rand.org/topics/delphi-method.html
• Gupta, U. G., & Clarke, R. E. (1996). Theory and applications of the Delphi technique: A bibliography (1975–1994).  Technological forecasting and social change ,  53 (2), 185-211.
• Franklin, K. K., & Hart, J. K. (2007). Idea generation and exploration: Benefits and limitations of the policy Delphi research method.  Innovative Higher Education ,  31 (4), 237-246.
• The Real Wisdom of the Crowds. (2021). Retrieved 11 February 2021, from https://www.nationalgeographic.com/science/phenomena/2013/01/31/the-real-wisdom-of-the-crowds/
• Surowiecki, J. (2005).  The wisdom of crowds . Anchor.
• Beekers, E. (2021). Are citizens’ assemblies the future of participation? – CitizenLab’s Blog. Retrieved 15 March 2021, from https://www.citizenlab.co/blog/civic-engagement/are-citizens-assemblies-the-future-of-participation/
• Mukherjee, N., Huge, J., Sutherland, W. J., McNeill, J., Van Opstal, M., Dahdouh‐Guebas, F., & Koedam, N. (2015). The Delphi technique in ecology and biological conservation: applications and guidelines.  Methods in Ecology and Evolution ,  6 (9), 1097-1109.
• Rowe, G., & Wright, G. (1999). The Delphi technique as a forecasting tool: issues and analysis. International journal of forecasting, 15(4), 353-375.
• RAND Corporation. (2021). Giving Patients a Voice in Medical Guidelines. Retrieved 11 February 2021, from https://www.rand.org/blog/rand-review/2019/12/giving-patients-a-voice-in-medical-guidelines.html

Prisoner’s Dilemma

Perceptual Barriers

Leadership Grid

Speech Recognition

Eager to learn about how behavioral science can help your organization?

Get new behavioral science insights in your inbox every month..

Delphi Technique Explained with Examples

Editorial Team

Delphi technique allows group members to submit anonymous feedback to encourage open dialogue from group members based on their own experiences and perspectives. Organizations use Delphi technique when they want to include and implement opinions from many people.

Delphi technique allows for more feedback while allowing people who can’t make it to an in-person meeting to participate in the conversation. It’s also used with significant effect by marketers in developing surveys and questionnaires to garner honest feedback from clients or customers. In this article, we will explain the Delphi technique and how organizations can use it to address their complex and controversial issues through a structured debate.

What is the Delphi Technique?

Delphi technique refers to a process where organizations collect information from experts using several rounds of questions. It’s a systematic and qualitative method that relies on experts to forecast the future outcome where they are much knowledgeable about.

Delphi technique comprises several written questionnaires that utilize the opinions of experts. Experts give their answers to each round-up of questionnaires. Then the facilitator gathers all answers and issues a summary report of the answers to each expert.

The experts analyze the summary reports and either agree or disagree with other expert’s answers. After analyzing the summary reports, the experts fill out another questionnaire that allows them to offer opinions based on their understanding of the summary reports. It only becomes complete when the experts reach an agreement on their forecasts.

When is the Delphi Technique Used?

Delphi technique is a significant methodology when an organization doesn’t know a real answer to a problem. Experts give a wide range of opinions, thus useful and prevent biases that result from relying on a single expert. Delphi technique applies when:

• A company desires subjective statements that come from a collective basis
• There are large sample sizes, and face to face interaction becomes unreliable
• It’s crucial to hide identity, especially on issues that are political or intractable
• There are high chances that one member would dominate the group discussion

Delphi Technique Process

Out of all the various change management models, the Delphi technique is often only associated with the consensus approach. It’s in many ways similar to the nominal group process (NGP) in that it requires a facilitator and several rounds of communications between people with opposing views. However, its unique value proposition distinguishes it from the nominal group process model and other related approaches. It promotes a more objective opinion on controversial topics than similar approaches.

• Identify issues and objectives.

An organization needs to identify their issue at hand and its main goal for using the Delphi technique. They should research and ascertain the importance of the thing they want to forecast.

• Select a facilitator and a team of experts

Experts can include internal or external individuals in the company. They may consist of the project team or any individual who is knowledgeable about a specific topic. The facilitator needs to occupy a neutral position and be a person who has experience in data collection and research.

• Round one questionnaire

The facilitator issues the experts with their first questionnaire. The questions are usually open-ended to allow the experts to brainstorm their ideas. The facilitator takes the questionnaires from the experts and comes up with a summary report for the answers which he gives the experts. The experts don’t reveal their identity, as this helps them share their opinions without fear.

• Round two questionnaire

An organization needs to set the questionnaire basing on the answers the experts gave in round one. They should identify similarities in the responses and leave out solutions that were out of context. The second questionnaire needs to follow answers that show consensus among the experts.

The expert’s opinions and answers in this questionnaire may remain the same or have a few changes. They may change their views after reading the summary report of the first questionnaire. The facilitator then makes a summary report of the second questionnaire answers and give the experts.

• Round three questionnaire

An organization needs to create a third-round questionnaire using the same ideas as the second questionnaire. The questionnaire is a result of the analysis of the second questionnaire. The facilitator then collects the questionnaires and comes up with a summary report answers which he gives the experts.

You can opt to stop here if you feel you have gathered enough information or continue with the questionnaire until you achieve your desired answers.

• Act on your findings

After the round three questionnaires, the experts should have reached a consensus, and hence an organization have a clear view of future events. An organization should now evaluate the results and develop a strategy to handle future opportunities and risks to their project.

Tips for Effective Delphi Technique

Every successful business out there knows that marketing is the key to success. The name of the game is to get as many clients as possible. When a business has more clients, profits go up, and client retention also increases. And that’s where the Delphi technique process comes in handy to get the best results for handling clients. Below are practical tips which will help you to get great results from the Delphi technique.

• Assessments

The experts should assess a situation before coming up with strategies to manage the assessment, including current issues, threats, or company expectations. From the reviews, they can now come up with ways to fix the problems.

• Carrying capacity

An organization should consider the capacity of experts they can hold. The ability relies on the resources and cost of conducting the survey. It should be within the company’s power to avoid misunderstandings between the experts and the company.

A company needs to consider the context in which they will apply the solution the experts give. Context refers to the ecological, managerial, and social atmosphere in which the organization will use the plan.

• The desire for future conditions

An effective questionnaire should include the project the company wants to achieve. The prescriptions will give experts the basis for them to answer the questions and ensure their thoughts don’t differ.

• Financial planning

Ensure you have enough resources and money to facilitate the research. Proper financial planning prevents the company from pausing the Delphi process due to the inability to pay the experts.

Once the experts give their view and a company comes up with a plan to achieve the results, they should closely monitor the plan to determine if it’s in line with achieving the desired outcome.

Advantages of Delphi Technique

The Delphi Technique is a simple way to structure the conversation between users and potential buyers. A company can determine what people think and feel about their product and then decide what steps to improve it.

Additionally, it’s a consensus-building method that seeks the opinions of a panel of experts. Each expert makes independent predictions and judgments about the subjects of a study without knowing the forecasts or assessments of the other experts. Some of its advantages include:

• Ensures collaboration

Anonymity prevents the impacts of dominant individuals, thus reduce the peer pressure to conform. It allows experts to give their opinions freely, without any judgments.

• Equal opinions

The facilitator must rate all the expert opinions equally. Thus, an individual can’t shift the group’s views.

• Reduces conflicts

The facilitator provides controlled feedback on the expert’s opinions, reducing disagreements and allowing participants to reconsider their input based on the other expert’s opinions.

• Reduced drawbacks

Delphi technique allows an organization to use a committee with few drawbacks such as scheduling, lengthy discussion, and travel arrangements.

Disadvantages of Delphi Technique

• Insufficient methodological guidelines
• The experts must be committed to answering similar questions over and over again.
• No evidence of reliability, i.e., a facilitator may issue two experts with a set of the questionnaire and they may not come to the same consensus.
• Limits the experts from discussing with one another, thus hinders them from sharing their views on the matter.
• The consensus doesn’t mean that the experts have got a correct answer. It only implies that the experts have found an area that they consider crucial to the topic.
• Its time consuming since the facilitator needs to conduct repetitive questions to come up with a common consensus.
• The process may become complex due to an increase in multiple and repetitive rounds.

Example/Application of Delphi Technique

The united states government intended to gather information on the number of teen pregnancies in a particular municipality. To collect the data, the government appointed a project facilitator. It then selected a group of experts who knew the city. The experts were excellent in communicating with the residents.

The facilitator did an analysis and came up with a survey that clearly stated the problem to the experts. The experts did round one questionnaire to which the facilitator analyzed the answers and came up with a summary report answers which he gave to the experts.

In the second questionnaire, the facilitator developed questions basing on the first-round questionnaire. The experts also gave their answers, to which the facilitator also made a summary report answer and handed it to the experts. He did the same for the third questionnaire, in which the experts reached a consensus.

Through the help of the solutions and suggestions from experts, the facilitator found the most efficient and top-quality information, which helped solve the problem.

The second application of the Delphi technique is of a project manager who desires to achieve canvas opinions and reach a consensus of how their products can be successful in the marketplace. The manager settles on the Delphi technique and opts to invite the top 50 people in the organization to participate.

The manager sets the questionnaires and asks the participants to rate the following options for them to accomplish their goals:

• Improve team productivity
• Give a tiered product pricing
• Add more people to the sales team
• Give faster feedbacks to customers

The manager used the number of questions to decide the points the experts should assign each question. Since there are five questions, the participants had to give points from 1 to 5. Five being the highest and one the lowest.

Below is a summary of how the participants gave their rating.

From the above table, only two participants, Mary, and rose gave five points. The comment was for the company to expand into German. The facilitator collected all the questionnaires, analyzed the answers, came up with a summary report answers, and gave it to the experts.

The experts answer to the second questionnaire was as follows:

The company continued with the questionnaires until they achieved what they wanted or got a low standard deviation. The typical standard deviation shows that there is a low variance for an item in the list. In this case, there is a low variance for launch in Germany. That means that most people agree with the idea of launching the business in Germany.

The item with the highest mean value will impact its more crucial, and that the organization needs to focus on it to achieve higher sales. From the example, the company settled for launching their business in Germany since it had more votes. That means it’s the best goal for the company.

Some organizations run into problems or different challenges in accomplishing their goals. That’s because there is no (or a little) company-wide consensus on which direction the organization should pursue, given that there are several paths to reach the same ultimate goal. However, using the Delphi technique will help solve the problem. Delphi technique uses experts with a vast knowledge of the topic, thus allowing your organization to follow the right path to achieve its success.

• Cover Letter From A to Z: Short Guide
• Guide to Modernizing Supply Chain Management
• 10 Ways To Grow Your Business Online Using Videos
• 4 Ways To Improve Construction Project Progress Tracking

most recent

Innovative marketing efforts: case studies from leading brands, how to achieve marketing success: proven strategies, effective marketing strategies to boost your business growth.

© 2024 Copyright ProjectPractical.com

updated 16 Oct 2021

Delphi Technique a Step-by-Step Guide

Tools | By Duncan Haughey | Read time minutes

For a project manager, it is essential to think about what future events may impact your projects. These events may be positive or negative, so understanding them allows you to prepare and plan to deal with them. But how can you forecast the future with any degree of certainty? The Delphi Technique can help.

The Delphi Technique is a method used to estimate the likelihood and outcome of future events. A group of experts exchange views, and each independently gives estimates and assumptions to a facilitator who reviews the data and issues a summary report.

The group members, known as panellists, discuss and review the summary report and give updated forecasts to the facilitator, who again reviews the material and issues a second report. This process continues until all participants reach a consensus.

The experts at each round have a complete record of what forecasts other experts have made. Still, they do not know who made which forecast. Anonymity allows the experts to express their opinions freely, encourages openness and avoids admitting errors by revising earlier forecasts.

This article looks at how to run a Delphi session. Upon completing this guide, you will be able to run a session enabling you to predict future events and their likely impact on your projects.

The technique is an iterative process and first aims to get a broad range of opinions from a group of experts. The results of the first round of questions, when summarised, provide the basis for the second round of questions. Results from the second round of questions feed into the third and final round.

The aim is to clarify and expand on issues, identify areas of agreement or disagreement, and find consensus.

Step 1: Choose a Facilitator

The first step is to choose your facilitator. You may wish to take on this role yourself or find a neutral person within your organisation. It is helpful to have someone familiar with research and data collection.

Step 2: Identify Your Experts

The Delphi technique relies on a panel of experts. This panel might be your project team, including the customer or other experts within your organisation or industry. An expert is any individual with relevant knowledge and experience of a particular topic. [1]

Step 3: Define the Problem

What is the problem or issue you are seeking to understand? The experts need to know what situation they are commenting on, so ensure you provide a precise and comprehensive definition.

Step 4: Round One Questions

Ask general questions to gain a broad understanding of the experts' views on future events. The questions may go out in the form of a questionnaire or survey. Collate and summarise the responses, removing any irrelevant material and looking for common viewpoints.

Step 5: Round Two Questions

Based on the answers to the first questions, the following questions should delve deeper into the topic to clarify specific issues. These questions may also go out in the form of a questionnaire or survey. Again, collate and summarise the results, removing any irrelevant material and look for the common ground. Remember, we are seeking to build consensus.

Step 6: Round Three Questions

The final questionnaire aims to focus on supporting decision making. Hone in on the areas of agreement. What is it upon which the experts are all agreed?

You may wish to have more than three rounds of questioning to reach a closer consensus.

Step 7: Act on Your Findings

After this round of questions, your experts will have, we hope, reached a consensus, and you will have a view of future events. Analyse the findings and put plans in place to deal with future risks and opportunities to your project.

Use the Delphi Technique to create work breakdown structures, identify risks and opportunities, and compile a lessons learned report. Use it anytime you would usually conduct a brainstorming session.

Predicting the future is not an exact science, but the Delphi Technique can help you understand the likelihood of future events and what impact they may have on your project.

[1] Cantrill JA, SibbaldB, Buetow S. The Delphi and Nominal Group Techniques in Health Services Research. International Journal of Pharmacy Practice 1996;4:67-74

Olaf Helmer and Norman Dalkey of the Rand Corporation developed the Delphi Method in the 1950s to address a specific military problem.

Recommended read: The Top Five Software Project Risks by Mike Griffiths.

Advertisment

What's Next?

You may also be interested in, how to recruit an it project manager.

• Project Smart's step-by-step guide to recruiting an IT Project Manager with 70 example interview questions.

Project Plans: 10 Essential Elements

• A project plan is more than just a Gantt chart, but do you know what you must have in your plan? This article takes you through the ten essential elements.

Stealth Team Building

• The effective project manager takes advantage of every opportunity the team gets together to develop team synergy.

5 Habits of Highly Productive People

• If you want to know how to make the most of the time you have available and achieve your goals, start with these five essential habits.

An In-Depth Look at the Delphi Method: Techniques, Studies, and Forecasts

The Delphi method is a widely used and effective forecasting technique that utilizes the knowledge of a panel of experts. In this comprehensive guide, we will examine how the Delphi method works, look at examples of real-world Delphi studies, and explore why this qualitative approach is so useful for generating forecasts and consensus.

The Delphi method allows researchers to systematically collect and aggregate informed opinions from subject matter experts on topics where there is incomplete or imperfect information. By surveying experts through multiple rounds of questionnaires and controlled feedback, researchers can arrive at a credible group consensus forecast or judgement. Read on to learn more about the origins, processes, applications, and strengths of the Delphi technique.

What is the Delphi Method and How Does it Work?

The Delphi method originated in the 1950s and 1960s by Olaf Helmer and Norman Dalkey at the RAND Corporation as a forecasting technique to predict likely future scenarios. It was originally used to forecast likely outcomes and impacts of technology and science for military purposes.

The name Delphi refers to the Oracle of Delphi in Greek mythology, representing a source of wise counsel. The Delphi method is based on the principle that group opinion is more valid than individual opinion. It is a qualitative, iterative process used to survey and collect the judgements of experts on a particular topic through a series of questionnaires and controlled feedback.

The key characteristics that define the Delphi method include:

Structured information flow: The process is structured into different rounds of questionnaires or surveys where information is fed back to panelists between each round.

Regular participant feedback: Panelists complete multiple rounds of questionnaires, receiving controlled feedback between each round that includes statistical representation of the group response.

Anonymity of participants: Panelists responses are anonymous to each other to avoid bias and groupthink.

-aggregation of group opinion: The objective is to arrive at a consensus or aggregation of group judgement.

The Delphi process typically involves these key steps:

Select a facilitator: The facilitator designs the questionnaires, recruits experts, controls the feedback between rounds, and moderates the process.

Recruit experts: Researchers identify and select experts who have deep knowledge and experience with the topic area. Panel sizes vary but often range from 10 to 50 experts.

Design the questionnaire: The facilitator develops open-ended questions to explore the topic and achieve the objectives.

Test and revise the questionnaire: The questionnaire is piloted before full implementation to improve questions.

Distribute initial questionnaire: Experts complete the first round questionnaire, answering questions anonymously.

Analyze first responses: The facilitator reviews, summarizes and groups the first round responses.

Develop new questionnaire: Using the responses, a new questionnaire is designed for the second round.

Distribute second questionnaire: Experts fill in the second questionnaire, rating and revising their answers in light of the group feedback.

Analyze second responses: Steps 6-8 are repeated over additional questionnaire rounds until consensus starts to emerge.

Finalize results: The facilitator analyzes the data to produce final results and areas of consensus.

The number of rounds used can vary depending on the scope and complexity of the issue, but commonly ranges from 3 to 5 rounds. Rounds continue until a satisfactory level of consensus is reached amongst the experts.

Examples of the Delphi Method in Practice

Since its inception, the Delphi method has been applied extensively across a diverse range of fields. It is a popular technique used in social policy and public health research, but has also been used for business forecasting, determining priorities, scenario planning, and in other contexts where expert consensus is needed. Here are some examples of how the Delphi technique has been utilized:

Healthcare policy and clinical practice guidelines - The Delphi method is commonly used in health contexts. For instance, it has been applied to develop clinical practice guidelines and care standards in areas lacking consensus amongst clinicians.

Business and technology forecasting - Corporations have applied the Delphi method to forecast sales, determine investment priorities, or predict industry trends and evolution of technologies.

Education research - In education, it has been used to identify and rank pressing issues and challenges facing schools and universities.

Environmental impact analysis - Researchers have leveraged the Delphi technique to better understand potential future environmental impacts of policies and human activities.

Scenario planning - Organizations have used the Delphi process as an input to scenario planning exercises to prepare for multiple plausible futures.

Determining research priorities - The Delphi method helpsfunding agencies to determine high priority areas for research investment.

Program planning and needs assessment - Nonprofits and government agencies have applied the Delphi process for program planning and identifying community needs.

Overall, the Delphi method has proven highly versatile and has been adapted to meet the forecasting and estimation needs of many fields. Next, we will look at some specific real-world examples of Delphi studies.

Real-World Examples of Delphi Studies

To understand how the Delphi technique is applied, it is helpful to look at examples of actual studies:

Healthcare Policy Example

A 2014 study published in BMC Health Services Research aimed to generate consensus and develop a framework amongst stakeholders for community-based health programming in Canada. A panel of 18 experts participated in a three round Delphi study. Analysis of responses revealed consensus on 25 core elements for community program planning, including recommendations related to partnerships, target populations, cultural adaptation and evaluation. The resulting framework provides evidence-based guidance for developing localized community health programs.

Education Example

Researchers applied the Delphi method to identify and rank the biggest challenges facing higher education academic libraries in the next 5-15 years. A panel of 56 library deans and directors participated in four rounds of questionnaires. The study revealed consensus on the top five issues: budgetary constraints, increased diversity, rising costs, digitalization, and participation in scholarly communication. The results help library leadership prioritize strategic initiatives.

Long-Range Forecasting Example

A classic example is a series of Delphi studies conducted by the Rand Corporation in the 1950s forecasting likely scientific and technological developments over the next 50 years. For instance, a study on future electronic computers predicted applications including artificial intelligence, use in education, computer programming languages, and networking—anticipating many developments long before they occurred.

Business Example

A consulting firm conducted a Delphi study to help an agricultural company forecast corn and soybean yields over the next decade. They surveyed a panel of 25 leading experts in corn and soybean production, economics, and climate science over four rounds. The findings provided probabilities for different yield scenarios, helping the company anticipate future supply conditions and make strategic plans.

Benefits and Advantages of Using the Delphi Method

There are several key advantages that make the Delphi method a useful and popular forecasting and consensus-building approach across domains:

Leverages collective wisdom from experts - The Delphi technique allows researchers to tap into the insights of a panel of intelligent specialists rather than just one or two people.

Avoids groupthink - The anonymity of responses avoids groupthink and domination of views by certain personalities that can bias conventional group discussions.

Structured process - The controlled, multi-round process produces more thoughtful, nuanced perspectives than one-off surveys or discussions.

Well-suited for complex, ambiguous topics - Delphi is ideal for issues where there are no definitive answers or that require subjective judgements from experts.

Flexibility - Researchers have flexibility to modify and adapt the process based on the topic and objectives.

Cost-effectiveness - It provides rich qualitative data at a lower cost than in-person workshops or conferences.

Consensus - Delphi studies work toward building agreement on the problem or questions at hand.

Stability of group opinion - Opinions generated through the iterative process tend to be stable and consistent.

For issues that require synthesizing informed judgements or forecasting complex trends, the Delphi method is a go-to technique. The structured approach makes it more rigorous than casual polling or basic surveys. Next we will look at some limitations.

Limitations and Critiques of the Delphi Method

While the Delphi method has many benefits for gathering expert consensus, there are also some inherent limitations to consider:

Time-intensive process - The multi-round process requires significant time and sustained engagement from experts. Panelist attrition can also be an issue.

Resource intensive - The process demands considerable effort from facilitators in questionnaire design, distribution, analysis and coordination.

No definitive answers - The technique does not produce objectively "correct" answers, but rather collective subjective judgements.

Potential bias - Results can be swayed by how information is presented by facilitators or how panels are selected.

Conformity - Pressure to conform to group judgments may lead some experts to modify their views.

Methodological issues - There are inconsistencies in how Delphi studies are applied and reported.

Questionable reliability - The reliability of results has been scrutinized, as results can differ based on the composition of the expert panel.

While limitations exist, researchers contend the Delphi method remains a valuable tool when used appropriately. It offers a structured approach to gleaning insights from experts when no definitive data exists on complex, ambiguous topics.

Tips for Applying the Delphi Method

When preparing to undertake a Delphi study, consider these tips:

Clearly define the purpose and objectives early when designing the study. This guides development of research questions.

Take time to thoughtfully select panelists - the quality of experts greatly influences the credibility of results. Seek heterogeneity and diversity.

Keep questionnaires concise and focused. Avoid overly broad or double-barreled questions.

Analyze and summarize the group response between each round so experts can reassess their judgements.

Monitor response rates at each round to maximize participation. Follow up promptly with non-responders.

Use statistical analysis techniques to assess consensus between rounds. Define consensus criteria upfront.

Document methodological details and decisions to support reliability.

Have a plan upfront for disseminating results and applying findings from the study.

The Future of the Delphi Method

Some researchers have explored modifications and technological enhancements to the Delphi method aimed at improving its reliability and effectiveness. Some examples include:

Employing real-time Delphi methods where experts interact anonymously via web-based platforms and interfaces. This allows for faster turnaround between rounds.

Combining Delphi with prediction markets where panelists can trade predictions on online marketplaces. Market prices are interpreted as probabilities.

Using algorithmic facilitation to automate parts of the process like sentiment analysis and clustering to support human facilitators.

Leveraging artificial intelligence to potentially serve as virtual experts on panels or assist with parts of the process.

While adoption of these kinds of technological modifications is still relatively limited, researchers continue seeking ways to refine and improve the flexibility, rigor and reliability of the Delphi method.

Conclusion and Key Takeaways

In summary, key points to remember about the Delphi method:

It is a flexible, qualitative forecasting technique relying on expert consensus gained through multiple rounds of questionnaires and controlled feedback.

Originated in the 1950s-60s by the RAND Corporation for technological forecasting but has expanded to diverse applications.

Key strengths are leveraging collective wisdom while avoiding groupthink, and providing more thoughtful data than basic surveys.

Limitations include intensive process, conformity pressures, and methodological inconsistencies.

Proper panel selection, clear procedures, statistical analysis, and consensus definition help maximizing the rigor and reliability of findings.

Technological enhancements like real-time feedback, algorithms, and AI may refine Delphi methods in the future.

Overall, the Delphi technique remains a valuable tool for forecasting, estimating, and building consensus on complex topics through expert judgement.

The Delphi method provides a structured communication process to generate credible qualitative data through expert consensus. With its versatility across fields and rigorous application, the Delphi technique will continue serving an important role where quantitative modelling and historical data are not available or adequate. This comprehensive overview covers key aspects of the Delphi method - from its origins to real-world applications through to future directions.

• Professional Services
• Creative & Design
• See all teams
• Project Management
• Workflow Management
• Task Management
• Resource Management
• See all use cases

Apps & Integrations

• Microsoft Teams
• See all integrations

Explore Wrike

• Book a Demo
• Take a Product Tour
• Start With Templates
• Customer Stories
• ROI Calculator
• Find a Reseller
• Mobile & Desktop Apps
• Cross-Tagging
• Kanban Boards
• Project Resource Planning
• Gantt Charts
• Custom Item Types
• Dynamic Request Forms
• Integrations
• See all features

Learn and connect

• Resource Hub
• Educational Guides

Become Wrike Pro

• Submit A Ticket
• Help Center
• Premium Support
• Community Topics
• Training Courses
• Facilitated Services

How Is the Delphi Technique Used in Project Management?

April 11, 2022 - 7 min read

It’s no secret that project management requires decision-making, planning , and shared understanding amongst stakeholders, but sometimes that’s easier said than done. 

When it comes to reaching a consensus during a project , it can feel impossible. But what if there was a technique you could use to help solve the impossible?

That’s where the Delphi Technique comes in. We’ve rounded up the basics of the Delphi Technique, how it applies in project management, and the benefits and drawbacks of using it. 

What is the Delphi Technique?

According to the Project Management Institute (PMI) , the Delphi Technique was developed in the 1960s due to changing technological environments and the impact this had on assessing and forecasting the future. It is also referred as the Delphi Method or Delphi Studies.

To put it simply, the Delphi Technique is used to make decisions about complex issues based on individual opinions. In this method, a group of experts writes down and shares their thoughts about a problem with a facilitator. Each expert’s view is compiled into a summary report by the facilitator. 

The experts then review the information and provide updated predictions to the facilitator to produce a new report. This process continues with systematic reviews until participants reach a consensus or agreement on the topic.  

How to use the Delphi Technique

Sounds simple, doesn’t it? Here’s a rundown of how to use the Delphi Technique: 

• Determine the problem you want to solve. What is the problem you are trying to solve? Be as clear and specific as possible. Understanding the issue at hand will inform who the experts are.
• Identify and engage your panel of experts. Now that you know what you’re trying to accomplish or answer, you should have a good idea of who you need to include in the process.
• Choose a facilitator to manage the process. Select a neutral person who has enough understanding of the topic to understand the conversations and compile results without bias.
• Start the process. Use questionnaires and surveys to gain an understanding of experts’ views and consolidate information.
• Create a summary report. Once you’ve gathered the group’s responses, you should remove irrelevant information and consolidate the results. Circulate the report back to the group to contemplate their peers’ ideas before the next round of questioning.
• Ask more questions, summarize the results, and repeat. Cycle through as many rounds of questionnaires and surveys as you need until a consensus emerges. In between, continue to create anonymized reports to share back with the structured group to use in their considerations.
• Reach consensus and find the way forward. Once your group of experts reaches an agreement, you should analyze the results and create plans to address future risks and opportunities accordingly. 

The Delphi Technique isn’t an exact science — it’s a process. You might need only two rounds of questionnaires to reach a consensus for more minor problems but upwards of 10 rounds for more complex ones. Regardless, keep the desired goal in mind: achieving agreement based upon expert viewpoints and guidance.

Delphi Technique in project management

Sure, reaching consensus is essential in projects, but where exactly do we see the Delphi Technique used in project management? The method is commonly used for both scope management and risk management . 

Using the Delphi Technique for scope management is valuable because it can help stakeholders reach an agreement on the scope of any given project. This helps eliminate big reasons for project failure , such as lack of clear requirements and inadequate planning.

For risk management, the Delphi Technique can help project teams predict and better prepare for future risks . That’s what the method was created to do in the first place: to help forecast for the future and get ahead of potential issues.

Delphi Technique examples

You’re probably getting a grasp of using the Delphi Technique in projects, but let’s run through a quick example related to scope management. 

Let’s say your business is kicking off a multi-phase project . You are going to implement a Project Management Office (PMO) and a customized project management methodology across your organization. 

Your experts, or key stakeholders, will likely have different views on what’s needed. They might also want other elements from the PMO , such as resource allocation , a template library, or project reporting assistance. 

You would use the Delphi Technique to reach a consensus on the scope of the PMO implementation as the initial phase of the project and then prioritize the elements of the following phases. 

The advantages of using the Delphi method

Successful projects require stakeholders and project teams to be on the same page, and that’s what the Delphi method can do for you. Here are the advantages of the Delphi Technique : 

• There’s potential to meet consensus rapidly. Have you ever been in a project meeting and needed to make a decision quickly but couldn’t get everyone in the room to agree? It happens to all of us. The Delphi Technique can help you achieve consensus and do it quickly, depending on the issue at hand.
• Less “groupthink” and more individual contributions. Sharing opinions, especially in a professional setting, can be challenging, particularly when there are a couple of dominant speakers in the room. One major perk of this method is that anonymized answers allow everyone to express and share their opinions freely.
• It’s cheap and quick to conduct. It’s relatively easy and inexpensive to use the Delphi Technique. You don’t need a lot of fancy tools or expensive materials to achieve results — just some time and a way to write down and compile ideas.

The drawbacks of using the Delphi analysis

As with any technique, you might experience some challenges with the Delphi Technique along the way. 

One key challenge: While it’s possible that you can achieve consensus quickly, it could be a long and tedious process to reach an agreement. If multiple rounds of questionnaires are required before your group hits consensus, you could fall behind schedule and put your project at risk. 

Another hurdle is that the technique is designed to help you reach an expert consensus , but just because your group of experts agrees doesn’t necessarily mean you’ve found the best or the right way to solve a problem. Don’t be mistaken – level-setting opinions and getting everyone on the same page is a huge win, but it’s worth noting the difference between consensus and identifying the best possible solution. 

Using Wrike for the Delphi Technique 

As we mentioned, you don’t need any fancy tools or a big budget if you want to use the Delphi Technique. However, once you’ve reached a consensus about a particular problem related to a project, a project management tool may come in handy to help ensure you follow through with what all experts have agreed upon.

With Wrike, you can follow up on action items from your Delphi method analyses and put together full project schedules . The Delphi Technique can help you determine the next steps of a project, and Wrike will help ensure you stay on track to achieve those next steps without fail. 

Ready? Get started today with a free trial of Wrike .

Kat Boogaard

Kat is a Midwest-based contributing writer. She covers topics related to careers, self-development, and the freelance life. She is also a columnist for Inc., writes for The Muse, is Career Editor for The Everygirl, and a contributor all over the web.

Related articles

A Quick Guide to The Linear Scheduling Method

A linear schedule is a project management tool used to present a schedule in two dimensions, such as time and distance. This is normally used for projects with linear construction properties or repetitive tasks, such as the construction of roads, bridges, and railways.  In this article, we will explain what the linear scheduling method is, how it benefits projects, and how to execute it. Keep reading to discover linear scheduling examples and tools you can use to perfect your next project plans. Linear scheduling method explained The linear scheduling method is a process for developing project schedules that take into account the various elements of a work that are linear, continuous, or repeated. Most commonly used in construction management and engineering, the quantity of work performed on a project is used to determine the duration of a work relationship and the work rates for each crew member. Although the term “linear scheduling” is often used to describe the steps involved in preparing linear schedules, it can also be referred to as the development of a graphical representation of a project schedule. You may also hear the linear scheduling method referred to as “time chainage” or “line-of-balance”.  The various elements of the linear schedule are represented on a grid with coordinates that define when and where the work begins and finishes. A range of display methods is also used to present the varying activities in different formats. What benefits does a linear schedule provide to a project manager? The linear scheduling method is simple. Not only is it easier to read, but it’s also easier to organize.  It’s helpful for communication. Contractors, stakeholders, and clients can understand your project plans from this information alone. It covers everything they need to know about what’s needed, where and when the work will take place, and how productive the work will be.  Linear scheduling mitigates risk. Project managers can easily monitor and report on the various activities that occurred during the execution phase. Even if unforeseen circumstances happen during the execution phase, this can be easily revisited to improve the project's productivity using this method.  It’s flexible. The time-location chart is useful for extending project benchmarks as needed because it provides a list of the factors that can affect the project's schedule, so you know what to expect.  The linear scheduling method helps inform decision-making. This method helps the project manager identify and take action on the lapses that occurred during past projects. It also helps with preventing them from happening again. It uncovers task dependencies. A time-location diagram is a great tool to visualize the various interrelated activities in a project. It can help you understand the sequence of activities and their importance. For example, in construction, the arrival of material at a site may cause delays in the mobilization of heavy equipment, resulting in costly downtime if not forecasted.  Which industries use linear scheduling? Linear scheduling is most commonly used in the construction industry for straightforward projects such as standard pipelines, highways, and railroads. However, any industry with projects that have linear continuous and repetitive tasks can take advantage of this method. It’s also effective for managing compliance and continuity across repeat projects.  Linear scheduling method in construction management For construction management, this method involves repeatedly running a set of projects in each location for the duration of the work. Some examples of prominent projects that use this method include highway construction and airport runway projects. One of the most important factors involved in planning a linear construction project is ensuring that the work of the crew does not interfere with the activities of other workers. It’s important to note that traditional planning and scheduling techniques commonly used for linear construction projects do not account for the various factors that affect the planning and scheduling of such projects. Other methods, such as line-of-balance and vertical production methods, are often used to improve the planning and scheduling techniques for linear construction projects. Linear scheduling examples Tunnel construction with the linear scheduling method [caption id="attachment_471943" align="aligncenter" width="772"] Source: James Wonneberg, PE, CCM via Slideshare[/caption] The x-axis depicts the location between the two main construction points of this project. The y-axis represents the time period over which the work will take place. Project phases are color-coded in orange, blue, green, and red for easier interpretation.  Notice how each of the tasks is associated with location and time.  For example, during Q2 of 2017, tunnel site 10+00 will be prepped. Then, in Q3 of 2017, tunnel site 10+00 will be constructed while tunnel site 110+00 is prepped.  Not only is this easy to understand at a glance, but it also gives individual team leaders a solid understanding of expectations, deadlines, and tasks that need to be completed by their group.  Building a road with the linear scheduling method [caption id="attachment_471949" align="aligncenter" width="801"] Source: Columbia University[/caption] In this linear scheduling method example, “A” represents the beginning of the road, and “E” represents the end of the road. The lettered segments each represent a different section of the road that will be built in order. The numbered bars correspond with a different repetitive process associated with this example road’s construction, such as prepping the site and pouring concrete.  As you can see, each road segment will repeat the same set of repetitive tasks. After tasks one and two are completed on road section A, section B begins the same list, and the pattern continues onward.  This super simplistic model may need a corresponding document to explain the task lists and locations, but it does visualize the big-picture aspects of the project while also being actionable.  Why choose Wrike as your linear scheduling software? Wrike is a project management tool designed to streamline repetitive tasks and maximize productivity. Once you have your linear schedule in place, you can effectively execute it with Wrike using our Recurrent Tasks, Project Schedule Template, and Automation Engine features.  With Recurrent Tasks, project managers can easily create and duplicate tasks assignments complete with estimated timelines, important details, and assigned team members or approvers. And when you opt for recurrent tasks to be created monthly or yearly on a specific day of the month, you can choose to automatically reschedule the task if this day turns out to be a weekend which saves time you would spend combing through calendars to triple check dates.  In Wrike, the Project Schedule Template gives managers the ability to duplicate project plans from previous engagements. Not only does this save time, but it also ensures that projects remain compliant. And if you need to customize any of the details, Wrike makes it easy to do so.  And finally, Wrike’s Automation Engine automatically updates and simplifies processes. It works seamlessly with existing rules to enable users to easily modify and streamline their actions. It also notifies when project obstacles appear and manages your team's timeline with custom reminders and notifications about overdue tasks, task dependencies, and when it’s time to move on to the next step.  Ready to maximize the power of the linear scheduling method? Get started with Wrike’s two-week free trial. 

Risk vs Issue: Everything You Need to Know

Whether you realize it or not, we make risk vs. issue calculations all day long.  Let’s consider one issue: You’re running late for work. Why not drive through that red light? Well, because of the risk: Getting pulled over by police, or worse, putting peoples’ lives in danger. So, you wait out the red light. That’s an easy one. Managing risk vs. issue in project management isn’t always so cut-and-dry. But understanding the difference between risk and issue does boost the chances that your project will be a success. Here’s what you’ll need to know to navigate the world of risks and issues in complex projects.  Risk vs. issue: What’s the difference? An issue is an obstacle or challenge that’s already present. A risk is a potential obstacle that may arise in the future but doesn’t necessarily have to. It might sound as if issues are always greater challenges than risks. But the two can vary wildly in terms of degrees. For example, a broken pinky toe is a manageable medical issue. But ignoring a medical risk like a clogged artery can be far more serious. As a project manager, your job is to evaluate both, so you never lose the patient (or the project). Why is it important to note the difference between a risk and an issue? Effective risk analysis hinges on estimating the potential pitfalls in running your project. Ideally, the planning phase will tackle them in reverse: You’ll try to lower the risks at the outset to avoid them becoming full-blown issues. But you’ll need plans for both. To make your own complicated project run smoothly, you need to start backward. Think about the biggest risks with your team and develop plans for them. Run a risk assessment before the initial phases. Remember the old adage that “an ounce of prevention is worth more than a pound of cure.” This is all the more important when launching a complex project. Risk vs. issue examples in project management Still feeling a little confused? Let’s take a look at a couple of examples of risk vs. issue in project management.  Boeing’s 787 Dreamliner: Risk complexity leading to inevitable issues Consider the problems with the unveiling of Boeing’s 787 Dreamliner. It was “a more complicated airplane, with newer ideas, new features, new systems, new technologies,” according to a Boeing engineer. From the outset, this complexity made it difficult to foresee specific issues. But given the novelty of the project, some risks were already present. After several delays — and an additional $10 billion in budgeting — Boeing eventually “outran our ability to manage it effectively,” according to a release from the company. It’s tempting to look at this risk vs. issue example from the outside and imagine that it was an unlucky project with too many issues. But Boeing’s ambitions meant higher levels of risks. Without enough risk management, project issues become inevitable.  Bank of America: When risks prove more intense issues than you imagined In 2011, Bank of America rolled out a new pricing initiative: $5 per month for customers to get access to debit funds. Bank of America anticipated some customer blowback — after all, that’s common whenever prices go up. But customer reaction was far more intense than Bank of America imagined. One customer started “Bank Transfer Day” on Facebook to protest the fees. Customers moved their money out of their Bank of America accounts in droves. Efforts like these had enough impact that the brand was eventually forced to back down on its new fees.  In this case, it wasn’t that Bank of America hadn’t adequately foreseen the risk. They simply hadn’t imagined the risk would manifest as such an intense issue during the rollout phase. Had they spotted this issue earlier, the launch may have played out differently. Issue log vs. risk register: what is the difference? In project management, leaders typically turn to two tools to manage these variables. For risk, they turn to risk registers. This register lists the various risks along with their probabilities and potential timing hazards. (Note: A risk matrix is another effective tool for weighing and predicting risks.) For issues currently affecting the project, an issue log is simply a list of the current problems a team member needs to solve. In one research paper, project manager N.K. Shrivastava says that when a risk manifests as an issue, he makes two key changes: Adding the issue to the issue log to assign it to the relevant team member Flagging the risk in the risk register for better project planning in the future, especially noting which risks were most likely to manifest as issues By flagging the risks that become issues, Shrivastava also grows as a project manager. He learns which issues to spot in potential future projects, making him better at risk assessment for future work. One key difference here: A risk register can be a list of potential obstacles that you create at the outset, while an issue log is only a list of real obstacles. One is essentially in the future tense, while the other is in the present or past tense. How to use Wrike for managing risks and issues In project management, risks are everywhere. There are so many, in fact, that there are even positive risks: unforeseen events that help your project along.  Wrike can help you get started with risk assessment by providing the information and templates you need to begin your evaluation. The key is in using previous risks to inform your future work. If you’re starting your first project, use Wrike’s risk assessment templates to better predict, record, and measure the risks associated with your first project.  With any luck, your analysis at the outset will help you minimize issues along the way, so you can get to the project wrap-up celebration on time and on budget. Ready to foresee risks, manage issues, and have a more successful project? Start your free trial of Wrike today.

How to Conduct Pareto Analysis Using Pareto Charts

The 80/20 rule, also known as the Pareto Principle, states that 80% of the benefit can be achieved by 20% of the work. The Pareto analysis uses this concept to identify which parts of a project can be done efficiently and which can be avoided. It can be used to decide which problems should be solved first. In this article, we’ll explain how to perform Pareto chart analysis and how it can be used to improve any project. We’ll also get into some vital tools you can use to help teams work smarter, not harder.  What is a Pareto chart? The Pareto chart is a visual representation of the most important factors in a given field or project in order to identify and prioritize issues.  In general, this tool can be used to identify the most critical factors in a given product or process. For example, in quality assurance, the Pareto chart helps identify the most prevalent sources of defects. The Pareto chart itself is a bar graph with two axes. The left axis shows the frequency of occurrence, which is the sum of the total number of occurrences and the cost of doing so. The right axis shows the cumulative function of the total number of occurrences. The values for each category are depicted in descending order. And the final total is represented with a line drawn at 80% on the bar graph. Any bars rising above that line are considered the problems that, if solved, would have the biggest impact on the project.  The Pareto chart can be generated by various means, such as creating Excel spreadsheets, statistical software, and online quality charts. What is a Pareto chart used for?    A Pareto chart helps you identify the causes of the various problems and the issues that need to be solved to get the most significant improvement. Here are some of the many ways it can be used: Visually represent project issues to find which have the greatest impact Communicate priority levels to stakeholders  Isolate individual process hiccups so that they can be better understood  Find the most impactful problems and eliminate them before they cause issues  Reallocate workloads so that team members companywide are maximizing their impact and productivity  When to use a Pareto chart in project management A Pareto chart is a tool that many people use to analyze different types of problems. It can also identify the most significant issues in a process. In project management, this means everything from big-picture project phases to individual task workflows.  When roadblocks come up, managers can use the Pareto analysis to quickly identify what is causing bottlenecks or delays. From there, they can use their project management tool to delegate troubleshooting, adjust task lists, or shift priorities without interrupting the entire project.  In addition to making on-the-fly decisions based on real evidence possible, Pareto charts can be used in project management for post-project analysis for both PMOs and stakeholders. Teams can learn from each other and what went wrong in projects with data clearly laid out in this way. In the future, they’ll be able to replicate their successes and mitigate failures.  Stakeholders can easily pinpoint areas of investment that worked out and learn more about how this project was managed so they can feel good about the outcome. In particularly successful works, they’ll be able to see how little interference the project encountered along the way. Or they can see how well the team managed issues that did arise.  This persuasive tactic allows them to feel great about providing repeat business or, at the very least, enrich their understanding of key projects related to their goals for future reference.  How to calculate Pareto analysis The simplest formula for calculating a Pareto analysis is as follows:  [Your total unit of measurement per item, e.g., number of occurrences, hours, cost, etc.] / [the grand total of all items] x 100%  Use this formula for each category. Keep in mind that each result should be a percentage. Afterward, put them in order from highest to lowest before inputting them into your chart-making software.  How to create a Pareto chart Step 1: Collect your data A minimum of 30 data points is best for an accurate picture of the project as a whole.  Step 2: Create a frequency table Use the following headers in this order:  Issue Type Number of Occurrences (listed in descending order) Note: Some programs will automatically generate a Pareto chart for you once you’ve added the number of occurrences or frequency for each issue category.  The rest of your headers may be calculated for you, again depending on the program you’re using:  Cumulative Total  Cumulative Percentage  80% Cut Off Step 3: Label your chart Keep it simple. You can never go wrong with “Types of Project Errors.” Freel free to add a single sentence description that includes the time period of your measurement and any other details that are important to the people you’ll be sharing it with.  Step 4: Clearly label the Y-axis Frequency, total number of occurrences, or even price all work well here. Use whichever value best represents your data set or makes the most sense to you as a manager.  Step 5: Note the categories on the X-axis These should match the Issue Types you first listed in your frequency table.  Step 6: Interpret the chart  Again, the software you use should fill in the other components of the chart, including the bars, lines, and cut-off. From there, you can get to work analyzing the results.  The higher the bars are, the more of an impact they are having on your project.  You’ll see percentages listed on the right-hand side of the chart across from frequency. They should be listed from 100% and counted down in increments of 10 to the bottom. Any bars that cross the 80% line should be considered a top priority for problem-solving.  Pareto chart example This Pareto chart example was created by Clinical Excellence Commission and thoughtfully illustrates the key areas of focus project managers should be aware of.  The areas marked in red and bold are the spaces project managers should focus on when conducting their analysis.  We can also see that the categories on the bottom are great examples of types of medication errors. But project managers may use groupings such as scope creep, resource management, or communication to define a variety of issues that may come up.  The most important line on this graph is the green 80% cut-off, which symbolizes the Pareto principle. Any bar that reaches above that line should be considered the most important issue. In the above chart, that would be “dose missed” at 92% and “wrong time” at 83%. Although “wrong drug” clocks in at 76%, it’s not considered nearly as important as the first two.  For the next steps, the project manager in this particular example would likely solve the issues above the 80% line first before moving on to the next highest scoring category. Alternatively, they could choose to solve the above-the-line problems, then create a new Pareto chart and see if the values have changed. It’s also possible that solving the highest priority issues may fix less pressing issues on your chart down the road as a byproduct.  How Pareto chart analysis can improve your project In general, the Pareto chart helps project managers and team leaders identify the causes of various problems that are having the biggest impact on their work. By figuring out what they are, managers can take the necessary steps to solve them. It’s also easier to determine task and even project or goal prioritization with a chart like this.  If you’re working with third-party partners or stakeholders, the visual aspects of Pareto charts make them easy to understand and interpret. Not only is this highly effective for communicating with non-experts, but it’s also highly persuasive.  How to interpret Pareto analysis with Wrike So you’ve made your Pareto charts and conducted your analysis. Now what? Put your plan into action with Wrike.  Wrike is a project management solution that makes project plans manageable, efficient, and crystal clear. Now that you know what’s going wrong, you can easily add actionable next steps to your project plans without missing a beat.  Start by adding a detailed task to your project. Add a description, deadline, and task owner. Wrike also allows you to see the workload of individual team members across all active projects so you can double-check they’re available before assigning it.  You can also use Wrike’s custom reporting features to identify issue categories for your Pareto chart. Dissect active and past tasks to find the biggest areas in need of improvement during individual project phases or projects as a whole.  In addition to Pareto charts, Wrike also offers Gantt charts and PERT charts that can improve productivity.  A Gantt chart is a bar chart that shows the various tasks and deadlines for a project. It's a great tool for managing time and improving efficiency.  A PERT chart is a network diagram that shows all the project tasks in separate containers. The boxes that make up the PERT chart are organized with arrows to represent the time needed to complete the task. Combining the results of your Pareto, Gantt, and PERT charts will help you turbocharge your project troubleshooting plans and may even prevent future issues too.  Ready to get the biggest results from the least amount of effort? Get started with Wrike’s two-week free trial. 

Get weekly updates in your inbox!

You are now subscribed to wrike news and updates.

Let us know what marketing emails you are interested in by updating your email preferences here .

Sorry, this content is unavailable due to your privacy settings. To view this content, click the “Cookie Preferences” button and accept Advertising Cookies there.

Advisory boards aren’t only for executives. Join the LogRocket Content Advisory Board today →

• Product Management
• Solve User-Reported Issues
• Find Issues Faster
• Optimize Conversion and Adoption

Understanding and applying the Delphi technique

Product managers have to wear multiple hats at work. One of these hats requires you to facilitate meetings. Everyone appreciates an effective, efficient meeting, and the Delphi technique can help you do this.

Besides facilitation, the Delphi technique is a great tool for decision-making. It’s commonly used in various fields, including business, healthcare, technology, and public policy.

In this article, you’ll learn more about the Delphi technique including its five steps, benefits, and use.

What is the Delphi technique?

The Delphi technique is a systematic method for group decision-making. It’s used to gather opinions from a group of knowledgeable members to reach a group consensus. The process typically involves several rounds of questionnaires where the responses remain anonymous.

5 steps of the Delphi technique

You can incorporate the Delphi technique into your product team by following five easy steps.

1. Determine the problem

In product management, people tend to emphasize the “problem space” or the “problem statement.” You first have to understand the problem to solve, the purpose of the study, and the goal you want to achieve before conducting research.

In this step, the problem statement needs to be clear and specific. It outlines the gap between the current state and the desired state, providing a foundation for further investigation and problem-solving. With a clear problem statement, you can smooth out the process in later steps.

2. Identify a group of experts

The central tenet of the Delphi technique involves consulting with relevant stakeholders or individuals with expertise. Different perspectives can provide valuable insights and help the panel consider aspects that might have been overlooked.

In the previous step, you identified the problems to solve. Now you need to determine which “subject matter experts” to include, depending on the complexity of the problem.

3. Choose a facilitator

The facilitator of the Delphi process is usually, but not limited to, a product manager, project manager, or program manager. The facilitator should be neutral and have enough domain knowledge to understand the conversations.

Over 200k developers and product managers use LogRocket to create better digital experiences

4. Launch rounds of questionnaires

To complete the Delphi process, you need to run multiple rounds of questionnaires or surveys. In each round, you’ll ask the panel of experts to provide their opinions, predictions, or judgments on a list of questions for the problem determined in the first step:

The above illustration shows how the rounds of questionnaires are iteratively designed and answered:

• The facilitator hands out questionnaires
• The panel of experts answers the questions anonymously
• The facilitator reviews the answers to find the themes between the answers and then shows the themes to the experts.
• The panel of experts adds new answers or adjust their original answers based on the feedback they receive from the panel
• The facilitator designs questions for the next round based on answers from previous rounds

You can distribute your questionnaires before each round. Generally, there should be at least two rounds of questionnaires. The first round is usually for asking high-level questions to kickstart the whole process. The rest of the rounds are for follow-up questions and for experts to adjust their answers until a general group consensus is reached.

5. Analyze the feedback and results

Once the panel of experts reaches a consensus, we can analyze the results and evaluate risks. The team can create plans and action items.

What are the benefits of the Delphi technique?

The Delphi technique provides three key benefits for you as a product manager.

The responses of each expert are kept anonymous. This helps to avoid the influence of dominant personalities or opinions within the group. This setup also encourages open and honest sharing of opinions.

Systematic process of iteration

The Delphi technique is a structured and systematic process involving multiple rounds of feedback and iteration. The opinions of the subject matter experts can be refined over time, leading to a more reliable outcome.

Prevent groupthink

Groupthink is a psychological phenomenon where group members tend to conform to a dominant viewpoint without critical evaluation to avoid conflicts. In the Delphi process, by collecting individual opinions independently and collating them without direct interaction among participants, the risk of groupthink is minimized.

More great articles from LogRocket:

• How to implement issue management to improve your product
• 8 ways to reduce cycle time and build a better product
• What is a PERT chart and how to make one
• Discover how to use behavioral analytics to create a great product experience
• Explore six tried and true product management frameworks you should know
• Advisory boards aren’t just for executives. Join LogRocket’s Content Advisory Board. You’ll help inform the type of content we create and get access to exclusive meetups, social accreditation, and swag.

The disadvantages of the Delphi technique

Although there are benefits to the Delphi technique, you should be aware of the disadvantages that come along with them to prevent affecting your process.

Time-consuming

Multiple rounds of feedback and iteration take a lot of time. While choosing a decision-making process, you need to consider the time constraints. The Delphi technique may not be the best option if your organization needs to reach a quick decision in a short timeframe.

Expert bias

The success of the Delphi technique heavily depends on the expertise and knowledge of the participants. If the experts have biases or limited perspectives, it can impact the quality of the final outcome.

Difficulty in handling complex issues

The Delphi technique may be less effective in dealing with highly complex or technical issues where consensus is hard to achieve, or where there is a lack of relevant expertise among the participants.

How can product managers use the Delphi technique?

Product managers can leverage the Delphi technique to gather insights, make informed decisions, and forecast outcomes. Here are some areas PMs can try.

Market validation

In the early stage of product development, you have to conduct research to find product-market fit . While conducting market validation for your product, it’s crucial to gather expert opinions on market trends, demand for specific features, or predictions for future market conditions.

By using the Delphi technique, you can obtain insights into potential shifts in customer preferences and emerging market opportunities. On the other hand, PMs can explore potential opportunities and threats in the market with the Delphi technique as well.

Strategic planning

While conducting competitor analysis , insights from subject matter experts make the analysis more complete regarding competitor strategies, strengths, and weaknesses. The Delphi technique can help product managers forecast potential competitive moves and assess the impact on their long-term product strategy.

The Delphi technique is a valuable tool for product managers. The five-step method ensures a systematic approach to group decision-making by leveraging the expertise of subject matter experts while maintaining anonymity throughout the process.

The goal of the Delphi technique is to achieve a consensus or convergence of opinions among the experts. The final results are based on the collective judgment of the group. It helps to mitigate the impact of groupthink and interpersonal dynamics that can affect traditional group decision-making processes.

Featured image source: IconScout

LogRocket generates product insights that lead to meaningful action

Get your teams on the same page — try LogRocket today.

Share this:

• Click to share on Twitter (Opens in new window)
• Click to share on Reddit (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Facebook (Opens in new window)
• #collaboration and communication
• #project management

Stop guessing about your digital experience with LogRocket

Recent posts:.

10 tips for a great project kickoff

Step up your game and make your next kickoff the best meeting you’ve ever facilitated — a meeting that people will remember for months.

Associate product manager role and responsibilities

An associate product manager doesn’t have prior product management experience, but can grow into this role via training and mentorship.

Aligning product metrics with business objectives

Product metrics are quantitative measures used to evaluate product performance and identify actionable insights.

Leader Spotlight: Leveraging creative testing frameworks, with Sarah Carusona

Sarah Carusona talks about her creative testing framework and how she broke down goals for each team member for their Meta account.

Leave a Reply Cancel reply

Delphi Method: A Strategic Approach to Decision-Making in Business

In today's complex and rapidly evolving business environment, decision-makers are often faced with scenarios that require structured foresight and collective intelligence. The Delphi Method is a systematic interactive process used to gather opinions and insights from a group of experts to aid in decision-making and forecasting.

A technique steeped in history yet still profoundly relevant, the Delphi Method is utilized across various sectors to navigate uncertainties and generate consensus-driven outcomes. In employing a structured communication technique, the method effectively aligns expert opinion on complex issues not easily quantified or analyzed through traditional methods.

This article will delve into the intricacies of the Delphi Method, unravel its stages, and highlight its unique applications and inherent limitations in business and broader fields.

Understanding the Delphi Method

Explanation of the delphi process.

The Delphi Method operates through a series of iterative rounds designed to transform opinion into informed consensus. Initially, in the exploration phase , a facilitator, known as the Delphi manager, identifies a central question or series of problems and selects a panel of experts. These experts, typically diverse in background yet specialized in the issue at hand, are invited to provide their insights independently. This ensures that the Delphi process is not subject to the undue influence of a single, dominant personality, as can occur in traditional committee or board meetings.

As the method moves into the consultation phase , experts are given the opportunity to re-evaluate their views after being presented with a summary of the panel's responses. The anonymity of contributions allows each expert to consider the information provided by peers without the risk of groupthink or peer pressure influencing their own opinions. This iterative feedback loop is central to the process; it refines and narrows the range of answers as participants revise their earlier estimates in light of the group's collective intelligence.

In the conclusion phase , the process repeats until a predetermined stop criterion is met, such as a consensus or stability in responses. The final outcome synthesizes the collective expertise into a cohesive decision or forecast, refined through the successive rounds of discussion and revision. This phase solidifies the Delphi Method's utility in providing a clarified perspective on issues where individual judgments may have been too varied or complex to form a clear direction.

The Role of Experts in the Method

The success of the Delphi Method hinges upon the selection of experts who carry the knowledge and experience necessary to inform the issue at hand. Their role extends beyond mere participation, as they collectively constitute a de facto think-tank, offering depth and diversity to the conversation. The selection process must be diligent and strategic; experts are chosen not just for their expertise but also for their ability to contribute unique perspectives to enrich the collective understanding of the topic.

Once chosen, these experts become the driving force behind the Delphi process. The role of experts involves independent thought as well as a willingness to adapt and refine perspectives in light of group input. Each is tasked with considering the input of peers, highlighting areas of agreement and dissent, and aiding in the construction of a refined understanding of the complex subject matter. The iterative nature of the method ensures that the experts' roles evolve as the rounds advance, securing an integrated output reflective of the panel's combined wisdom.

Importance and Use Cases of the Delphi Method

Advantages of using the delphi method.

The primary advantages of the Delphi Method lie in its ability to transform subjective insights into a collective forecast or decision that is more accurate and reliable than what an individual expert might provide. The process excels in issues lacking clear data or where future events are uncertain. It offers a structured approach to harnessing expert opinion, mitigating the biases and pressures that may influence individual judgments. Moreover, by circumventing direct confrontation, it promotes the expression of independent thought and fosters an open exchange of ideas.

Industries that use the Delphi Method

The use of the Delphi Method transcends industries, demonstrating its versatility and effectiveness. In the medical and healthcare fields , the method is pivotal in developing clinical guidelines and shaping policy by consolidating expert medical opinion. Public policy and social research have benefited from its use in areas ranging from environmental forecasting to educational reforms. It allows policymakers to cast a wide net of expert opinion and distill it into actionable insights.

The method also has found firm footing in the arena of business and marketing . Companies use the Delphi Method for strategic planning, product development, and forecasting market trends. By bringing together leaders in the field, businesses can inoculate their strategies against biases and make data-informed decisions in the absence of complete information.

Specific Examples of the Delphi Method Application

The application of the Delphi Method can range from small-scale problems to grand strategic initiatives. Technology firms, for example, routinely employ the method to predict future trends or determine product feasibility. Consulting firms may facilitate a Delphi session to aid a client in identifying new market opportunities or in assessing risks related to new investments.

Specific examples include the use of the Delphi Method in anticipating the impact of regulatory changes in healthcare or in constructing scenario planning exercises for corporations. The method's adaptability to various informational needs makes it an invaluable tool in many annotative and predictive circumstances.

Criticisms and Limitations of the Delphi Method

Drawbacks of the delphi method.

Despite its methodological soundness, the Delphi Method is not without criticisms. Skeptics point out that the method's reliance on expert opinion does not guarantee accuracy—experts, after all, can be fallible. The drawbacks may also include the potential for a lack of accountability, as anonymity could lead to less investment in the process' outcomes. Also, the time-consuming nature of multiple rounds can be a deterrent, particularly for time-sensitive projects.

Areas where the Delphi Method Can be Ineffective

There are areas where the Delphi Method may be ineffective , specifically where issues are not amenable to expert judgment or where factual data are more reliable than informed speculation. Instances requiring immediate decision-making or those that involve personal values and preferences may not be suitable for this method. Additionally, when the scope of input required is too broad, the Delphi Method may become unwieldy and impractical.

Addressing the Limitations

To support the efficacy of the Delphi Method, careful attention to process management is crucial. The facilitator must ensure a diverse and balanced panel of experts, maintain the process's integrity, and set clear objectives and stopping criteria. Selection of experts needs to be approached methodically, aiming for a group that represents a comprehensive understanding of the topic. The careful design of the process can ameliorate many of the criticisms, maximizing the Delphi Method's relevance and reliability.

The Importance of the Delphi Method in Various Industries

Throughout its history and ongoing adaptations, the Delphi Method has proven to be a potent tool, its structured approach bringing order to the chaos of decision-making. By leveraging the collective intellect of experts, the method offers an innovative way to tackle complex and uncertain scenarios across various industries, providing clarity and foresight amidst ambiguity.

Navigating the Limitations of the Delphi Method

As we navigate the limitations inherent in any methodological approach, it becomes apparent that a method's worth is often determined by the diligence with which it is applied. Expert execution of the Delphi Method can yield insights and decisions that integrate nuanced understanding with strategic direction. It is a testament to the enduring nature of collective wisdom, a principle well-captured by the Delphi Method's thoughtful process. Whether used to procure a problem solving certificate or in designing online certificate programs , it holds a strategic position in the fabric of decision-making within the business world and beyond.

What are the key steps involved in the Delphi Method for strategic decision-making in business?

The Delphi Method stands as a structured communication process. Experts achieve consensus through it. It requires several rounds of questionnaires. Feedback is anonymous. This preserves independent thought. Each round refines the previous answers. The process continues until consensus emerges.

Key Steps in the Delphi Method

1. select participants.

Identify and recruit subject matter experts. Diversity in knowledge enhances the process. Ensure participants have relevant experience. Their willingness to engage is crucial.

2. Design Questionnaires

Craft clear, concise questions. The initial questionnaire gathers basic information. Later rounds delve deeper into the topic. Each questionnaire reflects the previous round's insights.

3. Conduct Multiple Rounds

Expect several rounds for consensus-building. Feedback from one informs the next. Anonymity allows free expression of ideas. Critiques don't target individuals, only ideas.

4. Provide Controlled Feedback

Summarize the responses. Share with all participants. Maintain anonymity. Allow members to reassess their position. Feedback informs subsequent rounds.

5. Reach Consensus

Strive for agreement among the experts. Consensus does not mean unanimity. It signals general agreement. The process ends when stability of opinion occurs.

6. Report Results

Summarize the findings. A coherent report captures the consensus. Detail where agreement and disagreements lie. Highlight actionable items for strategic decisions.

Benefits of the Delphi Method

Flexibility marks the Delphi Method. It adapts across disciplines. Participants contribute without the need for face-to-face meetings. Time-efficiency improves as geographical constraints disappear. Anonymity reduces the impact of dominant personalities. More reticent members participate openly.

Challenges to Consider

Complexity can arise without careful design. Questionnaires must remain clear throughout the process. Time requirements can stretch with multiple rounds. Participants' engagement may wane over time. Interpretation of consensus needs care. Ensuring all understand what consensus means is essential.

The Delphi Method offers a structured approach. It enables expert-driven strategic decision-making. Through careful design and execution, it promotes a depth of insight. Aligning this method with clear business objectives can lead to robust, strategic outcomes.

In what specific ways can the Delphi Method enhance the decision-making process within a business context?

The delphi method in business decision-making, structured communication.

The Delphi Method uses rounds of feedback. It starts with a question. Experts give anonymous responses. Anonymity reduces bias. It stops dominant personalities from controlling discussions. Each round refines the answers. Feedback evolves toward a collective agreement.

Consensus Building

Consensus is key in decision-making. The Delphi Method encourages it. Experts see what others think. They can adjust their views. Iterative rounds lead to a middle ground. This builds a shared solution. Agreed-upon decisions likely see better implementation.

Wide Range of Knowledge

Business challenges are often complex. The Delphi Method brings together diverse expertise. Each participant contributes special knowledge. This cross-pollination enriches the decision-making process. It leads to well-informed business decisions.

Future Scanning

Forecasting is tough. The Delphi Method excels here. Experts predict future trends. Their collective intelligence foresees potential outcomes. Businesses prepare better for what lies ahead.

Mitigating Groupthink

Groupthink can derail good decision-making. The Delphi Method counters this. Participants remain separate. They consider the issue independently. The structured nature of the method prevents premature consensus. It promotes critical thinking and solution diversity.

Actionable Outcomes

Decisions must lead to action. The Delphi Method produces clear outcomes. Each round sharpens the focus. The final consensus is actionable. Businesses know the next steps to take.

Cost-Effective

Meetings can be expensive and time-consuming. The Delphi Method needs no physical meetings. Experts participate remotely. This saves time and money. Decisions happen without costly disruptions.

In conclusion, the Delphi Method streamlines business decision-making. It brings experts together effectively. It encourages critical and independent thinking. It helps forecast future trends. It avoids groupthink. It produces clear, actionable outcomes. It also saves time and money. Businesses do well to embrace this powerful tool.

What are the potential challenges or limitations when implementing the Delphi Method in a business environment?

The Delphi method is a forecasting process framework. Experts answer questionnaires in two or more rounds. After each round, a facilitator provides an anonymous summary of the forecasts. They include reasons for judgments. Thus, the range of answers decreases. The group converges towards the "correct" answer.

Challenges of Implementing the Delphi Method

Expert selection.

Choosing the right experts is crucial. The outcome depends on their expertise. Diversity in expertise is necessary as well. It can prevent groupthink. Experts must be willing to revise their views. Stubbornness can hinder the process.

Time Considerations

The Delphi method is time-consuming. Multiple rounds are necessary for consensus. Experts need time for thoughtful responses. Rushed decisions can compromise the process's integrity. Coordinating schedules poses another challenge. Time zones can create delays.

Feedback Synthesis

Facilitator roles are critical in the Delphi method. They must summarize feedback effectively. Biased summaries can affect the results. Summaries should stimulate informed discussions. They should not influence them unduly.

Participation Motivation

Encouraging continued expert involvement is difficult. Initial enthusiasm can wane. Participation may drop in later rounds. Ensuring high response rates is challenging. The process requires sustained motivation.

Information Overload

The process can generate vast amounts of data. Handling this information is tough. Participants may have difficulty processing all the data. This can lead to suboptimal decision-making.

Cost Implications

The method can be costly. It involves expert time and resources. Budget constraints can limit the number of rounds. Consequently, this affects the quality of the output.

Technology Dependence

The Delphi method relies on technology. Online platforms are common for surveys. Any technical hiccups can disrupt the process. Reliable tech support is essential.

Anonymity Issues

Anonymity is both a strength and a challenge. It can reduce the effects of dominant personalities. But it can also lead to a lack of accountability. Participants may not feel responsible for their contributions.

Consensus vs. Agreement

Aiming for consensus is a challenge. True consensus is difficult to achieve. The facilitator’s definition of 'consensus' affects the outcome. Some may settle for general agreement. This might overlook minority opinions.

Limitations of the Delphi Method

Limited scope.

The method focuses on specific questions. It might not capture the complexity of the business environment. Contextual nuances can be lost.

Qualitative Data

The Delphi method generates qualitative data. Interpreting this data can be subjective. Numeric data is easier to measure. Qualitative data analysis requires careful handling.

Changing Environments

Business environments evolve rapidly. The Delphi method may become outdated. This happens before the process completes. New information may render previous rounds obsolete.

Complexity Management

Managing the complexity of the method is daunting. It requires meticulous planning. Also, a keen understanding of the process is necessary.

Potential Biases

Despite anonymity, biases can influence the process. Confirmation bias can surface in feedback interpretation. Facilitators need to remain neutral.

Adaptability Limitations

The Delphi method is less adaptable. Once the process starts, changing course is hard. This can be problematic in dynamic business settings.

The Delphi method has both merits and drawbacks. It provides structured expert judgment. Yet, the challenges and limitations warrant careful consideration. Implementing it in business demands a strategic approach. With prudence, it can be a powerful tool for decision-making.

He is a content producer who specializes in blog content. He has a master's degree in business administration and he lives in the Netherlands.

Checking Accuracy: Mastering Problem Solving Skills

7 Problem Solving Skills You Need to Succeed

Game Theory: Strategic Analysis and Practical Applications

Plato: A Beacon of Influence Through Desire, Emotion, & Knowledge

Delphi Technique in Project Management

Project managers are often required to make decisions, plan, and share their understandings with stakeholders. Sometimes, these tasks may seem impossible when they all need to reach a consensus during a meeting.

However, there is a technique you could use for this problem: the Delphi Technique. This technique can help you anticipate setbacks and address them before they happen. Explore how the Delphi Technique can revolutionize your project planning and execution in our latest blog post.

What is the Delphi Technique in Project Management?

The Delphi Technique is a virtual brainstorming session in which experts share their thoughts on a project without knowing who said what. It helps teams make smart decisions by considering many different opinions without any office politics getting in the way.

This method is used in project management to gather insights and achieve consensus among a panel of experts through multiple rounds of anonymous questionnaires. It reduces the impact of individual biases and encourages independent thought, leading to more accurate forecasting and decision-making.

Developed in the 1950s by the RAND Corporation for military applications, the Delphi Estimating Technique has since been widely adopted across various fields for its effectiveness in tackling complex issues and strategic planning without the need for face-to-face meetings.

Delphi Method Steps

Here's a glimpse into the step-by-step procedure that makes the Delphi approach a powerful tool for decision-making and forecasting in project management.

• Problem Definition: Clearly outline the issue or question that requires expert input.
• Expert Panel Selection: Choose a diverse group of experts familiar with the problem's domain.
• First-Round Questionnaire: Develop and send a questionnaire to the panel, asking for their opinions or predictions regarding the problem.
• Summarize Responses: Collect and summarize the responses, identifying areas of agreement and divergence without attributing opinions to specific individuals.
• Second-Round Questionnaire: Refine the original questions based on the summary and possibly add new questions. Send this second questionnaire to the panelists, encouraging them to reconsider their initial responses considering the group's feedback.
• Repeat Rounds as Necessary: Continue the feedback and revision cycle until convergence is achieved, typically after two or three rounds.
• Final Report: Compile the consensus opinions into a final report, including conclusions and recommendations based on the panel's insights.

Delphi Method Characteristics

At its core, the Delphi Method is:

• Iterative: It involves several rounds of questionnaires, with each cycle refining and clarifying responses to reach a well-informed consensus.
• Anonymous: Participants provide their insights independently, free from peer pressure and the influence of dominant personalities.
• Expert-driven: The method relies on the knowledge and expertise of selected panelists who are specialists in the topic under discussion.
• Consensus-seeking: Its primary goal is to converge on a common understanding or solution among all participants through structured communication.
• Remote participation: Experts participate from various locations, making it a flexible and inclusive approach that transcends geographical barriers.

Tips for Implementing the Delphi Technique

Implementing the Delphi Estimating Technique requires careful planning and execution. These practical tips will ensure its success:

• Clearly Define the Objective: Start with a clear, concise statement of the problem or decision. A well-defined objective guides the entire process and keeps the panel focused.
• Choose the Right Experts: Select diverse participants with relevant expertise. Consider different backgrounds, experiences, and perspectives to enrich the process.
• Design Thoughtful Questionnaires: Develop open-ended questions that encourage detailed, thoughtful responses. The quality of your questionnaires significantly impacts the quality of the insights gathered.
• Maintain Anonymity: Ensure the anonymity of responses to prevent bias and influence. This encourages open and honest feedback, allowing for a more accurate aggregation of expert opinions.
• Foster an Iterative Process: Be prepared for multiple rounds of questionnaires. Each round should refine and narrow the focus based on previous responses, driving toward a clearer consensus.
• Provide Summary Feedback After Each Round: Share a summary of the results from each questionnaire round with the participants. This helps experts reconsider their views regarding the group's collective input.
• Be Patient: The Delphi process is not a quick fix. Allow enough time between rounds for participants to digest the feedback and respond thoughtfully.
• Analyze and Present Results Clearly: After the final round, compile and analyze the results to present clear conclusions and actionable insights. This final report should reflect the collective wisdom of the expert panel.
• Use Technology Wisely: Leverage online survey tools and platforms for efficiently distributing and collecting questionnaire responses. This can also help maintain anonymity and convenience for participants.
• Prepare for Challenges: Be ready to address potential challenges such as participant dropout or data overload. A plan to manage these issues will help keep the process on track.

Delphi Diagrams

Visual representation, notably through Delphi diagrams, is an integral part of effectively understanding and implementing this method. These diagrams serve as a roadmap, elucidating each process phase, from problem identification to achieving consensus. They simplify the method's conceptualization for new learners and enhance the strategic planning and execution for experienced practitioners.

Delphi Forecasting

Delphi forecasting is a consensus-based technique in which a panel of experts anonymously provides estimates and predictions on future events through structured questionnaires. After each round, summaries of the forecasts and justifications are shared among the participants, encouraging them to revise their previous answers based on the collective feedback.

When is the Delphi Technique Useful?

Delphi Technique can be useful on several occasions:

• Forecasting Future Trends: When predicting future trends or technological advancements, especially in fields where hard data may be scarce or nonexistent.
• Problem-Solving: For complex problems that benefit from diverse perspectives, where the collective insight of experts can lead to innovative solutions.
• Policy and Planning: In developing policies, strategies, or planning scenarios where multiple stakeholders' opinions are valuable for comprehensive understanding.
• Consensus Building: When it is critical to achieve consensus among experts or stakeholders on contentious or ambiguous issues.
• Risk Management: This is used to identify and assess risks, especially when expert judgment is needed to evaluate potential impacts and likelihoods.
• Resource Allocation: Deciding on allocating limited resources in scenarios where expert opinions can guide prioritization based on potential outcomes or needs.

Delphi Technique Example

Context: A global software company plans to develop a revolutionary project management tool for remote teams. Given the diversity in team structures and workflows and the rapid evolution of remote work dynamics, the project poses unique challenges, including technical complexities, user experience considerations, and integration capabilities.

Application of the Delphi Technique:

Initiation: The project lead forms a panel of experts comprising experienced project managers, software developers, UI/UX designers, cybersecurity experts, and select end-users from various industries.

Round 1 Gathering Requirements: The first round involves collecting detailed input on essential features, potential security concerns, and critical integrations the new tool should offer to support remote teams effectively.

Feedback and Analysis: After collecting the responses, a facilitator summarizes the feedback, highlighting common themes such as the need for high-level security features, intuitive design for diverse user groups, and seamless integration with existing productivity tools.

Round 2 Prioritization: Experts are then asked to prioritize the identified features and integrations based on their perceived importance and feasibility, considering the project's scope and timeline.

Consolidated Feedback: The second summary reveals consensus on priority features, such as real-time collaboration capabilities, robust security protocols, and flexible integration options, along with a suggested prioritization list.

Round 3 Addressing Technical Challenges: With a clear understanding of what needs to be built, the panel focuses on discussing technical challenges and potential solutions, especially concerning scalability, data privacy, and user accessibility.

Final Consensus and Planning: The final round solidifies expert recommendations on overcoming technical hurdles alongside a refined feature list and development roadmap.

Outcome: Leveraging the insights gained through the Delphi research, the project team creates a detailed project plan that aligns with expert consensus on critical features and technical solutions. This structured approach ensures that the development of the project management tool is guided by comprehensive expertise, enhancing its potential to meet the diverse needs of remote teams globally.

Delphi Technique Using ActiveCollab

ActiveCollab, with its suite of collaboration tools, facilitates each step of the Delphi process, from questionnaire distribution to consensus building.

Step 1: Define the Objective

Task Creation: Start by defining the main objective of your Delphi study in ActiveCollab. Create a task with a clear title and description outlining the issue or decision point.

Step 2: Select Your Panel of Experts

Team Invites: Use ActiveCollab to invite your selected panel of experts. These can be team members within your organization or external consultants who have been given access to your ActiveCollab workspace.

Step 3: First Round of Questionnaires

Questionnaire Distribution: Distribute the first round of questionnaires by creating a task list labeled "Round 1" and assigning tasks (questions) to each expert. Alternatively, use the discussions or notes features to share the questionnaire and collect responses in a centralized location.

Step 4: Summarize Responses and Provide Feedback

Feedback Compilation: Summarize the key points and feedback once all responses are in. You can create a summary document in ActiveCollab's files section or start a discussion thread where you highlight the consensus points and areas of divergence.

Step 5: Second Round (and Subsequent Rounds)

Iterative Review: Based on the feedback, create a new task list for the second round (e.g., "Round 2"), refine the questions, or ask for more detailed insights on specific points. Continue this process, utilizing discussions and files to share summaries and foster dialogue until a consensus is reached.

Step 6: Achieving Consensus

Consensus Documentation: Once the final round concludes, fully document the consensus or decision. Use ActiveCollab's note-taking feature or upload a final document outlining the agreed-upon strategies, solutions, or forecasts.

Step 7: Implementation

Action Tasks: Translate the consensus into actionable tasks within ActiveCollab. Assign responsibilities, set deadlines, and track progress to ensure the implementation aligns with the outcomes of the Delphi process.

Benefits of the Delphi Technique

The Delphi analysis offers a structured communication method that brings numerous benefits to decision-making and forecasting processes.

• Reduction of Bias: Anonymity minimizes personal biases and the influence of dominant individuals.
• Access to Diverse Expertise: It harnesses a wide range of expert knowledge and perspectives.
• Reduction of Groupthink: The method encourages independent thinking by preventing peer pressure and conformity.
• Deliberate Feedback Loop: Iterative rounds allow for thoughtful reflection and refinement of ideas.
• Anonymity Fostering Honest Feedback: Participants are likelier to provide genuine feedback without fear of reprisal.
• Remote Collaboration Benefits: Facilitates input from globally dispersed experts without physical meetings.

Disadvantages and Challenges of Delphi Technique

The Delphi technique, while useful, faces several disadvantages and challenges:

• Time-Consuming: Requires multiple rounds of feedback, making the process lengthy.
• Dependence on Expert Selection: Outcomes heavily rely on the diversity and expertise of the selected panel.
• Risk of Misinterpretation: Written communication can lead to misunderstandings.
• Limited Feedback Interaction: Lacks dynamic exchange of ideas in direct discussions.
• Information Overload: The volume of feedback may overwhelm participants.

Fundamentals of Agile Project Management

*Enter your email address and subscribe to our newsletter to get your hands on this, as well as many other free project management guides.

We weren't able to subscribe you to the newsletter. Please double-check your email address. If the issue persists, let us know by sending an email to [email protected]

Newsletter subscribers can download all free materials

Start your trial today, free for 14 days ! Onboard your team, plan, collaborate, organize your work, and get paid.

By signing up you are agreeing to the ActiveCollab Terms of Service & Privacy Policy.

Choose your favorite topics and we’ll send our stories from the tech front lines straight to your inbox.

Unsubscribe at any time * ActiveCollab Privacy Policy

Just a second

Awesome! Thank you for subscribing to our newsletter.

Oops, something went wrong! Please try again later.

Related Articles

We detected that you already have an ActiveCollab account

You can log in to an excisting account or you may start a new one

Great, just a few seconds and you're in.

All done! Redirecting you to your account.

We've sent you an email to confirm that it's you. Please check your email to complete the trial account creation.

Sorry, we could not create an account for you at this moment. Please double check your email address. If the issue still persists, please let us know by sending an email to [email protected]

• The Delphi Method: Meaning, Pros, Cons & Uses

Introduction

The Delphi Method is an iterative process often used for gathering and developing consensus among a group of experts. It is a process that combines the use of anonymous surveys and group discussions in order to reach a consensus. 

The Delphi Method is often used in situations where there is a lack of agreement or an absence of knowledge or expertise. In this article, we will explore the Delphin method, its pros and cons, and its importance.

What is the Delphi Method? 

The Delphi Method is a technique used to solicit and compile input from a group of experts in order to better understand a problem or to develop a solution. It is a process that combines the use of anonymous surveys and group discussions in order to reach a consensus.

The Delphi Method is most commonly used when there is a lack of agreement or an absence of knowledge or expertise. It is well suited to situations in which the experts are geographically dispersed and/or have different backgrounds and/or interests. 

This means that it is an invaluable tool for complex decision-making and problem-solving, and it is a great way to gain insight from experts without the need for face-to-face meetings. It is also beneficial for developing a well-rounded view of an issue, as the process allows for the inclusion of a variety of perspectives.

This method also allows experts to provide input without feeling pressure from the group. Furthermore, the Delphi Method was developed by the RAND Corporation in the 1950s and has been used in a variety of fields including business, public policy, health care, and engineering.

It is no surprise that it has become an increasingly popular decision-making tool for groups and organizations, as it allows for a diverse range of opinions to be considered without a single person dominating the discussion. It can also be conducted remotely and over a short period of time.

Characteristics of the Delphi method

The Delphi Method has several key characteristics that make it an effective tool for gathering and developing consensus among a group of experts.

• Anonymity. All responses are anonymous and the identity of the participants is kept confidential. This allows experts to provide input without feeling pressure from the group.
• Iteration. The Delphi Method is an iterative process. After the initial round of surveys are distributed, there may be several rounds of surveys sent out as the experts’ responses are collected and discussed.
• The use of experts: The Delphi Method requires the use of experts who have knowledge or experience in the area being discussed. The experts are typically selected based on their expertise and knowledge.
• Consensus. The Delphi Method is designed to reach a consensus among the experts. This is accomplished by having the experts provide input, discussing the results, and then revising the survey questions or making changes to the process.

Uses of the Delphi Method

The Delphi Method is a useful tool in situations where there is a need to reach a consensus on a complex issue. It has been used in a variety of contexts, including policy development, strategic planning, and organizational decision-making. It can also be used to generate innovative ideas and to support the development of creative solutions to problems.

The experts are asked a series of questions, and the responses are then used to form the basis of the market research. The experts are asked to provide their opinions on a particular subject, and their responses are used to form the opinion of the panel as a whole.

• The Delphi Method can be used to help companies make decisions about their marketing strategies. For example, a company could use the Delphi Method to gain insight into how their customers perceive their products or services. The panel of experts could provide valuable insights into the customer’s opinions on the company’s offerings. This information can then be used to develop more effective marketing strategies.
• The Delphi Method can also be used to assess potential risks and opportunities in the market. By gathering opinions from experts, companies can gain a better understanding of the potential threats and opportunities that may exist in the market. This information can then be used to develop strategies to mitigate these risks and take advantage of the opportunities.
• The Delphi Method can be used to assess customer satisfaction levels. Companies can use the responses from the panel of experts to determine how satisfied their customers are with their products and services. This information can then be used to make changes to the product or service in order to increase customer satisfaction.

Steps To Carry Out the Delphi Method

• Define the Problem: The first step in the Delphi Method is to define the problem or issue that needs to be addressed. This should include a clear description of the problem as well as the goal that needs to be achieved.
• Identify Experts: Once the problem has been defined, the next step is to identify experts who can provide valuable insights into the issue. This can include people from outside the organization, such as consultants or industry experts.
• Gather Data: The third step is to collect data from the identified experts. This can include surveys, interviews, or other methods of gathering information.
• Analyze the Data: Once the data has been gathered, it should be analyzed to identify patterns and trends in the information. This can help to identify potential solutions to the issue.
• Develop Solutions: After the data has been analyzed, the next step is to develop potential solutions to the problem. This should include a range of ideas and suggestions from the experts.
• Reach Consensus: Finally, a consensus should be reached on the best solution to the issue. This is done through a series of iterations between the experts, where they discuss and debate the different solutions until they reach an agreement.

The Delphi method has been used for decades across a wide range of industries, from healthcare and finance to education and engineering. By leveraging the collective knowledge and insights of experts, the Delphi method is able to generate a consensus-based understanding of the situation or topic. 

However, just like any other decision-making process, the Delphi method has both advantages and disadvantages. 

Advantages of the Delphi Method 

• Cost-Effective: The Delphi method is a cost-effective way of gathering opinions and insights from experts. It eliminates the need for costly in-person meetings or travels. 
• Anonymity: This is one of the best advantages of the Delphi method as it allows experts to provide their opinions anonymously, thus eliminating the potential for bias. 
• Reach: Another advantage of the Delphi method is that it can be used to reach experts who are geographically dispersed and would not be able to attend an in-person meeting. 
• Efficiency: It is an efficient way to generate a consensus opinion without needing to bring all experts together in one place. 
• Flexibility: The Delphi method can be used in a variety of situations and adapted to the specific needs of the issue or topic at hand.

Disadvantages of the Delphi Method

• Time-Consuming: The Delphi method can be time-consuming, as it requires multiple rounds of feedback from experts. 
• Lack of Face-to-Face Interaction: This method does not allow for face-to-face interaction between experts, which can lead to a lack of understanding of each other’s perspectives. 
• Potential for Groupthink: The Delphi method can lead to groupthink, as experts may be reluctant to express dissenting opinions for fear of being seen as not aligning with the group. 
• Inaccurate Interpretation of Results: This occurs because the Delphi method relies on the interpretation of the facilitator, which can lead to inaccurate results if the facilitator is not trained in the method. 
• Subjectivity: As the Delphi method is based on the expertise and opinions of the participants, it can be subjective and potentially lead to biased results. 

In conclusion, it is an efficient, cost-effective way to gain insight from experts, while developing a well-rounded view of an issue. The Delphi method also provides insight from experts that can be geographically dispersed and/or have different backgrounds and/or interests. 

This insight is invaluable for organizations looking to make informed decisions and this is why the Delphi method is efficient.

Connect to Formplus, Get Started Now - It's Free!

• Anonymous survey
• group interviews
• survey research
• the delphi method

You may also like:

Pricing Surveys: Methods, Tips & Templates

Pricing is a major motivator for most customers when purchasing a product or service. Customers are always looking for the best deal, an...

Matrix Question Surveys: Types, Examples, Pros & Cons

Introduction Matrix questions are a type of survey question that allows respondents to answer multiple statements using rates in rows...

What is a Control Survey? Types, Importance + Process

Introduction A control survey refers to a survey conducted to map out the positions of existing control points in the survey area. It is...

Survey Straightlining: Definition, Implications & Mitigation

Introduction Survey straight lining occurs when the respondents of a survey in haste, select the same response every time....

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

• Wed. Jul 3rd, 2024

HRM Leaders

Leading the Leaders...

Delphi Technique a Step-by-Step Guide

By Mayghan Dolmy

As a project manager, it is important to think about what future events may impact your projects. These events may be positive or negative, so understanding them allows you to prepare, and put plans in place to deal with them. But how can you forecast the future with any degree of certainty? The Delphi Technique can help.

The Delphi Technique is a method used to estimate the likelihood and outcome of future events. A group of experts exchange views, and each independently gives estimates and assumptions to a facilitator who reviews the data and issues a summary report.

The group members discuss and review the summary report, and give updated forecasts to the facilitator, who again reviews the material and issues a second report. This process continues until all participants reach a consensus.

The experts at each round have a full record of what forecasts other experts have made, but they do not know who made which forecast. Anonymity allows the experts to express their opinions freely, encourages openness and avoids admitting errors by revising earlier forecasts.

This article looks at how to run a Delphi session. On completion of this guide, you will be able to run a session enabling you to predict future events and their likely impact on your projects.

The technique is an iterative process, and first aims to get a broad range of opinions from the group of experts. The results of the first round of questions, when summarised, provide the basis for the second round of questions. Results from the second round of questions feed into the third and final round.

The aim is to clarify and expand on issues, identify areas of agreement or disagreement and begin to find consensus.

Step 1: Choose a Facilitator

The first step is to choose your facilitator. You may wish to take on this role yourself, or find a neutral person within your organisation. It is useful to have someone that is familiar with research and data collection.

Step 2: Identify Your Experts

The Delphi technique relies on a panel of experts. This panel may be your project team, including the customer, or other experts from within your organisation or industry. An expert is, any individual with relevant knowledge and experience of a particular topic. ¹

Step 3: Define the Problem

What is the problem or issue you are seeking to understand? The experts need to know what problem they are commenting on, so ensure you provide a precise and comprehensive definition.

Step 4: Round One Questions

Ask general questions to gain a broad understanding of the experts view on future events. The questions may go out in the form of a questionnaire or survey. Collate and summarise the responses, removing any irrelevant material and looking for common viewpoints.

Step 5: Round Two Questions

Based on the answers to the first questions, the next questions should delve deeper into the topic to clarify specific issues. These questions may also go out in the form of a questionnaire or survey. Again, collate and summarise the results, removing any irrelevant material and look for the common ground. Remember, we are seeking to build consensus.

Step 6: Round Three Questions

The final questionnaire aims to focus on supporting decision making. Hone in on the areas of agreement. What is it the experts are all agreed upon?

You may wish to have more than three rounds of questioning to reach a closer consensus.

Step 7: Act on Your Findings

After this round of questions, your experts will have, we hope, reached a consensus and you will have a view of future events. Analyse the findings and put plans in place to deal with future risks and opportunities to your project.

Use the Delphi Technique for creating Work Breakdown Structures, identifying risks and opportunities, compiling lessons learned and anytime you would usually conduct a brainstorming session.

Predicting the future is not an exact science, but the Delphi Technique can help you understand the likelihood of future events and what impact they may have on your project.

Maygan Dolmy, - Human Resources Consultant at Rite Hire, UK - London, United Kingdom - https://www.linkedin.com/in/maygan-dolmy-a8a1971a9/ The opinions expressed in this publication are those of the authors. They do not purport to reflect the opinions or views of the HRM Leaders or its members.

Related Post

Designing an ai-assisted model for hrm strategy: a comprehensive framework, hr policy manual- free sample, employee motivation questionnaires or surveys, leave a reply cancel reply.

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

Administrative Role of HR in Qatar

Employee absenteeism, mental health at workplace: guide for using wap.

How to use the nominal group and Delphi techniques

• Review Paper
• Open access
• Published: 05 February 2016
• Volume 38 , pages 655–662, ( 2016 )

Cite this article

You have full access to this open access article

• Sara S. McMillan 1 ,
• Michelle King 1 &
• Mary P. Tully 2 , 3  

37k Accesses

837 Citations

33 Altmetric

Explore all metrics

Introduction The Nominal Group Technique (NGT) and Delphi Technique are consensus methods used in research that is directed at problem-solving, idea-generation, or determining priorities. While consensus methods are commonly used in health services literature, few studies in pharmacy practice use these methods. This paper provides an overview of the NGT and Delphi technique, including the steps involved and the types of research questions best suited to each method, with examples from the pharmacy literature. Methodology The NGT entails face-to-face discussion in small groups, and provides a prompt result for researchers. The classic NGT involves four key stages: silent generation, round robin, clarification and voting (ranking). Variations have occurred in relation to generating ideas, and how ‘consensus’ is obtained from participants. The Delphi technique uses a multistage self-completed questionnaire with individual feedback, to determine consensus from a larger group of ‘experts.’ Questionnaires have been mailed, or more recently, e-mailed to participants. When to use The NGT has been used to explore consumer and stakeholder views, while the Delphi technique is commonly used to develop guidelines with health professionals. Method choice is influenced by various factors, including the research question, the perception of consensus required, and associated practicalities such as time and geography. Limitations The NGT requires participants to personally attend a meeting. This may prove difficult to organise and geography may limit attendance. The Delphi technique can take weeks or months to conclude, especially if multiple rounds are required, and may be complex for lay people to complete.

Similar content being viewed by others

Consensus Methodologies and Producing the Evidence

Consensus Methods for Health Research in a Global Setting

Avoid common mistakes on your manuscript.

Introduction

The Nominal Group Technique (NGT) and the Delphi Technique are commonly referred to as consensus methods [ 1 ]. They aim to achieve a general agreement or convergence of opinion around a particular topic. Consensus methods are used in research that is directed at problem-solving, idea-generation, or determining priorities [ 2 ]. How consensus is defined and operationalised will vary from study to study, depending on the research objectives [ 3 ].

Consensus techniques such as the NGT and Delphi Technique are superficially similar to focus groups, a commonly used method in pharmacy practice research. All methods involve interaction within a group of participants, yet they can provide different outcomes. Focus groups are useful for investigating an issue in-depth, including the identification of problems, questions or significant issues. Consensus methods, however, raise potential solutions or answers to a question, which can then be prioritised or agreed upon. A key strength of consensus methods is the balanced participation from group members, unlike a focus group, whereby the facilitator must control for, and minimise the risk of, a dominant participant influencing the discussion. The structured format of consensus methods avoids this issue.

The aim of this paper is to provide an overview of the NGT and Delphi technique, including the steps involved and the types of research questions best suited to each method, with examples from the pharmacy literature. Therefore, it provides a useful starting point for pharmacy practice researchers new to consensus methods. Initially it describes how to conduct the NGT and Delphi Technique and provides examples of their use within the pharmacy context. Then, it considers the choice of experts for the panels and which types of research questions are best suited to which method.

• Nominal group technique

The NGT is a highly structured face-to-face group interaction, which empowers participants by providing an opportunity to have their voices heard and opinions considered by other members [ 4 ]. It was designed by Delbecq and Van de Ven and comprises four key stages: silent generation, round robin, clarification and voting (ranking or rating) [ 2 ]. These stages are briefly explained below.

How to run the nominal group technique

While groups of between two and fourteen participants have been used in nominal group research (Table  1 ), a maximum of seven has been recommended [ 5 ]. A nominal group generally involves one to two questions which are sent to participants in advance. At the beginning of the meeting, participants are given up to twenty minutes to silently reflect or record their individual ideas in response to a question, i.e. silent generation [ 6 ]. The facilitator then asks one participant at a time to state a single idea to the group in a ‘round robin’ fashion. Participants are able to think of new ideas during this process, but must wait their turn before they can share with the group. This stage takes as much time as needed until no new ideas are forthcoming. It is recommended that there be no discussion at this stage and ideas are merely recorded verbatim on, for example, a flipchart or white board [ 2 ].

The third stage is clarification of the ideas, which also provides the opportunity for a grouping step, where similar ideas are grouped together with agreement from all participants. Participants may also exclude, include or alter ideas, as well as generate grouping themes [ 7 ]. All ideas should be discussed to ensure participant understanding [ 2 ], thus enabling them to make an informed decision when they come to voting on ideas. Facilitators should emphasise that participants do not have to agree with all ideas listed as, at the end of the clarification stage participants are able to ignore ideas by voting on personal preferences. The round robin [ 8 ] and clarification phase [ 9 ] can take up to 30 min each. Facilitators should not direct participants during the clarification process, which may make this stage particularly difficult.

Participants are then provided with a ranking sheet, where they are asked to select their top preferences from the generated ideas. The number of items chosen by participants depends on the topic, but the ranking of five ideas is common in the literature [ 2 , 5 , 10 ]. The facilitator should specify that a number should be allocated to each selected item, with larger numbers reflecting greater importance [ 2 , 5 ]. For example, for five ideas, the most important idea is scored five points. Although there is no anonymity for participants during nominal group discussions, individual scoring on a ranking sheet is confidential. Finally, the scores for each idea are summed and presented to the group for discussion. The timing for this stage is likely to depend on a number of factors, including the complexity of the topic and how many items need to be prioritised (the more items to rank, the harder the process and more time consuming it can become). Dening et al. [ 10 ] noted that voting could take up to 10 min to complete.

Ultimately, the time to complete one nominal group is variable, and depends on group size, how many questions are asked, and the type of participants involved. For example, Bradley et al. [ 11 ] documented a 2-h time limit to conduct a NGT for one question, whereas Hutchings and colleagues allocated half a day to conduct a NGT for two questions, followed by another half-day for a forum event [ 12 ] (see “ Variations on the nominal group technique ”).

Variations on the nominal group technique

The NGT is a highly adaptable method, and can be used in addition to [ 7 ] or to inform, other methods, e.g. a discrete choice experiment [ 13 ]. NGT variations may be influenced by the available research and participant time, or the level of clarification, consensus or generalisability required for the topic. Ultimately, researchers need to ensure that the NGT is working for each participant group; it may be that stages need to be adapted. For example, for indigenous or culturally and linguistically diverse populations, it may be the cultural norm to discuss ideas as a group. Thus, a more appropriate variation to the process for generating ideas has been to combine the round robin and clarification stages [ 5 ]. Other variations could be in direct response to participant ability. If it is too difficult for participants to group similar ideas at the clarification stage then this grouping step could be avoided altogether [ 5 ]. While this may make it harder for participants to vote, i.e. there is a longer list of ideas to consider, it may cause less frustration for participants.

Generally, variations are seen in relation to generating ideas, and how ‘consensus’ is obtained from participants, i.e. the ranking process (Fig.  1 ):

A simplified model of the NGT process and possible adaptions from the literature. *Traditional nominal group process is given in bold

Generating ideas instead of silent generation followed by a round robin, ideas are obtained from a literature review[ 13 ], or exploratory surveys are used which could be viewed as a way to achieve greater consultation [ 14 , 15 ];

Ranking this may be completed by either allocating a score[ 16 ] or by a rating on a Likert scale [ 15 ];

Re - ranking allowing participants to revise their original ranking, i.e. re-ranking, either in the original NGT meeting [ 9 ], via a secondary survey [ 14 ], or obtaining validation by sending a survey of nominal group results to other participants [ 15 ]. Alternatively, the re-ranking process could continue until no further changes are seen with the most important ideas [ 13 ].

Where separate nominal groups are held for similar participants, e.g. consumer groups, health professional groups or stakeholder groups, a mixed-forum event can provide the opportunity for consensus to be achieved by forming new groups with different participant types [ 12 ]. In a study that exemplified the use of a mixed forum event, Hutchings and colleagues asked previous participants to individually review the overall NGT results (overarching themes), and to rank the themes (pre-forum responses). At the forum, participants were asked to discuss the pre-forum responses in their newly allocated groups, which consisted of participants from differing backgrounds. Individuals were then asked to re-rank themes for a third time.

Other researchers have provided valuable information on important nominal group design considerations [ 3 ], its practical application [ 4 ] and method of analysis [ 5 ]. Black et al. [ 3 ] reviewed the literature to identify the evidence for certain ‘best practice recommendations’ for consensus methods. While that review provides some important considerations for researchers wishing to use these methods, the articles included are, at a minimum, over 15 years old. Using specific examples, Tully and Cantrill [ 4 ] discuss the steps involved in a nominal group, and guidance for researchers with respect to group composition. While a discussion of qualitative and quantitative analysis is also included, McMillan et al. [ 5 ] take this one step further in their paper by detailing the entire analysis process for researchers who undertake more than ten nominal groups.

Applications to pharmacy research

The NGT has been applied in numerous healthcare settings, to develop guidelines [ 17 ] or explore opinions of different health professionals [ 18 ], lay people and carers [ 10 , 19 , 20 ], or to compare views of both parties [ 9 , 21 ]. It is gradually building traction within the pharmacy setting, as seen in Table  1 . Researchers have generated evidence based guidelines or criteria for pharmacy practice situations [ 7 , 22 , 23 ], informed practice change [ 11 , 24 ] and the profession [ 12 , 16 ] about particular topics, and identified attributes to be included when interviewing pharmacy students [ 25 ].

The Delphi technique

Like the NGT, the Delphi Technique is a highly structured group interaction. However, the Delphi Technique uses interactions between group (called panel) members via questionnaires rather than face-to-face communication. This means that it preserves participant anonymity, if that is relevant. The Delphi Technique was developed by the Rand Corporation in 1953 [ 26 ] and uses a multistage self-completed questionnaire with individual feedback.

How to run the Delphi technique

There is no standard method to calculate a panel size for the Delphi Technique; however, the aim of the study and available resources are important [ 27 ]. A sample of about fifteen has been suggested [ 26 ] but larger panels have also been used (Table  2 ). Inviting more participants increases the variety of expertise, but eventually leads to diminishing returns [ 3 ].

The first-round questionnaire will present a series of statements that the respondent is asked to rate on a clearly defined Likert scale. The content of the statements may come from a variety of sources, singly or in combination, including the literature [ 28 – 30 ], clinical practice [ 31 , 32 ], or from previous research findings, including NGT studies [ 30 , 33 ]. Respondents are asked both to rate the item and to write free-text comments that, for example, explain their rating or express disagreement with the statement’s relevance. Reminders are sent to non-responders in the usual way.

The responses to the first-round questionnaires are collated and used to create the second-round questionnaire. The latter presents the same statements as before, together with both the individual respondent’s rating and the median rating from the entire panel. A selection of the free-text responses is given, to represent the breadth of opinion. Respondents to the previous round thus get a personalised, unique questionnaire. Figure  2 provides an example of a statement from a second-round questionnaire seeking consensus on indicators for assessing medicines reconciliation processes [ 34 ]. After considering the group median and free-text comments, respondents re-rate the statements, by either giving the same rating as before or an amended rating. Respondents may give further comments about the statements if they wish.

Example of individual feedback for a second-round questionnaire in a Delphi study [ 34 ]. Respondents were given a definition of appropriateness and asked to assess the appropriateness of indicators of medicines reconciliation

The number of survey rounds is usually decided in advance and is dependent upon the level of dissension expected. In most studies, two rounds are used but occasionally, only a single round has been run [ 35 ]. More than two rounds increases panel attrition, so this is rarely done. The minimum time for a two-round Delphi can be as long as 30 days, although it may well take longer if multiple reminders are needed. The time required for the collation of responses and the creation of personalised second-round questionnaires should not be underestimated.

Often a 9-point Likert scale is used for the rating [ 29 – 31 , 34 ], although 3-point [ 36 ], 5-point [ 28 , 37 ] and 7-point [ 33 ] scales have also been used. The decision as to when consensus will have been reached must be made at the beginning of the study. For example, if the aim is to develop assessment criteria using the RAND 9-point scale [ 38 ], then consensus is reached that a statement is appropriate if the median score is greater or equal to 7, and it is inappropriate if the median score is less than or equal to 3. Disagreement is defined as where at least one third of respondents rate the statement at the opposite end of the scale to their peers. Such a finding would mean that consensus had not been reached.

Variations on the Delphi technique

A common variant is for the initial study questionnaire to collect ideas in response to open questions [ 26 ]. Only subsequent questionnaires then ask respondents to conduct the rating process described above [ 33 , 37 , 39 ].

Examples of other modifications include researchers including only items that had failed to reach consensus in the second questionnaire (rather than all items, regardless of the ratings they received initially [ 29 , 36 ]) and asking respondents to choose between alternatives (rather than rate items) for each question [ 32 ]. Traditionally, the questionnaire was sent by post, but more recently, e-mail has been used for the so-called e-Delphi Technique [ 40 ]. Emailed questionnaires per se are now so commonplace, that this is probably the new norm.

The RAND appropriateness method has been described as a variant of both the Delphi technique [ 38 ] and the NGT [ 3 ], as it has features of both. It involves participants reading a detailed literature review, followed by a traditional Delphi questionnaire. However, participants discuss the first-round results at a face-to-face meeting, followed by a second-round Delphi questionnaire and re-rating of the items.

An early use of the Delphi Technique in pharmacy practice research was in forecasting the future of hospital pharmacy in Australia [ 41 ]. It has been used to gain consensus on indicators for assessing prescribing appropriateness [ 33 ] or quality [ 31 ], criteria for safety features [ 36 ], clinically significant interactions [ 28 ] or aspects of student education [ 37 ] including communication skills [ 30 ] and professional engagement [ 35 ], or definitions, such as prescribing error [ 29 ]. This range of topics reflects the common use of the technique for the generation of clinical guidelines within the wider healthcare arena [ 3 ].

Choice of experts

Experts, in the context of consensus methods, are those people who have knowledge about the topic of concern. Understandably, this is dependent upon the research aims and objectives, but such experts may not always be healthcare professionals. Given that greater importance has been placed on involving health consumers in research, consensus methods can be used to identify what is currently important to, or valued by, these experts. McMillan and colleagues, for example, explored the views of both the public and pharmacy staff on ideal community pharmacy services [ 16 ]. Therefore, their experts included people living with chronic conditions, their carers, and pharmacy staff who provided the relevant services. Campbell and colleagues, on the other hand, identified prescribing indicators that used data from dispensed prescriptions [ 31 ]. Therefore, their experts were the medical and pharmaceutical advisors who would be using the resultant indicators.

The NGT appears to be used more commonly with lay people than the Delphi Technique, although the reason why is not clear. Lay people may feel more comfortable participating in a face-to-face meeting, than in a relatively complex survey. For example, the NGT can be adapted to accommodate people with poor literacy [ 5 ]. The Delphi Technique has been used with patients in a small number of studies to prioritise outcome measures for clinical trials [ 42 ] and has begun to be used with members of the public (in this case, parents of children with attention deficit/hyperactivity disorder) in pharmacy practice research [ 43 ].

Power differentials between people in the NGT may mean that people with less power may feel unable to contribute their own views or contradict the views of someone more powerful. Therefore, it is usual for the experts in each meeting to be relatively homogeneous in status (see Table  1 ), such as running separate meetings for consumers and pharmacists [ 5 ]. This power differential may be less relevant for the Delphi Technique, as the experts are anonymous. Nonetheless, in those few Delphi studies that included both lay people and healthcare professionals, only patient data from the first-round questionnaire was sent to patients in the second-round questionnaire [ 44 ].

Choice of consensus method

The decision whether to use the NGT or the Delphi Technique is influenced by various factors, including the research question, the perception of consensus required, and the associated practicalities and limitations such as time and geography.

If researchers are seeking to explore ideas in relation to a problem or question, this best aligns with the NGT, as idea generation is an integral part of this method. If researchers want to develop guidelines, a Delphi Technique involving experts who are likely to use the guidelines in question would be more suitable. The development of guidelines requires a more rigorous process, with consensus needed from a larger number of experts, which is easier with the Delphi Technique. This larger group may be needed to give authority to the final decision [ 3 ].

While some researchers have specified a numerical level of consensus when using the NGT, this is not well documented and would likely require further re-ranking beyond the initial steps. Alternatively, most researchers using the Delphi method will explicitly refer to a consensus value, i.e. a numerical level of agreement, determined by researchers in advance. Thus, it could be viewed that these two techniques sit along a spectrum of consensus, with a clearer description of the level of agreement thought to be given by the Delphi Technique [ 3 ].

As the NGT involves participants for only a few hours, results can be obtained quickly, suiting researchers who require a prompt result. It is particularly suited if participants are likely to only want to attend a single session compared to answering multiple questionnaires several weeks apart. The NGT requires face-to-face meetings, but this may be more culturally appropriate even if participants are at a distance. However, it may be more difficult to organise a nominal group meeting for a time that suits everyone. In contrast, the Delphi Technique is more flexible. The Delphi Technique, especially if conducted by email, is accessible to participants regardless of location, thereby avoiding travel expenses. Yet, this method can take weeks or months to conclude, especially if multiple rounds are undertaken.

The NGT and Delphi Technique are both consensus methods that involve a group of ‘experts’ to generate ideas and determine priorities. The NGT has been used to explore consumer and stakeholder views, while the Delphi technique is commonly used to develop guidelines with health professionals. The NGT requires face-to-face discussion in small groups, and provides a prompt result for researchers. Alternatively, the Delphi technique uses questionnaires to preserve participant anonymity, can involve more participants but takes place over a longer time period.

Jones J, Hunter D. Consensus methods for medical and health services research. BMJ. 1995;311:376–80.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Delbecq AL, van de Ven AH, Gustafson DH. group techniques for program planning, a guide to nominal group and Delphi processes. Glenview, IL: Scott, Foresman and Company; 1975.

Google Scholar  

Black N, Murphy M, Lamping D, McKee M, Sanderson C, Askham J, et al. Consensus development methods: a review of best practice in creating clinical guidelines. J Health Serv Res Policy. 1999;4(4):236–48.

CAS   PubMed   Google Scholar  

Tully MP, Cantrill JA. The use of the Nominal group technique in pharmacy practice research: processes and practicalities. J Soc Admin Pharm. 1997;14:93–104.

McMillan SS, Kelly F, Sav A, Kendall E, King MA, Whitty JA, et al. Using the nominal group technique: how to analyse across multiple groups. Health Serv Outcomes Res Method. 2014;14:92–108.

Article   Google Scholar  

Claxton JD, Ritchie JRB, Zaichkowsky J. The nominal group technique: its potential for consumer research. J Consum Res. 1980;7:308–13.

Bissell P, Ward PR, Noyce PR. Appropriateness measurement: application to advice-giving in community pharmacies. Soc Sci Med. 2000;51(3):343–59.

Article   CAS   PubMed   Google Scholar  

Potter M, Gordon S, Hamer P. The nominal group technique: a useful consensus methodology in physiotherapy research. N Z J Physiother. 2004;32:126–30.

Gallagher M, Hares T, Spencer J, Bradshaw C, Webb I. The nominal group technique: A research tool for general practice? Fam Pract. 1993;10:76–81.

Dening KH, Jones L, Sampson EL. Preferences for end-of-life care: a nominal group study of people with dementia and their family carers. Palliat Med. 2012;27(5):409–17.

Article   PubMed   Google Scholar  

Bradley F, Schafheutle EI, Willis SC, Noyce PR. Changes to supervision in community pharmacy: pharmacist and pharmacy support staff views. Health Soc Care Community. 2013;21(6):644–54.

PubMed   Google Scholar  

Hutchings HA, Rapport FL, Wright S, Doel MA, Wainwright P. Obtaining consensus regarding patient-centred professionalism in community pharmacy: nominal group work activity with professionals, stakeholders and members of the public. Int J Pharm Pract. 2010;18(3):149–58.

Hiligsmann M, van Durme C, Geusens P, Dellaert BG, Dirksen CD, van der Weijden T, et al. Nominal group technique to select attributes for discrete choice experiments: an example for drug treatment choice in osteoporosis. Patient Prefer Adherence. 2013;7:133–9.

Article   PubMed   PubMed Central   Google Scholar  

Allen J, Dyas J, Jones M. Building consensus in health care: a guide to using the nominal group technique. Br J Community Nurs. 2004;9(3):110–4.

Vella K, Goldfrad C, Rowan K, Bion J, Black N. Use of consensus development to establish national research priorities in critical care. BMJ. 2000;320(7240):976–80.

McMillan SS, Kelly F, Sav A, Kendall E, King MA, Whitty JA, et al. Consumers and carers versus pharmacy staff: Do their priorities for Australian Pharmacy Services align? Patient. 2015;8:411–22.

Duncan E. The nature and use of consensus methodology in practice. In: Kielhofner G, editor. Research in occupational therapy: methods of inquiry for enhancing practice. Philidelphia: F.A. Davis; 2006. p. 401–10.

Harvey N, Holmes CA. Nominal group technique: an effective method for obtaining group consensus. Int J Nurs Pract. 2012;18(2):188–94.

Elliott T, Shewchuk R. Using the nominal group technique to identify the problems experienced by persons living with severe physical disabilities. J Clin Psychol Med S. 2002;9(2):65–76.

Miller D, Shewchuk R, Elliot TR, Richards S. Nominal group technique: a process for identifying diabetes self-care issues among patients and caregivers. Diabetes Educ 2000;26(2):305–10, 312, 314.

Drennan V, Walters K, Lenihan P, Cohen S, Myerson S, Iliffe S, et al. Priorities in identifying unmet need in older people attending general practice: a nominal group technique study. Fam Pract. 2007;24(5):454–60.

Bond CM, Watson MC. The development of evidence-based guidelines for over-the-counter treatment of vulvovaginal candidiasis. Pharm World Sci. 2003;25(4):177–81.

Tully MP, Cantrill JA. Exploring the domains of appropriateness of drug therapy, using the nominal group technique. Pharm World Sci. 2002;24:128–31.

Gastelurrutia MA, Benrimoj SI, Castrillon CC, de Amezua MJ, Fernandez-Llimos F, Faus MJ. Facilitators for practice change in Spanish community pharmacy. Pharm World Sci. 2009;31(1):32–9.

Cameron AJ, MacKeigan LD. Development and pilot testing of a multiple mini-interview for admission to a pharmacy degree program. Am J Pharm Educ 2012;76(1):10.

Linstone HA, Turoff M. The Delphi survey: method techniques and applications. Reading: Addison-Wesley; 1975.

Campbell SM, Cantrill JA. Consensus methods in prescribing research. J Clin Pharm Ther. 2001;26(1):5–14.

Chan A, Tan SH, Wong CM, Yap KY, Ko Y. Clinically significant drug–drug interactions between oral anticancer agents and nonanticancer agents: a Delphi survey of oncology pharmacists. Clin Ther. 2009;31(Pt 2):2379–86.

Dean B, Barber N, Schachter M. What is a prescribing error? Qual Health Care. 2000;9:232–7.

Mackellar A, Ashcroft DM, Bell D, James DH, Marriott J. Identifying criteria for the assessment of pharmacy students’ communication skills with patients. Am J Pharm Educ. 2007;71(3):50.

Campbell SM, Cantrill JA, Roberts R. Prescribing indicators for UK general practice: Delphi consultation study. BMJ. 2000;321:1–5.

Baysari MT, Westbrook JI, Egan B, Day RO. Identification of strategies to reduce computerized alerts in an electronic prescribing system using a Delphi approach. Stud Health Technol Inform. 2013;192:8–12.

Cantrill JA, Sibbald B, Buetow S. Indicators of the appropriateness of long term prescribing in general practice in the United Kingdom: consensus development, face and content validity, feasibility and reliability. Qual Health Care. 1998;7:130–5.

Aljamal M, Ashcroft DM, Tully MP. Development of indicators to assess the quality of medicines reconciliation at hospital admission: an e-Delphi study. Int J Pharm Pract 2016 (in press).

Aronson BD, Janke KK, Traynor AP. Investigating student pharmacist perceptions of professional engagement using a modified Delphi process. Am J Pharm Educ. 2012;76(7):125.

Cassar Flores A, Marshall S, Cordina M. Use of the Delphi technique to determine safety features to be included in a neonatal and paediatric prescription chart. Int J Clin Pharm. 2014;36(6):1179–89.

McDermott JH, Caiola SM, Kuhn KF, Stritter FT, Beza J. A delphi survey to identify the components of a community pharmacy clerkship. Am J Pharm Educ. 1995;59(4):334–41.

Fitch K, Bernstein SJ, Aguilar MS, Burnand B, LaCalle JR, Lazaro P, et al. The RAND/UCLA Appropriateness Method User’s Manual. RAND corporation 2001 [cited 2015 July 11]; Available from: URL: http://www.rand.org/pubs/monograph_reports/MR1269 .

McBride AJ, Pates R, Ramadan R, McGowan C. Delphi survey of experts’ opinions on strategies used by community pharmacists to reduce over-the-counter drug misuse. Addiction. 2003;98(4):487–97.

Avery AJ, Savelyich BSP, Sheikh A, Cantrill J, Morris CJ, Fernando B, et al. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study. Inform Prim Care. 2005;13:3–11.

Plumridge RJ. Forecast of the future of hospital pharmacy in Australia. Am J Hosp Pharm. 1981;38(10):1469–72.

Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393.

Ahmed R, McCaffery KJ, Aslani P. Development and validation of a question prompt list for parents of children with attention-deficit/hyperactivity disorder: a Delphi study. Health Expect 2015 Jan 19. doi: 10.1111/hex.12341 .

Mease PJ, Arnold LM, Crofford LJ, Williams DA, Russell IJ, Humphrey L, et al. Identifying the clinical domains of fibromyalgia: contributions from clinician and patient Delphi exercises. Arthritis Rheum. 2008;59(7):952–60.

Download references

Author information

Authors and affiliations.

School of Pharmacy, Menzies Health Institute Queensland, Griffith University, Gold Coast Campus, Gold Coast, QLD, Australia

Sara S. McMillan & Michelle King

Manchester Pharmacy School, University of Manchester, Oxford Road, Manchester, UK

Mary P. Tully

Manchester Academic Health Sciences Centre, Oxford Road, Manchester, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Mary P. Tully .

Ethics declarations

Conflicts of interest, rights and permissions.

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

McMillan, S.S., King, M. & Tully, M.P. How to use the nominal group and Delphi techniques. Int J Clin Pharm 38 , 655–662 (2016). https://doi.org/10.1007/s11096-016-0257-x

Download citation

Received : 13 July 2015

Accepted : 17 January 2016

Published : 05 February 2016

Issue Date : June 2016

DOI : https://doi.org/10.1007/s11096-016-0257-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Consensus methods
• Delphi technique
• Find a journal
• Publish with us
• Track your research

• li:not(.tcb-excluded-from-group-item)"> Home
• li:not(.tcb-excluded-from-group-item)"> About
• li:not(.tcb-excluded-from-group-item)"> Blog
• li:not(.tcb-excluded-from-group-item)"> RMP Courses

How to Use the Delphi Technique

PMI-RMP Exam Prep

•  Minute Read

In today's article, I'll test your understanding of the Delphi Technique. We'll also explore the technique, when you might wish to use it, and how to facilitate this process.

Which of the following statements describes the Delphi Technique most correctly? The Delphi Technique is used:

A. In face to face meetings when experts need to share their opinions.

B. To minimize groupthink and to reach consensus through anonymous expert input.

C. When groups need to make decisions quickly, as in a vote.

D. To create virtual mind maps.

Answer:  B

A common problem in decision-making is groupthink. A powerful, dominating individual drives the project decisions. One way to counter groupthink is to reach a consensus by asking for input anonymously using the Delphi Technique. This technique may also be used to brainstorm and capture ideas.

What is the Delphi Technique?

Project managers can gather information through anonymous surveys and questionnaires. Project managers share the results after each round. The process continues until a decision is made .

When to Use the Delphi Technique

Need to brainstorm ideas ? Perhaps you want to identify risks related to your project schedule. Or you want to gather weaknesses of a business process. You can use the Delphi Technique in countless ways.

How to Facilitate the Delphi Technique

So, how do you actually facilitate this process? There are many ways to use this technique, but here's an example of identifying and analyzing risks:

• Identify a facilitator.
• Identify the experts.
• The facilitator creates and distributes an online survey asking the experts to identify risks.
• Facilitator distributes a list of 7 identified risks and asks the experts to force rank the risks from highest (1) to lowest (7).
• Facilitator compiles and shares the results with the experts as a list of the risks from highest (lowest total) to lowest (highest total).
• The experts discuss the results to determine if any adjustments are needed.

Notes:  

• In this example, we used the Forced Ranking Technique. However, you may use any ranking/prioritization technique you desire. For example, you could simply have the experts rank each risk as low, medium, or high.
• When you ask participants to force rank a list, individuals prioritize the list in order one after another from 1 to the highest number (7 in this example). The number 1 indicates the most significant risk.
• In this example, the Facilitator distributed a list of 7 identified risks. This could be any number of risks. However, the Forced Ranking Technique works best with 10 or fewer items.

Forced Ranking Example

Note: The lowest total indicates the most significant risk. Therefore Requirement #5 matters most, followed by Requirement #2, and so forth.

You may also like

What are the project meeting roles, how to master project meeting minutes.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• VA Author Manuscripts

A Practical Guide to Applying the Delphi Technique in Mental Health Treatment Adaptation: The Example of Enhanced Problem-Solving Training (E-PST)

Paul r. king, jr..

1 VA Center for Integrated Healthcare, Buffalo, NY

2 Department of Counseling, School, and Educational Psychology, Graduate school of Education, University at Buffalo, Buffalo, NY

Gregory P. Beehler

3 Department of Community Health and Health Behavior, School of Public Health and Health Professions, University at Buffalo, Buffalo, NY

Kerry Donnelly

4 VA Western New York Healthcare System, Buffalo, NY

5 Department of Psychiatry, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, NY

Jennifer S. Funderburk

6 VA Center for Integrated Healthcare, Syracuse, NY

7 Department of Psychology, Syracuse University, Syracuse, NY;

Laura O. Wray

8 Division of Geriatrics/ Gerontology, Jacobs School of Medicine and Biomedical Sciences, University at Buffalo, Buffalo, NY

9 Department of Psychology, College of Arts and Sciences, University at Buffalo, Buffalo, NY

Expert consensus methods, such as the Delphi procedure, are commonly employed in consumer, education, and health services research. However, the utility of this methodology has not widely been described in relation to mental health treatment adaptation efforts. This gap is noteworthy given that evidence-based treatments are often modified in terms of core intervention content, method of delivery, and target populations. Expert consensus methods such as the Delphi procedure offer multiple practical benefits (e.g., flexibility, resource-efficiency) for psychologists who need to adapt existing treatments to meet new research and clinical practice needs. The purpose of this paper is to provide a brief overview of the Delphi procedure, and to offer a practical guide to using this method for treatment adaptation. An example is offered using our team’s application of a three-round Delphi procedure to render content and context modifications to an existing problem-solving intervention to optimize its use with a new treatment population. Data were collected from Department of Veterans Affairs clinical subject matter experts. Round 1 utilized semi-structured interviews to determine necessary protocol features and modifications. Rounds 2-3 utilized a forced-choice survey and feedback loop to evaluate expert consensus. More than 91% of rated items reached consensus following Round 2, with the remainder following Round 3. Recommended modifications included minor structural and content edits, and re-balancing time allotments. We conclude that consensus methods may facilitate treatment adaptation efforts, enhance treatment feasibility, and promote content and ecological validity. Considerations for future Delphi-based treatment adaptations are offered.

Psychologists often face demands to alter evidence-based treatments to meet novel clinical and research needs. Common areas of modification include core intervention content (e.g., adding, blending, reducing or tailoring treatment elements) or treatment context, including treatment method (e.g., individual vs . group; face-to-face vs . virtual) and target of delivery (i.e., new or unique service population) ( Stirman, Miller, Toder, & Calloway, 2013 ). Though these modifications may prove necessary to address real-world practice demands ( Stirman et al., 2013 ) and to improve cultural relevance ( Dinos, 2015 ), rendering nonsystematic adaptations runs the risk of compromising treatment fidelity by fundamentally altering essential therapeutic ingredients. However, it is not feasible to conduct rigorous large-scale studies with multiple permutations of significant content or context edits. Use of an expert consensus method is one potential way to guide treatment adaptation efforts in a systematic, but feasible and resource-efficient manner.

Expert consensus methods are commonly employed in consumer, education, and health services research ( Hsu & Sanford, 2007 ). Although specific procedures vary, the primary aim is to distill opinions from a group of subject matter experts (SMEs; i.e., individuals with advanced or extensive knowledge, abilities, or experience in the topic of interest) to facilitate decision-making ( Hasson, Keeney, & McKenna, 2000 ; McKenna, 1994 ). General steps include problem identification, data gathering and evaluation of agreement/disagreement, resolution of disagreement, and ultimately convergence of opinion. These methods are perhaps most useful in addressing two key data problems that lead to uncertainty in decision-making, namely an information shortage (i.e., insufficient scientific evidence exists to chart a path forward), or information overload (e.g., contradictory scientific evidence exists rendering clinical or business decision-making difficult) ( Hasson et al., 2000 ; Jones & Hunter, 1995 ; Powell, 2003 ).

The Delphi technique ( Dalkey & Helmer, 1963 ) is one specific example of an expert consensus method. This approach is a flexible, iterative means to gather and consolidate multiple rounds of SME opinions in a given domain ( McKenna, 1994 ). Prototypical procedures include: a) identification of a research question/problem statement; b) identification and selection of SMEs based on a priori criteria (e.g., years of clinical/research experience, relevant scientific publications); c) data collection and analysis; d) implementation of a multi-round, structured feedback loop between evaluators and SMEs; and e) a final decision-making process ( Hasson et al., 2000 ; Hsu & Sanford, 2007 ). Common recommendations for this approach indicate that diversity among SMEs is favorable over homogeneity ( Powell, 2003 ) and that the selected procedures purposefully align with the overall study objective(s) ( Keeney, Hasson, & McKenna, 2006 ). The general structure of the feedback loop is such that SMEs first review and respond to a specified stimulus (Round 1). In turn, the evaluators consolidate and interpret SMEs’ ratings and corrective feedback, edit the stimulus accordingly, and return a revised stimulus to the SME panel along with a summary of their individual and group-level responses. Round 2 ratings are then based on the revised stimulus materials, following a similar review and rating procedure. Subsequent rounds follow the same process, continuing until the predetermined consensus threshold is reached. Typically, two to three rounds are required ( Hsu & Sanford, 2007 ; McMillan, King, & Tully, 2016 ).

Because of the flexible nature of the Delphi method, the specific questions and procedures used by evaluators vary widely. For instance, variability has been noted in terms of identification and selection of SMEs (e.g., panel size, member anonymity) ( McMillan et al., 2016 ); thresholds used to determine consensus (e.g., percent agreement vs . mean/median item-ratings); and implementation of the feedback loop (e.g., approach used to develop stimulus materials, number and duration of rounds) ( Keeney et al., 2006 ). Quantitative assessment of agreement is most commonly employed, though, the process may also incorporate qualitative components, though these applications also vary (e.g., use of SME interviews/focus groups vs . open-text survey fields) ( Jorm, 2015 ; Keeley et al., 2016 ; Skulmoski, Hartman, & Krahn, 2007 ). Exemplar uses of the Delphi procedure include, but are not limited to, development of assessment tools ( Beehler, Funderburk, Possemato, & Vair, 2013 ; Biondo, Nekolaichuk, Stiles, Fainsinger, & Hagen, 2008 ); selection of health outcome measures ( Santaguida et al., 2018 ) and quality of care indicators ( Boulkedid, Abdoul, Loustau, Sibony, & Alberti, 2011 ); specifying clinical practice standards ( Goodyear et al., 2015 ; Hill, Shand, Torok, Halliday, & Reavley, 2019 ; van der Linde, Hofstad, van Limbeek, Postema, & Geertzen, 2005 ); generating policy ( Aarts, Schuit, van de Goor, & Oers, 2011 ); and identifying health care research priorities ( Mulligan & Conteh, 2016 ; Turner, Ollerhead, & Cook, 2017 ; Yotebieng et al., 2019 ).

In relation to mental health (MH) research, a 2015 review by Jorm documented 176 discrete consensus studies with topics ranging from the conceptual (e.g., operationally defining “relapse”) to practical (e.g., ways to improve MH practices and training standards). Missing from extant MH applications ( Jorm, 2015 ) are treatment adaptation efforts. This gap is noteworthy, given the ubiquity of adaptation in this arena ( Stirman et al., 2013 ). Although the use of expert consensus methods, such as the Delphi technique, does not substitute for an empirical test of effectiveness, it does offer several practical benefits for clinicians and researchers who find need to adapt MH treatments under time or resource-constrained circumstances. Advantages include the method’s flexibility, ability to capitalize on multiple sources of expertise without excessive sample size demands, and time and resource-efficiency: no complicated analytic methods, tools, or software are required; the procedure is easily learned; consensus interpretations are often simple and intuitive (e.g., percent agreement, mean/median Likert-type ratings); and the process can often be completed in several months ( Keeney et al., 2006 ). In our recent work, we have emphasized enhanced access to mental health care for U.S. veterans through primary care-mental health integration, and found the Delphi procedure to be a suitable fit to achieve a study aim to adapt an intervention to address population health concerns related to chronic post-concussion-like symptoms and co-occurring MH disorders in this cohort. In this context, where the need for rapid turnaround MH services is often great ( Gale et al., 2019 ) in order to meet organizational, operational, and consumer demands, use of the Delphi procedure was especially advantageous. The purpose of this paper is to provide a practical example of the rationale and use of the Delphi procedure employed in our team’s MH treatment adaptation and testing research so that others might use a similar methodology.

Applying the Delphi Framework in Treatment Adaptation

The sections below present a step-by-step guide to using the Delphi procedure for treatment adaption, along with a description of how we operationalized each step in our study. An overall summary of the Delphi framework and our exemplar procedure can be found in Table 1 .

Proposed Delphi Flow Process for Treatment Adaptation

Step 1: Identify Research Question/ Problem Statement

The first step in the process is to identify the question that will be addressed by using the Delphi method, and any deliverables that will be created. In the case of treatment adaptation, the primary question would generally pertain to the scope and sufficiency of content and context edits proposed to implement a treatment in a new setting or with a new population. Defining the question in this manner will point directly to the nature of subject matter expertise that will be articulated in Step 2 below. The specific deliverable would be an adapted treatment manual that reflects the aforementioned modifications. If a research application, this early stage would also represent when necessary regulatory approvals are garnered.

Step 1 Example

Our expert consensus study comprised the first phase in an ongoing, multi-part MH treatment adaptation and evaluation project. The overall aim of the parent project is to adapt and test a brief, skills-focused intervention that blends compensatory cognitive skills training with a pre-existing problem-solving intervention (i.e., Problem-Solving Training in Primary Care [PST-PC]; Miller et al., 2015 ). The specific deliverable for this project was an integrated primary care MH treatment manual that a) blended select content from two behavioral interventions (content modifications), and b) was tailored to address combat veterans with history of concussion (context modifications). We sought SME feedback to ensure the integrity of the blended treatment and to promote feasibility in the intended clinical setting (i.e., integrated primary care). Regulatory approvals were granted prior to commencing work: this study was deemed Institutional Review Board-exempt, though scientific methods were approved by the Research and Development committee at the VA Western New York Healthcare System.

Step 2: Identify and Select Qualified SMEs

The second step in the process is to identify and select members of the SME panel. Ideally, the panel will be comprised of individuals with requisite clinical and/or scientific knowledge and/or experience to address the question at hand, though the precise depth and breadth of credentials is unique to a given application. As noted by Powell (2003 , p . 379), “credibility with the target audience” is an important consideration. Thus, in some applications it may be sufficient for SMEs to be proficient in a given methodological or therapeutic technique. Others may require more extensive credentials, such as additional experience in implementing or evaluating policy in the same area. Pragmatic indicators of expertise (e.g., years of experience, number of publications, positions of leadership) may also be deemed critical. It is up to the evaluation team to operationalize the definition of SME in a manner suitable to their aims, and to fashion the panel with individuals likely to make the best use of existing empirical or clinical information. Nonrandom sampling procedures are especially useful in this regard as a statistically representative sample is not needed. Worth mention, however, is that the chosen sampling method will likely guide the minimum sample size determination, and the team may need to identify specific plans to address retention and attrition considerations. In the case of treatment adaptation, representation from SMEs with knowledge in the specific intervention or a substantively similar content area, treatment milieu (e.g., if treatment setting is altered from the original intended setting or feasibility questions exist), and service population would be exemplar areas, though the list could be expanded or contracted if indicated. Early identification of SMEs, along with establishing their commitment to provide feedback, is advantageous. Maintaining anonymity among experts is not required per se but can reduce the likelihood of a consensus built on only the input of highly influential opinion leaders ( McMillan et al., 2016 ).

Step 2 Example

We used purposeful sampling ( Palinkas et al., 2015 ) to identify the SME pool which included seven licensed MH professionals with knowledge and experience in three domains that we determined were relevant to our research question: two neuropsychologists with expertise in concussion rehabilitation; two psychiatrists and one psychologist with expertise in integrated primary care; and two practicing integrated primary care psychologists with expertise in brief problem-solving interventions. All SMEs were full-time employees of the Department of Veterans Affairs (VA), which we deemed essential as the intervention was designed for delivery in the VA health care system. These professionals were considered “critical cases” ( Palinkas, 2015 ) in the sense that their feedback would allow for logical ( not statistical) generalization to others with similar expertise. As such, a relatively small panel sufficed. In terms of individual qualifications, we recognized that many different credentials could establish expertise in the areas of interest to our study. Thus, upon identifying candidates, the team met to discuss fit and ensure that sufficient levels of expertise were represented. For instance, each neuropsychologist evidenced years of clinical experience in neuropsychology and cognitive rehabilitation (range: 15 to nearly 30 years at the time of interview) and had been the principal or co-investigator on at least one grant-funded concussion study. Each integrated primary care expert evidenced a leadership role in the field, as indicated by a local or national leadership designation (e.g., a clinic or program director), as well as at least one scientific paper in the area of integrated care; all also evidenced many years of clinical practice in their respective domains (range: 12 to more than 25 years at the time of interview). Each problem-solving expert served as a trainer/consultant in an existing national clinician training experience in PST-PC ( Miller et al., 2015 ). We determined a priori that SMEs would remain anonymous to one another throughout the study to prevent the possibility of social influence.

Step 3: Establish Data Collection and Analytic Procedures

Typical data collection and analytic decisions pertain to general design considerations (i.e., quantitative, qualitative, or mixed-method approach), determination of personnel roles (e.g., if interviews will be used, who will conduct them and how will data be consolidated), operational definitions of agreement (e.g., percent agreement threshold, item mean), and development of any necessary stimulus materials that would serve as points of reference to SMEs. These decisions are common to any Delphi procedure. A unique consideration for treatment adaptation is that stimulus materials (for instance, a treatment manual) may be voluminous and/or highly context dependent, and therefore rating paradigms need to specify precisely what raters will be providing feedback on and how. That is to say, unlike a Delphi procedure to create a psychometric item pool (where raters would provide ratings for individual items), a treatment adaptation study might prompt raters to evaluate content in larger segments, for instance therapist scripting/ exemplars, content of patient education materials, format of handouts, as well as potentially more abstract or complex topics such as session progression. Thus it is incumbent on the research team to attend carefully to orienting SMEs to their roles ( Hasson et al., 2000 ), providing them with adequate content primers, and prepare feedback and rating forms (i.e., interview schedules, surveys) in a manner that aligns the type of question with the specificity or depth of feedback needed from SMEs. Content primers, which may be brief or bulleted summaries of existing evidence, can help orient them to any necessary practice guidelines, policies, or theoretical/ conceptual referents that would be useful to consider as they develop and articulate their opinions. A critical component of the Delphi procedure is the feedback loop . Thus, a procedure is needed to ensure that once SME feedback is obtained, a mechanism is in place to consolidate the variety of opinions that have been gathered and share it with members of the SME panel. Generally, the only required resources are access to a means of communication (i.e., phone, email); basic calculator, spreadsheet and word processing software; and if deemed necessary, a recording medium for any qualitative interviews. Although estimates of the time required for each planned Delphi round vary, eight weeks has been described as “a realistic approximation” ( Keeney et al. 2006 , p. 209), however thoughtful planning, scheduling, and intentional communication with SMEs can potentially reduce the actual time required.

Step 3 Example

Our overall design was a mixed-method Delphi study that utilized both SME interviews and survey procedures in successive rounds. As such plans were established for managing both qualitative and quantitative data. For each round, the first author (a licensed psychologist leading the intervention adaptation) would schedule and conduct all interviews. A trained research assistant provided support with data management and analysis between each round. We decided in advance to consolidate qualitative feedback using a rapid analytic process ( Gale et al., 2019 ). Unlike traditional qualitative methods, the purpose of rapid analysis is quick turnaround of explanatory data, often in the context of preliminary or time-constrained studies. Data derived in this study were amenable to rapid analysis because SME interview feedback was intended to be both concise and actionable, with questions geared toward information that would guide initial adaptation and refinement of a treatment manual. In subsequent rounds, the main goals were to evaluate agreement. As such, descriptive statistics were emphasized as the primary means of analysis. For the survey, we developed a force-choice rating scale with three options: content is acceptable as-is ; content is unacceptable ; or content would become acceptable-with-modification . For items rated as acceptable-with-modification , we included an open-text space for SME’s to provide specific commentary on changes needed to make the treatment manual content acceptable in a future iteration. The final survey segment included five open-ended questions on additional ways to improve the treatment manual. We defined SME consensus a priori as ≥80% agreement ( Green, Jones, Hughes, & Williams, 1999 ) that rated manual content was acceptable as-is . We opted for telephone and email communication with SMEs to facilitate data collection.

Step 4: Collect & Analyze Data

Step 4 is essentially execution of the plan specified in Step 3. Due to the multi-round nature of the Delphi process, data from each round are analyzed prior to advancing to the next. This allows results from one round to guide the development or modification of stimulus materials used at the next. For this reason, results of each round may lead to alterations in data collection in the subsequent round (e.g., an unexpected survey result may prompt the research team to gather more in depth qualitative information from SMEs if needed; or conversely, reaching agreement earlier than expected may make additional qualitative feedback unnecessary). Where an adapted treatment manual is the primary deliverable, various revisions would be made between each round, and each subsequent SME feedback round would utilize the most up-to-date version. Any outlying or divergent feedback quickly becomes apparent, but is typically self-correcting due to the nature of the consensus rating and feedback process. Assuming that SME feedback trends toward consensus, the breadth and depth of feedback decreases with each round.

Step 4 Example

We used a three-round Delphi procedure that progressed from an initial qualitative round, followed by two survey rounds. Specific procedures for each round are described separately below.

Goals of Round 1 were twofold. First, to prepare SMEs for the Delphi process using pre-specified stimulus materials, and second, to collect initial qualitative data to guide development of the blended treatment manual. Initial stimulus materials were designed to orient SMEs to the study and prepare them to complete their eventual feedback and rating tasks. Following confirmation of participation, SMEs were emailed a three-part stimulus packet: a) a study overview that included the purpose and rationale for the project, and a description of SMEs’ role in the treatment adaptation process; b) a brief professional background survey; and c) content primers” consisting of excerpts from a published clinical practice guideline for managing concussions ( The Management of Concussion-mTBI Working Group, 2016 ), the PST-PC therapist manual ( Miller et al., 2015 ) that would be adapted in this study, and an outline of content that we sought to blend into this treatment for our newly-adapted intervention (which we dubbed Enhanced Problem-Solving Training ; E-PST).

The next action involved a series of semi-structured telephone interviews with seven SMEs over a two-day period. The interview schedule ( Appendix A ) emphasized features of the combined intervention (e.g., psychoeducational content, approaches to apply behavioral skills in-session); changes that should be made to the current PST-PC protocol (e.g., tailoring, adding, or removing content to improve fit for the clinical population); and considerations to maintain treatment integrity of the two interventions. The core content of each interview was largely the same, though items were tailored for different SME groups (e.g., neuropsychologists provided relatively greater feedback on the valence of concussion education; problem-solving experts provided relatively greater feedback on specific problem-solving skills). Interviews required approximately 30-40 minutes.

Following completion of the interviews, the study team analyzed and discussed results. In executing the rapid analysis, we treated each interview question as a discrete content domain (e.g., essential educational content), with text responses organized into a domain-by-respondent matrix that allowed for comparison across interview respondents. Once organized, we identified actionable corrective feedback as well as eight common themes across respondents. The first three identified the essential intervention elements for: (1) any patient-centered integrated primary care intervention; (2) concussion education; and (3) PST-PC. The next four areas of feedback included more specific structural commentary on: (4) the overall composition of the blended intervention; (5) behavioral skills-training conducted in-session; (6) participants’ at-home practice activities; (7) and strategies to ensure user-friendly intervention materials. The final area (8) included miscellaneous recommendations and editorial comments that were addressed to improve clarity and population-specific content (e.g., augmenting veteran and military-specific language). Open text feedback was then distilled into a bulleted feedback list. This analysis was completed within two weeks.

Following completion of Round 1 data analysis, the study team generated a preliminary version of the adapted treatment manual using the feedback provided. The preliminary manual included a) a 64-page therapist guide, which contained therapist orientation and education materials, exemplar scripting, in-session visual aids and note templates; and b) a 77-page participant workbook, which included written education, activities, and notes pages. Both the therapist guide and participant workbook were adapted from existing materials, which in some cases were edited to reflect tailoring and content recommendations, or repackaged (e.g., multiple handouts were compiled into a single participant workbook). The bulk of content changes appeared in two of the six sessions.

The goals of Round 2 were to gather SME feedback on the acceptability of the adapted treatment manual and determine whether additional changes were required to maintain treatment fidelity or address feasibility concerns. Stimulus items for this stage were designed to evaluate consensus. We provided SMEs with the following materials by e-mail: a) a bulleted summary of qualitative feedback gathered during the Round 1 interviews; b) a draft of the newly-adapted E-PST treatment manual (i.e., therapist guide and participant workbook); and c) a 58-item survey that mapped onto the content in the manual. Given the overall volume of content in our application (>140 total pages), we opted to partition ratings into eight core domains. The first seven directly aligned with content in the treatment manual, each with six items relevant to the manual’s front matter and introduction, five-to-six items for each session’s therapist guide, and three items for each session’s participant workbook material. Exemplar ratings within individual sessions included overall session structure (1 item), essential content/in-session activities (3-4 items), and relevant content from the participant workbook (3 items). The eighth and final domain included five open-text questions related to other potential modifications or means to improve the intervention.

Five of seven SMEs participated in Round 2 (two were lost to schedule conflicts), and percent agreement was calculated for each rated domain. Fifty-three of 58 (91.4%) possible rating areas achieved the minimum consensus threshold of ≥80% agreement during Round 2 (Table 3). Of the five areas that did not achieve consensus, two pertained to introductory content in the E-PST therapist guide, two pertained to altering content and time considerations for treatment session one, and one pertained to altering content and time considerations for session two. SME feedback on necessary modifications was analyzed using the rapid analytic process described above and was completed in less than two weeks. We subsequently revised the treatment manual in accord with SME recommendations.

Round 3 focused on establishing final consensus for items that required changes following Round 2. Stimulus materials were comparable to Round 2, except that the summary of SME ratings and the new survey tool were limited to areas that required additional feedback. We sent SMEs the following materials by e-mail: a) a copy of the Round 2 survey results, which included a personalized summary of individual rating for each item and specific changes we made to the therapist guide and participant workbook; b) the revised E-PST treatment manual; and c) a 5-item survey focused only on items that did not achieve consensus during Round 2.

Data analytic procedures were identical to Round 2. Five of seven SMEs again participated in Round 3. Each area of disagreement was resolved during this round, with 100% consensus reached. Analysis was completed with negligible time commitment (i.e., minutes). As no new actionable feedback was received, the primary focus of this stage was to finalize/copyedit the treatment manual. Had additional feedback been received, we budgeted time for a fourth Delphi round (teleconference) to resolve any lingering areas of disagreement, though this ultimately proved unnecessary.

Step 5: Determining Next Steps

The final step in the process is an overall evaluation of outcomes, and to determine any necessary next steps. In the context of treatment adaptation, this determination essentially is whether a suitable deliverable (e.g., an acceptable and feasible treatment manual or protocol) now exists. This would be the most likely scenario, as the previous steps were specifically designed to identify problems in the treatment manual and move toward consensus of what an acceptable product would be. Assuming that consensus has been reached, the team would move to implement the next planned step, which may be additional study or implementation. If disagreement lingers for some reason, next steps would involve a re-evaluation of the aim and product, and potentially designing a new study (or phase of study) to address the issue.

Step 5 Example

Enthusiastic responses from SMEs suggested the overall feasibility and utility of our approach. We agreed a priori that once consensus was achieved, we would move toward further study. In this case, the next stages of study would involve an open trial to glean patient acceptability data.

Though some have critiqued Delphi methods for over-relying on SMEs opinion ( McKenna, 1994 ; Powell, 2003 ), Delphi procedures can nonetheless be used as a rigorous process to improve clinical and research decision-making when there is a lack of clear empirical guidance. Among the primary advantages of the Delphi procedure is its ability to develop actionable findings by way of structured interactions with SMEs, progressing from initially diverse opinions to relative consensus. In our example, little empirical evidence existed to guide our determination of necessary ingredients of a brief problem-solving intervention combined with compensatory cognitive skills, thus SME feedback provided means to a very specific end: systematic generation of a blended treatment manual that will be used in a subsequent clinical trial.

General considerations for using the Delphi technique have been well-described elsewhere (e.g., McKenna, 1994 ; Powell, 2003 ). In terms of using the Delphi process specifically for MH treatment adaptation, our systematic approach yielded timely responses from diverse SMEs that suggested that resulted in key improvements to our intervention protocol. Our findings suggested that minor but specific structural and content edits, as well as re-balancing of time allotments, were needed to maintain treatment integrity and promote feasibility for delivery in integrated primary care settings. In terms of lessons learned, we offer the following recommendations to others who might find need to adapt an existing MH treatment protocol under resource or time-constrained circumstances, and feel that an expert consensus approach may be fitting:

• Seek additional information on the process. Though this manuscript provides a summary and exemplar application of the Delphi technique, a number of other works have described the history and variety of Delphi applications in relatively more detail. These manuscripts, several of which are cited here, would be useful referents for those seeking to apply this method.
• Consider other consensus-building options. A full discussion of other consensus techniques is beyond the scope of this paper, however it is worth mention that other options do exist. For instance, the nominal group technique is a type of structured but adaptable group interaction which may prove more expedient than the Delphi technique (e.g., days vs . weeks to complete) in instances where there are a small group of questions, and stimulus materials require a lower time-investment for SMEs to appraise (i.e., minutes vs . hours) ( McMillan et al., 2016 ). Benefits of expedience may, however, come at the cost of participant anonymity, and an increased likelihood that group dynamics may become a factor, as this approach is typically conducted via in-person gatherings. Cost-benefit determinations of whether an expert consensus, or another systematic decision-making approach, are project-specific.
• Remember that context matters. It is worth mention that the results from a Delphi study are not typically conclusive but often a starting point for a next study or study phase, or clinical initiative. Relative to treatment adaptation, the ultimate aim would typically be to point to actionable changes to improve or extend an intervention. In research applications, the larger context is likely a multiphase effort, with some combination of treatment adaptation and testing. As such, a Delphi study may be a fitting component for grant-funded work aimed in this area, perhaps leading to or culminating in a pilot randomized trial as is often the case in mental health career development (e.g., an NIH K award) or treatment development grants (e.g., NIH R34 mechanism). For clinical applications, the completion of the final step of the Delphi process may lead to a clinic-level training or implementation effort, potentially followed by further quality assurance checks (e.g., program evaluation). In particular for larger-scale implementation efforts, a systematic expert consensus approach such as that which we have outlined here could be beneficial.
• Identify relevant stakeholders and establish early buy-in. This recommendation is especially relevant in large systems. Other stakeholders might include administrative leadership, clinic managers, and other practitioners who might be asked to implement the modified treatment protocol. Though patients are important stakeholders, they would not typically be included in the Delphi stage. Despite our use of the Delphi process in the course of a research study, we nonetheless sought institutional support prior to seeking grant funding and also maintained consultation with SMEs and administrators as the project progressed. We also assured them that requests for their time would be kept to a minimum. Although this recommendation may seem obvious, a lack of institutional or team support would serve as a clear barrier to an intervention’s future implementation.
• Purposefully construct stimulus items and data collection tools. Careful attention to stimulus items and data collection tools has an obvious link to data quality. However, assuring clear and direct instructions for SMEs can also reduce the cognitive load required to evaluate what may amount to a large compilation of intervention materials. Similarly, be cautious not to overwhelm raters with too much content to review. A thorough role induction will prepare them for the requisite level of investment, and with luck, reduce the likelihood of attrition.
• Consider the benefits of incorporating interview data. A benefit of treatment adaptation is that a referent protocol exists. However, substantive changes to content or context, as well as those that might have wide-reaching implications (e.g., regional, national implementation) likely require a greater investment from SMEs. We observed benefits from ensuring diverse SME membership and collecting early interview feedback to guide development of the initial treatment manual. We largely attribute the high rate of consensus following Round 2 to having a clear decision-making path outlined following the Round 1 interviews. In retrospect, time taken to conduct interviews and analyze qualitative data was well-spent in this regard: most of the modifications later required or suggested were modest content additions or clarifications; none required substantial debate among the study team or extensive edits to the treatment manual. Adaptations that do not make as substantive changes (e.g., minor to moderate tailoring edits, content changes to one session, a setting change) may not experience the same return on investment from stakeholder interviews as we did, and may justifiably rely on other methods (e.g., exclusive use of survey methods).
• Plan for flexibility in time and effort. We observed that the actual time requirement for direct study work was low. For instance, study interviews required just two days, and data analysis for each round was completed in less than two weeks. Nonetheless, delays emerged. Sources included: 1) time spent awaiting regulatory approvals for each round of data collection (a nuance of our procedure which involved staff participation in a research protocol); 2) nuances in staff/SME schedules (e.g., vacations/holidays, other professional obligations) which also resulted in attrition of two SMEs between Rounds 1-2; and 3) honoring a subset of SMEs requests for additional time to complete manual reviews due to their own schedule limitations. Although we lost two SMEs to attrition between Round 1 and 2, we were nonetheless able to incorporate their more substantive interview feedback into our final product. Though possible, we do not have evidence to suggest that their additional ratings from them at Round 2 and 3 would have altered our outcome. Our study timeline also was not derailed as we accounted for the possibility of delays up front. For a more time-constrained study or clinic-level modification, relatively less flexibility may exist, or possibly more demand to turn results around quickly.

Unique clinical and research needs can drive innovation in treatment adaptation. The use of expert consensus methods, such as the Delphi technique, may facilitate treatment adaptation efforts, enhance treatment feasibility, and promote content and ecological validity. Our future research will evaluate patient-level acceptability and feasibility of the resulting E-PST protocol in a clinical trial.

Public Significance Statement

Evidence-based mental health treatments are often modified in terms of core intervention content, method of delivery, and target populations. This paper provides an overview and practical example of how an expert consensus technique known as the Delphi procedure can be used to facilitate mental health treatment adaptation efforts, enhance treatment feasibility, and promote content and ecological validity.

Acknowledgments

The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article. This work was supported by Career Development/Capacity Building Award Number IK2 RX002796 from the United States (U.S.) Department of Veterans Affairs Rehabilitation R&D (Rehabilitation Research and Development) Service, as well as with the resources and use of facilities at the VA Center for Integrated Healthcare and the VA Western New York Healthcare System. The views expressed in this article are those of the authors and do not represent the position or policy of the Department of Veterans Affairs or the United States Government. All authors are full-time employees of the Department of Veterans Affairs, and the work performed was part of usual duties. We wish to thank Brenda Jeffries-Silmon for her assistance with data collection, and Carrie Pengelly for her assistance with manuscript preparation.

Author Biographies

PAUL R KING JR received his PhD in counseling psychology from the University at Buffalo/SUNY. He is currently a clinical research psychologist at the Veterans Health Administration Center for Integrated Healthcare at the VA Western New York Health Care System in Buffalo, NY. He is also an adjunct assistant professor in the Department of Counseling, School, and Educational Psychology at the University at Buffalo. His areas of research include post-deployment health care, particularly primary care-based management of concussion and common mental health conditions such as post-traumatic stress disorder and depression, and processes pertaining to integrated primary care.

GREGORY P BEEHLER received his PhD in counseling psychology from the University at Buffalo/SUNY. He is currently the Associate Director for Research at the Veterans Health Administration Center for Integrated Healthcare at the VA Western New York Health Care System in Buffalo, NY. He is also a research assistant professor in the Department of Community Health and Health Behavior in the School of Public Health and Health Professions at the University at Buffalo. His research addresses integrated care settings broadly, with special emphasis on management of chronic pain and related comorbidities, methods of fidelity assessment, and identification and implementation of behavioral health provider best practices.

KERRY DONNELLY received her PhD in Counseling Psychology from the University at Buffalo/SUNY. She is currently a clinical neuropsychologist at the VA Western New York Health Care System in Buffalo, NY. Her research interests focus on neuropsychological functioning associated with traumatic brain injury and post-traumatic stress disorder.

JENNIFER S FUNDERBURK received her PhD in clinical psychology from Syracuse University. She is currently a clinical research psychologist at the Veterans Health Administration Center for Integrated Healthcare at the Syracuse VA Medical Center in Syracuse, NY. She is also an adjunct assistant professor in the Department of Psychology at Syracuse University and the Department of Psychiatry at the University of Rochester Medical Center. Her areas of professional interest include the integration of behavioral health in primary care with a special interest in the development and implementation of brief interventions.

LAURA O. WRAY received her PhD in clinical psychology from Stony Brook University. She is currently the director of the Veterans Health Administration Center for Integrated Healthcare which supports the integration of mental health care into veterans’ medical care through research, education, and implementation support. She is also an associate professor in the Jacobs School of Medicine and Biomedical Sciences at the University at Buffalo. Dr. Wray is a health services researcher whose portfolio focuses on implementation science of novel health care practice and care for veterans with dementia.

Appendix A. Interview Schedule.

Problem-solving therapy (pst) experts, essential elements of a brief pst intervention & education.

• 1. What would you consider to be essential features of a PST-inspired intervention that is adapted for use with primary care patients with a history of mild traumatic brain injury?
• 2. What education on problem-solving would you deem to be essential in working with Veterans with history of mild traumatic brain injury?

Composition, Proportion, & Feasibility of PST

• 3. What sorts of PST skills would you prioritize in a brief course of treatment for Veterans with mild traumatic brain injury? Why?
• 4. Are there specific types of skills that you would avoid delivering in a brief course of treatment? Why?
• 5. Over the four-to-six session course of treatment course, we are proposing a two-meeting sequence (sessions 2-3) that expounds upon externalizing skills, and simplification of goals and organizational strategies. How would you envision spending that time?

Functional Application

• 6. What approach would you advise in teaching Veterans to apply basic problem-solving and organizational skills in their day-to-day lives?
• 7. What types of at-home (between-session) practice would you recommend? Is this sort of practice any different from what you would recommend in a general population? If so, why?

General Questions on Protocol Adaptation

• 8. What, if any, elements of the current PST-PC protocol need to be changed to tailor this protocol to Veterans with history of mild traumatic brain injury? How would you suggest changing them?
• 9. As a clinician, what would you expect to see in a treatment manual for a brief PST intervention that emphasizes cognitive skills such as use of calendars and journals in integrated primary care (PC-MHI)?
• 10. Looking over the proposed content for our new intervention, what gaps exist in the content that we have proposed? Are there content areas that you would change, for example, adding new content, or eliminating content?
• 11. Are there any areas that I have not asked about today that you feel are critical considerations for adapting this treatment protocol?

Neuropsychology/ Rehabilitation Experts

Essential elements of primary care-based intervention & education.

• 1. What would you consider to be essential features of a primary care-based, and behaviorally-focused, management plan for persistent symptoms that Veterans might commonly attribute to mild traumatic brain injury?
• 2. What would you consider to be essential features of brief mild traumatic brain injury education?

Composition, Proportion, & Feasibility of Compensatory Skill Training

• 3. What sorts of cognitive and behavioral skills would you prioritize in a brief course of treatment for Veterans with mild traumatic brain injury? Why?
• 4. Are there skills that you would avoid delivering in a brief course of treatment? Why?
• 5. Over the four-to-six session course of treatment course, we are proposing one meeting that the bulk of mild traumatic brain injury education be delivered in the first session. Is that sufficient and if so, how would you recommend structuring that time?
• 6. What approach would you advise in teaching Veterans with history of mild traumatic brain injury to apply cognitive and behavioral skills in their day-to-day lives?
• 7. What types of at-home (between-session) practice would you recommend?
• 9. What would you expect to see in a treatment manual for a brief, behaviorally-focused intervention for mild traumatic brain injury?

Integrated Care Experts

Essential elements of integrated primary care (pc-mhi) intervention & education.

• 1. What would you consider to be essential features of a primary care-based intervention to address mild traumatic brain injury?
• 2. What types of educational topics would you deem to be essential in working with Veterans with history of mild traumatic brain injury?

Composition, Proportion, & Feasibility of PST/ Compensatory Skill Training

• 3. What sorts of skills would you prioritize in a brief course of treatment for Veterans with mild traumatic brain injury? Why?
• 4. Are there specific types of interventions that you would avoid delivering in a brief course of treatment? Why?
• 5. Over the four-to-six session course of treatment course, we are proposing that the first meeting focus on motivational interviewing, goal-setting and education. How would you envision spending that time?
• 9. As a clinician, what would you expect to see in a treatment manual for a brief intervention that incorporates PST and compensatory cognitive skill-training in PC-MHI?
• Aarts MJ, Schuit A, van de Goor IAM, & van Oers HAM, (2011). Feasibility of multi-sector policy measures that create activity-friendly environments for children: Results of a Delphi study . Implementation Science , 6 :128. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Beehler GP, Funderburk JS, Possemato K, & Vair CL, (2013). Developing a measure of provider adherence to improve implementation of behavioral health services in primary care: A Delphi study . Implementation Science , 8 ( 19 ). [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Biondo PD, Nekolaichuk CL, Stiles C, Fainsinger R, & Hagen NA, (2008). Applying the Delphi process to palliative care tool development: Lessons learned . Supportive Care in Cancer , 16 , 935–942. [ PubMed ] [ Google Scholar ]
• Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C, (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review . PLoS ONE , 6 ( 6 ): e20476. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Dalkey NC, & Helmer O, (1963). An experimental application of the Delphi method to the use of experts . Managerial Science , 9 ( 3 ), 458–467. [ Google Scholar ]
• Dinos S, (2015). Culturally adapted mental healthcare: Evidence, problems, and recommendations . BJPsych Bulletin , 39 , 153–155. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, & Midboe AM, (2019). Comparison of rapid vs. in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration . Implementation Science , 14 ( 11 ), 1–12. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Goodyear M, Hill TL, Allchin B, McCormick F, Hine R, Cuff R, & O’Hanlon B, (2015). Standards of practice for the adult mental health workforce: Meeting the needs of families where a parent has mental illness . International Journal of Mental Health Nursing , 24 ( 2 ), 169–180. [ PubMed ] [ Google Scholar ]
• Green B, Jones M, Hughes D, & Williams A, (1999). Applying the Delphi technique in a study of GPs’ information requirements . Health and Social Care in the Community , 7 ( 3 ), 198–205. [ PubMed ] [ Google Scholar ]
• Hasson F, Keeney S, & McKenna H, (2000). Research guidelines for the Delphi survey technique . Journal of Advanced Nursing , 32 ( 4 ), 1008–1015. [ PubMed ] [ Google Scholar ]
• Hill NTM, Shand F, Torok M, Halliday L, & Reavley NJ, (2019). Development of best practice guidelines for suicide-related crisis response and aftercare in the emergency department or other acute settings: a Delphi expert consensus study . BMC Psychiatry , 19 :6. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Hsu C, & Sanford BA, (2007). The Delphi technique: Making sense of consensus . Practical Assessment, Research, and Evaluation , 12 ( 10 ), 1–10. [ Google Scholar ]
• Jones J, & Hunter D, (1995). Qualitative research: Consensus methods for medical and health services research . BMJ , 311 :376. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Jorm AF, (2015). Using the Delphi expert consensus method in mental health research . Australian & New Zealand Journal of Psychiatry , 49 ( 10 ), 887–897. [ PubMed ] [ Google Scholar ]
• Keeley T, Williamson P, Callery P, Jones LL, Mathers J, Jones J, … & Calvert M, (2016). The use of qualitative methods to inform Delphi surveys in core outcome set development . Trials , 17 :230. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Keeney S, Hasson F, & McKenna H, (2006). Consulting the oracle: Ten lessons from using the Delphi technique in nursing research . Journal of Advanced Nursing , 53 ( 2 ), 205–212. [ PubMed ] [ Google Scholar ]
• McKenna HP, (1994). The Delphi technique: A worthwhile approach for nursing? Journal of Advanced Nursing , 19 , 1221–1225. [ PubMed ] [ Google Scholar ]
• McMillan SS, King M, & Tully MP, (2016). How to use the nominal group and Delphi techniques . International Journal of Clinical Pharmacy , 38 ( 3 ), 655–652. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Miller SA, Aspnes A, DeMuth L, McGill K, King P, Nezu AM, & Nezu CM, (2015). Problem-Solving Training in Primary Care: A problem-solving approach to achieving life’s goals . Washington, D.C.: Department of Veterans Affairs. [ Google Scholar ]
• Mulligan J, & Conteh L, (2016). Global priorities for research and the relative importance of different research outcomes: An international Delphi survey of malaria research experts . Malaria Journal , 15 : 585 , 1–12. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Palinkas LA, Horwitz SM, Green CA, Wisdom JP, Duan N, & Hoagwood K, (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research . Administration and Policy in Mental Health and Mental Health Services Research , 42 , 533–544. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Powell C, (2003). The Delphi technique: Myths and realities . Journal of Advanced Nursing , 41 ( 4 ), 376–382. [ PubMed ] [ Google Scholar ]
• Santaguida P, Dolvich L, Oliver D, Lamarche L, Gilsing A, Griffith LE, … & Raina P, (2018). Protocol for a Delphi consensus exercise to identify a core set of criteria for selecting health related outcome measures (HROM) to be used in primary health care . BMC Family Practice , 19 :152. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Skulmoski GJ, Hartman FT, & Krahn J, (2007). The Delphi method for graduate research . Journal of Information Technology Education , 6 , 1–21. [ Google Scholar ]
• Stirman SW, Miller CJ, Toder K, & Calloway A, (2013). Development of a framework and coding system for modifications and adaptations of evidence-based interventions . Implementation Science , 8 :65. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• The Management of Concussion-mTBI Working Group. (2016). VA/DoD Clinical Practice Guideline for the Management of Concussion-Mild Traumatic Brain Injury . Washington, D.C.: Department of Veterans Affairs and Department of Defense. [ Google Scholar ]
• Turner S, Ollerhead E, & Cook A, (2017). Identifying research priorities for public health research to address health inequalities: Use of Delphi-like survey methods . Health Research Policy and Systems , 15 :87. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• van der Linde H, Hofstad CJ, van Limbeek J, Postema K, Geertzen JHB, (2005). Use of the Delphi technique for developing national clinical guidelines for prescription of lower-limb prostheses . Journal of Rehabilitation Research & Development , 42 ( 5 ), 693–704. [ PubMed ] [ Google Scholar ]
• Yotebieng M, Brazier E, Addison D, Kimmel AD, Cornell M, Keiser O, … & The IeDEA Treat All in sub-Saharan Africa Consensus Statement Working Group. (2019). Research priorities to inform “Treat All” policy implementation for people living with HIV in sub-Saharan Africa: A consensus statement from the International epidemiology Databases to Evaluate AIDS (IeDEA) . Journal of the International AIDS Society , 22 :e25218. [ PMC free article ] [ PubMed ] [ Google Scholar ]

In order to continue enjoying our site, we ask that you confirm your identity as a human. Thank you very much for your cooperation.

Lorem ipsum test link amet consectetur a

The Delphi Technique | Method

In this article:

When groups come together to make decisions, very often group dynamics can result in the team making the wrong decision because they fall into the trap of groupthink . Groupthink is essentially a type of behavior which can be exhibited by groups whereby they reach a perceived consensus on the majority view, without critically thinking or testing alternative hypothesis. Groupthink is particularly likely to occur in cohesive groups, groups with a directional leader, and groups working in isolation from outside information.

Of course, you could argue that you don’t need to reach consensus in these examples, after all, a political party has a leader to make decisions, and a company has a CEO. But if you want to avoid the problems of group dynamics and groupthink, and the decision or problem is large enough, then it makes sense to consult with the experts within your organization and beyond to try to ensure that the best available decision is made. This is where the Delphi Method comes in.

Origins of the Delphi Technique

The delphi process.

Before we go into the general steps, it should be pointed out that the Delphi Model is a set (or family) of techniques, rather than one single clearly understood set of steps. Having said that, the key features of the Delphi Model are:

The Delphi Technique can be difficult to understand from these three components alone, but a simple example should help clarify how it works.

Delphi Technique Example

Suppose we are a project manager or program manager and want to canvas opinion and then reach a consensus on how our product can be the most successful product in the marketplace. We decide we’re going to use the Delphi Method and that we’re going to invite the top 100 people within the organization to participate. We construct our questionnaire as follows, asking participants to rate each of the following options to achieve our goal:

Note that two of the participants commented that the best way to have the most successful product would be to expand into China, this is why you see the two 5 scores against the Other option. Ignore the Mean and Standard Deviation for now – we will examine those shortly. The facilitator collates the answers and resends resends the questionnaire (plus previous answers) to include the China option and also any pertinent comments from the participants. The next round of questions may have been responded to as follows:

We repeat this process of receiving feedback and sending out questionnaires until an agreed number of rounds have been completed, or the standard deviations are very low. Low standard deviations will tell us that we have a low variance for each particular item in the list. In our example above we can see that there is low variance for the “Launch in China” option, telling us that most people are in agreement as to how well this option will help us achieve our goal.

Once we get to the stage where we have low variance, we can then pick the items with the highest mean values to focus our effort on, to either repeat the process until we get consensus, or to discuss until we get consensus. The items in the list with the highest mean values will be those which the group of participants believes are the most crucial to ensuring the success of the organization.

The Delphi Technique is a tool which can be used to reach consensus amongst a group of people. It is suited to situations when you have many participants in the group and/or when you’re looking to avoid the problems of groupthink to enable the group to agree on the best possible way forward. The Delphi Technique originates from the Cold War period, and is a family of techniques rather than one single repeatable procedure.

Cite this article

No votes so far! Be the first to rate this post.

Related Tools

The ladder of inference, force field analysis, the kano model, the ooda loop, the dmaic model, fishbone diagram tutorial, elaboration likelihood model, pareto analysis (the 80/20 rule), the action priority matrix, rapid decision making model, in our course you will learn how to:.

SPCT Journey Webinar by Jayaprakash Prabhakar (JP) : Register Now

Delphi Technique in PMP – Explained!

The Delphi Technique is a structured communication method originally developed as a systematic, interactive forecasting method that relies on a panel of experts. The RAND Corporation developed it during the 1950s to address complex problems involving subjective judgments. The technique has been widely adopted in various fields, including project management, business forecasting, and policy making.

At its core, the Delphi Technique involves a series of questionnaires sent to a panel of selected experts. Participants are asked to provide their opinions and predictions, and after each round, a facilitator provides an anonymous summary of the experts’ forecasts and reasons for their judgments. 

This process is repeated for several rounds, allowing experts to revise their earlier answers in light of the replies of other members of their panel. It is through this iterative process that the Delphi method seeks to reach the most accurate or consensus-based answer.

The Delphi Technique is particularly useful in situations where individual judgments need to be combined to address a lack of certainty or where subjective insights are valuable. It’s a tool for efficiently harnessing the wisdom of a group of experts to forecast future events or make decisions about complex issues.

Understanding the Delphi Technique

The Delphi Technique is a method used for gathering and synthesizing knowledge from a group of experts through a series of structured questionnaires. It is primarily used for forecasting, problem-solving, and decision-making in complex scenarios where direct consensus is difficult to achieve.

Expert Panel: It involves selecting a panel of experts who have knowledge and experience relevant to the subject matter. The number of experts can vary depending on the complexity of the issue.

Anonymity: One of the critical features of the Delphi Technique is the anonymity of the participants. This ensures that the opinions of the experts are not influenced by the identity or reputation of other panel members, reducing the bias that can occur in group dynamics.

Iterative Questionnaires: The process involves multiple rounds of questionnaires. In the first round, experts are asked to provide their opinions or predictions on the issue at hand. Their responses are collected and summarized to identify points of agreement and disagreement.

Controlled Feedback: After each round, a summary of the responses is shared with the panel, and experts are asked to reconsider their earlier responses in light of the feedback. This iterative process, with controlled feedback, allows experts to refine their views and move towards a consensus.

Convergence of Opinion: The goal of the Delphi Technique is to reach a more accurate or consensus-based conclusion. Through successive rounds, the range of answers typically narrows, and the group converges towards the “best” response.

Application: The Delphi Technique is used in various fields like business forecasting, project management, public policy, and healthcare. It is particularly useful in situations where individual judgments need to be aggregated to form a collective opinion about future events or complex issues.

Applications of the Delphi Technique

Business Forecasting and Planning: Companies use the Delphi Technique for predicting future market trends, identifying potential areas of growth, and planning business strategies. It helps in gathering diverse expert opinions to make informed predictions about market behavior.

Project Management: In project management, the Delphi Technique is used for risk assessment, resource allocation, and scheduling. It helps project managers in obtaining consensus among team members or stakeholders, especially when they are geographically dispersed.

Policy Making and Public Planning: Governments and public organizations use the Delphi Technique for policy formulation and urban planning. It aids in understanding the impact of policies and in making decisions that require a long-term perspective.

Healthcare and Medical Research: In the healthcare sector, the Delphi Technique is used for developing clinical guidelines, setting research priorities, and decision-making in public health policies. It helps in consolidating expert opinions on complex medical issues.

Education and Curriculum Development: Educators and academic researchers use the Delphi Technique to develop curriculum frameworks, set educational standards, and identify future trends in education.

Environmental and Technological Forecasting: The technique is also applied in environmental studies for assessing the impact of environmental policies, and in technology fields for predicting future technological advancements and trends.

Step-by-Step Guide to Implementing the Delphi Technique

Define the Problem or Question: Clearly articulate the issue or question that needs to be addressed. This step is crucial as it sets the direction for the entire process.

Choose a Facilitator: Select an impartial facilitator who will manage the process, communicate with participants, compile responses, and provide feedback. The facilitator should be skilled in summarizing information and maintaining confidentiality.

Identify and Select Experts: Assemble a panel of experts who have knowledge and experience relevant to the topic. The number of experts can vary, but they should represent a diverse range of perspectives.

First Round of Questionnaires: Develop and send out an initial questionnaire to the experts. These questions are usually open-ended to gather broad insights.

Collect and Summarize Responses: After the first round, the facilitator collects the responses, summarizes them, and identifies common themes, as well as areas of disagreement.

Second Round of Questionnaires: Based on the summary, the facilitator prepares a second questionnaire, often including the summarized information. This round aims to refine the responses and move closer to a consensus.

Iterative Rounds: Repeat the process of sending questionnaires, summarizing responses, and refining questions for several rounds. With each round, the group should move closer to agreement.

Reach Consensus: The process continues until a consensus is reached or until it becomes clear that consensus is not possible. The final round should provide a clear understanding of the collective opinion of the expert panel.

Report Findings: The facilitator compiles a final report detailing the conclusions and, if relevant, the degree of consensus among the experts.

Feedback to Participants: Optionally, the facilitator can share the findings with the participants, allowing them to see the outcome of their collective input.

Importance in PMP Exam Preparation

Understanding Project Management Tools and Techniques: The Delphi Technique is a vital tool in the arsenal of project management methodologies. Knowledge of this technique is essential for PMP candidates, as the exam covers various project management tools and techniques.

Risk Management and Decision Making: The Delphi Technique is particularly relevant in the context of risk management and decision-making processes in project management. Understanding this technique can help PMP candidates in answering exam questions related to these areas.

Stakeholder Management: The Delphi Technique is an effective method for achieving consensus among stakeholders, which is a key aspect of stakeholder management in projects. PMP exam aspirants need to be familiar with various stakeholder management techniques, including the Delphi Technique.

Enhancing Analytical Skills: Preparing for the PMP exam involves developing analytical and critical thinking skills. Understanding how the Delphi Technique works helps in honing these skills, as the technique involves analyzing expert opinions and reaching a consensus.

Real-world Application: The PMP exam tests not only theoretical knowledge but also the practical application of project management principles. Knowledge of the Delphi Technique is beneficial for real-world project management scenarios, which is a focus of the PMP exam.

Comprehensive Project Management Knowledge: The Delphi Technique is part of a broader set of project management methodologies. Familiarity with this technique contributes to a more comprehensive understanding of project management, which is crucial for PMP exam success.

Key Components of the Delphi Method

Expert Panel: The core of the Delphi Method is a panel of experts chosen for their knowledge and experience in the relevant field. The diversity and expertise of the panel are crucial for the success of the method.

Anonymity of Participants: Experts participate anonymously, which helps prevent the dominance of certain individuals and reduces the influence of authority or reputation. This encourages open and honest feedback.

Iterative Rounds of Questionnaires: The Delphi Method involves multiple rounds of questionnaires where experts provide their opinions or forecasts. After each round, the responses are summarized and shared with the panel, allowing experts to revise their opinions in light of the group’s feedback.

Controlled Feedback: The facilitator plays a key role in controlling the feedback. They summarize the responses from each round and present them back to the panel, ensuring that the feedback is structured and relevant.

Statistical Aggregation of Group Response: Responses are often quantitatively analyzed to find the median, mean, or mode of the experts’ opinions, providing a statistical basis for conclusions.

Convergence Towards Consensus: The iterative process, combined with controlled feedback, aims to move the group towards a consensus. The goal is not necessarily unanimous agreement but a convergence of opinions.

Stop Criterion: The process continues until a predefined stop criterion is met, which could be a certain number of rounds, a level of consensus, or a stability of results.

Structured Information Flow: The method relies on a structured process for collecting and disseminating information, which is managed by the facilitator.

Flexibility in Application: While the Delphi Method has a defined structure, it is flexible in terms of the types of questions asked, the number of rounds, and the composition of the expert panel, allowing it to be adapted to different contexts.

Advantages of the Delphi Technique

Reduction of Group Bias: The anonymity of the Delphi Technique helps minimize the effects of dominant personalities or group pressure. This leads to more honest and unbiased input from all participants.

Access to Diverse Expertise: It allows for the gathering of a wide range of opinions and insights from experts in different fields, leading to a more comprehensive understanding of the subject matter.

Consensus Building: Through iterative rounds of questioning, the Delphi Technique facilitates a gradual convergence of opinions, aiding in reaching a consensus among experts on complex issues.

Structured Communication: The method provides a structured framework for communication, which is particularly useful in managing discussions on complex or controversial topics.

Geographical Flexibility: Experts can participate from anywhere, as the process does not require face-to-face meetings. This is particularly advantageous for engaging a diverse group of international experts.

Time Efficiency: Although the Delphi Technique involves multiple rounds, it can be more time-efficient compared to traditional meetings or committees, as it eliminates the need for synchronizing schedules for physical meetings.

Focus on the Issue: The structured nature of the questionnaires helps in keeping the discussion focused on the specific issue or question at hand, reducing the likelihood of digression.

Documentation and Analysis: The process generates a lot of documented qualitative and quantitative data, which can be useful for further analysis and record-keeping.

Flexibility in Application: The Delphi Technique can be adapted to a wide range of topics and is not limited to any specific field, making it a versatile tool.

Resolution of Conflicting Views : By allowing experts to revise their opinions after seeing the responses of others, the technique helps in resolving conflicts and misunderstandings.

The Delphi Technique is practically applied in various fields for decision-making, forecasting, and problem-solving. For instance, in project management, it’s used for risk assessment and resource planning, where a consensus on uncertain future events is crucial. In public policy, it aids in developing policies by synthesizing expert opinions on complex societal issues. In business, it’s employed for strategic planning and market forecasting, gathering insights from various stakeholders to make informed decisions.

The technique involves gathering a panel of experts, conducting iterative rounds of questionnaires, and refining opinions based on collective feedback. This process, managed by a facilitator, ensures that each expert’s opinion is considered, leading to a well-rounded conclusion.

Leanwisdom is the best curated professional course provider for Project Management, Agile, and Scrum. With a desire to guide professionals in their Agile Journey. In a short span we have trained many professionals and supporting them with Cross-Skilling and upskilling, the Agile, Scrum and other Project Management domain.

You May Also Like

Risk Management Process and Plans in PMP

Five reasons to apply for a PMP certification

Difference Between PMP and PMI ACP ( PMP Vs PMI ACP)

9 Steps To Becoming a PMP® Certified Professional

PMP vs MBA: Advantages of PMP Over MBA

ITIL vs PMP Certification: Which One is Right for You?

Free Top 70+ PMP Exam Questions and Answers

CAPM vs PMP Certification: Which is Best in 2023?

PMP Salary In India 2024

Drop Your Query