how to conduct a quasi experimental research

• Quasi-Experimental Research: Types, Examples & Application

Let’s say you want to study the effects of a new drug on lowering blood pressure. You could randomly assign half of the participants to receive the drug and the other half to receive a placebo. However, this isn’t fair to the patients in the placebo group. 

So, instead of experimenting to find out the effect of the new drug, you use a quasi-experiment to evaluate the drug. The purpose of quasi-experimental research is to establish a causal relationship between an independent variable and a dependent variable.

This guide will discuss the different types of quasi-experimental research, their practical applications, and the best practices for conducting successful quasi-experimental research.

Understanding Quasi-Experimental Research

Quasi-experimental research is a way of finding out if there’s a cause-and-effect relationship between variables when true experiments are not possible because of practical or ethical constraints.

For example, you want to know if a new medicine is effective for migraines. Instead of giving the medication to some people and not to others, you use quasi-experimental research to compare who takes the medication and people who haven’t.

What’s The Difference Between True Experiments and Quasi-Experiments

Quasi-experimental research doesn’t always have the same level of internal validity as real experiments. Pre-selected samples can be biased and not represent the real population.

In a true experiment, the participants are assigned randomly to the experimental or control groups. This ensures that groups are as homogeneous as possible, except for the treatment they receive.

Quasi-experiments don’t randomly assign participants to groups, so the differences within the groups could affect the groups.

Read – Experimental Research Designs: Types, Examples & Methods  

• Types of Quasi-Experimental Designs

Pretest-Posttest Design

This design captures changes by measuring participants before and after an intervention. The pretest measures the dependent variable before the intervention, while the posttest measures it after the intervention.

The difference between the two measurements is the change that occurred due to the intervention. 

However, it is important to be aware of the potential threats to the internal validity of the pretest-postest design. One threat is selection bias , which happens when the group is not homogenous. 

Another is maturation, which occurs when participants change naturally over time. You can mitigate these threats using a control group, randomization, and blinding techniques.

Posttest-Only Design with Nonequivalent Groups

The posttest-only design with nonequivalent groups is similar to the pretest-posttest design, but it does not include a pretest. You see the effect of the intervention by comparing groups to see if there is a difference in their scores.

The difference in scores determines the impact of the intervention. This design is less powerful than the pretest-posttest design because it does not account for potential differences between the groups at baseline. 

However, the posttest design is still a valuable tool for research, especially when you can’t collect pretest data. You can mitigate its limitations by using matching participants on important factors, statistical analysis, and sensitivity analysis

• Regression Discontinuity Design

The regression discontinuity design uses naturally occurring cutoff points to assign participants to treatment and control groups. It’s widely adopted in education and social policy research. 

For example, a talent recruiter might use the cutoff score on a standardized test to move candidates to the next phase of the application.

Interrupted Time Series Design

The Interrupted Time Series design is a type of study that uses time series data to figure out how interventions or events affect the population. In this study, you measure the dependent variable multiple times over time, before and after the intervention.

The interrupted time series design is most commonly used to study the impact of policies or programs. For example, you are studying the impact of a new law on traffic accidents.

You could collect data on the number of traffic accidents each month for a year before the law was passed and a year after the law was passed. If the number of traffic accidents decreases after the law was passed, then you could conclude that the law had a positive impact on traffic safety.

Rigor in Quasi-Experimental Research

Matching techniques.

Matching techniques are a way to create balanced treatment and control groups in quasi-experimental research. This is done by matching participants on important characteristics, such as age, gender, or socioeconomic status.

Propensity score matching is one of the most popular matching methods. It works by using a statistical model to figure out how likely it is that each person in the study would have been in the treatment group if they were selected randomly. Then, people are randomly assigned according to their propensity scores, making sure that the treatment group and the control group are as close to the same as possible.

• Creates balanced treatment and control groups, which reduces bias in the results.
• Versatile- you can use them in various research settings, especially where randomization is not possible.
• Relatively easy to implement.
• Computationally complex.
• Sensitive to the choice of the matching algorithm.
• Does not perfectly balance the treatment and control groups on all relevant characteristics.

Instrumental Variables

An instrumental variable (IV) in quasi-experimental research is a variable that’s related to the independent variable, but not to the error term. It’s a variable that can be used to measure how the independent variable affects the dependent variable.

Let’s say you want to investigate how a new drug reduces the risk of heart attack. You can use the number of days a person has taken aspirin as the instrumental variable.

Aspirin is associated with an independent variable (new drug), however, it is not associated with a dependent variable (risk of a heart attack). This is because people who take aspirin are more likely to take other medications, such as statins, which also lower the risk of heart attack.

• Addresses endogeneity issues.
• Versatile-you can use it in different research settings
• It is relatively hard to find an instrumental variable that meets all of the criteria.
• May not be perfectly correlated with the independent variable.
• Tends to be affected by the dependent variable.

Difference-in-Differences Analysis

Difference-in-differences analysis is a statistical technique that can be used to compare changes in treatment and control groups over time. It is typically used in quasi-experimental research to estimate the causal effect of an intervention.

You have to first define two groups when using the difference-in-differences analysis: the treatment group and the control group. A treatment group is a group that receives an intervention, while a control group doesn’t receive an intervention.

Next, collect data on the dependent variable for both groups before and after the intervention. The difference-in-differences estimate is then calculated by comparing the change in the dependent variable for the treatment group to the change in the dependent variable for the control group.

For example, in a study by David Card and Alan Krueger , they compared the effect of increasing the minimum wage in a particular region to the employment rate. They found that the minimum wage increase in New Jersey did not lead to job losses.

• Addresses selection bias.
• Control for time-invariant confounders.
• Requires a rigorous research design.
• Sensitive to measurement errors.
• Difficult to interpret when multiple interventions or events are happening during the study period.

Challenges and Best Practices

• Validity Threat
• Selection bias : This occurs when the groups being compared are not equivalent. This can happen if participants are self-selected into the groups, or if the groups are not randomly assigned.
• History effects : These are events that happen during the study period that could affect the dependent variable. For example, if there is a natural disaster during the study period, it could affect the results.
• Maturation : This refers to the natural changes that occur over time. For example, students may naturally improve their test scores over time, regardless of whether or not they receive an intervention.
• Testing effects : These are the effects of taking a test on subsequent test scores. For example, if students take a pretest before an intervention, they may learn from the test and do better on the posttest, even if the intervention had no effect.
• Instrumentation : This refers to changes in the way the dependent variable is measured. For example, if the test used to measure student achievement is changed during the study period, it could affect the results.

Strategies for Minimizing Validity Threats

• Matching participants on important characteristics such as age, education level, and more to reduce selection bias.
• Use control groups to avoid history effects, maturation effects, and testing effects.
• Use multiple methods to measure dependent variables to reduce instrumentation effects.
• Conduct a pilot study to identify and address potential validity threats 

Read Also – Internal Validity in Research: Definition, Threats, Examples 

Sample Size and Power

Sample size is the number of participants in a study, while power is the probability of detecting a meaningful effect if it exists.

You have to carefully consider sample size and power when designing a quasi-experiment. This is because groups may not be 100% homogeneous at the start of the study, which can limit the strength of the design.

Using power analysis, you can figure out the sample size you need to see a significant effect. Power analysis looks at the magnitude of the effect, and the standard deviation of the dependent variable, and determines the alpha level.

Generalizability

A major downside of the quasi-experimental design is that it’s usually not generalizable or applicable to other environments. It is typically done in natural environments, so you can’t control factors that could influence the results.

Carefully consider the context of the study before you generalize a quasi-experimental. Also, try replicating the study in other settings to see if the results are consistent.

Real-World Applications

• Healthcare Interventions:

A study by the Universiti Kebangsaan, Malaysia used a quasi-experimental design to assess the effectiveness of a new program for preventing childhood obesity. The study found that the program was effective in reducing the risk of obesity, but it was also expensive.

• Education Policy Changes

A study by Raj Chetty and his colleagues found that students who attended charter schools in California were more likely to attend college than students who did not attend charter schools. However, this study arguably promoted academically underqualified students being admitted to colleges.

• Social and Economic Interventions

A study by the RAND Corporation used a quasi-experimental design to assess the effects of a job training program on employment and earnings. 

The study found that job training programs were effective in increasing employment and earnings, but they also found that the impact varied depending on the characteristics of the participants and the design of the program.

Ethical Considerations

• Informed consent : Provide full information about the purpose of the study, the procedures, the risks and benefits of participating, and their right to withdraw from the study at any time.
• Confidentiality : Do not collect any personal information that is irrelevant to the study and keep participant information confidential.
• Risks to participants : Quasi-experimental research may pose some risks to participants, such as the risk of harm or discomfort. Minimize these risks as much as possible and only perform the research if the benefits outweigh the risks.
• Benefits to participants : Quasi-experimental research should offer some potential benefits to participants, such as access to new treatments or interventions. Researchers should carefully consider the potential benefits and risks of participating in the study before recruiting participants
• Balance of research goals and participant welfare : Do not conduct research that is likely to harm participants, even if the research has the potential to benefit society.

Read: What are Ethical Practices in Market Research?

Quasi-experimental research is a valuable tool for understanding the causal effects of interventions. It is particularly useful when you can’t conduct actual experiments because of ethical or practical constraints.

However, it is important to be aware of the limitations of this type of research. Carefully design the study and consider the limitations to ensure that the findings are accurate and reliable.

Connect to Formplus, Get Started Now - It's Free!

• Quasi-Experimental Research
• Quasi-Experiments
• True Experiments
• Moradeke Owa

You may also like:

Market Research: Types, Methods & Survey Examples

A complete guide on market research; definitions, survey examples, templates, importance and tips.

Recall Bias: Definition, Types, Examples & Mitigation

This article will discuss the impact of recall bias in studies and the best ways to avoid them during research.

What is Pure or Basic Research? + [Examples & Method]

Simple guide on pure or basic research, its methods, characteristics, advantages, and examples in science, medicine, education and psychology

Exploratory Research: What are its Method & Examples?

Overview on exploratory research, examples and methodology. Shows guides on how to conduct exploratory research with online surveys

Formplus - For Seamless Data Collection

Collect data the right way with a versatile data collection tool. try formplus and transform your work productivity today..

An official website of the United States government

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock Locked padlock icon ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

• Publications
• Account settings
• Advanced Search
• Journal List

The Use and Interpretation of Quasi-Experimental Studies in Medical Informatics

Anthony d harris , md, mph, jessina c mcgregor , phd, eli n perencevich , md, ms, jon p furuno , phd, jingkun zhu , ms, dan e peterson , md, mph, joseph finkelstein , md.

• Author information
• Article notes
• Copyright and License information

Correspondence and reprints: Anthony D. Harris, MD, MPH, Division of Healthcare Outcomes Research, Department of Epidemiology and Preventive Medicine, University of Maryland School of Medicine, 100 N. Greene Street, Lower Level, Baltimore, MD; e-mail: < [email protected] >.

Received 2004 Nov 19; Accepted 2005 Aug 12.

Quasi-experimental study designs, often described as nonrandomized, pre-post intervention studies, are common in the medical informatics literature. Yet little has been written about the benefits and limitations of the quasi-experimental approach as applied to informatics studies. This paper outlines a relative hierarchy and nomenclature of quasi-experimental study designs that is applicable to medical informatics intervention studies. In addition, the authors performed a systematic review of two medical informatics journals, the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI), to determine the number of quasi-experimental studies published and how the studies are classified on the above-mentioned relative hierarchy. They hope that future medical informatics studies will implement higher level quasi-experimental study designs that yield more convincing evidence for causal links between medical informatics interventions and outcomes.

Quasi-experimental studies encompass a broad range of nonrandomized intervention studies. These designs are frequently used when it is not logistically feasible or ethical to conduct a randomized controlled trial. Examples of quasi-experimental studies follow. As one example of a quasi-experimental study, a hospital introduces a new order-entry system and wishes to study the impact of this intervention on the number of medication-related adverse events before and after the intervention. As another example, an informatics technology group is introducing a pharmacy order-entry system aimed at decreasing pharmacy costs. The intervention is implemented and pharmacy costs before and after the intervention are measured.

In medical informatics, the quasi-experimental, sometimes called the pre-post intervention, design often is used to evaluate the benefits of specific interventions. The increasing capacity of health care institutions to collect routine clinical data has led to the growing use of quasi-experimental study designs in the field of medical informatics as well as in other medical disciplines. However, little is written about these study designs in the medical literature or in traditional epidemiology textbooks. 1 , 2 , 3 In contrast, the social sciences literature is replete with examples of ways to implement and improve quasi-experimental studies. 4 , 5 , 6

In this paper, we review the different pretest-posttest quasi-experimental study designs, their nomenclature, and the relative hierarchy of these designs with respect to their ability to establish causal associations between an intervention and an outcome. The example of a pharmacy order-entry system aimed at decreasing pharmacy costs will be used throughout this article to illustrate the different quasi-experimental designs. We discuss limitations of quasi-experimental designs and offer methods to improve them. We also perform a systematic review of four years of publications from two informatics journals to determine the number of quasi-experimental studies, classify these studies into their application domains, determine whether the potential limitations of quasi-experimental studies were acknowledged by the authors, and place these studies into the above-mentioned relative hierarchy.

The authors reviewed articles and book chapters on the design of quasi-experimental studies. 4 , 5 , 6 , 7 , 8 , 9 , 10 Most of the reviewed articles referenced two textbooks that were then reviewed in depth. 4 , 6

Key advantages and disadvantages of quasi-experimental studies, as they pertain to the study of medical informatics, were identified. The potential methodological flaws of quasi-experimental medical informatics studies, which have the potential to introduce bias, were also identified. In addition, a summary table outlining a relative hierarchy and nomenclature of quasi-experimental study designs is described. In general, the higher the design is in the hierarchy, the greater the internal validity that the study traditionally possesses because the evidence of the potential causation between the intervention and the outcome is strengthened. 4

We then performed a systematic review of four years of publications from two informatics journals. First, we determined the number of quasi-experimental studies. We then classified these studies on the above-mentioned hierarchy. We also classified the quasi-experimental studies according to their application domain. The categories of application domains employed were based on categorization used by Yearbooks of Medical Informatics 1992–2005 and were similar to the categories of application domains employed by Annual Symposiums of the American Medical Informatics Association. 11 The categories were (1) health and clinical management; (2) patient records; (3) health information systems; (4) medical signal processing and biomedical imaging; (5) decision support, knowledge representation, and management; (6) education and consumer informatics; and (7) bioinformatics. Because the quasi-experimental study design has recognized limitations, we sought to determine whether authors acknowledged the potential limitations of this design. Examples of acknowledgment included mention of lack of randomization, the potential for regression to the mean, the presence of temporal confounders and the mention of another design that would have more internal validity.

All original scientific manuscripts published between January 2000 and December 2003 in the Journal of the American Medical Informatics Association (JAMIA) and the International Journal of Medical Informatics (IJMI) were reviewed. One author (ADH) reviewed all the papers to identify the number of quasi-experimental studies. Other authors (ADH, JCM, JF) then independently reviewed all the studies identified as quasi-experimental. The three authors then convened as a group to resolve any disagreements in study classification, application domain, and acknowledgment of limitations.

Results and Discussion

What is a quasi-experiment.

Quasi-experiments are studies that aim to evaluate interventions but that do not use randomization. Similar to randomized trials, quasi-experiments aim to demonstrate causality between an intervention and an outcome. Quasi-experimental studies can use both preintervention and postintervention measurements as well as nonrandomly selected control groups.

Using this basic definition, it is evident that many published studies in medical informatics utilize the quasi-experimental design. Although the randomized controlled trial is generally considered to have the highest level of credibility with regard to assessing causality, in medical informatics, researchers often choose not to randomize the intervention for one or more reasons: (1) ethical considerations, (2) difficulty of randomizing subjects, (3) difficulty to randomize by locations (e.g., by wards), (4) small available sample size. Each of these reasons is discussed below.

Ethical considerations typically will not allow random withholding of an intervention with known efficacy. Thus, if the efficacy of an intervention has not been established, a randomized controlled trial is the design of choice to determine efficacy. But if the intervention under study incorporates an accepted, well-established therapeutic intervention, or if the intervention has either questionable efficacy or safety based on previously conducted studies, then the ethical issues of randomizing patients are sometimes raised. In the area of medical informatics, it is often believed prior to an implementation that an informatics intervention will likely be beneficial and thus medical informaticians and hospital administrators are often reluctant to randomize medical informatics interventions. In addition, there is often pressure to implement the intervention quickly because of its believed efficacy, thus not allowing researchers sufficient time to plan a randomized trial.

For medical informatics interventions, it is often difficult to randomize the intervention to individual patients or to individual informatics users. So while this randomization is technically possible, it is underused and thus compromises the eventual strength of concluding that an informatics intervention resulted in an outcome. For example, randomly allowing only half of medical residents to use pharmacy order-entry software at a tertiary care hospital is a scenario that hospital administrators and informatics users may not agree to for numerous reasons.

Similarly, informatics interventions often cannot be randomized to individual locations. Using the pharmacy order-entry system example, it may be difficult to randomize use of the system to only certain locations in a hospital or portions of certain locations. For example, if the pharmacy order-entry system involves an educational component, then people may apply the knowledge learned to nonintervention wards, thereby potentially masking the true effect of the intervention. When a design using randomized locations is employed successfully, the locations may be different in other respects (confounding variables), and this further complicates the analysis and interpretation.

In situations where it is known that only a small sample size will be available to test the efficacy of an intervention, randomization may not be a viable option. Randomization is beneficial because on average it tends to evenly distribute both known and unknown confounding variables between the intervention and control group. However, when the sample size is small, randomization may not adequately accomplish this balance. Thus, alternative design and analytical methods are often used in place of randomization when only small sample sizes are available.

What Are the Threats to Establishing Causality When Using Quasi-experimental Designs in Medical Informatics?

The lack of random assignment is the major weakness of the quasi-experimental study design. Associations identified in quasi-experiments meet one important requirement of causality since the intervention precedes the measurement of the outcome. Another requirement is that the outcome can be demonstrated to vary statistically with the intervention. Unfortunately, statistical association does not imply causality, especially if the study is poorly designed. Thus, in many quasi-experiments, one is most often left with the question: “Are there alternative explanations for the apparent causal association?” If these alternative explanations are credible, then the evidence of causation is less convincing. These rival hypotheses, or alternative explanations, arise from principles of epidemiologic study design.

Shadish et al. 4 outline nine threats to internal validity that are outlined in ▶ . Internal validity is defined as the degree to which observed changes in outcomes can be correctly inferred to be caused by an exposure or an intervention. In quasi-experimental studies of medical informatics, we believe that the methodological principles that most often result in alternative explanations for the apparent causal effect include (a) difficulty in measuring or controlling for important confounding variables, particularly unmeasured confounding variables, which can be viewed as a subset of the selection threat in ▶ ; (b) results being explained by the statistical principle of regression to the mean . Each of these latter two principles is discussed in turn.

Threats to Internal Validity

Adapted from Shadish et al. 4

An inability to sufficiently control for important confounding variables arises from the lack of randomization. A variable is a confounding variable if it is associated with the exposure of interest and is also associated with the outcome of interest; the confounding variable leads to a situation where a causal association between a given exposure and an outcome is observed as a result of the influence of the confounding variable. For example, in a study aiming to demonstrate that the introduction of a pharmacy order-entry system led to lower pharmacy costs, there are a number of important potential confounding variables (e.g., severity of illness of the patients, knowledge and experience of the software users, other changes in hospital policy) that may have differed in the preintervention and postintervention time periods ( ▶ ). In a multivariable regression, the first confounding variable could be addressed with severity of illness measures, but the second confounding variable would be difficult if not nearly impossible to measure and control. In addition, potential confounding variables that are unmeasured or immeasurable cannot be controlled for in nonrandomized quasi-experimental study designs and can only be properly controlled by the randomization process in randomized controlled trials.

Example of confounding. To get the true effect of the intervention of interest, we need to control for the confounding variable.

Another important threat to establishing causality is regression to the mean. 12 , 13 , 14 This widespread statistical phenomenon can result in wrongly concluding that an effect is due to the intervention when in reality it is due to chance. The phenomenon was first described in 1886 by Francis Galton who measured the adult height of children and their parents. He noted that when the average height of the parents was greater than the mean of the population, the children tended to be shorter than their parents, and conversely, when the average height of the parents was shorter than the population mean, the children tended to be taller than their parents.

In medical informatics, what often triggers the development and implementation of an intervention is a rise in the rate above the mean or norm. For example, increasing pharmacy costs and adverse events may prompt hospital informatics personnel to design and implement pharmacy order-entry systems. If this rise in costs or adverse events is really just an extreme observation that is still within the normal range of the hospital's pharmaceutical costs (i.e., the mean pharmaceutical cost for the hospital has not shifted), then the statistical principle of regression to the mean predicts that these elevated rates will tend to decline even without intervention. However, often informatics personnel and hospital administrators cannot wait passively for this decline to occur. Therefore, hospital personnel often implement one or more interventions, and if a decline in the rate occurs, they may mistakenly conclude that the decline is causally related to the intervention. In fact, an alternative explanation for the finding could be regression to the mean.

What Are the Different Quasi-experimental Study Designs?

In the social sciences literature, quasi-experimental studies are divided into four study design groups 4 , 6 :

Quasi-experimental designs without control groups

Quasi-experimental designs that use control groups but no pretest

Quasi-experimental designs that use control groups and pretests

Interrupted time-series designs

There is a relative hierarchy within these categories of study designs, with category D studies being sounder than categories C, B, or A in terms of establishing causality. Thus, if feasible from a design and implementation point of view, investigators should aim to design studies that fall in to the higher rated categories. Shadish et al. 4 discuss 17 possible designs, with seven designs falling into category A, three designs in category B, and six designs in category C, and one major design in category D. In our review, we determined that most medical informatics quasi-experiments could be characterized by 11 of 17 designs, with six study designs in category A, one in category B, three designs in category C, and one design in category D because the other study designs were not used or feasible in the medical informatics literature. Thus, for simplicity, we have summarized the 11 study designs most relevant to medical informatics research in ▶ .

Relative Hierarchy of Quasi-experimental Designs

O = Observational Measurement; X = Intervention Under Study. Time moves from left to right.

In general, studies in category D are of higher study design quality than studies in category C, which are higher than those in category B, which are higher than those in category A. Also, as one moves down within each category, the studies become of higher quality, e.g., study 5 in category A is of higher study design quality than study 4, etc.

The nomenclature and relative hierarchy were used in the systematic review of four years of JAMIA and the IJMI. Similar to the relative hierarchy that exists in the evidence-based literature that assigns a hierarchy to randomized controlled trials, cohort studies, case-control studies, and case series, the hierarchy in ▶ is not absolute in that in some cases, it may be infeasible to perform a higher level study. For example, there may be instances where an A6 design established stronger causality than a B1 design. 15 , 16 , 17

Quasi-experimental Designs without Control Groups

Here, X is the intervention and O is the outcome variable (this notation is continued throughout the article). In this study design, an intervention (X) is implemented and a posttest observation (O1) is taken. For example, X could be the introduction of a pharmacy order-entry intervention and O1 could be the pharmacy costs following the intervention. This design is the weakest of the quasi-experimental designs that are discussed in this article. Without any pretest observations or a control group, there are multiple threats to internal validity. Unfortunately, this study design is often used in medical informatics when new software is introduced since it may be difficult to have pretest measurements due to time, technical, or cost constraints.

This is a commonly used study design. A single pretest measurement is taken (O1), an intervention (X) is implemented, and a posttest measurement is taken (O2). In this instance, period O1 frequently serves as the “control” period. For example, O1 could be pharmacy costs prior to the intervention, X could be the introduction of a pharmacy order-entry system, and O2 could be the pharmacy costs following the intervention. Including a pretest provides some information about what the pharmacy costs would have been had the intervention not occurred.

The advantage of this study design over A2 is that adding a second pretest prior to the intervention helps provide evidence that can be used to refute the phenomenon of regression to the mean and confounding as alternative explanations for any observed association between the intervention and the posttest outcome. For example, in a study where a pharmacy order-entry system led to lower pharmacy costs (O3 < O2 and O1), if one had two preintervention measurements of pharmacy costs (O1 and O2) and they were both elevated, this would suggest that there was a decreased likelihood that O3 is lower due to confounding and regression to the mean. Similarly, extending this study design by increasing the number of measurements postintervention could also help to provide evidence against confounding and regression to the mean as alternate explanations for observed associations.

This design involves the inclusion of a nonequivalent dependent variable ( b ) in addition to the primary dependent variable ( a ). Variables a and b should assess similar constructs; that is, the two measures should be affected by similar factors and confounding variables except for the effect of the intervention. Variable a is expected to change because of the intervention X, whereas variable b is not. Taking our example, variable a could be pharmacy costs and variable b could be the length of stay of patients. If our informatics intervention is aimed at decreasing pharmacy costs, we would expect to observe a decrease in pharmacy costs but not in the average length of stay of patients. However, a number of important confounding variables, such as severity of illness and knowledge of software users, might affect both outcome measures. Thus, if the average length of stay did not change following the intervention but pharmacy costs did, then the data are more convincing than if just pharmacy costs were measured.

The Removed-Treatment Design

This design adds a third posttest measurement (O3) to the one-group pretest-posttest design and then removes the intervention before a final measure (O4) is made. The advantage of this design is that it allows one to test hypotheses about the outcome in the presence of the intervention and in the absence of the intervention. Thus, if one predicts a decrease in the outcome between O1 and O2 (after implementation of the intervention), then one would predict an increase in the outcome between O3 and O4 (after removal of the intervention). One caveat is that if the intervention is thought to have persistent effects, then O4 needs to be measured after these effects are likely to have disappeared. For example, a study would be more convincing if it demonstrated that pharmacy costs decreased after pharmacy order-entry system introduction (O2 and O3 less than O1) and that when the order-entry system was removed or disabled, the costs increased (O4 greater than O2 and O3 and closer to O1). In addition, there are often ethical issues in this design in terms of removing an intervention that may be providing benefit.

The Repeated-Treatment Design

The advantage of this design is that it demonstrates reproducibility of the association between the intervention and the outcome. For example, the association is more likely to be causal if one demonstrates that a pharmacy order-entry system results in decreased pharmacy costs when it is first introduced and again when it is reintroduced following an interruption of the intervention. As for design A5, the assumption must be made that the effect of the intervention is transient, which is most often applicable to medical informatics interventions. Because in this design, subjects may serve as their own controls, this may yield greater statistical efficiency with fewer numbers of subjects.

Quasi-experimental Designs That Use a Control Group but No Pretest

An intervention X is implemented for one group and compared to a second group. The use of a comparison group helps prevent certain threats to validity including the ability to statistically adjust for confounding variables. Because in this study design, the two groups may not be equivalent (assignment to the groups is not by randomization), confounding may exist. For example, suppose that a pharmacy order-entry intervention was instituted in the medical intensive care unit (MICU) and not the surgical intensive care unit (SICU). O1 would be pharmacy costs in the MICU after the intervention and O2 would be pharmacy costs in the SICU after the intervention. The absence of a pretest makes it difficult to know whether a change has occurred in the MICU. Also, the absence of pretest measurements comparing the SICU to the MICU makes it difficult to know whether differences in O1 and O2 are due to the intervention or due to other differences in the two units (confounding variables).

Quasi-experimental Designs That Use Control Groups and Pretests

The reader should note that with all the studies in this category, the intervention is not randomized. The control groups chosen are comparison groups. Obtaining pretest measurements on both the intervention and control groups allows one to assess the initial comparability of the groups. The assumption is that if the intervention and the control groups are similar at the pretest, the smaller the likelihood there is of important confounding variables differing between the two groups.

The use of both a pretest and a comparison group makes it easier to avoid certain threats to validity. However, because the two groups are nonequivalent (assignment to the groups is not by randomization), selection bias may exist. Selection bias exists when selection results in differences in unit characteristics between conditions that may be related to outcome differences. For example, suppose that a pharmacy order-entry intervention was instituted in the MICU and not the SICU. If preintervention pharmacy costs in the MICU (O1a) and SICU (O1b) are similar, it suggests that it is less likely that there are differences in the important confounding variables between the two units. If MICU postintervention costs (O2a) are less than preintervention MICU costs (O1a), but SICU costs (O1b) and (O2b) are similar, this suggests that the observed outcome may be causally related to the intervention.

In this design, the pretests are administered at two different times. The main advantage of this design is that it controls for potentially different time-varying confounding effects in the intervention group and the comparison group. In our example, measuring points O1 and O2 would allow for the assessment of time-dependent changes in pharmacy costs, e.g., due to differences in experience of residents, preintervention between the intervention and control group, and whether these changes were similar or different.

With this study design, the researcher administers an intervention at a later time to a group that initially served as a nonintervention control. The advantage of this design over design C2 is that it demonstrates reproducibility in two different settings. This study design is not limited to two groups; in fact, the study results have greater validity if the intervention effect is replicated in different groups at multiple times. In the example of a pharmacy order-entry system, one could implement or intervene in the MICU and then at a later time, intervene in the SICU. This latter design is often very applicable to medical informatics where new technology and new software is often introduced or made available gradually.

Interrupted Time-Series Designs

An interrupted time-series design is one in which a string of consecutive observations equally spaced in time is interrupted by the imposition of a treatment or intervention. The advantage of this design is that with multiple measurements both pre- and postintervention, it is easier to address and control for confounding and regression to the mean. In addition, statistically, there is a more robust analytic capability, and there is the ability to detect changes in the slope or intercept as a result of the intervention in addition to a change in the mean values. 18 A change in intercept could represent an immediate effect while a change in slope could represent a gradual effect of the intervention on the outcome. In the example of a pharmacy order-entry system, O1 through O5 could represent monthly pharmacy costs preintervention and O6 through O10 monthly pharmacy costs post the introduction of the pharmacy order-entry system. Interrupted time-series designs also can be further strengthened by incorporating many of the design features previously mentioned in other categories (such as removal of the treatment, inclusion of a nondependent outcome variable, or the addition of a control group).

Systematic Review Results

The results of the systematic review are in ▶ . In the four-year period of JAMIA publications that the authors reviewed, 25 quasi-experimental studies among 22 articles were published. Of these 25, 15 studies were of category A, five studies were of category B, two studies were of category C, and no studies were of category D. Although there were no studies of category D (interrupted time-series analyses), three of the studies classified as category A had data collected that could have been analyzed as an interrupted time-series analysis. Nine of the 25 studies (36%) mentioned at least one of the potential limitations of the quasi-experimental study design. In the four-year period of IJMI publications reviewed by the authors, nine quasi-experimental studies among eight manuscripts were published. Of these nine, five studies were of category A, one of category B, one of category C, and two of category D. Two of the nine studies (22%) mentioned at least one of the potential limitations of the quasi-experimental study design.

Systematic Review of Four Years of Quasi-designs in JAMIA

JAMIA = Journal of the American Medical Informatics Association; IJMI = International Journal of Medical Informatics.

Could have been analyzed as an interrupted time-series design.

In addition, three studies from JAMIA were based on a counterbalanced design. A counterbalanced design is a higher order study design than other studies in category A. The counterbalanced design is sometimes referred to as a Latin-square arrangement. In this design, all subjects receive all the different interventions but the order of intervention assignment is not random. 19 This design can only be used when the intervention is compared against some existing standard, for example, if a new PDA-based order entry system is to be compared to a computer terminal–based order entry system. In this design, all subjects receive the new PDA-based order entry system and the old computer terminal-based order entry system. The counterbalanced design is a within-participants design, where the order of the intervention is varied (e.g., one group is given software A followed by software B and another group is given software B followed by software A). The counterbalanced design is typically used when the available sample size is small, thus preventing the use of randomization. This design also allows investigators to study the potential effect of ordering of the informatics intervention.

Although quasi-experimental study designs are ubiquitous in the medical informatics literature, as evidenced by 34 studies in the past four years of the two informatics journals, little has been written about the benefits and limitations of the quasi-experimental approach. As we have outlined in this paper, a relative hierarchy and nomenclature of quasi-experimental study designs exist, with some designs being more likely than others to permit causal interpretations of observed associations. Strengths and limitations of a particular study design should be discussed when presenting data collected in the setting of a quasi-experimental study. Future medical informatics investigators should choose the strongest design that is feasible given the particular circumstances.

Supplementary Material

Dr. Harris was supported by NIH grants K23 AI01752-01A1 and R01 AI60859-01A1. Dr. Perencevich was supported by a VA Health Services Research and Development Service (HSR&D) Research Career Development Award (RCD-02026-1). Dr. Finkelstein was supported by NIH grant RO1 HL71690.

• 1. Rothman KJ, Greenland S. Modern epidemiology. Philadelphia: Lippincott–Raven Publishers, 1998.
• 2. Hennekens CH, Buring JE. Epidemiology in medicine. Boston: Little, Brown, 1987.
• 3. Szklo M, Nieto FJ. Epidemiology: beyond the basics. Gaithersburg, MD: Aspen Publishers, 2000.
• 4. Shadish WR, Cook TD, Campbell DT. Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin, 2002.
• 5. Trochim WMK. The research methods knowledge base. Cincinnati: Atomic Dog Publishing, 2001.
• 6. Cook TD, Campbell DT. Quasi-experimentation: design and analysis issues for field settings. Chicago: Rand McNally Publishing Company, 1979.
• 7. MacLehose RR, Reeves BC, Harvey IM, Sheldon TA, Russell IT, Black AM. A systematic review of comparisons of effect sizes derived from randomised and non-randomised studies. Health Technol Assess. 2000;4:1–154. [ PubMed ] [ Google Scholar ]
• 8. Shadish WR, Heinsman DT. Experiments versus quasi-experiments: do they yield the same answer? NIDA Res Monogr. 1997;170:147–64. [ PubMed ] [ Google Scholar ]
• 9. Grimshaw J, Campbell M, Eccles M, Steen N. Experimental and quasi-experimental designs for evaluating guideline implementation strategies. Fam Pract. 2000;17(Suppl 1):S11–6. [ DOI ] [ PubMed ] [ Google Scholar ]
• 10. Zwerling C, Daltroy LH, Fine LJ, Johnston JJ, Melius J, Silverstein BA. Design and conduct of occupational injury intervention studies: a review of evaluation strategies. Am J Ind Med. 1997;32:164–79. [ DOI ] [ PubMed ] [ Google Scholar ]
• 11. Haux RKC, editor. Yearbook of medical informatics 2005. Stuttgart: Schattauer Verlagsgesellschaft, 2005, 563.
• 12. Morton V, Torgerson DJ. Effect of regression to the mean on decision making in health care. BMJ. 2003;326:1083–4. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 13. Bland JM, Altman DG. Regression towards the mean. BMJ. 1994;308:1499. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 14. Bland JM, Altman DG. Some examples of regression towards the mean. BMJ. 1994;309:780. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 15. Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L, Naylor CD, et al. Users' guides to the medical literature: XXV. Evidence-based medicine: principles for applying the users' guides to patient care. Evidence-Based Medicine Working Group. JAMA. 2000;284:1290–6. [ DOI ] [ PubMed ] [ Google Scholar ]
• 16. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20:21–35. [ DOI ] [ PubMed ] [ Google Scholar ]
• 17. Harbour R, Miller J. A new system for grading recommendations in evidence based guidelines. BMJ. 2001;323:334–6. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 18. Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. J Clin Pharm Ther. 2002;27:299–309. [ DOI ] [ PubMed ] [ Google Scholar ]
• 19. Campbell DT. Counterbalanced design. In: Company RMCP, editor. Experimental and Quasiexperimental Designs for Research. Chicago: Rand-McNally College Publishing Company, 1963, 50–5.
• 20. Staggers N, Kobus D. Comparing response time, errors, and satisfaction between text-based and graphical user interfaces during nursing order tasks. J Am Med Inform Assoc. 2000;7:164–76. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 21. Schriger DL, Baraff LJ, Buller K, Shendrikar MA, Nagda S, Lin EJ, et al. Implementation of clinical guidelines via a computer charting system: effect on the care of febrile children less than three years of age. J Am Med Inform Assoc. 2000;7:186–95. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 22. Patel VL, Kushniruk AW, Yang S, Yale JF. Impact of a computer-based patient record system on data collection, knowledge organization, and reasoning. J Am Med Inform Assoc. 2000;7:569–85. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 23. Borowitz SM. Computer-based speech recognition as an alternative to medical transcription. J Am Med Inform Assoc. 2001;8:101–2. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 24. Patterson R, Harasym P. Educational instruction on a hospital information system for medical students during their surgical rotations. J Am Med Inform Assoc. 2001;8:111–6. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 25. Rocha BH, Christenson JC, Evans RS, Gardner RM. Clinicians' response to computerized detection of infections. J Am Med Inform Assoc. 2001;8:117–25. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 26. Lovis C, Chapko MK, Martin DP, Payne TH, Baud RH, Hoey PJ, et al. Evaluation of a command-line parser-based order entry pathway for the Department of Veterans Affairs electronic patient record. J Am Med Inform Assoc. 2001;8:486–98. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 27. Hersh WR, Junium K, Mailhot M, Tidmarsh P. Implementation and evaluation of a medical informatics distance education program. J Am Med Inform Assoc. 2001;8:570–84. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 28. Makoul G, Curry RH, Tang PC. The use of electronic medical records: communication patterns in outpatient encounters. J Am Med Inform Assoc. 2001;8:610–5. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 29. Ruland CM. Handheld technology to improve patient care: evaluating a support system for preference-based care planning at the bedside. J Am Med Inform Assoc. 2002;9:192–201. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 30. De Lusignan S, Stephens PN, Adal N, Majeed A. Does feedback improve the quality of computerized medical records in primary care? J Am Med Inform Assoc. 2002;9:395–401. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 31. Mekhjian HS, Kumar RR, Kuehn L, Bentley TD, Teater P, Thomas A, et al. Immediate benefits realized following implementation of physician order entry at an academic medical center. J Am Med Inform Assoc. 2002;9:529–39. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 32. Ammenwerth E, Mansmann U, Iller C, Eichstadter R. Factors affecting and affected by user acceptance of computer-based nursing documentation: results of a two-year study. J Am Med Inform Assoc. 2003;10:69–84. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 33. Oniki TA, Clemmer TP, Pryor TA. The effect of computer-generated reminders on charting deficiencies in the ICU. J Am Med Inform Assoc. 2003;10:177–87. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 34. Liederman EM, Morefield CS. Web messaging: a new tool for patient-physician communication. J Am Med Inform Assoc. 2003;10:260–70. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 35. Rotich JK, Hannan TJ, Smith FE, Bii J, Odero WW, Vu N, Mamlin BW, et al. Installing and implementing a computer-based patient record system in sub-Saharan Africa: the Mosoriot Medical Record System. J Am Med Inform Assoc. 2003;10:295–303. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 36. Payne TH, Hoey PJ, Nichol P, Lovis C. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc. 2003;10:322–9. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 37. Hoch I, Heymann AD, Kurman I, Valinsky LJ, Chodick G, Shalev V. Countrywide computer alerts to community physicians improve potassium testing in patients receiving diuretics. J Am Med Inform Assoc. 2003;10:541–6. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 38. Laerum H, Karlsen TH, Faxvaag A. Effects of scanning and eliminating paper-based medical records on hospital physicians' clinical work practice. J Am Med Inform Assoc. 2003;10:588–95. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 39. Devine EG, Gaehde SA, Curtis AC. Comparative evaluation of three continuous speech recognition software packages in the generation of medical reports. J Am Med Inform Assoc. 2000;7:462–8. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 40. Dunbar PJ, Madigan D, Grohskopf LA, Revere D, Woodward J, Minstrell J, et al. A two-way messaging system to enhance antiretroviral adherence. J Am Med Inform Assoc. 2003;10:11–5. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 41. Lenert L, Munoz RF, Stoddard J, Delucchi K, Bansod A, Skoczen S, et al. Design and pilot evaluation of an Internet smoking cessation program. J Am Med Inform Assoc. 2003;10:16–20. [ DOI ] [ PMC free article ] [ PubMed ] [ Google Scholar ]
• 42. Koide D, Ohe K, Ross-Degnan D, Kaihara S. Computerized reminders to monitor liver function to improve the use of etretinate. Int J Med Inf. 2000;57:11–9. [ DOI ] [ PubMed ] [ Google Scholar ]
• 43. Gonzalez-Heydrich J, DeMaso DR, Irwin C, Steingard RJ, Kohane IS, Beardslee WR. Implementation of an electronic medical record system in a pediatric psychopharmacology program. Int J Med Inf. 2000;57:109–16. [ DOI ] [ PubMed ] [ Google Scholar ]
• 44. Anantharaman V, Swee Han L. Hospital and emergency ambulance link: using IT to enhance emergency pre-hospital care. Int J Med Inf. 2001;61:147–61. [ DOI ] [ PubMed ] [ Google Scholar ]
• 45. Chae YM, Heon Lee J, Hee Ho S, Ja Kim H, Hong Jun K, Uk Won J. Patient satisfaction with telemedicine in home health services for the elderly. Int J Med Inf. 2001;61:167–73. [ DOI ] [ PubMed ] [ Google Scholar ]
• 46. Lin CC, Chen HS, Chen CY, Hou SM. Implementation and evaluation of a multifunctional telemedicine system in NTUH. Int J Med Inf. 2001;61:175–87. [ DOI ] [ PubMed ] [ Google Scholar ]
• 47. Mikulich VJ, Liu YC, Steinfeldt J, Schriger DL. Implementation of clinical guidelines through an electronic medical record: physician usage, satisfaction and assessment. Int J Med Inf. 2001;63:169–78. [ DOI ] [ PubMed ] [ Google Scholar ]
• 48. Hwang JI, Park HA, Bakken S. Impact of a physician's order entry (POE) system on physicians' ordering patterns and patient length of stay. Int J Med Inf. 2002;65:213–23. [ DOI ] [ PubMed ] [ Google Scholar ]
• 49. Park WS, Kim JS, Chae YM, Yu SH, Kim CY, Kim SA, et al. Does the physician order-entry system increase the revenue of a general hospital? Int J Med Inf. 2003;71:25–32. [ DOI ] [ PubMed ] [ Google Scholar ]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

• View on publisher site
• PDF (115.2 KB)
• Collections

Similar articles

Cited by other articles, links to ncbi databases.

• Download .nbib .nbib
• Format: AMA APA MLA NLM

Add to Collections

Want to create or adapt books like this? Learn more about how Pressbooks supports open publishing practices.

Quasi-Experimental Research

The prefix  quasi  means “resembling.” Thus quasi-experimental research is research that resembles experimental research but is not true experimental research. Recall with a true between-groups experiment, random assignment to conditions is used to ensure the groups are equivalent and with a true within-subjects design counterbalancing is used to guard against order effects. Quasi-experiments are missing one of these safeguards. Although an independent variable is manipulated, either a control group is missing or participants are not randomly assigned to conditions (Cook & Campbell, 1979) [1] .

Because the independent variable is manipulated before the dependent variable is measured, quasi-experimental research eliminates the directionality problem associated with non-experimental research. But because either counterbalancing techniques are not used or participants are not randomly assigned to conditions—making it likely that there are other differences between conditions—quasi-experimental research does not eliminate the problem of confounding variables. In terms of internal validity, therefore, quasi-experiments are generally somewhere between non-experimental studies and true experiments.

Quasi-experiments are most likely to be conducted in field settings in which random assignment is difficult or impossible. They are often conducted to evaluate the effectiveness of a treatment—perhaps a type of psychotherapy or an educational intervention. There are many different kinds of quasi-experiments, but we will discuss just a few of the most common ones in this chapter. 

• Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues in field settings . Boston, MA: Houghton Mifflin. ↵

Research Methods in Psychology Copyright © 2019 by Rajiv S. Jhangiani, I-Chant A. Chiang, Carrie Cuttler, & Dana C. Leighton is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License , except where otherwise noted.

Share This Book

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Guide to Experimental Design | Overview, Steps, & Examples

Guide to Experimental Design | Overview, 5 steps & Examples

Published on December 3, 2019 by Rebecca Bevans . Revised on June 21, 2023.

Experiments are used to study causal relationships . You manipulate one or more independent variables and measure their effect on one or more dependent variables.

Experimental design create a set of procedures to systematically test a hypothesis . A good experimental design requires a strong understanding of the system you are studying.

There are five key steps in designing an experiment:

• Consider your variables and how they are related
• Write a specific, testable hypothesis
• Design experimental treatments to manipulate your independent variable
• Assign subjects to groups, either between-subjects or within-subjects
• Plan how you will measure your dependent variable

For valid conclusions, you also need to select a representative sample and control any  extraneous variables that might influence your results. If random assignment of participants to control and treatment groups is impossible, unethical, or highly difficult, consider an observational study instead. This minimizes several types of research bias, particularly sampling bias , survivorship bias , and attrition bias as time passes.

Table of contents

Step 1: define your variables, step 2: write your hypothesis, step 3: design your experimental treatments, step 4: assign your subjects to treatment groups, step 5: measure your dependent variable, other interesting articles, frequently asked questions about experiments.

You should begin with a specific research question . We will work with two research question examples, one from health sciences and one from ecology:

To translate your research question into an experimental hypothesis, you need to define the main variables and make predictions about how they are related.

Start by simply listing the independent and dependent variables .

Then you need to think about possible extraneous and confounding variables and consider how you might control  them in your experiment.

Finally, you can put these variables together into a diagram. Use arrows to show the possible relationships between variables and include signs to show the expected direction of the relationships.

Here we predict that increasing temperature will increase soil respiration and decrease soil moisture, while decreasing soil moisture will lead to decreased soil respiration.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Now that you have a strong conceptual understanding of the system you are studying, you should be able to write a specific, testable hypothesis that addresses your research question.

The next steps will describe how to design a controlled experiment . In a controlled experiment, you must be able to:

• Systematically and precisely manipulate the independent variable(s).
• Precisely measure the dependent variable(s).
• Control any potential confounding variables.

If your study system doesn’t match these criteria, there are other types of research you can use to answer your research question.

How you manipulate the independent variable can affect the experiment’s external validity – that is, the extent to which the results can be generalized and applied to the broader world.

First, you may need to decide how widely to vary your independent variable.

• just slightly above the natural range for your study region.
• over a wider range of temperatures to mimic future warming.
• over an extreme range that is beyond any possible natural variation.

Second, you may need to choose how finely to vary your independent variable. Sometimes this choice is made for you by your experimental system, but often you will need to decide, and this will affect how much you can infer from your results.

• a categorical variable : either as binary (yes/no) or as levels of a factor (no phone use, low phone use, high phone use).
• a continuous variable (minutes of phone use measured every night).

How you apply your experimental treatments to your test subjects is crucial for obtaining valid and reliable results.

First, you need to consider the study size : how many individuals will be included in the experiment? In general, the more subjects you include, the greater your experiment’s statistical power , which determines how much confidence you can have in your results.

Then you need to randomly assign your subjects to treatment groups . Each group receives a different level of the treatment (e.g. no phone use, low phone use, high phone use).

You should also include a control group , which receives no treatment. The control group tells us what would have happened to your test subjects without any experimental intervention.

When assigning your subjects to groups, there are two main choices you need to make:

• A completely randomized design vs a randomized block design .
• A between-subjects design vs a within-subjects design .

Randomization

An experiment can be completely randomized or randomized within blocks (aka strata):

• In a completely randomized design , every subject is assigned to a treatment group at random.
• In a randomized block design (aka stratified random design), subjects are first grouped according to a characteristic they share, and then randomly assigned to treatments within those groups.

Sometimes randomization isn’t practical or ethical , so researchers create partially-random or even non-random designs. An experimental design where treatments aren’t randomly assigned is called a quasi-experimental design .

Between-subjects vs. within-subjects

In a between-subjects design (also known as an independent measures design or classic ANOVA design), individuals receive only one of the possible levels of an experimental treatment.

In medical or social research, you might also use matched pairs within your between-subjects design to make sure that each treatment group contains the same variety of test subjects in the same proportions.

In a within-subjects design (also known as a repeated measures design), every individual receives each of the experimental treatments consecutively, and their responses to each treatment are measured.

Within-subjects or repeated measures can also refer to an experimental design where an effect emerges over time, and individual responses are measured over time in order to measure this effect as it emerges.

Counterbalancing (randomizing or reversing the order of treatments among subjects) is often used in within-subjects designs to ensure that the order of treatment application doesn’t influence the results of the experiment.

Prevent plagiarism. Run a free check.

Finally, you need to decide how you’ll collect data on your dependent variable outcomes. You should aim for reliable and valid measurements that minimize research bias or error.

Some variables, like temperature, can be objectively measured with scientific instruments. Others may need to be operationalized to turn them into measurable observations.

• Ask participants to record what time they go to sleep and get up each day.
• Ask participants to wear a sleep tracker.

How precisely you measure your dependent variable also affects the kinds of statistical analysis you can use on your data.

Experiments are always context-dependent, and a good experimental design will take into account all of the unique considerations of your study system to produce information that is both valid and relevant to your research question.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Likert scale

Research bias

• Implicit bias
• Framing effect
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic

Experimental design means planning a set of procedures to investigate a relationship between variables . To design a controlled experiment, you need:

• A testable hypothesis
• At least one independent variable that can be precisely manipulated
• At least one dependent variable that can be precisely measured

When designing the experiment, you decide:

• How you will manipulate the variable(s)
• How you will control for any potential confounding variables
• How many subjects or samples will be included in the study
• How subjects will be assigned to treatment levels

Experimental design is essential to the internal and external validity of your experiment.

The key difference between observational studies and experimental designs is that a well-done observational study does not influence the responses of participants, while experiments do have some sort of treatment condition applied to at least some participants by random assignment .

A confounding variable , also called a confounder or confounding factor, is a third variable in a study examining a potential cause-and-effect relationship.

A confounding variable is related to both the supposed cause and the supposed effect of the study. It can be difficult to separate the true effect of the independent variable from the effect of the confounding variable.

In your research design , it’s important to identify potential confounding variables and plan how you will reduce their impact.

In a between-subjects design , every participant experiences only one condition, and researchers assess group differences between participants in various conditions.

In a within-subjects design , each participant experiences all conditions, and researchers test the same participants repeatedly for differences between conditions.

The word “between” means that you’re comparing different conditions between groups, while the word “within” means you’re comparing different conditions within the same group.

An experimental group, also known as a treatment group, receives the treatment whose effect researchers wish to study, whereas a control group does not. They should be identical in all other ways.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Bevans, R. (2023, June 21). Guide to Experimental Design | Overview, 5 steps & Examples. Scribbr. Retrieved October 21, 2024, from https://www.scribbr.com/methodology/experimental-design/

Is this article helpful?

Rebecca Bevans

Other students also liked, random assignment in experiments | introduction & examples, quasi-experimental design | definition, types & examples, how to write a lab report, what is your plagiarism score.