Organizing Your Social Sciences Research Assignments

  • Annotated Bibliography
  • Analyzing a Scholarly Journal Article
  • Group Presentations
  • Dealing with Nervousness
  • Using Visual Aids
  • Grading Someone Else's Paper
  • Types of Structured Group Activities
  • Group Project Survival Skills
  • Leading a Class Discussion
  • Multiple Book Review Essay
  • Reviewing Collected Works
  • Writing a Case Analysis Paper
  • Writing a Case Study
  • About Informed Consent
  • Writing Field Notes
  • Writing a Policy Memo
  • Writing a Reflective Paper
  • Writing a Research Proposal
  • Generative AI and Writing
  • Acknowledgments

Definition and Introduction

Journal article analysis assignments require you to summarize and critically assess the quality of an empirical research study published in a scholarly [a.k.a., academic, peer-reviewed] journal. The article may be assigned by the professor, chosen from course readings listed in the syllabus, or you must locate an article on your own, usually with the requirement that you search using a reputable library database, such as, JSTOR or ProQuest . The article chosen is expected to relate to the overall discipline of the course, specific course content, or key concepts discussed in class. In some cases, the purpose of the assignment is to analyze an article that is part of the literature review for a future research project.

Analysis of an article can be assigned to students individually or as part of a small group project. The final product is usually in the form of a short paper [typically 1- 6 double-spaced pages] that addresses key questions the professor uses to guide your analysis or that assesses specific parts of a scholarly research study [e.g., the research problem, methodology, discussion, conclusions or findings]. The analysis paper may be shared on a digital course management platform and/or presented to the class for the purpose of promoting a wider discussion about the topic of the study. Although assigned in any level of undergraduate and graduate coursework in the social and behavioral sciences, professors frequently include this assignment in upper division courses to help students learn how to effectively identify, read, and analyze empirical research within their major.

Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students make the most of Scholarly Articles." Library Management 33 (2012): 525-535.

Benefits of Journal Article Analysis Assignments

Analyzing and synthesizing a scholarly journal article is intended to help students obtain the reading and critical thinking skills needed to develop and write their own research papers. This assignment also supports workplace skills where you could be asked to summarize a report or other type of document and report it, for example, during a staff meeting or for a presentation.

There are two broadly defined ways that analyzing a scholarly journal article supports student learning:

Improve Reading Skills

Conducting research requires an ability to review, evaluate, and synthesize prior research studies. Reading prior research requires an understanding of the academic writing style , the type of epistemological beliefs or practices underpinning the research design, and the specific vocabulary and technical terminology [i.e., jargon] used within a discipline. Reading scholarly articles is important because academic writing is unfamiliar to most students; they have had limited exposure to using peer-reviewed journal articles prior to entering college or students have yet to gain exposure to the specific academic writing style of their disciplinary major. Learning how to read scholarly articles also requires careful and deliberate concentration on how authors use specific language and phrasing to convey their research, the problem it addresses, its relationship to prior research, its significance, its limitations, and how authors connect methods of data gathering to the results so as to develop recommended solutions derived from the overall research process.

Improve Comprehension Skills

In addition to knowing how to read scholarly journals articles, students must learn how to effectively interpret what the scholar(s) are trying to convey. Academic writing can be dense, multi-layered, and non-linear in how information is presented. In addition, scholarly articles contain footnotes or endnotes, references to sources, multiple appendices, and, in some cases, non-textual elements [e.g., graphs, charts] that can break-up the reader’s experience with the narrative flow of the study. Analyzing articles helps students practice comprehending these elements of writing, critiquing the arguments being made, reflecting upon the significance of the research, and how it relates to building new knowledge and understanding or applying new approaches to practice. Comprehending scholarly writing also involves thinking critically about where you fit within the overall dialogue among scholars concerning the research problem, finding possible gaps in the research that require further analysis, or identifying where the author(s) has failed to examine fully any specific elements of the study.

In addition, journal article analysis assignments are used by professors to strengthen discipline-specific information literacy skills, either alone or in relation to other tasks, such as, giving a class presentation or participating in a group project. These benefits can include the ability to:

  • Effectively paraphrase text, which leads to a more thorough understanding of the overall study;
  • Identify and describe strengths and weaknesses of the study and their implications;
  • Relate the article to other course readings and in relation to particular research concepts or ideas discussed during class;
  • Think critically about the research and summarize complex ideas contained within;
  • Plan, organize, and write an effective inquiry-based paper that investigates a research study, evaluates evidence, expounds on the author’s main ideas, and presents an argument concerning the significance and impact of the research in a clear and concise manner;
  • Model the type of source summary and critique you should do for any college-level research paper; and,
  • Increase interest and engagement with the research problem of the study as well as with the discipline.

Kershaw, Trina C., Jennifer Fugate, and Aminda J. O'Hare. "Teaching Undergraduates to Understand Published Research through Structured Practice in Identifying Key Research Concepts." Scholarship of Teaching and Learning in Psychology . Advance online publication, 2020; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students make the most of Scholarly Articles." Library Management 33 (2012): 525-535; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946.

Structure and Organization

A journal article analysis paper should be written in paragraph format and include an instruction to the study, your analysis of the research, and a conclusion that provides an overall assessment of the author's work, along with an explanation of what you believe is the study's overall impact and significance. Unless the purpose of the assignment is to examine foundational studies published many years ago, you should select articles that have been published relatively recently [e.g., within the past few years].

Since the research has been completed, reference to the study in your paper should be written in the past tense, with your analysis stated in the present tense [e.g., “The author portrayed access to health care services in rural areas as primarily a problem of having reliable transportation. However, I believe the author is overgeneralizing this issue because...”].

Introduction Section

The first section of a journal analysis paper should describe the topic of the article and highlight the author’s main points. This includes describing the research problem and theoretical framework, the rationale for the research, the methods of data gathering and analysis, the key findings, and the author’s final conclusions and recommendations. The narrative should focus on the act of describing rather than analyzing. Think of the introduction as a more comprehensive and detailed descriptive abstract of the study.

Possible questions to help guide your writing of the introduction section may include:

  • Who are the authors and what credentials do they hold that contributes to the validity of the study?
  • What was the research problem being investigated?
  • What type of research design was used to investigate the research problem?
  • What theoretical idea(s) and/or research questions were used to address the problem?
  • What was the source of the data or information used as evidence for analysis?
  • What methods were applied to investigate this evidence?
  • What were the author's overall conclusions and key findings?

Critical Analysis Section

The second section of a journal analysis paper should describe the strengths and weaknesses of the study and analyze its significance and impact. This section is where you shift the narrative from describing to analyzing. Think critically about the research in relation to other course readings, what has been discussed in class, or based on your own life experiences. If you are struggling to identify any weaknesses, explain why you believe this to be true. However, no study is perfect, regardless of how laudable its design may be. Given this, think about the repercussions of the choices made by the author(s) and how you might have conducted the study differently. Examples can include contemplating the choice of what sources were included or excluded in support of examining the research problem, the choice of the method used to analyze the data, or the choice to highlight specific recommended courses of action and/or implications for practice over others. Another strategy is to place yourself within the research study itself by thinking reflectively about what may be missing if you had been a participant in the study or if the recommended courses of action specifically targeted you or your community.

Possible questions to help guide your writing of the analysis section may include:

Introduction

  • Did the author clearly state the problem being investigated?
  • What was your reaction to and perspective on the research problem?
  • Was the study’s objective clearly stated? Did the author clearly explain why the study was necessary?
  • How well did the introduction frame the scope of the study?
  • Did the introduction conclude with a clear purpose statement?

Literature Review

  • Did the literature review lay a foundation for understanding the significance of the research problem?
  • Did the literature review provide enough background information to understand the problem in relation to relevant contexts [e.g., historical, economic, social, cultural, etc.].
  • Did literature review effectively place the study within the domain of prior research? Is anything missing?
  • Was the literature review organized by conceptual categories or did the author simply list and describe sources?
  • Did the author accurately explain how the data or information were collected?
  • Was the data used sufficient in supporting the study of the research problem?
  • Was there another methodological approach that could have been more illuminating?
  • Give your overall evaluation of the methods used in this article. How much trust would you put in generating relevant findings?

Results and Discussion

  • Were the results clearly presented?
  • Did you feel that the results support the theoretical and interpretive claims of the author? Why?
  • What did the author(s) do especially well in describing or analyzing their results?
  • Was the author's evaluation of the findings clearly stated?
  • How well did the discussion of the results relate to what is already known about the research problem?
  • Was the discussion of the results free of repetition and redundancies?
  • What interpretations did the authors make that you think are in incomplete, unwarranted, or overstated?
  • Did the conclusion effectively capture the main points of study?
  • Did the conclusion address the research questions posed? Do they seem reasonable?
  • Were the author’s conclusions consistent with the evidence and arguments presented?
  • Has the author explained how the research added new knowledge or understanding?

Overall Writing Style

  • If the article included tables, figures, or other non-textual elements, did they contribute to understanding the study?
  • Were ideas developed and related in a logical sequence?
  • Were transitions between sections of the article smooth and easy to follow?

Overall Evaluation Section

The final section of a journal analysis paper should bring your thoughts together into a coherent assessment of the value of the research study . This section is where the narrative flow transitions from analyzing specific elements of the article to critically evaluating the overall study. Explain what you view as the significance of the research in relation to the overall course content and any relevant discussions that occurred during class. Think about how the article contributes to understanding the overall research problem, how it fits within existing literature on the topic, how it relates to the course, and what it means to you as a student researcher. In some cases, your professor will also ask you to describe your experiences writing the journal article analysis paper as part of a reflective learning exercise.

Possible questions to help guide your writing of the conclusion and evaluation section may include:

  • Was the structure of the article clear and well organized?
  • Was the topic of current or enduring interest to you?
  • What were the main weaknesses of the article? [this does not refer to limitations stated by the author, but what you believe are potential flaws]
  • Was any of the information in the article unclear or ambiguous?
  • What did you learn from the research? If nothing stood out to you, explain why.
  • Assess the originality of the research. Did you believe it contributed new understanding of the research problem?
  • Were you persuaded by the author’s arguments?
  • If the author made any final recommendations, will they be impactful if applied to practice?
  • In what ways could future research build off of this study?
  • What implications does the study have for daily life?
  • Was the use of non-textual elements, footnotes or endnotes, and/or appendices helpful in understanding the research?
  • What lingering questions do you have after analyzing the article?

NOTE: Avoid using quotes. One of the main purposes of writing an article analysis paper is to learn how to effectively paraphrase and use your own words to summarize a scholarly research study and to explain what the research means to you. Using and citing a direct quote from the article should only be done to help emphasize a key point or to underscore an important concept or idea.

Business: The Article Analysis . Fred Meijer Center for Writing, Grand Valley State University; Bachiochi, Peter et al. "Using Empirical Article Analysis to Assess Research Methods Courses." Teaching of Psychology 38 (2011): 5-9; Brosowsky, Nicholaus P. et al. “Teaching Undergraduate Students to Read Empirical Articles: An Evaluation and Revision of the QALMRI Method.” PsyArXi Preprints , 2020; Holster, Kristin. “Article Evaluation Assignment”. TRAILS: Teaching Resources and Innovations Library for Sociology . Washington DC: American Sociological Association, 2016; Kershaw, Trina C., Jennifer Fugate, and Aminda J. O'Hare. "Teaching Undergraduates to Understand Published Research through Structured Practice in Identifying Key Research Concepts." Scholarship of Teaching and Learning in Psychology . Advance online publication, 2020; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Reviewer's Guide . SAGE Reviewer Gateway, SAGE Journals; Sego, Sandra A. and Anne E. Stuart. "Learning to Read Empirical Articles in General Psychology." Teaching of Psychology 43 (2016): 38-42; Kershaw, Trina C., Jordan P. Lippman, and Jennifer Fugate. "Practice Makes Proficient: Teaching Undergraduate Students to Understand Published Research." Instructional Science 46 (2018): 921-946; Gyuris, Emma, and Laura Castell. "To Tell Them or Show Them? How to Improve Science Students’ Skills of Critical Reading." International Journal of Innovation in Science and Mathematics Education 21 (2013): 70-80; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36; MacMillan, Margy and Allison MacKenzie. "Strategies for Integrating Information Literacy and Academic Literacy: Helping Undergraduate Students Make the Most of Scholarly Articles." Library Management 33 (2012): 525-535.

Writing Tip

Not All Scholarly Journal Articles Can Be Critically Analyzed

There are a variety of articles published in scholarly journals that do not fit within the guidelines of an article analysis assignment. This is because the work cannot be empirically examined or it does not generate new knowledge in a way which can be critically analyzed.

If you are required to locate a research study on your own, avoid selecting these types of journal articles:

  • Theoretical essays which discuss concepts, assumptions, and propositions, but report no empirical research;
  • Statistical or methodological papers that may analyze data, but the bulk of the work is devoted to refining a new measurement, statistical technique, or modeling procedure;
  • Articles that review, analyze, critique, and synthesize prior research, but do not report any original research;
  • Brief essays devoted to research methods and findings;
  • Articles written by scholars in popular magazines or industry trade journals;
  • Pre-print articles that have been posted online, but may undergo further editing and revision by the journal's editorial staff before final publication; and
  • Academic commentary that discusses research trends or emerging concepts and ideas, but does not contain citations to sources.

Journal Analysis Assignment - Myers . Writing@CSU, Colorado State University; Franco, Josue. “Introducing the Analysis of Journal Articles.” Prepared for presentation at the American Political Science Association’s 2020 Teaching and Learning Conference, February 7-9, 2020, Albuquerque, New Mexico; Woodward-Kron, Robyn. "Critical Analysis and the Journal Article Review Assignment." Prospect 18 (August 2003): 20-36.

  • << Previous: Annotated Bibliography
  • Next: Giving an Oral Presentation >>
  • Last Updated: Mar 6, 2024 1:00 PM
  • URL: https://libguides.usc.edu/writingguide/assignments

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base
  • Research paper

How to Write a Research Paper | A Beginner's Guide

A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research.

Research papers are similar to academic essays , but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research. Writing a research paper requires you to demonstrate a strong knowledge of your topic, engage with a variety of sources, and make an original contribution to the debate.

This step-by-step guide takes you through the entire writing process, from understanding your assignment to proofreading your final draft.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

upload-your-document-ai-proofreader

Table of contents

Understand the assignment, choose a research paper topic, conduct preliminary research, develop a thesis statement, create a research paper outline, write a first draft of the research paper, write the introduction, write a compelling body of text, write the conclusion, the second draft, the revision process, research paper checklist, free lecture slides.

Completing a research paper successfully means accomplishing the specific tasks set out for you. Before you start, make sure you thoroughly understanding the assignment task sheet:

  • Read it carefully, looking for anything confusing you might need to clarify with your professor.
  • Identify the assignment goal, deadline, length specifications, formatting, and submission method.
  • Make a bulleted list of the key points, then go back and cross completed items off as you’re writing.

Carefully consider your timeframe and word limit: be realistic, and plan enough time to research, write, and edit.

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

  • Missing commas and periods
  • Incorrect usage of “et al.”
  • Ampersands (&) in narrative citations
  • Missing reference entries

journal analysis research paper

There are many ways to generate an idea for a research paper, from brainstorming with pen and paper to talking it through with a fellow student or professor.

You can try free writing, which involves taking a broad topic and writing continuously for two or three minutes to identify absolutely anything relevant that could be interesting.

You can also gain inspiration from other research. The discussion or recommendations sections of research papers often include ideas for other specific topics that require further examination.

Once you have a broad subject area, narrow it down to choose a topic that interests you, m eets the criteria of your assignment, and i s possible to research. Aim for ideas that are both original and specific:

  • A paper following the chronology of World War II would not be original or specific enough.
  • A paper on the experience of Danish citizens living close to the German border during World War II would be specific and could be original enough.

Note any discussions that seem important to the topic, and try to find an issue that you can focus your paper around. Use a variety of sources , including journals, books, and reliable websites, to ensure you do not miss anything glaring.

Do not only verify the ideas you have in mind, but look for sources that contradict your point of view.

  • Is there anything people seem to overlook in the sources you research?
  • Are there any heated debates you can address?
  • Do you have a unique take on your topic?
  • Have there been some recent developments that build on the extant research?

In this stage, you might find it helpful to formulate some research questions to help guide you. To write research questions, try to finish the following sentence: “I want to know how/what/why…”

A thesis statement is a statement of your central argument — it establishes the purpose and position of your paper. If you started with a research question, the thesis statement should answer it. It should also show what evidence and reasoning you’ll use to support that answer.

The thesis statement should be concise, contentious, and coherent. That means it should briefly summarize your argument in a sentence or two, make a claim that requires further evidence or analysis, and make a coherent point that relates to every part of the paper.

You will probably revise and refine the thesis statement as you do more research, but it can serve as a guide throughout the writing process. Every paragraph should aim to support and develop this central claim.

A research paper outline is essentially a list of the key topics, arguments, and evidence you want to include, divided into sections with headings so that you know roughly what the paper will look like before you start writing.

A structure outline can help make the writing process much more efficient, so it’s worth dedicating some time to create one.

Your first draft won’t be perfect — you can polish later on. Your priorities at this stage are as follows:

  • Maintaining forward momentum — write now, perfect later.
  • Paying attention to clear organization and logical ordering of paragraphs and sentences, which will help when you come to the second draft.
  • Expressing your ideas as clearly as possible, so you know what you were trying to say when you come back to the text.

You do not need to start by writing the introduction. Begin where it feels most natural for you — some prefer to finish the most difficult sections first, while others choose to start with the easiest part. If you created an outline, use it as a map while you work.

Do not delete large sections of text. If you begin to dislike something you have written or find it doesn’t quite fit, move it to a different document, but don’t lose it completely — you never know if it might come in useful later.

Paragraph structure

Paragraphs are the basic building blocks of research papers. Each one should focus on a single claim or idea that helps to establish the overall argument or purpose of the paper.

Example paragraph

George Orwell’s 1946 essay “Politics and the English Language” has had an enduring impact on thought about the relationship between politics and language. This impact is particularly obvious in light of the various critical review articles that have recently referenced the essay. For example, consider Mark Falcoff’s 2009 article in The National Review Online, “The Perversion of Language; or, Orwell Revisited,” in which he analyzes several common words (“activist,” “civil-rights leader,” “diversity,” and more). Falcoff’s close analysis of the ambiguity built into political language intentionally mirrors Orwell’s own point-by-point analysis of the political language of his day. Even 63 years after its publication, Orwell’s essay is emulated by contemporary thinkers.

Citing sources

It’s also important to keep track of citations at this stage to avoid accidental plagiarism . Each time you use a source, make sure to take note of where the information came from.

You can use our free citation generators to automatically create citations and save your reference list as you go.

APA Citation Generator MLA Citation Generator

The research paper introduction should address three questions: What, why, and how? After finishing the introduction, the reader should know what the paper is about, why it is worth reading, and how you’ll build your arguments.

What? Be specific about the topic of the paper, introduce the background, and define key terms or concepts.

Why? This is the most important, but also the most difficult, part of the introduction. Try to provide brief answers to the following questions: What new material or insight are you offering? What important issues does your essay help define or answer?

How? To let the reader know what to expect from the rest of the paper, the introduction should include a “map” of what will be discussed, briefly presenting the key elements of the paper in chronological order.

The major struggle faced by most writers is how to organize the information presented in the paper, which is one reason an outline is so useful. However, remember that the outline is only a guide and, when writing, you can be flexible with the order in which the information and arguments are presented.

One way to stay on track is to use your thesis statement and topic sentences . Check:

  • topic sentences against the thesis statement;
  • topic sentences against each other, for similarities and logical ordering;
  • and each sentence against the topic sentence of that paragraph.

Be aware of paragraphs that seem to cover the same things. If two paragraphs discuss something similar, they must approach that topic in different ways. Aim to create smooth transitions between sentences, paragraphs, and sections.

The research paper conclusion is designed to help your reader out of the paper’s argument, giving them a sense of finality.

Trace the course of the paper, emphasizing how it all comes together to prove your thesis statement. Give the paper a sense of finality by making sure the reader understands how you’ve settled the issues raised in the introduction.

You might also discuss the more general consequences of the argument, outline what the paper offers to future students of the topic, and suggest any questions the paper’s argument raises but cannot or does not try to answer.

You should not :

  • Offer new arguments or essential information
  • Take up any more space than necessary
  • Begin with stock phrases that signal you are ending the paper (e.g. “In conclusion”)

There are four main considerations when it comes to the second draft.

  • Check how your vision of the paper lines up with the first draft and, more importantly, that your paper still answers the assignment.
  • Identify any assumptions that might require (more substantial) justification, keeping your reader’s perspective foremost in mind. Remove these points if you cannot substantiate them further.
  • Be open to rearranging your ideas. Check whether any sections feel out of place and whether your ideas could be better organized.
  • If you find that old ideas do not fit as well as you anticipated, you should cut them out or condense them. You might also find that new and well-suited ideas occurred to you during the writing of the first draft — now is the time to make them part of the paper.

The goal during the revision and proofreading process is to ensure you have completed all the necessary tasks and that the paper is as well-articulated as possible. You can speed up the proofreading process by using the AI proofreader .

Global concerns

  • Confirm that your paper completes every task specified in your assignment sheet.
  • Check for logical organization and flow of paragraphs.
  • Check paragraphs against the introduction and thesis statement.

Fine-grained details

Check the content of each paragraph, making sure that:

  • each sentence helps support the topic sentence.
  • no unnecessary or irrelevant information is present.
  • all technical terms your audience might not know are identified.

Next, think about sentence structure , grammatical errors, and formatting . Check that you have correctly used transition words and phrases to show the connections between your ideas. Look for typos, cut unnecessary words, and check for consistency in aspects such as heading formatting and spellings .

Finally, you need to make sure your paper is correctly formatted according to the rules of the citation style you are using. For example, you might need to include an MLA heading  or create an APA title page .

Scribbr’s professional editors can help with the revision process with our award-winning proofreading services.

Discover our paper editing service

Checklist: Research paper

I have followed all instructions in the assignment sheet.

My introduction presents my topic in an engaging way and provides necessary background information.

My introduction presents a clear, focused research problem and/or thesis statement .

My paper is logically organized using paragraphs and (if relevant) section headings .

Each paragraph is clearly focused on one central idea, expressed in a clear topic sentence .

Each paragraph is relevant to my research problem or thesis statement.

I have used appropriate transitions  to clarify the connections between sections, paragraphs, and sentences.

My conclusion provides a concise answer to the research question or emphasizes how the thesis has been supported.

My conclusion shows how my research has contributed to knowledge or understanding of my topic.

My conclusion does not present any new points or information essential to my argument.

I have provided an in-text citation every time I refer to ideas or information from a source.

I have included a reference list at the end of my paper, consistently formatted according to a specific citation style .

I have thoroughly revised my paper and addressed any feedback from my professor or supervisor.

I have followed all formatting guidelines (page numbers, headers, spacing, etc.).

You've written a great paper. Make sure it's perfect with the help of a Scribbr editor!

Open Google Slides Download PowerPoint

Is this article helpful?

Other students also liked.

  • Writing a Research Paper Introduction | Step-by-Step Guide
  • Writing a Research Paper Conclusion | Step-by-Step Guide
  • Research Paper Format | APA, MLA, & Chicago Templates

More interesting articles

  • Academic Paragraph Structure | Step-by-Step Guide & Examples
  • Checklist: Writing a Great Research Paper
  • How to Create a Structured Research Paper Outline | Example
  • How to Write a Discussion Section | Tips & Examples
  • How to Write Recommendations in Research | Examples & Tips
  • How to Write Topic Sentences | 4 Steps, Examples & Purpose
  • Research Paper Appendix | Example & Templates
  • Research Paper Damage Control | Managing a Broken Argument
  • What Is a Theoretical Framework? | Guide to Organizing

What is your plagiarism score?

U.S. flag

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

  • Publications
  • Account settings
  • Advanced Search
  • Journal List
  • Perspect Clin Res
  • v.12(2); Apr-Jun 2021

Critical appraisal of published research papers – A reinforcing tool for research methodology: Questionnaire-based study

Snehalata gajbhiye.

Department of Pharmacology and Therapeutics, Seth GS Medical College and KEM Hospital, Mumbai, Maharashtra, India

Raakhi Tripathi

Urwashi parmar, nishtha khatri, anirudha potey.

1 Department of Clinical Trials, Serum Institute of India, Pune, Maharashtra, India

Background and Objectives:

Critical appraisal of published research papers is routinely conducted as a journal club (JC) activity in pharmacology departments of various medical colleges across Maharashtra, and it forms an important part of their postgraduate curriculum. The objective of this study was to evaluate the perception of pharmacology postgraduate students and teachers toward use of critical appraisal as a reinforcing tool for research methodology. Evaluation of performance of the in-house pharmacology postgraduate students in the critical appraisal activity constituted secondary objective of the study.

Materials and Methods:

The study was conducted in two parts. In Part I, a cross-sectional questionnaire-based evaluation on perception toward critical appraisal activity was carried out among pharmacology postgraduate students and teachers. In Part II of the study, JC score sheets of 2 nd - and 3 rd -year pharmacology students over the past 4 years were evaluated.

One hundred and twenty-seven postgraduate students and 32 teachers participated in Part I of the study. About 118 (92.9%) students and 28 (87.5%) faculties considered the critical appraisal activity to be beneficial for the students. JC score sheet assessments suggested that there was a statistically significant improvement in overall scores obtained by postgraduate students ( n = 25) in their last JC as compared to the first JC.

Conclusion:

Journal article criticism is a crucial tool to develop a research attitude among postgraduate students. Participation in the JC activity led to the improvement in the skill of critical appraisal of published research articles, but this improvement was not educationally relevant.

INTRODUCTION

Critical appraisal of a research paper is defined as “The process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context.”[ 1 ] Since scientific literature is rapidly expanding with more than 12,000 articles being added to the MEDLINE database per week,[ 2 ] critical appraisal is very important to distinguish scientifically useful and well-written articles from imprecise articles.

Educational authorities like the Medical Council of India (MCI) and Maharashtra University of Health Sciences (MUHS) have stated in pharmacology postgraduate curriculum that students must critically appraise research papers. To impart training toward these skills, MCI and MUHS have emphasized on the introduction of journal club (JC) activity for postgraduate (PG) students, wherein students review a published original research paper and state the merits and demerits of the paper. Abiding by this, pharmacology departments across various medical colleges in Maharashtra organize JC at frequent intervals[ 3 , 4 ] and students discuss varied aspects of the article with teaching faculty of the department.[ 5 ] Moreover, this activity carries a significant weightage of marks in the pharmacology university examination. As postgraduate students attend this activity throughout their 3-year tenure, it was perceived by the authors that this activity of critical appraisal of research papers could emerge as a tool for reinforcing the knowledge of research methodology. Hence, a questionnaire-based study was designed to find out the perceptions from PG students and teachers.

There have been studies that have laid emphasis on the procedure of conducting critical appraisal of research papers and its application into clinical practice.[ 6 , 7 ] However, there are no studies that have evaluated how well students are able to critically appraise a research paper. The Department of Pharmacology and Therapeutics at Seth GS Medical College has developed an evaluation method to score the PG students on this skill and this tool has been implemented for the last 5 years. Since there are no research data available on the performance of PG Pharmacology students in JC, capturing the critical appraisal activity evaluation scores of in-house PG students was chosen as another objective of the study.

MATERIALS AND METHODS

Description of the journal club activity.

JC is conducted in the Department of Pharmacology and Therapeutics at Seth GS Medical College once in every 2 weeks. During the JC activity, postgraduate students critically appraise published original research articles on their completeness and aptness in terms of the following: study title, rationale, objectives, study design, methodology-study population, inclusion/exclusion criteria, duration, intervention and safety/efficacy variables, randomization, blinding, statistical analysis, results, discussion, conclusion, references, and abstract. All postgraduate students attend this activity, while one of them critically appraises the article (who has received the research paper given by one of the faculty members 5 days before the day of JC). Other faculties also attend these sessions and facilitate the discussions. As the student comments on various sections of the paper, the same predecided faculty who gave the article (single assessor) evaluates the student on a total score of 100 which is split per section as follows: Introduction –20 marks, Methodology –20 marks, Discussion – 20 marks, Results and Conclusion –20 marks, References –10 marks, and Title, Abstract, and Keywords – 10 marks. However, there are no standard operating procedures to assess the performance of students at JC.

Methodology

After seeking permission from the Institutional Ethics Committee, the study was conducted in two parts. Part I consisted of a cross-sectional questionnaire-based survey that was conducted from October 2016 to September 2017. A questionnaire to evaluate perception towards the activity of critical appraisal of published papers as research methodology reinforcing tool was developed by the study investigators. The questionnaire consisted of 20 questions: 14 questions [refer Figure 1 ] graded on a 3-point Likert scale (agree, neutral, and disagree), 1 multiple choice selection question, 2 dichotomous questions, 1 semi-open-ended questions, and 2 open-ended questions. Content validation for this questionnaire was carried out with the help of eight pharmacology teachers. The content validity ratio per item was calculated and each item in the questionnaire had a CVR ratio (CVR) of >0.75.[ 8 ] The perception questionnaire was either E-mailed or sent through WhatsApp to PG pharmacology students and teaching faculty in pharmacology departments at various medical colleges across Maharashtra. Informed consent was obtained on E-mail from all the participants.

An external file that holds a picture, illustration, etc.
Object name is PCR-12-100-g001.jpg

Graphical representation of the percentage of students/teachers who agreed that critical appraisal of research helped them improve their knowledge on various aspects of research, perceived that faculty participation is important in this activity, and considered critical appraisal activity beneficial for students. The numbers adjacent to the bar diagrams indicate the raw number of students/faculty who agreed, while brackets indicate %

Part II of the study consisted of evaluating the performance of postgraduate students toward skills of critical appraisal of published papers. For this purpose, marks obtained by 2 nd - and 3 rd -year residents during JC sessions conducted over a period of 4 years from October 2013 to September 2017 were recorded and analyzed. No data on personal identifiers of the students were captured.

Statistical analysis

Marks obtained by postgraduate students in their first and last JC were compared using Wilcoxon signed-rank test, while marks obtained by 2 nd - and 3 rd -year postgraduate students were compared using Mann–Whitney test since the data were nonparametric. These statistical analyses were performed using GraphPad Prism statistical software, San Diego, Calfornia, USA, Version 7.0d. Data obtained from the perception questionnaire were entered in Microsoft Excel sheet and were expressed as frequencies (percentages) using descriptive statistics.

Participants who answered all items of the questionnaire were considered as complete responders and only completed questionnaires were analyzed. The questionnaire was sent through an E-mail to 100 students and through WhatsApp to 68 students. Out of the 100 students who received the questionnaire through E-mail, 79 responded completely and 8 were incomplete responders, while 13 students did not revert back. Out of the 68 students who received the questionnaire through WhatsApp, 48 responded completely, 6 gave an incomplete response, and 14 students did not revert back. Hence, of the 168 postgraduate students who received the questionnaire, 127 responded completely (student response rate for analysis = 75.6%). The questionnaire was E-mailed to 33 faculties and was sent through WhatsApp to 25 faculties. Out of the 33 faculties who received the questionnaire through E-mail, 19 responded completely, 5 responded incompletely, and 9 did not respond at all. Out of the 25 faculties who received the questionnaire through WhatsApp, 13 responded completely, 3 were incomplete responders, and 9 did not respond at all. Hence, of a total of 58 faculties who were contacted, 32 responded completely (faculty response rate for analysis = 55%). For Part I of the study, responses on the perception questionnaire from 127 postgraduate students and 32 postgraduate teachers were recorded and analyzed. None of the faculty who participated in the validation of the questionnaire participated in the survey. Number of responses obtained region wise (Mumbai region and rest of Maharashtra region) have been depicted in Table 1 .

Region-wise distribution of responses

Number of responses obtained from students/faculty belonging to Mumbai colleges and rest of Maharashtra colleges. Brackets indicate percentages

As per the data obtained on the Likert scale questions, 102 (80.3%) students and 29 (90.6%) teachers agreed that critical appraisal trains the students in doing a review of literature before selecting a particular research topic. Majority of the participants, i.e., 104 (81.9%) students and 29 (90.6%) teachers also believed that the activity increases student's knowledge regarding various experimental evaluation techniques. Moreover, 112 (88.2%) students and 27 (84.4%) faculty considered that critical appraisal activity results in improved skills of writing and understanding methodology section of research articles in terms of inclusion/exclusion criteria, endpoints, and safety/efficacy variables. About 103 (81.1%) students and 24 (75%) teachers perceived that this activity results in refinement of the student's research work. About 118 (92.9%) students and 28 (87.5%) faculty considered the critical appraisal activity to be beneficial for the students. Responses to 14 individual Likert scale items of the questionnaire have been depicted in Figure 1 .

With respect to the multiple choice selection question, 66 (52%) students and 16 (50%) teachers opined that faculty should select the paper, 53 (41.7%) students and 9 (28.1%) teachers stated that the papers should be selected by the presenting student himself/herself, while 8 (6.3%) students and 7 (21.9%) teachers expressed that some other student should select the paper to be presented at the JC.

The responses to dichotomous questions were as follows: majority of the students, that is, 109 (85.8%) and 23 (71.9%) teachers perceived that a standard checklist for article review should be given to the students before critical appraisal of journal article. Open-ended questions of the questionnaire invited suggestions from the participants regarding ways of getting trained on critical appraisal skills and of improving JC activity. Some of the suggestions given by faculty were as follows: increasing the frequency of JC activity, discussion of cited articles and new guidelines related to it, selecting all types of articles for criticism rather than only randomized controlled trials, and regular yearly exams on article criticism. Students stated that regular and frequent article criticism activity, practice of writing letter to the editor after criticism, active participation by peers and faculty, increasing weightage of marks for critical appraisal of papers in university examinations (at present marks are 50 out of 400), and a formal training for research criticism from 1 st year of postgraduation could improve critical appraisal program.

In Part II of this study, performance of the students on the skill of critical appraisal of papers was evaluated. Complete data of the first and last JC scores of a total of 25 students of the department were available, and when these scores were compared, it was seen that there was a statistically significant improvement in the overall scores ( P = 0.04), as well as in the scores obtained in methodology ( P = 0.03) and results section ( P = 0.02). This is depicted in Table 2 . Although statistically significant, the differences in scores in the methodology section, results section, and overall scores were 1.28/20, 1.28/20, and 4.36/100, respectively, amounting to 5.4%, 5.4%, and 4.36% higher scores in the last JC, which may not be considered educationally relevant (practically significant). The quantum of difference that would be considered practically significant was not decided a priori .

Comparison of marks obtained by pharmacology residents in their first and last journal club

Marks have been represented as mean±SD. The maximum marks that can be obtained in each section have been stated as maximum. *Indicates statistically significant ( P <0.05). IQR=Interquartile range, SD=Standard deviation

Scores of two groups, one group consisting of 2 nd -year postgraduate students ( n = 44) and second group consisting of 3 rd -year postgraduate students ( n = 32) were compared and revealed no statistically significant difference in overall score ( P = 0.84). This is depicted in Table 3 . Since the quantum of difference in the overall scores was meager 0.84/100 (0.84%), it cannot be considered practically significant.

Comparison of marks obtained by 2 nd - and 3 rd -year pharmacology residents in the activity of critical appraisal of research articles

Marks have been represented as mean±SD. The maximum marks that can be obtained in each section have been stated as maximum. P <0.05 was considered to be statistically significant. IQR=Interquartile range, SD=Standard deviation

The present study gauged the perception of the pharmacology postgraduate students and teachers toward the use of critical appraisal activity as a reinforcing tool for research methodology. Both students and faculties (>50%) believed that critical appraisal activity increases student's knowledge on principles of ethics, experimental evaluation techniques, CONSORT guidelines, statistical analysis, concept of conflict of interest, current trends and recent advances in Pharmacology and trains on doing a review of literature, and improves skills on protocol writing and referencing. In the study conducted by Crank-Patton et al ., a survey on 278 general surgery program directors was carried out and more than 50% indicated that JC was important to their training program.[ 9 ]

The grading template used in Part II of the study was based on the IMRaD structure. Hence, equal weightage was given to the Introduction, Methodology, Results, and Discussion sections and lesser weightage was given to the references and title, abstract, and keywords sections.[ 10 ] While evaluating the scores obtained by 25 students in their first and last JC, it was seen that there was a statistically significant improvement in the overall scores of the students in their last JC. However, the meager improvement in scores cannot be considered educationally relevant, as the authors expected the students to score >90% for the upgrade to be considered educationally impactful. The above findings suggest that even though participation in the JC activity led to a steady increase in student's performance (~4%), the increment was not as expected. In addition, the students did not portray an excellent performance (>90%), with average scores being around 72% even in the last JC. This can be probably explained by the fact that students perform this activity in a routine setting and not in an examination setting. Unlike the scenario in an examination, students were aware that even if they performed at a mediocre level, there would be no repercussions.

A separate comparison of scores obtained by 44 students in their 2 nd year and 32 students in their 3 rd year of postgraduation students was also done. The number of student evaluation sheets reviewed for this analysis was greater than the number of student evaluation sheets reviewed to compare first and last JC scores. This can be spelled out by the fact that many students were still in 2 nd year when this analysis was done and the score data for their last JC, which would take place in 3 rd year, was not available. In addition, few students were asked to present at JC multiple times during the 2 nd /3 rd year of their postgraduation.

While evaluating the critical appraisal scores obtained by 2 nd - and 3 rd -year postgraduate students, it was found that although the 3 rd -year students had a mean overall score greater than the 2 nd -year students, this difference was not statistically significant. During the 1 st year of MD Pharmacology course, students at the study center attend JC once in every 2 weeks. Even though the 1 st -year students do not themselves present in JC, they listen and observe the criticism points stated by senior peers presenting at the JC, and thereby, incur substantial amount of knowledge required to critically appraise papers. By the time, they become 2 nd -year students, they are already well versed with the program and this could have led to similar overall mean scores between the 2 nd -year students (71.50 ± 10.71) and 3 rd -year students (72.34 ± 10.85). This finding suggests that attentive listening is as important as active participation in the JC. Moreover, although students are well acquainted with the process of criticism when they are in their 3 rd year, there is certainly a scope for improvement in terms of the mean overall scores.

Similar results were obtained in a study conducted by Stern et al ., in which 62 students in the internal medicine program at the New England Medical Center were asked to respond to a questionnaire, evaluate a sample article, and complete a self-assessment of competence in evaluation of research. Twenty-eight residents returned the questionnaire and the composite score for the resident's objective assessment was not significantly correlated with the postgraduate year or self-assessed critical appraisal skill.[ 11 ]

Article criticism activity provides the students with practical experience of techniques taught in research methodology workshop. However, this should be supplemented with activities that assess the improvement of designing and presenting studies, such as protocol and paper writing. Thus, critical appraisal plays a significant role in reinforcing good research practices among the new generation of physicians. Moreover, critical appraisal is an integral part of PG assessment, and although the current format of conducting JCs did not portray a clinically meaningful improvement, the authors believe that it is important to continue this activity with certain modifications suggested by students who participated in this study. Students suggested that an increase in the frequency of critical appraisal activity accompanied by the display of active participation by peers and faculty could help in the betterment of this activity. This should be brought to attention of the faculty, as students seem to be interested to learn. Critical appraisal should be a two-way teaching–learning process between the students and faculty and not a dire need for satisfying the students' eligibility criteria for postgraduate university examinations. This activity is not only for the trainee doctors but also a part of the overall faculty development program.[ 12 ]

In the present era, JCs have been used as a tool to not only teach critical appraisal skills but also to teach other necessary aspects such as research design, medical statistics, clinical epidemiology, and clinical decision-making.[ 13 , 14 ] A study conducted by Khan in 2013 suggested that success of JC program can be ensured if institutes develop a defined JC objective for the development of learning capability of students and also if they cultivate more skilled faculties.[ 15 ] A good JC is believed to facilitate relevant, meaningful scientific discussion, and evaluation of the research updates that will eventually benefit the patient care.[ 12 ]

Although there is a lot of literature emphasizing the importance of JC, there is a lack of studies that have evaluated the outcome of such activity. One such study conducted by Ibrahim et al . assessed the importance of critical appraisal as an activity in surgical trainees in Nigeria. They reported that 92.42% trainees considered the activity to be important or very important and 48% trainees stated that the activity helped in improving literature search.[ 16 ]

This study is unique since it is the first of its kind to evaluate how well students are able to critically appraise a research paper. Moreover, the study has taken into consideration the due opinions of the students as well as faculties, unlike the previous literature which has laid emphasis on only student's perception. A limitation of this study is that sample size for faculties was smaller than the students, as it was not possible to convince the distant faculty in other cities to fill the survey. Besides, there may be a variation in the manner of conduct of the critical appraisal activity in pharmacology departments across the various medical colleges in the country. Another limitation of this study was that a single assessor graded a single student during one particular JC. Nevertheless, each student presented at multiple JC and thereby came across multiple assessors. Since the articles addressed at different JC were disparate, interobserver variability was not taken into account in this study. Furthermore, the authors did not make an a priori decision on the quantum of increase in scores that would be considered educationally meaningful.

Pharmacology students and teachers acknowledge the role of critical appraisal in improving the ability to understand the crucial concepts of research methodology and research conduct. In our institute, participation in the JC activity led to an improvement in the skill of critical appraisal of published research articles among the pharmacology postgraduate students. However, this improvement was not educationally relevant. The scores obtained by final-year postgraduate students in this activity were nearly 72% indicating that there is still scope of betterment in this skill.

Financial support and sponsorship

Conflicts of interest.

There are no conflicts of interest.

Acknowledgments

We would like to acknowledge the support rendered by the entire Department of Pharmacology and Therapeutics at Seth GS Medical College.

Detail of a painting depicting the landscape of New Mexico with mountains in the distance

Explore millions of high-quality primary sources and images from around the world, including artworks, maps, photographs, and more.

Explore migration issues through a variety of media types

  • Part of The Streets are Talking: Public Forms of Creative Expression from Around the World
  • Part of The Journal of Economic Perspectives, Vol. 34, No. 1 (Winter 2020)
  • Part of Cato Institute (Aug. 3, 2021)
  • Part of University of California Press
  • Part of Open: Smithsonian National Museum of African American History & Culture
  • Part of Indiana Journal of Global Legal Studies, Vol. 19, No. 1 (Winter 2012)
  • Part of R Street Institute (Nov. 1, 2020)
  • Part of Leuven University Press
  • Part of UN Secretary-General Papers: Ban Ki-moon (2007-2016)
  • Part of Perspectives on Terrorism, Vol. 12, No. 4 (August 2018)
  • Part of Leveraging Lives: Serbia and Illegal Tunisian Migration to Europe, Carnegie Endowment for International Peace (Mar. 1, 2023)
  • Part of UCL Press

Harness the power of visual materials—explore more than 3 million images now on JSTOR.

Enhance your scholarly research with underground newspapers, magazines, and journals.

Explore collections in the arts, sciences, and literature from the world’s leading museums, archives, and scholars.

  • Search Menu
  • Supplements
  • Advance articles
  • Editor's Choice
  • Special Issues
  • Author Guidelines
  • Submission Site
  • Why Publish With Us?
  • Open Access
  • About Nicotine & Tobacco Research
  • About Society for Nicotine & Tobacco Research
  • Editorial Board
  • Advertising and Corporate Services
  • Journals Career Network
  • Self-Archiving Policy
  • Dispatch Dates
  • Journals on Oxford Academic
  • Books on Oxford Academic

Article Contents

Meta-analysis, declaration of interests, author contributions, the impact of menthol cigarette bans: a systematic review and meta-analysis.

ORCID logo

  • Article contents
  • Figures & tables
  • Supplementary Data

Sarah D Mills, Snigdha Peddireddy, Rachel Kurtzman, Frantasia Hill, Victor Catalan, Jennifer S Bissram, Kurt M Ribisl, The Impact of Menthol Cigarette Bans: A Systematic Review and Meta-Analysis, Nicotine & Tobacco Research , 2024;, ntae011, https://doi.org/10.1093/ntr/ntae011

  • Permissions Icon Permissions

This review investigates the impacts of banning the sale of menthol cigarettes at stores.

A systematic search of studies published in English up to November 2022 was conducted. The following databases were searched: PubMed/Medline, CINAHL, PsycINFO, Web of Science, and Embase, as well as a non-indexed journal. Studies evaluating either the impact of real-world or hypothesized menthol cigarette bans were included. Primary outcomes include tobacco use behaviors. Secondary outcomes include cigarette sales, retailer compliance, and the tobacco industry’s response to a menthol ban. Data on tobacco use behavior after a menthol ban were pooled using random-effects models. Two pairs of reviewers independently extracted data and assessed study quality.

Of the 964 articles that were identified during the initial search, 78 were included in the review and 16 were included in the meta-analysis. Cessation rates among menthol cigarette smokers were high after a menthol ban. Pooled results show that 24% (95% confidence interval [95% CI]: 20%, 28%) of menthol cigarette smokers quit smoking after a menthol ban, 50% (95% CI: 31%, 68%) switched to non-menthol cigarettes, 12% (95% CI: 3%, 20%) switched to other flavored tobacco products, and 24% (95% CI: 17%, 31%) continued smoking menthol cigarettes. Hypothesized quitting and switching rates were fairly close to real-world rates. Studies found the tobacco industry attempts to undermine menthol bans. National menthol bans appear more effective than local or state menthol bans.

Menthol cigarette bans promote smoking cessation suggesting their potential to improve public health.

Findings from this review suggest that menthol cigarette bans promote smoking cessation among menthol cigarette smokers and have the potential to improve public health.

Menthol cigarettes are of particular public health concern because studies show that the anesthetic and cooling effects of menthol mask the harshness of cigarettes, making it easier for youth to initiate smoking. 1 Menthol in cigarettes has been found to increase the bioavailability of nicotine, which is hypothesized to result in greater dependence, and smokers have greater difficulty in quitting menthol cigarettes compared to non-menthol cigarettes. 1

Prevalence rates of menthol cigarette use among cigarette smokers vary globally. 2 In a 2016 study of eight European countries, 7.4% of smokers, on average, used menthol cigarettes. 2 Prevalence rates ranged from 0.4% in Spain to 12.4% in England. In Kenya and Zambia, rates of menthol cigarette use among smokers were 21% (2012) and 43% (2014), respectively. 3 In the United States (US) 43.4% of adult past-month smokers used menthol cigarettes in 2020. 4 Menthol cigarettes are disproportionately used by youth, racial/ethnic minority, and lower-income smokers in the US. 4 Approximately 81% of non-Hispanic Black smokers in the US use menthol cigarettes, as compared to 34% of non-Hispanic White smokers. 4

More than 170 US localities and two states, several countries (eg, Canada, Ethiopia), and the European Union ban the sale of menthol cigarettes. 5 , 6 Research on the impacts of policies that ban the sale of menthol cigarettes is emerging. To the best of our knowledge, this study is the first systematic review to incorporate a meta-analysis of research examining the impact of menthol cigarette bans (also referred to as menthol bans) on tobacco use behaviors. Secondary outcomes for the review include impacts on cigarette sales, retailer compliance, and the tobacco industry’s response to menthol bans. A prior scoping review of studies published until November 2019 suggested that banning flavored tobacco product sales would promote smoking cessation. 7 A systematic review of studies published through May 2020 concluded there was insufficient evidence to make definitive conclusions about the effects of flavor bans on tobacco use behavior. 8 The number of localities with menthol bans has grown rapidly in the US and internationally. 5 , 6 An updated comprehensive review of studies is needed to summarize the latest research. In addition, a meta-analysis may provide more precise quantitative estimates of changes in tobacco use behaviors after a menthol ban. Meta-analyses increase statistical power and provide more robust summary estimates with greater generalizability than the results of individual studies. 9

Data Sources and Search

A literature search of PubMed/Medline, CINAHL, PsycINFO, Web of Science, and Embase was conducted on May 20, 2020, and updated on November 3, 2022. A manual search of Tobacco Regulatory Science was conducted in 2020, as the journal was not indexed in electronic databases at that time. The search strategy was developed for PubMed/Medline and translated for use in the other databases ( Appendix Table 1 ). This study was registered (CRD42020156087) with the International Prospective Register of Systematic Reviews (PROSPERO), an online database of review protocols, and was guided by the standards of the Preferred Reporting Items for Systematic reviews and Meta-Analyses Statement (PRISMA). 10 Prospectively registering a review aims to increase transparency and reduce bias in the conduct of research.

Study Selection

Studies investigating either the real-world or hypothesized impacts of menthol cigarette bans on tobacco use behavior (ie, quitting, switching to other products, continued use of menthol cigarettes) were included. Studies of hypothesized tobacco use behavior refer to studies where smokers in settings without menthol bans were asked to consider how their tobacco use behavior would change in the presence of a menthol ban (eg, “If menthol cigarettes were no longer sold in US stores, would you quit smoking?”). Grey literature was included, and studies were not restricted by methodology (eg, experimental, cross-sectional). Studies were also included if they examined any of the following secondary outcomes: impact of a menthol ban on cigarette sales, retailer compliance, or the tobacco industry’s response to a menthol ban. Opinion pieces, commentaries, and articles published in languages other than English were excluded.

Two pairs of investigators independently evaluated studies for potential inclusion in Covidence, a web-based tool that supports the screening and data extraction process in systematic reviews. 11 Titles/abstracts were screened to identify relevant articles. Next, full-text articles were reviewed to determine eligibility for inclusion. Disagreements between investigators were resolved by a third investigator. Inter-rater agreement between the two investigators was good for the title/abstract screen (2020 screen: 96% agreement; Cohen’s κ = 0.77; 2022 screen: 90% agreement; Cohen’s κ = 0.79) and moderate to good for the full text review (2020 screen: 83% agreement; Cohen’s κ = 0.46; 2022 screen: 88% agreement; Cohen’s κ = 0.67).

Data Extraction and Study Assessment

The following information was extracted from studies that met inclusion criteria: (1) sample characteristics, (2) location, (3) study design, (4) ban information, (5) study period, and (6) results. We used the PREFS checklist, a tool for assessing the quality of stated preference studies, to evaluate stated preference experimental studies and studies of hypothesized tobacco use behavior. 12 The PREFS checklist evaluates study quality based on the study purpose, respondent sampling, description of methods, bias in reporting of results, and use of significance testing. Scores range from zero to five, and higher scores indicate higher study quality. 12 The quality of the other studies in the review was assessed using the Study Quality Assessment Tools, an online set of tools specific to individual study designs focused on appraising internal validity. 13 The tools include items that assess for potential flaws in methodology, sources of bias (eg, patient selection), strength of causality in the association between the intervention and outcome, and other factors. 13 Items include, “Was the study population clearly specified and defined?” and “Were the outcome measures clearly defined, valid, reliable, and implemented consistently across all study participants?” 13 Some items in the tool were not applicable to the studies evaluated (eg, “For exposures that can vary in amount or level, did the study examine different levels of the exposure as related to the outcome”). Two pairs of reviewers independently assessed the quality of studies. For the Study Quality Assessment Tools, reviewers provided a rating of “good,” “fair,” or “poor” based on an overall assessment of quality informed by their evaluation of relevant items in the assessment tool. Discrepancies in quality ratings were resolved by a third reviewer.

Statistical Analysis

Meta-analytic techniques were used to pool estimates of menthol cigarette smokers’ real-world or hypothesized tobacco use behaviors in the setting of a menthol ban. Studies examining real-world menthol bans in the meta-analysis were restricted to longitudinal studies among pre-ban menthol cigarette smokers. Studies on sales and cross-sectional studies examining the prevalence of menthol cigarette use or other tobacco products in the setting of menthol bans, without accounting for individuals’ pre-ban tobacco use behavior, were excluded. These studies did not provide information on tobacco use behavior specific to pre-ban menthol cigarette smokers in the setting of a menthol ban.

When at least two studies provided prevalence estimates for a tobacco use behavior outcome (eg, percentage of menthol smokers that quit), pooled prevalence estimates were obtained across studies using random effects models. Studies were pooled if they examined tobacco use behavior in similar policy environments (eg, national vs. local menthol bans) and time periods after the ban. In each study included in the meta-analysis, the prevalence estimate of the tobacco use behavior outcome was extracted. The standard error of each estimate was calculated using standard formulas based on the 95% confidence intervals (95% CI) of the estimate or the sample sizes reported in the study. The meta-analysis was conducted using metan in STATA version 18. The logits of the prevalence estimates and their standard errors were used to obtain the pooled prevalence estimates. For the forest plots, random effect regression logits and 95% confidence intervals were back transformed so absolute prevalence estimates, instead of logits, are presented. Statistical heterogeneity in the estimates were assessed using the I 2 statistic.

Systematic Review

The search yielded 964 unique articles. Seventy-eight articles met inclusion criteria ( Figure 1 ). The majority examined real-world or hypothesized tobacco use behaviors after a ban (real-world: k (number of studies) = 20; hypothesized: k  = 32). Fewer studies assessed the industry’s response ( k  = 9), retailer compliance ( k  = 6), and cigarette sales after a ban ( k  = 11). The quality of most studies was high ( Appendix Table 2 ). Informed by the studies identified in the review and health behavior theories that propose behavior is determined in part by attitudes and beliefs, intentions, and environmental factors, we developed a theoretical model describing the relationship between a menthol ban and tobacco use behavior ( Figure 2 ). 14

Literature flow diagram. *Additional records were identified from Tobacco Regulatory Science, a journal that was not indexed in an electronic database at the time of the initial search.

Literature flow diagram. *Additional records were identified from Tobacco Regulatory Science , a journal that was not indexed in an electronic database at the time of the initial search.

Theoretical model describing the relationship between a menthol cigarette ban and tobacco use behavior. The model was informed by studies identified in the review and health behavior theories that propose health behavior is determined, in part, by attitudes and beliefs, intentions, and environmental factors. The model indicates that a menthol ban impacts tobacco use behavior and sales by affecting individuals’ attitudes and beliefs and quit/use intentions. In addition, retailer and consumer compliance and the tobacco industry’s response to a menthol ban moderate the relationship between a menthol ban and individuals’ attitudes and beliefs. For example, studies identified in the review suggest the tobacco industry interferes with menthol bans by introducing new replacement products for menthol smokers to the market. Studies about attitudes and beliefs about menthol bans were not reviewed or summarized in this review.

Theoretical model describing the relationship between a menthol cigarette ban and tobacco use behavior. The model was informed by studies identified in the review and health behavior theories that propose health behavior is determined, in part, by attitudes and beliefs, intentions, and environmental factors. The model indicates that a menthol ban impacts tobacco use behavior and sales by affecting individuals’ attitudes and beliefs and quit/use intentions. In addition, retailer and consumer compliance and the tobacco industry’s response to a menthol ban moderate the relationship between a menthol ban and individuals’ attitudes and beliefs. For example, studies identified in the review suggest the tobacco industry interferes with menthol bans by introducing new replacement products for menthol smokers to the market. Studies about attitudes and beliefs about menthol bans were not reviewed or summarized in this review.

Tobacco Use Behaviors

Quitting and reducing consumption.

Compared to non-menthol cigarette smokers, menthol cigarette smokers quit smoking at higher rates after a menthol ban.

Canada . Seven studies assessed quitting among menthol cigarette smokers after menthol bans in Canada. 15–21 In a longitudinal study, Chaiton et al. 15 assessed smoking behavior 1 month after the 2017 ban in Ontario among a sample ( N  = 325) of smokers who had used at least one menthol cigarette in the past year. 15 In October 2017 menthol cigarettes and most flavored cigars were banned nationally. 22 Flavored e-cigarettes were excluded from the bans. Chaiton et al. 15 found that 29.1% of pre-ban menthol smokers reported they attempted to quit, whereas only 14.5% of respondents believed they would attempt to quit prior to the ban. Twelve percent of pre-ban menthol smokers were not smoking at 1-month follow-up. 15 In a cross-sectional study of lower quality, 3 months after policy implementation in Ontario, Soule et al. 16 surveyed past-year menthol smokers ( N  = 67) and found the majority (76.1%) reported using cigarettes most days or every day; 7.5% reported using cigarettes (menthol or non-menthol) “not at all.” Among pre-ban menthol smokers who had quit post-ban, 30.7% of smokers reported the menthol ban helped with smoking cessation. 18 Pooling data from a cohort study 17 of smokers in Ontario and another cohort study of smokers in provinces across Canada, 19 Fong et al. 20 found that 22.3% of pre-ban menthol smokers successfully quit when surveyed approximately 1 to 2 years post-ban. 20 Only 15.0% of non-menthol smokers successfully quit, indicating that a menthol ban was associated with a net cessation effect size of 7.3 percentage points. 20 Another study following the Ontario cohort 2 years post-ban found similar effect sizes. 21 Twelve percent of daily and 10% of occasional menthol smokers reported having quit smoking, as compared to 3% of non-menthol smokers. 21

European Union . Two studies examined tobacco use behavior among smokers in the Netherlands and England after the menthol ban in the European Union (EU). 23 , 24 In May 2020 the EU banned menthol cigarettes. 6 , 25 Flavored cigarillos, cigars, smokeless tobacco products, and e-cigarettes were exempted from the ban. 6 , 25 In a cohort study of adult smokers living in the Netherlands, 26.1% of menthol cigarette smokers quit approximately 1 year after the EU menthol ban was implemented, as compared to 14.1% of non-menthol cigarette smokers (% difference = 12.0, p  = .002). 23 In a repeated cross-sectional survey of youth in England, 12.1% of youth reported smoking a menthol (including capsule) cigarette brand prior to the menthol ban in February 2020. 24 Approximately 3 months after the ban, 3% of youth reported smoking a menthol cigarette brand. 24

United States . In a small longitudinal study of pre-ban menthol cigarette smokers in Massachusetts ( N  = 14), 50% (7 out of 14) reported making a quit attempt because of the 2020 flavored tobacco products ban in the state, and two reported successfully quitting 6 months post-ban. 26 The Massachusetts policy bans the sale of menthol cigarettes, other flavored tobacco products (eg, flavored cigars), and flavored e-cigarettes in most retailers. 27 The use of flavored tobacco products is restricted to smoking bars for onsite consumption only. 27 A San Francisco, California, study found only one of 20 (5%) pre-ban menthol smokers quit approximately 1 year after the comprehensive flavored tobacco products ban on menthol cigarettes, e-cigarettes, and other flavored tobacco products in the city. 28

A cross-sectional study compared trends in tobacco use among youth in Minneapolis and St. Paul, Minnesota, where menthol bans were implemented in 2018, to trends in the rest of the state that did not have a menthol ban. 29 Cigarette use declined at a faster rate from 2016 (pre-policy) to 2019 (post-policy) in Minneapolis and St. Paul as compared to the rest of the state. 29 In a qualitative study conducted in Minneapolis and Duluth, Minnesota, youth reported that the impact of the menthol bans may have been limited because many youth regularly travel to adjacent cities without bans. 30 The local policies in Minnesota ban sales of fruit-, candy-, and menthol-flavored tobacco products but have exemptions for adult-only tobacco stores and liquor stores in Minneapolis and St. Paul and exemptions for adult-only tobacco shops in Duluth. 29 , 30

Hypothesized behavior . Ten studies assessed hypothesized smoking behaviors in the event of a menthol ban in the US. Across studies, between 7% and 64.6% of menthol smokers reported they would quit or try to quit smoking. 31–40 Smokers who identified as Black, female, had less than a high school education, smoked less than a pack per day, were not a daily smoker, did not smoke soon after waking, and had current intentions to quit were more likely to express intentions to quit after a menthol ban. 33–36 , 41 One study 36 found that being older was associated with greater intentions to quit after a potential ban, while another study 41 found the opposite. In a qualitative study conducted among young adult (18–24 years) menthol smokers in New Jersey, Wackowski et al. 42 found that, despite not supporting a menthol ban, participants reported that a ban would help them quit smoking.

Quasi-experimental studies and other studies that assessed hypothesized behavior suggest that banning menthol cigarettes may also encourage smokers to reduce the number of cigarettes smoked per day 2 , 43–49 and support smoking cessation by reducing immediate relapse after a cessation attempt. 50 For example, to simulate the effect of a menthol ban, Bold et al. 43 examined changes in tobacco use behavior when study participants who used menthol cigarettes were switched to non-menthol cigarettes for 2 weeks. After switching to non-menthol cigarettes, participants on average smoked 2.2 fewer cigarettes per day and reported lower nicotine dependence and cravings. One of the studies that found a menthol ban would encourage smokers to reduce the number of cigarettes smoked was of lower quality because only an abstract with limited study information was published. 46

Three studies from the same research group used simulation modeling to estimate the potential effects of a menthol ban in the US. 51–53 If a menthol ban were implemented in 2021, overall smoking prevalence was estimated to decline by 16% within 5 years post-ban. 52 Among non-Hispanic Black adults, smoking prevalence would decline by 25.3%. 53 An earlier study conducted by Levy et al. 51 simulated the impact of a menthol ban if implemented in 2011. These earlier predictions were more conservative. The model predicted a 4.8%–9.7% relative reduction in smoking prevalence in the US population 40 years post-ban. 51 The relative reduction among Black individuals was also higher than the general population in this study, at 9.1%–24.8%. 51

Levy et al. 54 also conducted an expert elicitation to estimate the impact of a menthol ban on tobacco use in the US. Expert elicitation is a process to integrate knowledge among experts to estimate unknown parameters. 54 Experts hypothesized that 2 years after a menthol ban, compared to pre-ban rates, combustible tobacco product use would decline by 20% among menthol cigarette users 35–54 years old. Among those no longer using combustible tobacco products post ban, half were expected to quit smoking and half would switch to non-combustible products such as e-cigarettes. 54 Young adult (18–24) menthol cigarette smokers were expected to reduce combustible tobacco product use by 30% in the setting of a ban. 54

Zeng et al. 55 simulated the impact of a menthol ban in Singapore. They estimated that 50 years after a menthol ban smoking prevalence would decrease by 2.1 percentage points. In the status quo scenario with no menthol ban, smoking prevalence was expected to increase from 12.7% in 2018 to 15.2% in 2068. 55

Switching to Non-menthol Cigarettes

Among pre-ban menthol cigarette smokers, rates of switching to non-menthol cigarettes were higher in settings of national menthol bans in Canada and the Netherlands as compared to switching rates found in the setting of a statewide menthol ban in the US. 15 , 19 , 23 , 26

Canada . In a longitudinal study in Ontario, 1 month after ban implementation in the province, 28.2% of menthol smokers had switched to non-menthol cigarettes compared to 59.7% who hypothesized they would switch prior to the ban. 15 In a separate study following a Canadian cohort, 59.1% of pre-ban menthol cigarette smokers switched to non-menthol cigarettes 1 to 2 years post-ban. 19

European Union . In a longitudinal study in the Netherlands, 1 year after the EU menthol ban 40.0% of menthol cigarette smokers switched to non-menthol cigarettes. 23

United States . In a small longitudinal study examining the 2020 Massachusetts ban, two of 14 (14%) pre-ban menthol smokers reported starting to use non-menthol cigarettes 6 months post-ban. 26 Another study examined the impact of the 2018 comprehensive flavored tobacco products ban in San Francisco, California. 56 A difference-in-difference analysis was conducted using data from the cross-sectional Youth Risk Behavior Surveillance System survey. Contrary to expectations, the ban was associated with 2.24 [95% CI, 1.42, 3.53] higher odds of past 30-day cigarette smoking among high school students. 56 The authors reported that their results suggest youth substituted e-cigarettes with non-menthol cigarettes after the flavor ban. 56 Some researchers have noted that the results of this study are misleading. 57 , 58 Among other critiques, the flavor ban in San Francisco was not enforced at the time of data collection for the study. 58

Hypothesized behavior . In studies conducted in the US, 10.7%–53.6% of menthol cigarette smokers reported they would switch to non-menthol cigarettes in the event of a ban. 31 , 33–37 , 39–41 In a study of eight European countries, 20% of menthol cigarette smokers reported they would switch to a non-menthol brand. 2 In a study conducted in Brazil, 21.1% of menthol cigarette smokers reported they would switch to non-menthol cigarettes. 47

Switching to E-cigarettes and Other Flavored Tobacco Products

In the setting of a menthol ban, menthol cigarette smokers switch to e-cigarettes and other flavored tobacco products (OTPs; eg, flavored cigars) on the market. Studies suggest that a ban on menthol cigarettes without an accompanying ban on flavored e-cigarettes may increase e-cigarette use. 48 , 59–63

Canada . Three longitudinal studies by the same lead author assessed rates of e-cigarette and OTP use among pre-ban menthol cigarette smokers at different time periods after the ban. One month after implementation of the menthol cigarette ban in Ontario, Canada, 29.1% of menthol cigarette smokers had switched to flavored e-cigarettes and OTPs, whereas only 5.8% reported they would switch to flavored e-cigarettes or OTPs prior to the ban. 15 The menthol ban exempts flavored e-cigarettes and OTPs such as alcohol-flavored cigars without filters. 64 In the year after implementation of the ban in Ontario, 18% of daily menthol smokers reported using flavored tobacco products and 34% reported using e-cigarettes. 64 Approximately 2 years after the Ontario ban, 14.6% of pre-ban daily menthol smokers reported using additive cards, drops, or oil to add menthol flavoring. 65 Prior to the ban 4.4% of daily menthol smokers tried flavor additives. 65

United States . In a longitudinal study examining the 2020 flavored tobacco products ban in Massachusetts, one of 14 (7%) pre-ban menthol smokers reported starting to use e-cigarettes 6 months post-ban. 26

Hypothesized behavior . In two studies conducted in the US assessing hypothesized behavior, 8% and 22% of menthol cigarette smokers reported they would switch to OTPs and flavored e-cigarettes. 31 , 41 In another study, 12% of menthol cigarette smokers reported they would switch to flavored cigars. 32 An estimated 12.3%–25.6% of menthol cigarette smokers reported they would switch to e-cigarettes (flavored and/or non-flavored). 36 , 37 , 40 , 41 In a qualitative study of 35 menthol cigarette smokers in Rhode Island, the majority (23/35) reported they would begin using e-cigarettes at least some of the time. 39 Predictors of hypothesized product switching included greater nicotine dependence, 33 current use of OTPs, 35 and being White and male. 34

Continued Menthol Cigarette Use

Rates of continued menthol cigarette use were typically higher in settings of local or state menthol bans as compared to national menthol bans.

Canada . Three longitudinal studies from the same lead author examined the percentage of menthol cigarette smokers who continued using menthol cigarettes after the menthol ban in Ontario, Canada. In a sample of past-year menthol smokers, 14.1% reported using contraband menthol cigarettes (eg, purchasing cigarettes online or from another country) 1 month after the ban. 15 In another sample of menthol cigarette smokers in Ontario, 46.3% of participants reported using menthol cigarettes at least rarely 3 months after implementation of the ban. 16 In a separate sample surveyed at least a year after ban implementation, 22% of daily menthol smokers reported purchasing menthol cigarettes since the beginning of the ban. 17 Participants primarily reported purchasing on First Nations Reservations. 17 Stoklosa et al. 66 compared the number of illicit cigarettes seized by the Provincial Tax Commission in Nova Scotia, Canada, before and after the menthol ban and found no surge in illicit cigarettes. 66

European Union . In a cohort study of smokers in the Netherlands, 2.9% of menthol cigarette smokers reported using menthol cigarettes 1 year after the EU menthol ban. 23 In a cross-sectional study of smokers in England, 15.7% smoked menthol cigarettes between July 2020 and June 2021 after the EU ban (implemented in May 2020). 67

United States . In a longitudinal study examining the 2020 Massachusetts flavor ban, eight of 14 (57%) pre-ban menthol smokers reported continuing to smoke menthol cigarettes exclusively 6 months post-ban. 26 Most reported purchasing menthol cigarettes out of state. 26 A cross-sectional study of individuals in residential substance use disorder treatment in San Francisco, California, found 50% of menthol smokers reported recently purchasing menthol cigarettes in San Francisco approximately 1 year after the comprehensive flavor ban. 68 Another cross-sectional study found 70% of pre-ban menthol smokers continued using menthol cigarettes in San Francisco approximately 1 year post-ban. 28

Hypothesized behavior . In studies examining hypothesized behavior in the US, 24.1%–54.5% of menthol smokers reported they would find a way to buy a menthol brand or purchase from illicit sources. 32 , 39 , 40 In a qualitative study of 27 African American menthol cigarette smokers in Minneapolis and St. Paul, Minnesota, individuals reported they would find a way to obtain menthol cigarettes by asking friends to purchase them, traveling to other cities, or finding illegal sources. 69

In samples of European, Brazilian, and Canadian menthol smokers, 27%, 16.8%, and 11.1% reported they would find a way to get the banned product or use contraband menthol cigarettes, respectively. 2 , 15 , 47 Kulick et al. 70 modeled the unintended consequences of cigarette prohibition and suggested that a menthol ban would shift demand to illicit products. The study was contracted by Altria Client Services. 70

Canada . Three studies examined cigarette sales pre- and post-implementation of menthol cigarette bans in Canada. 22 , 71 , 72 Brown et al. 72 compared cigarette sales pre- (January–June 2016) and post-menthol ban (January–June 2017) in Ontario, Canada. Per capita sales of menthol cigarettes decreased by 93% in Ontario, compared with only a 2% decline in British Columbia, a comparison Canadian province without a ban. 72 The authors reported that substitution of menthol cigarettes with non-menthol cigarettes or other tobacco products appeared minimal. 72 Using wholesale cigarettes sales data from all 10 provinces in Canada between 2010 and 2018, Chaiton et al. 71 found that menthol cigarette sales increased from 2013 to 2017, prior to the nationwide ban implemented in October 2017. After the national ban, sales of menthol cigarettes fell to zero in all Canadian provinces. 22 , 71 Total cigarette sales declined by 4.6% as compared to the same month in the previous year. 22 , 71

United States . Six studies examined change in tobacco product sales associated with menthol bans in the US. 73–76 Four of these studies assessed sales in Massachusetts, 73–76 which implemented the first statewide menthol ban in the US in June 2020. One study assessed sales in San Francisco, California, 77 and another study assessed sales in St. Paul and Minneapolis, Minnesota. 78 The Massachusetts studies found that after ban implementation menthol cigarette sales decreased by more than 90% in the state compared to sales prior to the ban. 73–76 There was some evidence, including from one study of lower quality, that individuals were going to other states to purchase tobacco products. 73 , 75 However, overall, there was a net decline in total cigarette sales in Massachusetts and neighboring states. 79 In San Francisco, average weekly sales of menthol cigarettes declined by 96% from pre-policy to post-enforcement periods and at a higher rate than in comparison cities without menthol bans (11% decline in San Jose and 20% decline in San Diego). 77 In St. Paul and Minneapolis, weekly unit sales of menthol cigarettes decreased by 67.1% and 72.4% post-policy implementation, respectively, compared to approximately 2 years prior. These rates were higher than menthol cigarette sales reductions in the rest of the state and in the US. 78 The St. Paul and Minneapolis policies exempted some liquor stores and tobacco product shops, which may explain the lesser effect on sales as compared to the comprehensive menthol bans in Massachusetts and San Francisco. 77 , 78

European Union . Liber et al. 80 examined cigarette sales in Poland pre- and post-implementation of the 2020 EU menthol ban. Overall, there was no significant change in the sale of cigarettes (menthol and non-menthol) that could be attributed to the menthol ban. 80 Sales of menthol cigarettes declined by 97% from May 2018 to April 2021 in Poland but sales of non-flavored cigarettes increased by 38%. 80 In particular regions in Poland with relatively high menthol share of cigarettes before the ban (eg, 36.8% in Warsaw), there was a significant reduction in total cigarette sales. 80 On the other hand, in regions with below the nationwide average menthol share of cigarette sales pre-ban, there was no significant reduction in total cigarette sales. 80 Liber et al. 80 noted that the ban may have not had its intended impact in part because the tobacco industry released new alternative products for menthol smokers like cigarette pack inserts with a menthol flavor.

Retailer Compliance

Retailer compliance with menthol bans was evaluated in two Canadian provinces, Chicago, Illinois, and cities in Minnesota and California. See Table 1 for a summary of studies examining retailer compliance.

Summary of Studies Examining the Tobacco Industry’s Response to Menthol Cigarette Bans and Retailer Compliance

Canada . In province-wide menthol bans in Alberta and Nova Scotia, Canada, Brown et al. 83 purchased a sample of cigarette packs post-ban and found no packs labeled as “menthol.”

United States . Czaplicki et al. 90 evaluated retailer compliance in Chicago, Illinois, 1 year after implementation of their policy that restricted retailers within 500 feet of high schools from selling flavored tobacco products including menthol cigarettes. Only 57% of affected stores were compliant. Compared to larger/chain stores, gas stations had an 81% lower odds of compliance. 90 D’Silva et al. 91 found higher (97.5%–100%) compliance rates in a sample of retailers in Minneapolis, St. Paul, Duluth, and Falcon Heights, Minnesota, where sales of menthol cigarettes were restricted to adult-only tobacco shops and liquor stores. Two convenience stores in Minneapolis sampled added interior adult-only tobacco shops to circumvent the policy. 91

In three studies of California communities with menthol bans, 12.9%–35.4% of retailers sold menthol cigarettes or Newport menthol cigarettes, the most popular menthol cigarette brand. Pre-policy 87.9%–89.6% of stores sold menthol or Newport menthol cigarettes. 92–94 Vyas et al. 95 evaluated retailer compliance in San Francisco, California, before and after the San Francisco Department of Public Health began enforcing the policy with compliance inspections and penalties. Prior to enforcement 17% of retailers inspected were compliant with the ban. After enforcement began, 80% of retailers were compliant. 95

In Oakland, California, Kurti et al. 96 found that approximately half (46.0%) of discarded cigarette packs on streets and sidewalks were menthol 7 months after a menthol ban exempting adult-only tobacco shops went into effect. The authors concluded that a comprehensive ban is needed to reduce product availability. 96

Industry Response

See Table 1 for a summary of studies examining the industry’s response to menthol bans.

Canada . Borland et al. 81 conducted a content analysis of cigarette packs before and after a menthol ban in Ontario, Canada. Menthol descriptors were typically removed from “non-menthol alternative” packs post-ban. These packs did not have menthol but were promoted by tobacco companies as being the best replacement. Instead of using green as the prominent pack color, some replacement packs used blue as the prominent color and variant name (eg, LD Super Kings Blue) post-ban. 81 In addition, prior to the ban, new menthol products with prominent blue coloring emerged on the market and were cheaper than traditional menthol packs. 81 In a small study with lower-quality evidence, Schwartz et al. 82 also reported similar changes in products and packaging in Ontario. In Alberta, Canada, Brown et al. 83 found “menthol replacement” packs post-ban that used green as the prominent color, like menthol cigarette packs pre-ban. The majority (87.3%) of menthol replacement packs had a cellophane wrapper that stated “smooth taste without menthol.” 83

United States . Schroth et al. 97 discussed potential industry responses to a national menthol ban in the US. For example, Altria and Reynolds will likely challenge a menthol ban in court and claim that a ban on menthol cigarettes would increase illicit trade. 97 The authors report that scientific evidence supporting the ban will be sufficient for a court to uphold the policy. 97 In cities in Minnesota, the tobacco industry stated a menthol ban would have large negative financial impacts on tobacco retailers, including job loss, and a menthol ban would result in more policing of black men. 87 , 88 The industry disseminated messages opposing menthol bans by sending mailers to households, putting postcards about the ban at the point-of-sale and hosting forums. 87 , 88

European Union . In the United Kingdom, Hiscock et al. 84 and Branston et al. 85 observed that after the EU ban on menthol cigarettes was announced, the tobacco industry introduced new products including cigarillos with menthol capsules and menthol accessories that provide menthol flavor if inserted into cigarettes. Branston et al. 85 reported that to encourage retailers to stock a new cigarillo product with a menthol capsule, the industry offered a wholesale price that provided retailers a higher profit margin on the product. Similarly, Brink et al. 86 reported that tobacco companies introduced tobacco flavor accessories that could be inserted into cigarette packs, filters, and roll-your-own tobacco in Denmark. Tobacco companies included descriptors on packs that suggested the cigarette would have a menthol-like flavor, which the authors reported may violate the EU Tobacco Products Directive. 86 Prior to the EU ban in 2020, two new variants of cigarillos with menthol flavor were put on the market in Denmark, and cigarillo and cigar sales rose by 7% in 2019 and 2020. 86

Brazil . In Brazil, Oliveira da Silva et al. 89 reported that the industry used several strategies to delay ban implementation including political interference, litigation, and sponsoring research.

Findings from 16 studies were used for the meta-analysis ( Figures 3 and 4 ). Among studies of real-world behavior, we pooled estimates across three studies that assessed tobacco use behavior 1   to 2 years after national menthol bans. 17 , 19 , 23 We did not pool estimates from studies of local or state bans; there were not enough local or state-level studies identified in the review conducted around the same time period after a menthol ban to pool estimates. Studies assessing hypothesized behavior after a menthol ban asked about behavior broadly without specifying a particular time period and thus were pooled together without subdivision. One study of hypothesized behavior was excluded because it did not provide the information needed to calculate the standard error of the prevalence estimate. 36

Results from the meta-analysis of real-world studies examining the impact of menthol cigarette bans on tobacco use behavior among menthol cigarette smokers. 95% CI = 95% confidence interval. Studies examined tobacco use behavior among pre-ban menthol cigarette smokers 1 to 2 years after ban implementation. In the graphical display in the figure, each line represents a single study. The black diamond on the line symbolizes the point estimate of the effect. The width of the line extending through the black diamond shows the confidence interval for the point estimate. The unshaded diamond represents the pooled estimate.

Results from the meta-analysis of real-world studies examining the impact of menthol cigarette bans on tobacco use behavior among menthol cigarette smokers. 95% CI = 95% confidence interval. Studies examined tobacco use behavior among pre-ban menthol cigarette smokers 1 to 2 years after ban implementation. In the graphical display in the figure, each line represents a single study. The black diamond on the line symbolizes the point estimate of the effect. The width of the line extending through the black diamond shows the confidence interval for the point estimate. The unshaded diamond represents the pooled estimate.

Results from meta-analysis of studies examining the hypothesized impact of menthol cigarette bans on tobacco use behavior among menthol cigarette smokers. 95% CI = 95% confidence interval; E-cig = e-cigarettes; OTP = other flavored tobacco products (eg, flavored cigars). Studies examining hypothesized tobacco use behavior asked about behavior after a menthol ban without specifying a time period (eg, “If menthol cigarettes were no longer sold in US stores, would you quit smoking?”). In the graphical display in the figure, each line represents a single study. The black diamond on the line symbolizes the point estimate of the effect. The width of the line extending through the black diamond shows the confidence interval for the point estimate. The unshaded diamond represents the pooled estimate.

Results from meta-analysis of studies examining the hypothesized impact of menthol cigarette bans on tobacco use behavior among menthol cigarette smokers. 95% CI = 95% confidence interval; E-cig = e-cigarettes; OTP = other flavored tobacco products (eg, flavored cigars). Studies examining hypothesized tobacco use behavior asked about behavior after a menthol ban without specifying a time period (eg, “If menthol cigarettes were no longer sold in US stores, would you quit smoking?”). In the graphical display in the figure, each line represents a single study. The black diamond on the line symbolizes the point estimate of the effect. The width of the line extending through the black diamond shows the confidence interval for the point estimate. The unshaded diamond represents the pooled estimate.

Tobacco Use Behavior

Findings from the meta-analysis indicate that 24% (95% CI: 20%, 28%) of menthol cigarette smokers quit smoking approximately 1 to 2 years after a real-world menthol ban. Among studies examining hypothesized behavior, 33% (95% CI: 24%, 41%) of menthol smokers hypothesize they would quit or try to quit smoking.

Pooling across real-world studies, 50% (95% CI: 31%, 68%) of menthol smokers switched to non-menthol cigarettes approximately 1 to 2 years after a ban. Pooling across studies examining hypothesized behavior, 26% (95% CI: 18%, 34%) of menthol smokers hypothesize they would switch to non-menthol cigarettes after a ban.

Switching to Flavored E-cigarettes and Other Flavored Tobacco Products

There were not enough real-world studies to obtain a pooled estimate of the percentage of menthol smokers who switched to flavored e-cigarettes or OTPs post-ban. Among studies examining hypothesized behavior, 12% (95% CI: 3%, 20%) of menthol smokers hypothesize they would switch to flavored e-cigarettes or OTPs.

Pooling across real-world studies, 24% (95% CI: 17%, 31%) of menthol smokers reported continued use of menthol cigarettes after a menthol ban. Among studies examining hypothesized behavior, 28% (95% CI: 11%, 45%) of menthol smokers hypothesized continued use of menthol cigarettes.

There has been policy debate over menthol cigarette bans. 98 The Tobacco Products Scientific Advisory Committee, a Congressionally-mandated committee formed to advise the Food and Drug Administration, concluded that removing menthol cigarettes from the marketplace would benefit public health. 1 Critics of menthol bans, however, report that such policies will have little benefit because menthol smokers will not quit after a ban and instead switch to non-menthol cigarettes and OTPs. 98 Concern about an increase in illicit trade of cigarettes is another argument against a ban. 98 In contrast, findings from this review suggest that menthol cigarette smokers quit smoking at relatively high rates after a menthol ban. Research from Canada found no increase in the number of illicit cigarettes seized after the ban. 66

A prior review concluded that there was moderate evidence flavored tobacco sales bans decrease tobacco use prevalence and low-quality evidence that a flavored tobacco sales ban has intended effects on quit attempts and cessation. 8 At the time of the prior review, the authors identified only one study conducted on quitting behavior after a menthol ban. The study found only 1 of 20 pre-ban menthol smokers quit after the comprehensive flavor ban in San Francisco. 28 Unlike the prior review, which was limited to US studies, the present review also included studies examining menthol bans implemented internationally. In this review, compared to non-menthol cigarette smokers, quit rates among menthol smokers were 7 to 12 percentage-points higher 1 to 2 years after menthol bans in Canada and the Netherlands. 20 , 21 , 23 More studies conducted in US cities have been published since the prior review. Research finds cigarette use declined at a faster rate in US cities with menthol bans as compared to those without. 29 Studies examining sales found significant declines in menthol cigarette sales and total cigarette sales after a menthol ban. 22 , 71 , 73–76 As expected, rates of continued menthol cigarette use were lower in settings of national bans and highest in settings of local or statewide menthol bans. 26 , 28 , 68 Individuals can more easily access menthol cigarettes in settings of local menthol bans by traveling to nearby jurisdictions without a ban. The effects of a ban on tobacco use behavior were also less pronounced in settings with policies that exempted particular store types like liquor stores. 77 , 78 In Poland, there was no significant change in cigarette sales after the menthol ban. 80 This may have been due to the relatively low share of menthol cigarette sales in the country. 80

In the US, racial/ethnic minority and lower-income smokers use menthol cigarettes at higher rates than non-Hispanic White and higher-income smokers, respectively. 4 Therefore, a menthol ban may promote cessation disproportionately among racial/ethnic minority and lower-income groups. Using simulation modeling, Levy et al. 52 and Issabakhsh et al. 53 estimated a 16% reduction in smoking prevalence in the total US population within 5 years after a menthol ban and 25.3% reduction among non-Hispanic black adults. 52 , 53 A menthol ban appears to be a strong policy option to support equity-focused goals.

Studies in this review highlight the importance of promoting retailer compliance, implementing comprehensive flavored products bans and preparing for the tobacco industry’s response to undermine flavor bans. Policy enforcement coupled with retailer education may help increase compliance. 95 In addition, studies suggest a menthol ban that is not comprehensive may slightly increase e-cigarette and OTP use. 15 , 17 , 59 , 60 Presently, bans on the sale of menthol cigarettes vary across US localities, and some policies do not apply to e-cigarettes. 5 Also, policies should anticipate the industry’s response by prohibiting new products like menthol accessories for cigarettes, synthetic coolants, and regulating changes to packaging. 84 , 99

There are limitations to this review. Not all studies examining the impact of menthol bans may have been identified because our search strategy may not have identified all relevant articles. We included abstracts and grey literature that emerged from our search but did not conduct a separate review of industry or sales data. In addition, there was significant heterogeneity in many of the pooled estimates from the meta-analysis. This is, in part, a result of the limited number of studies available. To date, studies examining real-world menthol bans that could be pooled were limited to those conducted in Canada and the Netherlands. In addition, it is common for I 2 values to be high in meta-analyses of proportions. 100 Another study found a median I 2 of 96.9% in 134 meta-analyses of proportions. 100 Researchers note that high I 2 values do not necessarily mean that study estimates are inconsistent. 100 Considering the expected range of estimates is recommended. 100 Also, the majority of studies evaluated in the meta-analysis were derived from studies of hypothesized behavior. Although rates of switching to non-menthol cigarettes in real-world studies were higher than studies of hypothesized behavior, estimates from real-world and hypothesized studies for the other tobacco use behavior outcomes examined were similar. Findings from hypothesized studies were also typically consistent with studies examining cigarette sales. In addition, studies in this review focused on pre-ban menthol cigarette smokers. Future studies should assess tobacco use behavior in the setting of menthol bans among other tobacco product users.

Menthol bans promote smoking cessation. This review provides a snapshot of what is known about the impacts of menthol bans. This literature is an evolving area as more localities and countries implement menthol bans, the tobacco industry responds to the changing regulatory environment with new products, and the longer-term impacts of menthol bans emerge.

Research reported in this publication was supported by the National Cancer Institute and FDA Center for Tobacco Products (K01CA242530). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the Food and Drug Administration.

KMR has served as a paid expert consultant representing plaintiffs in litigation against e-cigarette and tobacco companies. All other authors have no conflicts of interest.

Sarah Mills (Conceptualization [lead], Data curation [lead], Formal analysis [lead], Funding acquisition [lead], Project administration [lead], Supervision [lead], Writing—original draft [lead], Writing—review & editing [lead]), Snigdha Peddireddy (Formal analysis [supporting], Writing—original draft [supporting], Writing—review & editing [supporting]), Rachel Kurtzman (Formal analysis [supporting], Writing—original draft [supporting], Writing—review & editing [supporting]), Frantasia Hill (Formal analysis [supporting], Writing—original draft [supporting], Writing—review & editing [supporting]), Victor Catalan (Formal analysis [supporting], Writing—original draft [supporting], Writing—review & editing [supporting]), Jennifer Bissram (Data curation [equal], Methodology [supporting]), and Kurt Ribisl (Conceptualization [supporting], Funding acquisition [supporting], Supervision [supporting], Writing—review & editing [supporting])

Tobacco Products Scientific Advisory Committee . Menthol cigarettes and public health: review of the scientific evidence and recommendations . 2011 . https://wayback.archive-it.org/7993/20170404143901/ https://www.fda.gov/AdvisoryCommittees/CommitteesMeetingMaterials/TobaccoProductsScientificAdvisoryCommittee/ucm237359.htm . Accessed January 4, 2023 .

Zatoński   M , Herbeć   A , Zatoński   W , et al.  ; EUREST-PLUS consortium . Characterising smokers of menthol and flavoured cigarettes, their attitudes towards tobacco regulation, and the anticipated impact of the Tobacco Products Directive on their smoking and quitting behaviours: the EUREST-PLUS ITC Europe Surveys . Tob Ind Dis.   2018 ; 16 : A4 – A4 .

Google Scholar

Kaai   SC , Ong’ang’o   J , Craig   L , et al.  . Prevalence, perceptions and predictors of menthol cigarettes among African smokers: findings from the ITC Kenya and Zambia Surveys . Tob Induc Dis.   2018 ; 16 ( 1 ): A487 .

Goodwin   RD , Ganz   O , Weinberger   AH , et al.  . Menthol cigarette use among adults who smoke cigarettes, 2008–2020: rapid growth and widening inequities in the United States . Nicotine Tob Res.   2022 ; 25 ( 4 ): ntac214 .

Campaign for Tobacco-Free Kids . States and localities that have restricted the sale of flavored tobacco products . 2022 . https://www.tobaccofreekids.org/assets/factsheets/0398.pdf . Accessed January 4, 2023 .

Erinoso   OA-O , Clegg Smith   KA-O , Iacobelli   M , et al.    Global review of tobacco product flavour policies . Tob Control.   2020 ; 30 ( 4 ): 373 – 379 .

Cadham   CJ , Sanchez-Romero   LM , Fleischer   NL , et al.  . The actual and anticipated effects of a menthol cigarette ban: a scoping review . BMC Public Health.   2020 ; 20 ( 1 ): 1055 .

Rogers   T , Brown   EM , Siegel-Reamer   L , et al.  . A comprehensive qualitative review of studies evaluating the impact of local US laws restricting the sale of flavored and menthol tobacco products . Nicotine Tob Res.   2022 ; 24 ( 4 ): 433 – 443 .

Paul   J , Barari   M.   Meta-analysis and traditional systematic literature reviews—what, why, when, where, and how ? Psychol Mark.   2022 ; 39 ( 6 ): 1099 – 1115 .

Page   MJ , McKenzie   JE , Bossuyt   PM , et al.  . The PRISMA 2020 statement: an updated guideline for reporting systematic reviews . Int J Surg . 2021 ; 88 : 105906 .

Covidence . Covidence systematic review software . www.covidence.org . Accessed January 4, 2023 .

Joy   SM , Little   E , Maruthur   NM , Purnell   TS , Bridges   JFP.   Patient preferences for the treatment of type 2 diabetes: a scoping review . PharmacoEcon.   2013 ; 31 ( 10 ): 877 – 892 .

National Heart L, and Blood Institute . Study quality assessment tools . https://www.nhlbi.nih.gov/health-topics/study-quality-assessment-tools . Accessed December 15, 2020 .

Glanz   K , Rimer   BK , Viswanath   K.   Health Behavior: Theory, Research, and Practice . San Francisco, CA : John Wiley & Sons ; 2015 .

Google Preview

Chaiton   M , Schwartz   R , Cohen   JE , Soule   E , Eissenberg   T.   Association of Ontario’s ban on menthol cigarettes with smoking behavior 1 month after implementation . JAMA Intern Med.   2018 ; 178 ( 5 ): 710 – 711 .

Soule   EK , Chaiton   M , Zhang   B , et al.  . Menthol cigarette smoker reactions to an implemented menthol cigarette ban . Tobacco Reg Sci.   2019 ; 5 ( 1 ): 50 – 64 .

Chaiton   MO , Nicolau   I , Schwartz   R , et al.  . Ban on menthol-flavoured tobacco products predicts cigarette cessation at 1 year: a population cohort study . Tob Control.   2020 ; 29 ( 3 ): 341 – 347 .

Soule   EK , Dubray   J , Cohen   JE , Schwartz   R , Chaiton   M.   Smoking cessation strategies used by former menthol cigarette smokers after a menthol ban . Addict Behav.   2021 ; 123 : 107046 .

Chung-Hall   J , Fong   GT , Meng   G , et al.  . Evaluating the impact of menthol cigarette bans on cessation and smoking behaviours in Canada: longitudinal findings from the Canadian arm of the 2016–2018 ITC four country smoking and vaping surveys . Tob Control.   2022 ; 31 ( 4 ): 556 – 563 .

Fong   GT , Chung-Hall   J , Meng   G , et al.  . Impact of Canada’s menthol cigarette ban on quitting among menthol smokers: pooled analysis of pre–post evaluation from the ITC Project and the Ontario Menthol Ban Study and projections of impact in the USA . Tob Control.   2022 ; 32 ( 6 ). doi: 10.1136/tobaccocontrol-2021-057227 .

Chaiton   M , Schwartz   R , Cohen   JE , et al.  . Prior daily menthol smokers more likely to quit 2 years after a menthol ban than non-menthol smokers: a population cohort study . Nicotine Tob Res.   2021 ; 23 ( 9 ): 1584 – 1589 .

Chaiton   M , Schwartz   R , Shuldiner   J , Tremblay   G , Nugent   R.   Evaluating a real world ban on menthol cigarettes: an interrupted time-series analysis of sales . Nicotine Tob Res.   2020 ; 22 ( 4 ): 576 – 579 .

Kyriakos   CN , Driezen   P , Fong   G , et al.  . Impact of the European Union’s menthol cigarette ban on smoking cessation outcomes: longitudinal findings from the 2020–2021 ITC Netherlands Surveys . Tob Control.   2022 . doi: 10.1136/tc-2022-057428 .

East   KA , Reid   JL , Burkhalter   R , et al.  . Evaluating the outcomes of the menthol cigarette ban in England by comparing menthol cigarette smoking among youth in England, Canada, and the US, 2018-2020 . JAMA Netw Open.   2022 ; 5 ( 5 ): e2210029 – e2210e29 .

Constantine   IV.   European Tobacco Products Directive (TPD): current impact and future steps . Tob Control.   2022 ; 31 ( 2 ): 198 .

Booras   A , Wiener   R , Maccarone   J , et al.  . Impact of the Massachusetts menthol ban on perceptions and cigarette use behavior at a large safety-net hospital: a longitudinal survey and qualitative study . 2022 . https://assets.researchsquare.com/files/rs-2078170/v1/d7be9801-0fcd-4b18-8876-084bb7a5a96c.pdf?c=1667329163

Commonwealth of Massachusetts . 2019 Tobacco Control Law . https://www.mass.gov/guides/2019-tobacco-control-law . Accessed September 29, 2023 .

Yang   Y , Lindblom   EN , Salloum   RG , Ward   KD.   The impact of a comprehensive tobacco product flavor ban in San Francisco among young adults . Addict Behav Rep.   2020 ; 11 : 100273 .

Olson   LT , Coats   EM , Rogers   T , et al.  . Youth tobacco use before and after local sales restrictions on flavored and menthol tobacco products in Minnesota . J Adolesc Health.   2022 ; 70 ( 6 ): 978 – 984 .

Moze   JM , O’Gara   EK , Fritz Fogel   K , et al.  . Youth perspectives on menthol sales restrictions in Minneapolis and Duluth, Minnesota . Health Promot Pract.   2022 ; 24 ( 6 ): 1163 – 1173 .

Guillory   J , Kim   AE , Nonnemaker   JM , et al.  . Effect of menthol cigarette and other menthol tobacco product bans on tobacco purchases in the RTI iShoppe virtual convenience store . Tob Control.   2020 ; 29 ( 4 ): 452 – 459 .

O’Connor   RJ , Bansal-Travers   M , Carter   LP , Cummings   KM.   What would menthol smokers do if menthol in cigarettes were banned? Behavioral intentions and simulated demand . Addiction.   2012 ; 107 ( 7 ): 1330 – 1338 .

Pearson   JL , Abrams   DB , Niaura   RS , Richardson   A , Vallone   DM.   A ban on menthol cigarettes: impact on public opinion and smokers’ intention to quit . Am J Public Health.   2012 ; 102 ( 11 ): e107 – e114 .

Rose   SW , Ganz   O , Zhou   Y , et al.  . Longitudinal response to restrictions on menthol cigarettes among young adult US menthol smokers, 2011–2016 . Am J Public Health.   2019 ; 109 ( 10 ): 1400 – 1403 .

Wackowski   OA , Manderski   MTB , Delnevo   CD.   Young adults’ behavioral intentions surrounding a potential menthol cigarette ban . Nicotine Tob Res.   2014 ; 16 ( 6 ): 876 – 880 .

Wackowski   OA , Delnevo   CD , Pearson   JL.   Switching to e-cigarettes in the event of a menthol cigarette ban . Nicotine Tob Res.   2015 ; 17 ( 10 ): 1286 – 1287 .

D’Silva   J , Amato   MS , Boyle   RG.   Quitting and switching: menthol smokers’ reactions to a menthol ban . Tob Reg Sci.   2015 ; 1 ( 1 ): 54 – 60 .

Harrell   MB , Loukas   A , Jackson   CD , Marti   CN , Perry   CL.   Flavored tobacco product use among youth and young adults: what if flavors didn’t exist ? Tob Reg Sci.   2017 ; 3 ( 2 ): 168 – 173 .

Denlinger-Apte   RL , Lockhart   DE , Strahley   AE , et al.  . “I think it’s a good idea for the people that’s young, the kids, but for someone like me it’sa bad idea.”—interviews about a US menthol cigarette ban with people who smoke menthol cigarettes . Drug Alcohol Depend.   2022 ; 232 : 109293 .

Yang   Y , Lindblom   EN , Ward   KD , Salloum   RG.   How smokers of menthol cigarettes and flavored cigars might respond to FDA’s proposed bans . Nicotine Tob Res.   2022 ; 24 ( 10 ): 1645 – 1653 .

Dearfield   CT , Horn   K , Crandell   I , Bernat   DH.   Behavioural intentions in response to a potential menthol cigarette sales ban: a survey examining smokers in Washington, DC public housing . BMJ Open.   2022 ; 12 ( 7 ): e059821 .

Wackowski   OA , Evans   KR , Harrell   MB , et al.  . In their own words: young adults’ menthol cigarette initiation, perceptions, experiences and regulation perspectives . Nicotine Tob Res.   2018 ; 20 ( 9 ): 1076 – 1084 .

Bold   KW , Jatlow   P , Fucito   LM , et al.  . Evaluating the effect of switching to non-menthol cigarettes among current menthol smokers: an empirical study of a potential ban of characterising menthol flavour in cigarettes . Tob Control.   2020 ; 29 ( 6 ): 624 – 630 .

Kotlyar   M , Mills   AM , Shanley   R , et al.  . Smoker response to a simulation ban of menthol cigarettes: a pilot study . Tob Reg Sci.   2016 ; 1 ( 3 ): 236 – 242 .

Kotlyar   M , Shanley   R , Dufresne   SR , et al.  . Effects on smoking behavior of switching menthol smokers to non-menthol cigarettes . Nicotine Tob Res.   2021 ; 23 ( 11 ): 1921 – 1927 .

Edwards   R , Waa   A , Stanley   J , et al.  . Achieving the tobacco endgame: the case for removing additives: findings from the ITC New Zealand Survey . Tobacco Induc Dis.   2018 ; 16 ( 1 ): A40 .

Kyriakos   CN , Fong   GT , de Abreu Perez   C , et al.  . Brazilian smokers are ready for the ban on flavour additives in tobacco to be implemented . Prev Med.   2022 ; 160 : 107074 .

Kotlyar   M , Shanley   R , Dufresne   SR , Corcoran   GA , Hatsukami   DK.   Effect of restricting menthol flavored cigarettes or e-cigarettes on smoking behavior in menthol smokers . Prev Med.   2022 ; 165 ( Pt B ): 107243 .

Oncken   C , Litt   MD , Thurlow   S , et al.  . Manipulation of menthol and nicotine content in cigarettes: effects on smoking behavior and toxicant exposure in women menthol smokers . Nicotine Tob Res.   2022 ; 25 ( 4 ): 665 – 673 .

Kotlyar   M , Shanley   R , Dufresne   SR , et al.  . Effects on time to lapse of switching menthol smokers to non-menthol cigarettes prior to a cessation attempt: a pilot study . Tob Control.   2021 ; 30 ( 5 ): 574 – 577 .

Levy   DT , Pearson   JL , Villanti   AC , et al.  . Modeling the future effects of a menthol ban on smoking prevalence and smoking-attributable deaths in the United States . Am J Public Health.   2011 ; 101 ( 7 ): 1236 – 1240 .

Levy   DT , Meza   R , Yuan   Z , et al.  . Public health impact of a US ban on menthol in cigarettes and cigars: a simulation study . Tob Control.   2021 ; 32 ( e1 ): e37 – e44 .

Issabakhsh   M , Meza   R , Li   Y , et al.  . Public health impact of a US menthol cigarette ban on the non-Hispanic black population: a simulation study . Tob Control.   2022 ; 33 ( e1 ): e37 – e44 .

Levy   DT , Cadham   CJ , Sanchez-Romero   LM , et al.  . An expert elicitation on the effects of a ban on menthol cigarettes and cigars in the United States . Nicotine Tob Res.   2021 ; 23 ( 11 ): 1911 – 1920 .

Zeng   Z , Cook   AR , Chen   JIP , van der Eijk   Y.   Evaluating the public health impact of partial and full tobacco flavour bans: a simulation study . Lancet Reg Health West Pac.   2022 ; 21 : 100414 .

Friedman   AS.   A difference-in-differences analysis of youth smoking and a ban on sales of flavored tobacco products in San Francisco, California . JAMA Pediatr.   2021 ; 175 ( 8 ): 863 – 865 .

Maa   J , Gardiner   P.   Further considerations on the association between flavored tobacco legislation and high school student smoking rates . JAMA Pediatr.   2021 ; 175 ( 12 ): 1289 – 1290 .

Glantz   S . Study claiming SF flavor ban increases youth smoking dissolves; it is not based on any data collected after enforcing the ban . 2022 . https://profglantz.com/2022/03/21/study-claiming-sf-flavor-ban-increased-youth-smoking-dissolves-it-is-not-based-on-any-data-collected-after-enforcing-the-ban/ . Accessed April 27, 2023.

Buckell   J , Marti   J , Sindelar   JL.   Should flavours be banned in cigarettes and e-cigarettes? Evidence on adult smokers and recent quitters from a discrete choice experiment . Tob Control.   2019 ; 28 ( 2 ): 168 .

Pacek   LR , Oliver   JA , Sweitzer   MM , McClernon   FJ.   Young adult dual combusted cigarette and e-cigarette users’ anticipated responses to a nicotine reduction policy and menthol ban in combusted cigarettes . Drug Alcohol Depend.   2019 ; 194 : 40 – 44 .

Buckell   J , Fucito   LM , Krishnan-Sarin   S , O'Malley   S , Sindelar   JL.   Harm reduction for smokers with little to no quit interest: can tobacco policies encourage switching to e-cigarettes ? Tob Control.   2022 ; 32 ( e2 ): e173 – e179 .

Kotlyar   M , Shanley   R , Dufresne   SR , Corcoran   GA , Hatsukami   DK.   Effect on tobacco use and subjective measures of including e-cigarettes in a simulated ban of menthol in combustible cigarettes . Nicotine Tob Res.   2022 ; 24 ( 9 ): 1448 – 1457 .

Denlinger-Apte   RL , Cassidy   RN , Carey   KB , et al.  . The impact of menthol flavoring in combusted tobacco on alternative product purchasing: a pilot study using the Experimental Tobacco Marketplace . Drug Alcohol Depend.   2021 ; 218 : 108390 .

Chaiton   M , Papadhima   I , Schwarz   R , et al.  . Product substitution after a real world menthol ban: a cohort Study . Tob Reg Sci.   2020 ; 6 ( 3 ): 205 – 212 .

Chaiton   MO , Schwartz   R , Cohen   JE , et al.  . The use of flavour cards and other additives after a menthol ban in Canada . Tob Control.   2021 ; 30 ( 5 ): 601 – 602 .

Stoklosa   M.   No surge in illicit cigarettes after implementation of menthol ban in Nova Scotia . Tob Control.   2019 ; 28 ( 6 ): 702 – 704 .

Kock   L , Shahab   L , Bogdanovica   I , Brown   J.   Profile of menthol cigarette smokers in the months following the removal of these products from the market: a cross-sectional population survey in England . Tob Control.   2021 ; 32 ( e1 ): e121 – e124 .

Guydish   JR , Straus   ER , Le   T , Gubner   N , Delucchi   KL.   Menthol cigarette use in substance use disorder treatment before and after implementation of a county-wide flavoured tobacco ban . Tob Control.   2021 ; 30 ( 6 ): 616 – 622 .

D’Silva   J , O’Gara   E , Fryer   CS , Boyle   RG.   “Because there’s just something about that menthol”: exploring African American smokers’ perspectives on menthol smoking and local menthol sales restrictions . Nicotine Tob Res.   2021 ; 23 ( 2 ): 357 – 363 .

Kulick   J , Prieger   J , Kleiman   MAR.   Unintended consequences of cigarette prohibition, regulation, and taxation . Int J Law Crime Justice   2016 ; 46 : 69 – 85 .

Chaiton   M , Schwartz   R , Kundu   A , Houston   C , Nugent   R.   Analysis of wholesale cigarette sales in Canada after menthol cigarette bans . JAMA Netw Open.   2021 ; 4 ( 11 ): e2133673 – e2133e73 .

Brown   EM , Gammon   DG , Rogers   T , et al.  . Changes in retail sales of tobacco products in Ontario after a menthol sales restriction . Tob Control.   2021 ; 31 ( e2 ): e134 – e139 .

Kingsley   M , McGinnes   H , Song   G , Doane   J , Henley   P.   Impact of Massachusetts’ statewide sales restriction on flavored and menthol tobacco products on tobacco sales in Massachusetts and surrounding states . Am J Public Health.   2022 ; 112 ( 8 ): 1147 – 1150 .

Asare   S , Majmundar   A , Westmaas   JL , et al.  . Association of cigarette sales with comprehensive menthol flavor ban in Massachusetts . JAMA Intern Med.   2022 ; 182 ( 2 ): 231 – 234 .

Rich   JJ.   Estimates of cross-border menthol cigarette sales following the comprehensive tobacco flavor ban in Massachusetts . medRxiv.   2022 .

Satchell   T , Diaz   MC , Stephens   D , et al.  . The impact of two state-level approaches to restricting the sale of flavored tobacco products . BMC Public Health.   2022 ; 22 ( 1 ): 1799 .

Gammon   DG , Rogers   T , Gaber   J , et al.  . Implementation of a comprehensive flavoured tobacco product sales restriction and retail tobacco sales . Tob Control.   2021 ; 31 . doi: 10.1136/tobaccocontrol-2021-056494 .

Olson   LT , Gammon   DG , Rogers   T , et al.  . Expanding local sales restrictions on flavoured tobacco products to include menthol: retail sales changes in two Minnesota cities . Tob Control.   2022 . doi: 10.1136/tc-2022-057392 .

Asare   S , Majmundar   A , Westmaas   JL , et al.  . Spatial analysis of changes in cigarette sales in Massachusetts and bordering states following the Massachusetts menthol flavor ban . JAMA Netw Open.   2022 ; 5 ( 9 ): e2232103 – e2232e03 .

Liber   AC , Stoklosa   M , Levy   DT , et al.  . An analysis of cigarette sales during Poland’s menthol cigarette sales ban: small effects with large policy implications . Eur J Public Health.   2022 ; 32 ( 5 ): 735 – 740 .

Borland   T , D’Souza   SA , O’Connor   S , Chaiton   MO , Schwartz   R.   Is blue the new green? Repackaging menthol cigarettes in response to a flavour ban in Ontario, Canada . Tob Control.   2019 ; 28 ( e1 ): e7 – e12 .

Schwartz   R , Chaiton   M , Borland   T , Diemert   L.   Tobacco industry tactics in preparing for menthol ban . Tob Control.   2018 ; 27 ( 5 ): 577 .

Brown   J , DeAtley   T , Welding   K , et al.  . Tobacco industry response to menthol cigarette bans in Alberta and Nova Scotia, Canada . Tob Control.   2017 ; 26 ( e1 ): e71 – e74 .

Hiscock   R , Silver   K , Zatoński   M , Gilmore   AB.   Tobacco industry tactics to circumvent and undermine the menthol cigarette ban in the UK . Tob Control.   2020 ; 29 ( e1 ): e138 – e142 .

Branston   JR , Hiscock   R , Silver   K , Arnott   D , Gilmore   AB.   Cigarette-like cigarillo introduced to bypass taxation, standardised packaging, minimum pack sizes, and menthol ban in the UK . Tob Control.   2021 ; 30 ( 6 ): 708 – 711 .

Brink   A-L , Glahn   AS , Kjaer   NT.   Tobacco companies’ exploitation of loopholes in the EU ban on menthol cigarettes: a case study from Denmark . Tob Control.   2022 ; 32 ( 6 ): 809 – 812 .

Ackert   K , Brock   B , Friedrichsen   SC , et al.  . Countering tobacco industry tactics on the economic costs of restricting menthol tobacco . Tob Control.   2020 ; 29 ( e1 ): e113 – e114 .

Bosma   LM , D’Silva   J , Moze   J , et al.  . Restricting sales of menthol tobacco products: lessons learned from policy passage and implementation in Minneapolis, St. Paul, and Duluth, Minnesota . Health Equity.   2021 ; 5 ( 1 ): 439 – 447 .

Oliveira da Silva   AL , Bialous   SA , Albertassi   PGD , et al.  . The taste of smoke: tobacco industry strategies to prevent the prohibition of additives in tobacco products in Brazil . Tob Control.   2019 ; 28 ( e2 ): e92 – e101 .

Czaplicki   L , Cohen   JE , Jones   MR , et al.  . Compliance with the City of Chicago’s partial ban on menthol cigarette sales . Tob Control.   2019 ; 28 ( 2 ): 161 – 167 .

D’Silva   J , Moze   J , Kingsbury   JH , et al.  . Local sales restrictions significantly reduce the availability of menthol tobacco: findings from four Minnesota cities . Tob Control.   2021 ; 30 ( 5 ): 492 – 497 .

Holmes   LM , Lempert   LK , Ling   PM.   Flavored tobacco sales restrictions reduce tobacco product availability and retailer advertising . Int J Environ Res Public Health.   2022 ; 19 ( 6 ): 3455 .

Welwean   RA , Andersen-Rodgers   E , Akintunde   A , Zhang   X.   Evaluating the impact of strong and weak California flavored tobacco sales restriction policies on the tobacco retail environment . Am J Health Promot.   2022 ; 36 ( 4 ): 687 – 696 .

Andersen-Rodgers   E , Zhang   X , Vuong   TD , et al.  . Are California’s local flavored tobacco sales restrictions effective in reducing the retail availability of flavored tobacco products? A multicomponent evaluation . Eval Rev.   2021 ; 45 ( 3-4 ): 134 – 165 .

Vyas   P , Ling   P , Gordon   B , et al.  . Compliance with San Francisco’s flavoured tobacco sales prohibition . Tob Control.   2020 ; 30 ( 2 ): 227 – 230 .

Kurti   MK , Schroth   KRJ , Ackerman   C , et al.  . Availability of menthol cigarettes in Oakland, California after a partial flavor ban . Prev Med Rep.   2020 ; 20 : 101200 .

Schroth   KRJ , Villanti   AC , Kurti   M , Delnevo   CD.   Why an FDA ban on menthol is likely to survive a tobacco industry lawsuit . Public Health Rep.   2019 ; 134 ( 3 ): 300 – 306 .

Ducharme   J.   The FDA is moving forward with a menthol cigarette ban. Here’s what the science says . 2022 . https://time.com/6173070/menthol-cigarette-ban-studies/ . Accessed April 27, 2023 .

Michelle   KP , Emily   EP , Noel   JL , et al.  . Still “Cool”: tobacco industry responds to state-wide menthol ban with synthetic coolants . Tob Control.   2023 . doi: 10.1136/tc-2023-058149 .

Migliavaca   CB , Stein   C , Colpani   V , et al.  ; Prevalence Estimates Reviews-Systematic Review Methodology Group (PERSyst) . Meta‐analysis of prevalence: I 2 statistic and how to deal with heterogeneity . Res Synth Methods.   2022 ; 13 ( 3 ): 363 – 367 .

  • tobacco use
  • menthol cigarettes
  • flavored tobacco products

Supplementary data

Email alerts, citing articles via.

  • About Nicotine & Tobacco Research
  • Recommend to your Library

Affiliations

  • Online ISSN 1469-994X
  • Copyright © 2024 Society for Research on Nicotine and Tobacco
  • About Oxford Academic
  • Publish journals with us
  • University press partners
  • What we publish
  • New features  
  • Open access
  • Institutional account management
  • Rights and permissions
  • Get help with access
  • Accessibility
  • Advertising
  • Media enquiries
  • Oxford University Press
  • Oxford Languages
  • University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

  • Copyright © 2024 Oxford University Press
  • Cookie settings
  • Cookie policy
  • Privacy policy
  • Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

journal analysis research paper

International Journal of Data Science and Analytics

  • Focuses on fundamental and applied research outcomes in data and analytics theories, technologies and applications.
  • Promotes new scientific and technological approaches for strategic value creation in data-rich applications.
  • Encourages transdisciplinary and cross-domain collaborations.
  • Strives to bring together researchers, industry practitioners, and potential users of data science and analytics.
  • Addresses challenges ranging from data capture, creation, storage, retrieval, sharing, analysis, optimization, and visualization.

journal analysis research paper

Latest issue

Volume 17, Issue 2

Latest articles

Genetic algorithm-based hybrid deep learning model for explainable alzheimer’s disease prediction using temporal multimodal cognitive data.

  • Hager Saleh
  • Nora ElRashidy
  • Shaker El-Sappagh

journal analysis research paper

Simultaneously feature selection and parameters optimization by teaching–learning and genetic algorithms for diagnosis of breast cancer

  • Alok Kumar Shukla

journal analysis research paper

An open science automatic workflow for multi-model species distribution estimation

  • Gianpaolo Coro
  • Lorenzo Sana
  • Pasquale Bove

journal analysis research paper

Efficient three-way SVM for three-class classification problems

  • Vivek Prakash Srivastava
  • Kapil Gupta

journal analysis research paper

Robust machine learning models: linear and nonlinear

  • Paolo Giudici
  • Emanuela Raffinetti
  • Marco Riani

journal analysis research paper

Journal updates

Cfp: theoretical and practical data science and analytics .

Submission Deadline: 15 April 2024

Guest Editor: Fragkiskos Malliaros

CfP: Innovative Hardware and Architectures for Ubiquitous Data Science

Submission Deadline: 10 September 2023

Guest Editors: Dr. Faheem Khan, Dr. Umme Laila, Dr. Muhammad Adnan Khan.

CfP: CCF BigData conference Journal Track on ‘Data Science in China’

Cfp: learning from temporal data.

Submission Deadline: 17 November 2023

Guest Editors: João Mendes-Moreira, Joydeep Chandra, Albert Bifet

Journal information

  • EI Compendex
  • Emerging Sources Citation Index
  • Google Scholar
  • Japanese Science and Technology Agency (JST)
  • OCLC WorldCat Discovery Service
  • TD Net Discovery Service
  • UGC-CARE List (India)

Rights and permissions

Springer policies

© Springer Nature Switzerland AG

  • Find a journal
  • Publish with us
  • Track your research

Fake academic papers are on the rise: why they’re a danger and how to stop them

journal analysis research paper

Professor of Methodology and Integrity, Vrije Universiteit Amsterdam

Disclosure statement

Lex Bouter is the founding chair of the World Conferences on Research Integrity Foundation and co-chair of the 8th WCRI in Athens, 2-5 June 2024.

View all partners

An illustration of a magnifying glass poised over two wooden discs. Fake is written on one; real on the other

In the 1800s, British colonists in India set about trying to reduce the cobra population, which was making life and trade very difficult in Delhi. They began to pay a bounty for dead cobras. The strategy very quickly resulted in the widespread breeding of cobras for cash .

This danger of unintended consequences is sometimes referred to as the “ cobra effect ”. It can also be well summed up by Goodhardt’s Law , named after British economist Charles Goodhart. He stated that, when a measure becomes a target, it ceases to be a good measure.

The cobra effect has taken root in the world of research. The “publish or perish” culture, which values publications and citations above all, has resulted in its own myriad of “cobra breeding programmes”. That includes the widespread practice of questionable research practices, like playing up the impact of research findings to make work more attractive to publishers.

It’s also led to the rise of paper mills, criminal organisations that sell academic authorship. A report on the subject describes paper mills as (the)

process by which manufactured manuscripts are submitted to a journal for a fee on behalf of researchers with the purpose of providing an easy publication for them, or to offer authorship for sale.

These fake papers have serious consequences for research and its impact on society. Not all fake papers are retracted. And even those that are often still make their way into systematic literature reviews which are, in turn, used to draw up policy guidelines, clinical guidelines, and funding agendas.

How paper mills work

Paper mills rely on the desperation of researchers — often young, often overworked, often on the peripheries of academia struggling to overcome the high obstacles to entry — to fuel their business model.

They are frighteningly successful. The website of one such company based in Latvia advertises the publication of more than 12,650 articles since its launch in 2012. In an analysis of just two journals jointly conducted by the Committee on Publications Ethics and the International Association of Scientific, Technical and Medical Publishers, more than half of the 3440 article submissions over a two-year period were found to be fake.

It is estimated that all journals, irrespective of discipline, experience a steeply rising number of fake paper submissions. Currently the rate is about 2%. That may sound small. But, given the large and growing amount of scholarly publications it means that a lot of fake papers are published. Each of these can seriously damage patients, society or nature when applied in practice.

The fight against fake papers

Many individuals and organisations are fighting back against paper mills.

The scientific community is lucky enough to have several “fake paper detectives” who volunteer their time to root out fake papers from the literature. Elizabeth Bik , for instance, is a Dutch microbiologist turned science integrity consultant. She dedicates much of her time to searching the biomedical literature for manipulated photographic images or plagiarised text. There are others doing this work , too.

Organisations such as PubPeer and Retraction Watch also play vital roles in flagging fake papers and pressuring publishers to retract them.

These and other initiatives, like the STM Integrity Hub and United2Act , in which publishers collaborate with other stakeholders, are trying to make a difference.

But this is a deeply ingrained problem. The use of generative artificial intelligence like ChatGPT will help the detectives – but will also likely result in more fake papers which are now more easy to produce and more difficult or even impossible to detect.

Stop paying for dead cobras

They key to changing this culture is a switch in researcher assessment.

Researchers must be acknowledged and rewarded for responsible research practices: a focus on transparency and accountability, high quality teaching, good supervision, and excellent peer review. This will extend the scope of activities that yield “career points” and shift the emphasis of assessment from quantity to quality.

Fortunately, several initiatives and strategies already exist to focus on a balanced set of performance indicators that matter. The San Francisco Declaration on Research Assessment , established in 2012, calls on the research community to recognise and reward various research outputs, beyond just publication. The Hong Kong Principles , formulated and endorsed at the 6th World Conference in Research Integrity in 2019, encourage research evaluations that incentivise responsible research practices while minimise perverse incentives that drive practices like purchasing authorship or falsifying data.

These issues, as well as others related to protecting the integrity of research and building trust in it, will also be discussed during the 8th World Conference on Research Integrity in Athens, Greece in June this year.

Practices under the umbrella of “ Open Science ” will be pivotal to making the research process more transparent and researchers more accountable. Open Science is the umbrella term for a movement consisting of initiatives to make scholarly research more transparent and equitable, ranging from open access publication to citizen science.

Open Methods, for example, involves the pre-registration of a study design’s essential features before its start. A registered report containing the introduction and methods section is submitted to a journal before data collection starts. It is subsequently accepted or rejected based on the relevance of the research, as well as the methodology’s strength.

The added benefit of a registered report is that reviewer feedback on the methodology can still change the study methods, as the data collection hasn’t started. Research can then begin without pressure to achieve positive results, removing the incentive to tweak or falsify data.

Peer review

Peer reviewers are an important line of defence against the publication of fatally flawed or fake papers. In this system, quality assurance of a paper is done on a completely voluntary and often anonymous basis by an expert in the relevant field or subject.

However, the person doing the review work receives no credit or reward. It’s crucial that this sort of “invisible” work in academia be recognised, celebrated and included among the criteria for promotion. This can contribute substantially to detecting questionable research practices (or worse) before publication.

It will incentivise good peer review, so fewer suspect articles pass through the process, and it will also open more paths to success in academia – thus breaking up the toxic publish-or-perish culture.

This article is based on a presentation given by the lead author at Stellenbosch University, South Africa on 12 February 2024. Natalie Simon, a communications consultant specialising in research who is part of the communications team for the 8th World Conference on Research Integrity and is also currently completing an MPhil in Science and Technology Studies at Stellenbosch University, co-authored this article.

  • Academic journals
  • Research integrity
  • Academic research
  • Publish or perish
  • Fake journals
  • Paper mills
  • Open Science movement

journal analysis research paper

School of Social Sciences – Academic appointment opportunities

journal analysis research paper

Union Organiser (part-time 0.8)

journal analysis research paper

Director, Indigenous Education Research Centre

journal analysis research paper

Communications Specialist

journal analysis research paper

AUSTRALIAN PESTICIDES AND VETERINARY MEDICINES AUTHORITY (APVMA) BOARD CHAIR

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • View all journals
  • Explore content
  • About the journal
  • Publish with us
  • Sign up for alerts
  • 04 March 2024
  • Clarification 05 March 2024

Millions of research papers at risk of disappearing from the Internet

You can also search for this author in PubMed   Google Scholar

Old documents and books stored on shelves in a library's archive.

A study identified more than two million articles that did not appear in a major digital archive, despite having an active DOI. Credit: Anna Berkut/Alamy

More than one-quarter of scholarly articles are not being properly archived and preserved, a study of more than seven million digital publications suggests. The findings, published in the Journal of Librarianship and Scholarly Communication on 24 January 1 , indicate that systems to preserve papers online have failed to keep pace with the growth of research output.

“Our entire epistemology of science and research relies on the chain of footnotes,” explains author Martin Eve, a researcher in literature, technology and publishing at Birkbeck, University of London. “If you can’t verify what someone else has said at some other point, you’re just trusting to blind faith for artefacts that you can no longer read yourself.”

Eve, who is also involved in research and development at digital-infrastructure organization Crossref, checked whether 7,438,037 works labelled with digital object identifiers (DOIs) are held in archives. DOIs — which consist of a string of numbers, letters and symbols — are unique fingerprints used to identify and link to specific publications, such as scholarly articles and official reports. Crossref is the largest DOI registration agency, allocating the identifiers to about 20,000 members, including publishers, museums and other institutions.

The sample of DOIs included in the study was made up of a random selection of up to 1,000 registered to each member organization. Twenty-eight per cent of these works — more than two million articles — did not appear in a major digital archive, despite having an active DOI. Only 58% of the DOIs referenced works that had been stored in at least one archive. The other 14% were excluded from the study because they were published too recently, were not journal articles or did not have an identifiable source.

Preservation challenge

Eve notes that the study has limitations: namely that it tracked only articles with DOIs, and that it did not search every digital repository for articles (he did not check whether items with a DOI were stored in institutional repositories, for example).

Nevertheless, preservation specialists have welcomed the analysis. “It’s been hard to know the real extent of the digital preservation challenge faced by e-journals,” says William Kilbride, managing director of the Digital Preservation Coalition, headquartered in York, UK. The coalition publishes a handbook detailing good preservation practice.

“Many people have the blind assumption that if you have a DOI, it’s there forever,” says Mikael Laakso, who studies scholarly publishing at the Hanken School of Economics in Helsinki. “But that doesn’t mean that the link will always work.” In 2021, Laakso and his colleagues reported 2 that more than 170 open-access journals had disappeared from the Internet between 2000 and 2019.

Kate Wittenberg, managing director of the digital archiving service Portico in New York City, warns that small publishers are at higher risk of failing to preserve articles than are large ones. “It costs money to preserve content,” she says, adding that archiving involves infrastructure, technology and expertise that many smaller organizations do not have access to.

Eve’s study suggests some measures that could improve digital preservation, including stronger requirements at DOI registration agencies and better education and awareness of the issue among publishers and researchers.

“Everybody thinks of the immediate gains they might get from having a paper out somewhere, but we really should be thinking about the long-term sustainability of the research ecosystem,” Eve says. “After you’ve been dead for 100 years, are people going to be able to get access to the things you’ve worked on?”

doi: https://doi.org/10.1038/d41586-024-00616-5

Updates & Corrections

Clarification 05 March 2024 : The headline of this story has been edited to reflect the fact that some of these papers have not entirely disappeared from the Internet. Rather, many papers are still accessible but have not been properly archived.

Eve, M. P. J. Libr. Sch. Commun. 12 , eP16288 (2024).

Article   Google Scholar  

Laakso, M., Matthias, L. & Jahn, N. J. Assoc. Inf. Sci. Technol. 72 , 1099–1112 (2021).

Download references

Reprints and permissions

  • Information technology
  • Scientific community

Why scientists trust AI too much — and what to do about it

Why scientists trust AI too much — and what to do about it

Editorial 06 MAR 24

Gender bias is more exaggerated in online images than in text

Gender bias is more exaggerated in online images than in text

News & Views 14 FEB 24

Computers make mistakes and AI will make things worse — the law must recognize that

Computers make mistakes and AI will make things worse — the law must recognize that

Editorial 23 JAN 24

Show off your science in Nature’s photo competition

Show off your science in Nature’s photo competition

Career News 08 MAR 24

Workplaces are failing Black women; they must do better

Workplaces are failing Black women; they must do better

World View 08 MAR 24

China promises more money for science in 2024

China promises more money for science in 2024

News 08 MAR 24

Superconductivity scandal: the inside story of deception in a rising star's physics lab

Superconductivity scandal: the inside story of deception in a rising star's physics lab

News Feature 08 MAR 24

China has a list of suspect journals and it’s just been updated

China has a list of suspect journals and it’s just been updated

News Q&A 06 MAR 24

Nature publishes too few papers from women researchers — that must change

Nature publishes too few papers from women researchers — that must change

Postdoc recruitment 2024, SUSTech Proteomics Research Group

Postdoctoral fellows or research assistant professors to join Prof. Ruijun Tian's team for MS-based proteomics and cell signal transduction.

Shenzhen, Guangdong (CN)

journal analysis research paper

Faculty Positions at SUSTech Department of Biomedical Engineering

We seek outstanding applicants for full-time tenure-track/tenured faculty positions. Positions are available for both junior and senior-level.

Shenzhen, Guangdong, China

Southern University of Science and Technology (Biomedical Engineering)

journal analysis research paper

Leading Talents Position in Shandong Institute of Brain Science and Brain-inspired Research

Shandong Institute of Brain Science and Brain-inspired Research is a strategic scientific initiative of Shandong Province, P.R. China.

Jinan, Shandong (CN)

Shandong First Medical University

journal analysis research paper

Permanent Researcher Positions for Materials Science

NIMS (Tsukuba, Japan) invites international applications from researchers who can conduct research in materials science.

National Institute for Materials Science

journal analysis research paper

Postdoctoral Research Fellow at the Dalian Institute of Chemical Physics

Located in the beautiful coastal city of Dalian, surrounded by mountains and sea, DICP seeks all talents from around the globe.

Dalian, Liaoning, China

The Dalian Institute of Chemical Physics (DICP)

journal analysis research paper

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

  • Explore articles by subject
  • Guide to authors
  • Editorial policies

Tax Policy and Investment in a Global Economy

We evaluate the 2017 Tax Cuts and Jobs Act. Combining reduced-form estimates from tax data with a global investment model, we estimate responses, identify parameters, and conduct counterfactuals. Domestic investment of firms with the mean tax change increases 20% versus a no-change baseline. Due to novel foreign incentives, foreign capital of U.S. multinationals rises substantially. These incentives also boost domestic investment, indicating complementarity between domestic and foreign capital. In the model, the long-run effect on domestic capital in general equilibrium is 7% and the tax revenue feedback from growth offsets only 2p.p. of the direct cost of 41% of pre-TCJA corporate revenue.

We thank Agustin Barboza, Emily Bjorkman, Walker Lewis, Anh-Huy Nguyen, Shivani Pandey, Sarah Robinson, Francesco Ruggieri, Sam Thorpe, and Caleb Wroblewski for excellent research assistance; our discussants Eyal Argov, Steven Bond, Manon François, Andrea Lanteri, and Jason Furman; and seminar and conference participants for comments, ideas, and help with data. We thank Michael Caballero, Anne Moore, and Laura Power for insights on multinational tax data and Tom Winberry for helpful discussions about his adjustment cost estimates. Chodorow-Reich gratefully acknowledges support from the Ferrante Fund and Chae fund at Harvard University. Zwick gratefully acknowledges financial support from the Booth School of Business at the University of Chicago. Zidar thanks the NSF for support under grant no. 1752431. Disclaimer: All data work for this project involving confidential taxpayer information was done at IRS facilities, on IRS computers, and at no time was confidential taxpayer data ever outside of the IRS computing environment. The views expressed herein are those of the authors and do not necessarily represent the views of the IRS, the U.S. Department of the Treasury, or the National Bureau of Economic Research. All results have been reviewed to ensure that no confidential information is disclosed. The model-implied revenue estimates are not revenue estimates of the TCJA.

MARC RIS BibTeΧ

Download Citation Data

Conferences

More from nber.

In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship  — as well as online conference reports , video lectures , and interviews .

15th Annual Feldstein Lecture, Mario Draghi, "The Next Flight of the Bumblebee: The Path to Common Fiscal Policy in the Eurozone cover slide

VIDEO

  1. Searching for journal articles in a database

  2. 11: Research writing

  3. Tips for Scanning Journal Articles

  4. Systematic Reviews In Research Universe

  5. How to publish your biological paper in a peer-reviewed journal within 3 months

  6. Systematic Review (the validity & search strategy & searching databases & importing by endnote)

COMMENTS

  1. Home

    The Journal of Analysis publishes high-quality, peer-reviewed original papers and surveys on mathematical analysis and its various applications. Covers a broad range of topics in real and complex analysis. Official publication of the Forum D'Analystes, promoting the study of mathematical analysis. Prioritizes significant results of topical ...

  2. Learning to Do Qualitative Data Analysis: A Starting Point

    For many researchers unfamiliar with qualitative research, determining how to conduct qualitative analyses is often quite challenging. Part of this challenge is due to the seemingly limitless approaches that a qualitative researcher might leverage, as well as simply learning to think like a qualitative researcher when analyzing data. From framework analysis (Ritchie & Spencer, 1994) to content ...

  3. A Step-by-Step Process of Thematic Analysis to Develop ...

    Thematic analysis is a research method used to identify and interpret patterns or themes in a data set; it often leads to new insights and understanding (Boyatzis, 1998; Elliott, 2018; Thomas, 2006).However, it is critical that researchers avoid letting their own preconceptions interfere with the identification of key themes (Morse & Mitcham, 2002; Patton, 2015).

  4. Organizing Your Social Sciences Research Assignments

    A journal article analysis paper should be written in paragraph format and include an instruction to the study, your analysis of the research, and a conclusion that provides an overall assessment of the author's work, along with an explanation of what you believe is the study's overall impact and significance.

  5. PDF Summary and Analysis of Scientific Research Articles

    The analysis shows that you can evaluate the evidence presented in the research and explain why the research could be important. Summary. The summary portion of the paper should be written with enough detail so that a reader would not have to look at the original research to understand all the main points. At the same time, the summary section ...

  6. Articles

    Chiranjib Choudhury. Original Research Paper 13 October 2023. 1. 2. …. 15. Next. The Journal of Analysis publishes high-quality, peer-reviewed original papers and surveys on mathematical analysis and its various applications. Covers a ...

  7. A practical guide to data analysis in general literature reviews

    This article is a practical guide to conducting data analysis in general literature reviews. The general literature review is a synthesis and analysis of published research on a relevant clinical issue, and is a common format for academic theses at the bachelor's and master's levels in nursing, physiotherapy, occupational therapy, public health and other related fields.

  8. Critical Analysis: The Often-Missing Step in Conducting Literature

    The research process for conducting a critical analysis literature review has three phases ; (a) the deconstruction phase in which the individually reviewed studies are broken down into separate discreet data points or variables (e.g., breastfeeding duration, study design, sampling methods); (b) the analysis phase that includes both cross-case ...

  9. How to conduct a meta-analysis in eight steps: a practical guide

    2.1 Step 1: defining the research question. The first step in conducting a meta-analysis, as with any other empirical study, is the definition of the research question. Most importantly, the research question determines the realm of constructs to be considered or the type of interventions whose effects shall be analyzed.

  10. How to conduct a bibliometric analysis: An overview and guidelines

    1. Introduction. Bibliometric analysis has gained immense popularity in business research in recent years (Donthu et al., 2020b, Donthu, Kumar, Pattnaik, & Lim, 2021, Khan et al., 2021), and its popularity can be attributed to (1) the advancement, availability, and accessibility of bibliometric software such as Gephi, Leximancer, VOSviewer, and scientific databases such as Scopus and Web of ...

  11. Analysis

    Brian Ellis and Analysis. in 1952 Analysis helped encourage a young academic to follow a career in philosophy: to be curious, to learn and to articulate ideas with clarity, precession and intellectual honesty, and to see where this leads. Learn more about the 70+ year career of philosopher Brian Ellis and browse a selection of his Analysis articles, made free to read for a limited time.

  12. Journal of Financial and Quantitative Analysis

    The Journal of Financial and Quantitative Analysis ( JFQA) publishes theoretical and empirical research in financial economics. Topics include corporate finance, investments, capital and security markets, and quantitative methods of particular relevance to financial researchers. With a circulation of 3000 libraries, firms, and individuals in 70 ...

  13. How to Write a Research Paper

    A research paper is a piece of academic writing that provides analysis, interpretation, and argument based on in-depth independent research. Research papers are similar to academic essays , but they are usually longer and more detailed assignments, designed to assess not only your writing skills but also your skills in scholarly research.

  14. Introduction to systematic review and meta-analysis

    It is easy to confuse systematic reviews and meta-analyses. A systematic review is an objective, reproducible method to find answers to a certain research question, by collecting all available studies related to that question and reviewing and analyzing their results. A meta-analysis differs from a systematic review in that it uses statistical ...

  15. Journal of Functional Analysis

    The Journal of Functional Analysis presents original research papers in all scientific disciplines in which modern functional analysis plays a basic role. Articles by scientists in a variety of interdisciplinary areas are published. Research Areas Include: • Significant applications of functional analysis, including those to other areas of ...

  16. Journal of Multivariate Analysis

    The Journal of Multivariate Analysis (JMVA) is the central venue for the publication of new, relevant methodology and particularly theoretical developments of multivariate statistics combined with innovative applications pertaining to the analysis and interpretation of multidimensional data. Papers making substantial contributions to regression or time series analysis for multidimensional ...

  17. Critical appraisal of published research papers

    INTRODUCTION. Critical appraisal of a research paper is defined as "The process of carefully and systematically examining research to judge its trustworthiness, value and relevance in a particular context."[] Since scientific literature is rapidly expanding with more than 12,000 articles being added to the MEDLINE database per week,[] critical appraisal is very important to distinguish ...

  18. JSTOR Home

    Harness the power of visual materials—explore more than 3 million images now on JSTOR. Enhance your scholarly research with underground newspapers, magazines, and journals. Explore collections in the arts, sciences, and literature from the world's leading museums, archives, and scholars. JSTOR is a digital library of academic journals ...

  19. Web of Science Master Journal List

    Browse, search, and explore journals indexed in the Web of Science. The Master Journal List is an invaluable tool to help you to find the right journal for your needs across multiple indices hosted on the Web of Science platform. Spanning all disciplines and regions, Web of Science Core Collection is at the heart of the Web of Science platform. Curated with care by an expert team of in-house ...

  20. Impact of Menthol Cigarette Bans: A Systematic Review and Meta-Analysis

    Statistical Analysis. Meta-analytic techniques were used to pool estimates of menthol cigarette smokers' real-world or hypothesized tobacco use behaviors in the setting of a menthol ban. Studies examining real-world menthol bans in the meta-analysis were restricted to longitudinal studies among pre-ban menthol cigarette smokers.

  21. Methodological Guidance Paper: High-Quality Meta-Analysis in a

    The term meta-analysis was first used by Gene Glass (1976) in his presidential address at the AERA (American Educational Research Association) annual meeting, though Pearson (1904) used methods to combine results from studies on the relationship between enteric fever and mortality in 1904. The 1980s was a period of rapid development of statistical methods (Cooper & Hedges, 2009) leading to the ...

  22. Journal of Pharmaceutical Analysis

    Read the latest articles of Journal of Pharmaceutical Analysis at ScienceDirect.com, Elsevier's leading platform of peer-reviewed scholarly literature ... Review paper. select article The potential of herbal drugs to treat heart failure: The roles of Sirt1/AMPK. ... Research article Open access

  23. Home

    Overview. The International Journal of Data Science and Analytics is a pioneering journal in data science and analytics, publishing original and applied research outcomes. Focuses on fundamental and applied research outcomes in data and analytics theories, technologies and applications. Promotes new scientific and technological approaches for ...

  24. (PDF) An Introduction to Water Quality Analysis

    International Research Journal of Engineering and Technology (IRJET) e-ISSN: 2395-0056 Volume: 06 Issue: 01 | Jan 2019 www.irjet.net p-ISSN: 23 95-0072

  25. Fake academic papers are on the rise: why they're a danger and how to

    Lex Bouter is the founding chair of the World Conferences on Research Integrity Foundation and co-chair of the 8th WCRI in Athens, 2-5 June 2024. In the 1800s, British colonists in India set about ...

  26. Millions of research papers at risk of disappearing from the Internet

    The findings, published in the Journal of Librarianship and Scholarly Communication on 24 January 1, indicate that systems to preserve papers online have failed to keep pace with the growth of ...

  27. Microplastics and Nanoplastics in Atheromas and Cardiovascular Events

    Stable isotope analysis was performed on 26 random patient plaque samples because petroleum-derived plastics have lower δ 13 C values (the ratio of carbon-13 to carbon-12) than human tissues. 18 ...

  28. Worldwide trends in underweight and obesity from 1990 to 2022: a pooled

    The combined burden of underweight and obesity has increased in most countries, driven by an increase in obesity, while underweight and thinness remain prevalent in south Asia and parts of Africa. A healthy nutrition transition that enhances access to nutritious foods is needed to address the remaining burden of underweight while curbing and reversing the increase in obesity.

  29. Sustainability

    The evaluation and analysis of the ecological vulnerability of urban green-space systems are conducive to the sustainable development of urban green-space systems. Taking the urban green-space system in the Beijing-Tianjin-Hebei region in 2010, 2015, and 2020 as the research object, an ESSR model was first constructed, and a total of ten types of impact factors were integrated into the ...

  30. Tax Policy and Investment in a Global Economy

    Founded in 1920, the NBER is a private, non-profit, non-partisan organization dedicated to conducting economic research and to disseminating research findings among academics, public policy makers, and business professionals.