How to Write a Bibliography for a Research Paper

Academic Writing Service

Do not try to “wow” your instructor with a long bibliography when your instructor requests only a works cited page. It is tempting, after doing a lot of work to research a paper, to try to include summaries on each source as you write your paper so that your instructor appreciates how much work you did. That is a trap you want to avoid. MLA style, the one that is most commonly followed in high schools and university writing courses, dictates that you include only the works you actually cited in your paper—not all those that you used.

Academic Writing, Editing, Proofreading, And Problem Solving Services

Get 10% off with 24start discount code, assembling bibliographies and works cited.

  • If your assignment calls for a bibliography, list all the sources you consulted in your research.
  • If your assignment calls for a works cited or references page, include only the sources you quote, summarize, paraphrase, or mention in your paper.
  • If your works cited page includes a source that you did not cite in your paper, delete it.
  • All in-text citations that you used at the end of quotations, summaries, and paraphrases to credit others for their ideas,words, and work must be accompanied by a cited reference in the bibliography or works cited. These references must include specific information about the source so that your readers can identify precisely where the information came from.The citation entries on a works cited page typically include the author’s name, the name of the article, the name of the publication, the name of the publisher (for books), where it was published (for books), and when it was published.

The good news is that you do not have to memorize all the many ways the works cited entries should be written. Numerous helpful style guides are available to show you the information that should be included, in what order it should appear, and how to format it. The format often differs according to the style guide you are using. The Modern Language Association (MLA) follows a particular style that is a bit different from APA (American Psychological Association) style, and both are somewhat different from the Chicago Manual of Style (CMS). Always ask your teacher which style you should use.

A bibliography usually appears at the end of a paper on its own separate page. All bibliography entries—books, periodicals, Web sites, and nontext sources such radio broadcasts—are listed together in alphabetical order. Books and articles are alphabetized by the author’s last name.

Most teachers suggest that you follow a standard style for listing different types of sources. If your teacher asks you to use a different form, however, follow his or her instructions. Take pride in your bibliography. It represents some of the most important work you’ve done for your research paper—and using proper form shows that you are a serious and careful researcher.

Bibliography Entry for a Book

A bibliography entry for a book begins with the author’s name, which is written in this order: last name, comma, first name, period. After the author’s name comes the title of the book. If you are handwriting your bibliography, underline each title. If you are working on a computer, put the book title in italicized type. Be sure to capitalize the words in the title correctly, exactly as they are written in the book itself. Following the title is the city where the book was published, followed by a colon, the name of the publisher, a comma, the date published, and a period. Here is an example:

Format : Author’s last name, first name. Book Title. Place of publication: publisher, date of publication.

  • A book with one author : Hartz, Paula.  Abortion: A Doctor’s Perspective, a Woman’s Dilemma . New York: Donald I. Fine, Inc., 1992.
  • A book with two or more authors : Landis, Jean M. and Rita J. Simon.  Intelligence: Nature or Nurture?  New York: HarperCollins, 1998.

Bibliography Entry for a Periodical

A bibliography entry for a periodical differs slightly in form from a bibliography entry for a book. For a magazine article, start with the author’s last name first, followed by a comma, then the first name and a period. Next, write the title of the article in quotation marks, and include a period (or other closing punctuation) inside the closing quotation mark. The title of the magazine is next, underlined or in italic type, depending on whether you are handwriting or using a computer, followed by a period. The date and year, followed by a colon and the pages on which the article appeared, come last. Here is an example:

Format:  Author’s last name, first name. “Title of the Article.” Magazine. Month and year of publication: page numbers.

  • Article in a monthly magazine : Crowley, J.E.,T.E. Levitan and R.P. Quinn.“Seven Deadly Half-Truths About Women.”  Psychology Today  March 1978: 94–106.
  • Article in a weekly magazine : Schwartz, Felice N.“Management,Women, and the New Facts of Life.”  Newsweek  20 July 2006: 21–22.
  • Signed newspaper article : Ferraro, Susan. “In-law and Order: Finding Relative Calm.”  The Daily News  30 June 1998: 73.
  • Unsigned newspaper article : “Beanie Babies May Be a Rotten Nest Egg.”  Chicago Tribune  21 June 2004: 12.

Bibliography Entry for a Web Site

For sources such as Web sites include the information a reader needs to find the source or to know where and when you found it. Always begin with the last name of the author, broadcaster, person you interviewed, and so on. Here is an example of a bibliography for a Web site:

Format : Author.“Document Title.” Publication or Web site title. Date of publication. Date of access.

Example : Dodman, Dr. Nicholas. “Dog-Human Communication.”  Pet Place . 10 November 2006.  23 January 2014 < http://www.petplace.com/dogs/dog-human-communication-2/page1.aspx >

After completing the bibliography you can breathe a huge sigh of relief and pat yourself on the back. You probably plan to turn in your work in printed or handwritten form, but you also may be making an oral presentation. However you plan to present your paper, do your best to show it in its best light. You’ve put a great deal of work and thought into this assignment, so you want your paper to look and sound its best. You’ve completed your research paper!

Back to  How To Write A Research Paper .

ORDER HIGH QUALITY CUSTOM PAPER

what is the bibliography in a research paper

  • Utility Menu

University Logo

fa3d988da6f218669ec27d6b6019a0cd

A publication of the harvard college writing program.

Harvard Guide to Using Sources 

  • The Honor Code
  • Bibliography

If you are using Chicago style footnotes or endnotes, you should include a bibliography at the end of your paper that provides complete citation information for all of the sources you cite in your paper. Bibliography entries are formatted differently from notes. For bibliography entries, you list the sources alphabetically by last name, so you will list the last name of the author or creator first in each entry. You should single-space within a bibliography entry and double-space between them. When an entry goes longer than one line, use a hanging indent of .5 inches for subsequent lines. Here’s a link to a sample bibliography that shows layout and spacing . You can find a sample of note format here .

Complete note vs. shortened note

Here’s an example of a complete note and a shortened version of a note for a book:

1. Karen Ho, Liquidated: An Ethnography of Wall Street (Durham: Duke University Press, 2009), 27-35.

1. Karen Ho, Liquidated , 27-35.

Note vs. Bibliography entry

The bibliography entry that corresponds with each note is very similar to the longer version of the note, except that the author’s last and first name are reversed in the bibliography entry. To see differences between note and bibliography entries for different types of sources, check this section of the Chicago Manual of Style .

For Liquidated , the bibliography entry would look like this:

Ho, Karen, Liquidated: An Ethnography of Wall Street . Durham: Duke University Press, 2009.

Citing a source with two or three authors

If you are citing a source with two or three authors, list their names in your note in the order they appear in the original source. In the bibliography, invert only the name of the first author and use “and” before the last named author.

1. Melissa Borja and Jacob Gibson, “Internationalism with Evangelical Characteristics: The Case of Evangelical Responses to Southeast Asian Refugees,” The Review of Faith & International Affairs 17, no. 3 (2019): 80-81, https://doi.org/10.1080/15570274.2019.1643983 .

Shortened note:

1. Borja and Gibson, “Internationalism with Evangelical Characteristics,” 80-81.

Bibliography:

Borja, Melissa, and Jacob Gibson. “Internationalism with Evangelical Characteristics: The Case of Evangelical Responses to Southeast Asian Refugees.” The Review of Faith & International Affairs 17. no. 3 (2019): 80–93. https://doi.org/10.1080/15570274.2019.1643983 .

Citing a source with more than three authors

If you are citing a source with more than three authors, include all of them in the bibliography, but only include the first one in the note, followed by et al. ( et al. is the shortened form of the Latin et alia , which means “and others”).

1. Justine M. Nagurney, et al., “Risk Factors for Disability After Emergency Department Discharge in Older Adults,” Academic Emergency Medicine 27, no. 12 (2020): 1271.

Short version of note:

1. Justine M. Nagurney, et al., “Risk Factors for Disability,” 1271.

Nagurney, Justine M., Ling Han, Linda Leo‐Summers, Heather G. Allore, Thomas M. Gill, and Ula Hwang. “Risk Factors for Disability After Emergency Department Discharge in Older Adults.” Academic Emergency Medicine 27, no. 12 (2020): 1270–78. https://doi.org/10.1111/acem.14088 .

Citing a book consulted online

If you are citing a book you consulted online, you should include a URL, DOI, or the name of the database where you found the book.

1. Karen Ho, Liquidated: An Ethnography of Wall Street (Durham: Duke University Press, 2009), 27-35, https://doi-org.ezp-prod1.hul.harvard.edu/10.1215/9780822391371 .

Bibliography entry:

Ho, Karen. Liquidated: An Ethnography of Wall Street . Durham: Duke University Press, 2009. https://doi-org.ezp-prod1.hul.harvard.edu/10.1215/9780822391371 .

Citing an e-book consulted outside of a database

If you are citing an e-book that you accessed outside of a database, you should indicate the format. If you read the book in a format without fixed page numbers (like Kindle, for example), you should not include the page numbers that you saw as you read. Instead, include chapter or section numbers, if possible.

1. Karen Ho, Liquidated: An Ethnography of Wall Street (Durham: Duke University Press, 2009), chap. 2, Kindle.

Ho, Karen. Liquidated: An Ethnography of Wall Street . Durham: Duke University Press, 2009. Kindle.

  • Citation Management Tools
  • In-Text Citations
  • Examples of Commonly Cited Sources
  • Frequently Asked Questions about Citing Sources in Chicago Format
  • Sample Bibliography

PDFs for This Section

  • Citing Sources
  • Online Library and Citation Tools
  • Bipolar Disorder
  • Therapy Center
  • When To See a Therapist
  • Types of Therapy
  • Best Online Therapy
  • Best Couples Therapy
  • Best Family Therapy
  • Managing Stress
  • Sleep and Dreaming
  • Understanding Emotions
  • Self-Improvement
  • Healthy Relationships
  • Student Resources
  • Personality Types
  • Verywell Mind Insights
  • 2023 Verywell Mind 25
  • Mental Health in the Classroom
  • Editorial Process
  • Meet Our Review Board
  • Crisis Support

How to Write a Bibliography in APA Format

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

what is the bibliography in a research paper

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

what is the bibliography in a research paper

  • APA Bibliography
  • How to Create One
  • Why You Need It

Sample Bibliography

An APA format bibliography lists all of the sources that might be used in a paper. A bibliography can be a great tool to help you keep track of information during the research and writing process. In some cases, your instructor may require you to include a bibliography as part of your assignment.

At a Glance

A well-written APA format bibliography can help you keep track of information and sources as you research and write your psychology paper. To create a bibliography, gather up all of the sources that you might use in your paper. Create an APA format reference for each source and then write a brief annotation. Your annotation should be a brief summary of what each reference is about. You can quickly refer to these annotations When writing your paper and determine which to include.

What Is an APA Format Bibliography?

An APA format bibliography is an alphabetical listing of all sources that might be used to write an academic paper, essay, article, or research paper—particularly work that is covering psychology or psychology-related topics. APA format is the official style of the American Psychological Association (APA). This format is used by many psychology professors, students, and researchers.

Even if it is not a required part of your assignment, writing a bibliography can help you keep track of your sources and make it much easier to create your final reference page in proper APA format.

Creating an APA Bibliography

A bibliography is similar in many ways to a reference section , but there are some important differences. While a reference section includes every source that was actually used in your paper, a bibliography may include sources that you considered using but may have dismissed because they were irrelevant or outdated.

Bibliographies can be a great way to keep track of information you might want to use in your paper and to organize the information that you find in different sources. The following are four steps you can follow to create your APA format bibliography.

Start on a New Page

Your working bibliography should be kept separate from the rest of your paper. Start it on a new page, with the title "Bibliography" centered at the top and in bold text. Some people use the title "References" instead, so it's best to check with your professor or instructor about which they prefer you to use.

Gather Your Sources

Compile all the sources you might possibly use in your paper. While you might not use all of these sources in your paper, having a complete list will make it easier later on when you prepare your reference section.

Gathering your sources can be particularly helpful when outlining and writing your paper.

By quickly glancing through your working bibliography, you will be able to get a better idea of which sources will be the most appropriate to support your thesis and main points.

Reference Each Source

Your references should be listed alphabetically by the author’s last name, and they should be double-spaced. The first line of each reference should be flush left, while each additional line of a single reference should be a few spaces to the right of the left margin, which is known as a hanging indent.

The format of each source is as follows for academic journals:

  • Last name of first author (followed by their first initial)
  • The year the source was published in parentheses
  • The title of the source
  • The journal that published the source (in italics)
  • The volume number, if applicable (in italics)
  • The issue number, if applicable
  • Page numbers (in parentheses)
  • The URL or "doi" in lowercase letters followed by a colon and the doi number, if applicable

The following examples are scholarly articles in academic journals, cited in APA format:

  • Kulacaoglu, F., & Kose, S. (2018). Borderline personality disorder (BPD): In the midst of vulnerability, chaos, and awe.  Brain sciences ,  8 (11), 201. doi:10.3390/brainsci8110201
  • Cattane, N., Rossi, R., & Lanfredi, M. (2017). Borderline personality disorder and childhood trauma: exploring the affected biological systems and mechanisms.  BMC Psychiatry,   18 (221). doi:10.1186/s12888-017-1383-2

Visit the American Psychological Association's website for more information on citing other types of sources including online media, audiovisual media, and more.

Create an Annotation for Each Source

Normally a bibliography contains only references' information, but in some cases you might decide to create an annotated bibliography. An annotation is a summary or evaluation of the source.

An annotation is a brief description of approximately 150 words describing the information in the source, your evaluation of its credibility, and how it pertains to your topic. Writing one of these for each piece of research will make your writing process faster and easier.

This step helpful in determining which sources to ultimately use in your paper. Your instructor may also require it as part of the assignment so they can assess your thought process and understanding of your topic.

Reasons to Write a Bibliography

One of the biggest reasons to create an APA format bibliography is simply to make the research and writing process easier.

If you do not have a comprehensive list of all of your references, you might find yourself scrambling to figure out where you found certain bits of information that you included in your paper.

A bibliography is also an important tool that your readers can use to access your sources.

While writing an annotated bibliography might not be required for your assignment, it can be a very useful step. The process of writing an annotation helps you learn more about your topic, develop a deeper understanding of the subject, and become better at evaluating various sources of information.

The following is an example of an APA format bibliography by the website EasyBib:

There are many online resources that demonstrate different formats of bibliographies, including the American Psychological Association website . Purdue University's Online Writing Lab also has examples of formatting an APA format bibliography.

Check out this video on their YouTube channel which provides detailed instructions on formatting an APA style bibliography in Microsoft Word.

You can check out the Purdue site for more information on writing an annotated APA bibliography as well.

What This Means For You

If you are taking a psychology class, you may be asked to create a bibliography as part of the research paper writing process. Even if your instructor does not expressly require a bibliography, creating one can be a helpful way to help structure your research and make the writing process more manageable.

For psychology majors , it can be helpful to save any bibliographies you have written throughout your studies so that you can refer back to them later when studying for exams or writing papers for other psychology courses.

American Psychological Association. Publication Manual of the American Psychological Association . 7th Edition. Washington, DC: American Psychological Association; 2020.

Masic I. The importance of proper citation of references in biomedical articles.   Acta Inform Med . 2013;21(3):148–155. doi:10.5455/aim.2013.21.148-155

American Psychological Association. How do you format a bibliography in APA Style?

Cornell University Library. How to prepare an annotated bibliography: The annotated bibliography .

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Banner

Citation Guide

  • What is a Citation?
  • Citation Generator
  • Chicago/Turabian Style
  • Paraphrasing and Quoting
  • Examples of Plagiarism

What is a Bibliography?

What is an annotated bibliography, introduction to the annotated bibliography.

  • Writing Center
  • Writer's Reference Center
  • Helpful Tutorials
  • the authors' names
  • the titles of the works
  • the names and locations of the companies that published your copies of the sources
  • the dates your copies were published
  • the page numbers of your sources (if they are part of multi-source volumes)

Ok, so what's an Annotated Bibliography?

An annotated bibliography is the same as a bibliography with one important difference: in an annotated bibliography, the bibliographic information is followed by a brief description of the content, quality, and usefulness of the source. For more, see the section at the bottom of this page.

What are Footnotes?

Footnotes are notes placed at the bottom of a page. They cite references or comment on a designated part of the text above it. For example, say you want to add an interesting comment to a sentence you have written, but the comment is not directly related to the argument of your paragraph. In this case, you could add the symbol for a footnote. Then, at the bottom of the page you could reprint the symbol and insert your comment. Here is an example:

This is an illustration of a footnote. 1 The number “1” at the end of the previous sentence corresponds with the note below. See how it fits in the body of the text? 1 At the bottom of the page you can insert your comments about the sentence preceding the footnote.

When your reader comes across the footnote in the main text of your paper, he or she could look down at your comments right away, or else continue reading the paragraph and read your comments at the end. Because this makes it convenient for your reader, most citation styles require that you use either footnotes or endnotes in your paper. Some, however, allow you to make parenthetical references (author, date) in the body of your work.

Footnotes are not just for interesting comments, however. Sometimes they simply refer to relevant sources -- they let your reader know where certain material came from, or where they can look for other sources on the subject. To decide whether you should cite your sources in footnotes or in the body of your paper, you should ask your instructor or see our section on citation styles.

Where does the little footnote mark go?

Whenever possible, put the footnote at the end of a sentence, immediately following the period or whatever punctuation mark completes that sentence. Skip two spaces after the footnote before you begin the next sentence. If you must include the footnote in the middle of a sentence for the sake of clarity, or because the sentence has more than one footnote (try to avoid this!), try to put it at the end of the most relevant phrase, after a comma or other punctuation mark. Otherwise, put it right at the end of the most relevant word. If the footnote is not at the end of a sentence, skip only one space after it.

What's the difference between Footnotes and Endnotes?

The only real difference is placement -- footnotes appear at the bottom of the relevant page, while endnotes all appear at the end of your document. If you want your reader to read your notes right away, footnotes are more likely to get your reader's attention. Endnotes, on the other hand, are less intrusive and will not interrupt the flow of your paper.

If I cite sources in the Footnotes (or Endnotes), how's that different from a Bibliography?

Sometimes you may be asked to include these -- especially if you have used a parenthetical style of citation. A "works cited" page is a list of all the works from which you have borrowed material. Your reader may find this more convenient than footnotes or endnotes because he or she will not have to wade through all of the comments and other information in order to see the sources from which you drew your material. A "works consulted" page is a complement to a "works cited" page, listing all of the works you used, whether they were useful or not.

Isn't a "works consulted" page the same as a "bibliography," then?

Well, yes. The title is different because "works consulted" pages are meant to complement "works cited" pages, and bibliographies may list other relevant sources in addition to those mentioned in footnotes or endnotes. Choosing to title your bibliography "Works Consulted" or "Selected Bibliography" may help specify the relevance of the sources listed.

This information has been freely provided by plagiarism.org and can be reproduced without the need to obtain any further permission as long as the URL of the original article/information is cited. 

How Do I Cite Sources? (n.d.) Retrieved October 19, 2009, from http://www.plagiarism.org/plag_article_how_do_i_cite_sources.html

The Importance of an Annotated Bibliography

An Annotated Bibliography is a collection of annotated citations. These annotations contain your executive notes on a source. Use the annotated bibliography to help remind you of later of the important parts of an article or book. Putting the effort into making good notes will pay dividends when it comes to writing a paper!

Good Summary

Being an executive summary, the annotated citation should be fairly brief, usually no more than one page, double spaced.

  • Focus on summarizing the source in your own words.
  • Avoid direct quotations from the source, at least those longer than a few words. However, if you do quote, remember to use quotation marks. You don't want to forget later on what is your own summary and what is a direct quotation!
  • If an author uses a particular term or phrase that is important to the article, use that phrase within quotation marks. Remember that whenever you quote, you must explain the meaning and context of the quoted word or text. 

Common Elements of an Annotated Citation

  • Summary of an Article or Book's thesis or most important points (Usually two to four sentences)
  • Summary of a source's methodological approach. That is, what is the source? How does it go about proving its point(s)? Is it mostly opinion based? If it is a scholarly source, describe the research method (study, etc.) that the author used. (Usually two to five sentences)
  • Your own notes and observations on the source beyond the summary. Include your initial analysis here. For example, how will you use this source? Perhaps you would write something like, "I will use this source to support my point about . . . "
  • Formatting Annotated Bibliographies This guide from Purdue OWL provides examples of an annotated citation in MLA and APA formats.

Youtube

  • << Previous: Examples of Plagiarism
  • Next: ACM Style >>
  • Last Updated: Feb 5, 2024 12:30 PM
  • URL: https://libguides.limestone.edu/citation
  • Try for free

How to Write a Bibliography (MLA, APA Examples)

TeacherVision Staff

Learn how to easily write a bibliography by following the format outlined in this article.

This resource will help your students properly cite different resources in the bibliography of a research paper, and how to format those citations, for books, encyclopedias, films, websites, and people.

What is a bibliography?

According to Infoplease.com, A bibliography is a list of the types of sources you used to get information for your report. It is included at the end of your report, on the last page (or last few pages).

What are the types of bibliography styles (MLA, APA, etc.)?

The 3 most common bibliography/citation styles are:

  • MLA Style: The Modern Language Association works cited page style
  • APA Style: The American Psychological Association style
  • Chicago Style: The bibliography style defined by the Chicago Manual of Style

We’ll give examples of how to create bibliography entries in various styles further down in this article. 

What sources do you put in a bibliography?

An annotated bibliography should include a reference list of any sources you use in writing a research paper. Any printed sources from which you use a text citation, including books, websites, newspaper articles, journal articles, academic writing, online sources (such as PDFs), and magazines should be included in a reference list. In some cases, you may need or want to cite conversations or interviews, works of art, visual works such as movies, television shows, or documentaries - these (and many others) can also be included in a reference list.

How to get started writing your bibliography

You will find it easier to prepare your MLA, APA, or Chicago annotated bibliography if you keep track of each book, encyclopedia, journal article, webpage or online source you use as you are reading and taking notes. Start a preliminary, or draft, bibliography by listing on a separate sheet of paper all your sources. Note down the full title, author’s last name, place of publication, web address, publisher, and date of publication for each source.

Haven't started your paper yet and need an outline? These sample essay outlines include a research paper outline from an actual student paper.

How to write a bibliography step-by-step (with examples)

General Format: Author (last name first). Title of the book. Publisher, Date of publication.

MLA Style: Sibley, David Allen. What It’s Like to Be a Bird. From Flying to Nesting, Eating to Singing, What Birds Are Doing, and Why. Alfred A. Knopf, 2020.

APA Style: Sibley, D.A. (2020). What It’s Like to Be a Bird. From Flying to Nesting, Eating to Singing, What Birds Are Doing, and Why . Alfred A. Knopf.

Notes: Use periods, not commas, to separate the data in the entry. Use a hanging indent if the entry is longer than one line. For APA style, do not use the full author’s first name.

Websites or webpages:

  MLA Style: The SB Nation Family of Sites. Pension Plan Puppets: A Toronto Maple Leafs Blog, 2022, www.pensionplanpuppets.com. Accessed 15 Feb. 2022.

APA Style: American Heart Association. (2022, April 11). How to keep your dog’s heart healthy. https://www.heart.org/en/news/2022/04/11/how-to-keep-your-dogs-heart-healthy

Online news article from a newspaper site:

APA Style: Duehren, A. (2022, April 9). Janet Yellen faces challenge to keep pressure on Russia. Wall Street Journal. https://www.wsj.com/articles/janet-yellen-faces-challenge-to-keep-pressure-on-russia-while-addressing-global-consequences-11650366000

Print journal articles:

MLA Style: Booch, Grady. "Patterns in Object-Oriented Design." IEEE Software Engineering, vol. 6, no. 6, 2006, pp. 31-50.

APA Style: Booch, G. (2006). Patterns in object-oriented design. IEEE Software Engineering, 6(6), 31–50.

Note: It is suggested that you include a DOI and a webpage address when referencing either a printed journal article, and electronic journal article, or an journal article that appears in both formats. 

MLA Style: Gamma, Eric, and Peter A. Coad. “Exceptions to the Unified Modeling Language in Python Patterns.” IEEE Software Engineering, vol. 2, no. 6, 8 Mar. 2006, pp. 190-194. O’Reilly Software Engineering Library, https://doi.org/10.1006/se.20061. Accessed 26 May 2009.

APA Style: Masters, H., Barron, J., & Chanda, L. (2017). Motivational interviewing techniques for adolescent populations in substance abuse counseling. NAADAC Notes, 7(8), 7–13. https://www.naadac.com/notes/adolescent-techniques

ML:A Style: @Grady_Booch. “That’s a bold leap over plain old battery power cars.” Twitter, 13 Mar. 2013, 12:06 p.m., https://twitter.com/Grady_Booch/status/1516379006727188483.

APA Style: Westborough Library [@WestboroughLib]. (2022, April 12). Calling all 3rd through 5th grade kids! Join us for the Epic Writing Showdown! Winner receives a prize! Space is limited so register, today. loom.ly/ypaTG9Q [Tweet; thumbnail link to article]. Twitter. https://twitter.com/WestboroughLib/status/1516373550415896588.

Print magazine articles:

General format: Author (last name first), "Article Title." Name of magazine. Volume number, (Date): page numbers.

MLA Style: Stiteler, Sharon. "Tracking Red-Breasted Grosbeak Migration." Minnesota Bird Journal, 7 Sept. 2019, pp. 7-11.

APA Style: Jordan, Jennifer, "Filming at the Top of the World." Museum of Science Magazine. Volume 47, No. 1, (Winter 1998): p. 11.

Print newspaper articles:

General format: Author (last name first), "Article Title." Name of newspaper, city, state of publication. (date): edition if available, section, page number(s).

MLA Style: Adelman, Martin. "Augustus Announces Departure from City Manager Post." New York Times, late ed., 15 February 2020, p. A1

APA Style: Adelman, M. (2020, February 15). Augustus announced departure from city manager post. New York Times, A1.

Encyclopedias:

General Format: Encyclopedia Title, Edition Date. Volume Number, "Article Title," page numbers.

MLA Style: “Gorillas.” The Encyclopedia Brittanica. 15th ed. 2010.

APA Style: Encyclopedia Brittanica, Inc. (1997.) Gorillas. In The Encyclopedia Brittanica (15th ed., pp. 50-51). Encyclopedia Brittanica, Inc.

Personal interviews:

General format: Full name (last name first). Personal Interview. (Occupation.) Date of interview.

MLA Style: Smithfield, Joseph. Personal interview. 19 May 2014.

APA Style: APA does not require a formal citation for a personal interview. Published interviews from other sources should be cited accordingly.

Films and movies:

General format: Title, Director, Distributor, Year.

MLA Style: Fury. Directed by David Ayer, performances by Brad Pitt, Shia LaBeouf, Jon Bernthal, Sony Pictures, 2014.

APA Style: Ayer, D. (Director). (2014). Fury [Film]. Sony Pictures.

Featured High School Resources

Vocabulary Building Activities and Templates

Related Resources

Writing an essay outline - with examples

About the author

TeacherVision Staff

TeacherVision Editorial Staff

The TeacherVision editorial team is comprised of teachers, experts, and content professionals dedicated to bringing you the most accurate and relevant information in the teaching space.

sandbbox logo

What Is a Bibliography?

  • Writing Research Papers
  • Writing Essays
  • English Grammar
  • M.Ed., Education Administration, University of Georgia
  • B.A., History, Armstrong State University

A bibliography is a list of books, scholarly  articles , speeches, private records, diaries, interviews, laws, letters, websites, and other sources you use when researching a topic and writing a paper. The bibliography appears at the end.

The main purpose of a bibliography entry is to give credit to authors whose work you've consulted in your research. It also makes it easy for a reader to find out more about your topic by delving into the research that you used to write your paper. In the academic world, papers aren't written in a vacuum; academic journals are the way new research on a topic circulates and previous work is built upon.

Bibliography entries must be written in a very specific format, but that format will depend on the particular style of writing you follow. Your teacher or publisher will tell you which style to use, and for most academic papers it will be either MLA , American Psychological Association (APA), Chicago (author-date citations or footnotes/endnotes format), or Turabian style .

The bibliography is sometimes also called the references, works cited, or works consulted page.

Components of a Bibliography Entry

Bibliography entries will compile:

  • Authors and/or editors (and translator, if applicable)
  • Title of your source (as well as edition, volume, and the book title if your source is a chapter or article in a multi-author book with an editor)
  • Publication information (the city, state, name of the publisher, date published, page numbers consulted, and URL or DOI, if applicable)
  • Access date, in the case of online sources (check with the style guide at the beginning of your research as to whether you need to track this information)

Order and Formatting

Your entries should be listed in alphabetical order by the last name of the first author. If you are using two publications that are written by the same author, the order and format will depend on the style guide.

In MLA, Chicago, and Turabian style, you should list the duplicate-author entries in alphabetical order according to the title of the work. The author's name is written as normal for his or her first entry, but for the second entry, you will replace the author's name with three long dashes. 

In APA style, you list the duplicate-author entries in chronological order of publication, placing the earliest first. The name of the author is used for all entries.

For works with more than one author, styles vary as to whether you invert the name of any authors after the first. Whether you use title casing or sentence-style casing on titles of sources, and whether you separate elements with commas or periods also varies among different style guides. Consult the guide's manual for more detailed information.

Bibliography entries are usually formatted using a hanging indent. This means that the first line of each citation is not indented, but subsequent lines of each citation are indented. Check with your instructor or publication to see if this format is required, and look up information in your word processor's help program if you do not know how to create a hanging indent with it.

Chicago's Bibliography vs. Reference System

Chicago has two different ways of citing works consulted: using a bibliography or a references page. Use of a bibliography or a references page depends on whether you're using author-date parenthetical citations in the paper or footnotes/endnotes. If you're using parenthetical citations, then you'll follow the references page formatting. If you're using footnotes or endnotes, you'll use a bibliography. The difference in the formatting of entries between the two systems is the location of the date of the cited publication. In a bibliography, it goes at the end of an entry. In a references list in the author-date style, it goes right after the author's name, similar to APA style.

  • What Is a Citation?
  • What Is a Senior Thesis?
  • Bibliography: Definition and Examples
  • Formatting Papers in Chicago Style
  • Turabian Style Guide With Examples
  • MLA Bibliography or Works Cited
  • How to Write a Bibliography For a Science Fair Project
  • APA In-Text Citations
  • MLA Sample Pages
  • How to Use Block Quotations in Writing
  • Writing a History Book Review
  • Definition of Appendix in a Book or Written Work
  • MLA Style Parenthetical Citations
  • Tips for Typing an Academic Paper on a Computer
  • Title Page Examples and Formats
  • Formatting APA Headings and Subheadings

University Library, University of Illinois at Urbana-Champaign

University of Illinois Library Wordmark

Bibliography and Historical Research

Introduction.

  • National Bibliography
  • Personal Bibliography
  • Corporate Bibliography
  • Subject Bibliography
  • Searching the Catalog for Bibliographies
  • Browsing the Catalog for Bibliographies
  • Other Tools for Finding Bibliographies
  • Return to HPNL Website

How to Write a Research Paper: Annotated Bibliography

  • Anatomy of a Research Paper
  • Developing a Research Focus
  • Background Research Tips
  • Searching Tips
  • Scholarly Journals vs. Popular Journals
  • Thesis Statement
  • Annotated Bibliography
  • Citing Sources
  • Evaluating Sources
  • Literature Review
  • Academic Integrity
  • Scholarship as Conversation
  • Understanding Fake News
  • Data, Information, Knowledge

What is an Annotated Bibliography?

UMary Writing Center

UST Writing Center

Check out the resources available from the  Writing Center . 

Write an Annotated Bibliography

What is an annotated bibliography?

It is a list of citations for various books, articles, and other sources on a topic. 

An annotation is a short summary and/or critical evaluation of a source.

Annotated bibliographies answer the question: "What would be the most relevant, most useful, or most up-to-date sources for this topic?"

 Annotated bibliographies can be part of a larger research project, or can be a stand-alone report in itself. 

Annotation versus abstracts 

An abstract is a paragraph at the beginning of the paper that discusses the main point of the original work. They typically do not include evaluation comments. 

Annotations can either be descriptive or evaluative. The annotated bibliography looks like a works cited page but includes an annotation after each source cited. 

Types of Annotations: 

Descriptive Annotations: Focuses on description. Describes the source by answering the following questions. 

Who wrote the document?

What does the document discuss?

When and where was the document written? 

Why was the document produced?

How was it provided to the public?

Evaluative Annotations: Focuses on description and evaluation. Includes a summary and critically assess the work for accuracy, relevance, and quality. 

Evaluative annotations help you learn about your topic, develop a thesis statement, decide if a specific source will be useful for your assignment, and determine if there is enough valid information available to complete your project.

What does the annotation include?

Depending on your assignment and style guide, annotations may include some or all of the following information. 

  • Should be no more than 150 words or 4 to 6 sentences long. 
  • What is the main focus or purpose of the work?
  • Who is the intended audience?
  • ​How useful or relevant was the article to your topic?
  • Was there any unique features that useful to you?
  • What is the background and credibility of the author?
  • What are any conclusions or observations that your reached about the article?

Which citation style to use?

There are many styles manuals with specific instructions on how to format your annotated bibliography. This largely depends on what your instructor prefers or your subject discipline. Check out our citation guides for more information. 

Additional Information

Why doesn't APA have an official APA-approved format for annotated bibliographies?

Always consult your instructor about the format of an annotated bibliography for your class assignments. These guides provide you with examples of various styles for annotated bibliographies and they may not be in the format required by your instructor. 

Citation Examples and Annotations

Book Citation with Descriptive Annotation

Liroff, R. A., & G. G. Davis. (1981). Protecting open space: Land use control in the Adirondack Park. Cambridge, MA: Ballinger.

This book describes the implementation of regional planning and land use regulation in the Adirondack Park in upstate New York. The authors provide program evaluations of the Adirondack Park Agency’s regulatory and local planning assistance programs.

Journal Article Citation with Evaluative Annotation

Gottlieb, P. D. (1995). The “golden egg” as a natural resource: Toward a normative theory of growth management. Society and Natural Resources, 8, (5): 49-56.

This article explains the dilemma faced by North American suburbs, which demand both preservation of local amenities (to protect quality of life) and physical development (to expand the tax base). Growth management has been proposed as a policy solution to this dilemma. An analogy is made between this approach and resource economics. The author concludes that the growth management debate raises legitimate issues of sustainability and efficiency.

Examples were taken from http://lib.calpoly.edu/support/how-to/write-an-annotated-bibliography/#samples

Book Citation

Lee, Seok-hoon, Yong-pil Kim, Nigel Hemmington, and Deok-kyun Yun. “Competitive Service Quality Improvement (CSQI): A Case Study in the Fast-Food Industry.” Food Service Technology 4 (2004): 75-84.

In this highly technical paper, three industrial engineering professors in Korea and one services management professor in the UK discuss the mathematical limitations of the popular SERVQUAL scales. Significantly, they also aim to measure service quality in the fast-food industry, a neglected area of study. Unfortunately, the paper’s sophisticated analytical methods make it inaccessible to all but the most expert of researchers.

Battle, Ken. “Child Poverty: The Evolution and Impact of Child Benefits.”  A Question of Commitment: Children's Rights in Canada . Ed. Katherine Covell and R.Brian Howe. Waterloo, ON: Wilfrid Laurier University Press. 2007. 21-44.

             Ken Battle draws on a close study of government documents, as well as his own research as an extensively-published policy analyst, to explain Canadian child benefit programs.  He outlines some fundamental assumptions supporting the belief that all society members should contribute to the upbringing of children.  His comparison of child poverty rates in a number of countries is a useful wake-up to anyone assuming Canadian society is doing a good job of protecting children.  Battle pays particular attention to the National Child Benefit (NCB), arguing that it did not deserve to be criticized by politicians and journalists.  He outlines the NCB’s development, costs, and benefits, and laments that the Conservative government scaled it back in favour of the inferior Universal Child Care Benefit (UCCB).  However, he relies too heavily on his own work; he is the sole or primary author of almost half the sources in his bibliography.  He could make this work stronger by drawing from others' perspectives and analyses.  However, Battle does offer a valuable source for this essay, because the chapter provides a concise overview of government-funded assistance currently available to parents.  This offers context for analyzing the scope and financial reality of child poverty in Canada.

Journal Article Example

  Kerr, Don and Roderic Beaujot. “Child Poverty and Family Structure in Canada, 1981-1997.”  Journal of Comparative Family Studies  34.3 (2003): 321-335.

             Sociology professors Kerr and Beaujot analyze the demographics of impoverished families.  Drawing on data from Canada’s annual Survey of Consumer Finances, the authors consider whether each family had one or two parents, the age of single parents, and the number of children in each household.  They analyze child poverty rates in light of both these demographic factors and larger economic issues.  Kerr and Beaujot use this data to argue that. 

Examples were taken from  http://libguides.enc.edu/writing_basics/ annotatedbib/mla

Check out these resources for more information about Annotated Bibliographies. 

  • Purdue Owl- Annotated Bibliographies
  • University of North Carolina at Chapel Hill- Annotated Bibliographies
  • << Previous: Thesis Statement
  • Next: Citing Sources >>
  • Last Updated: Feb 27, 2024 11:57 AM
  • URL: https://libguide.umary.edu/researchpaper

Banner

Research Process: Bibliographic Information

  • Selecting a Topic
  • Background Information
  • Narrowing the Topic
  • Library Terms
  • Generating Keywords
  • Boolean Operators
  • Search Engine Strategies
  • Google Searching
  • Basic Internet Terms
  • Research & The Web
  • Search Engines
  • Evaluating Books
  • Evaluating Articles
  • Evaluating Websites

Bibliographic Information

  • Off Campus Access
  • Periodical Locator

What is a bibliography?

A bibliography is a list of works on a subject or by an author that were used or consulted to write a research paper, book or article. It can also be referred to as a list of works cited. It is usually found at the end of a book, article or research paper. 

Gathering Information

Regardless of what citation style is being used, there are key pieces of information that need to be collected in order to create the citation.

For books and/or journals:

  • Author name
  • Title of publication 
  • Article title (if using a journal)
  • Date of publication
  • Place of publication
  • Volume number of a journal, magazine or encyclopedia
  • Page number(s)

For websites:

  • Author and/or editor name
  • Title of the website
  • Company or organization that owns or posts to the website
  • URL (website address)
  • Date of access 

This section provides two examples of the most common cited sources: a print book and an online journal retrieved from a research database. 

Book - Print

For print books, bibliographic information can be found on the  TITLE PAGE . This page has the complete title of the book, author(s) and publication information.

The publisher information will vary according to the publisher - sometimes this page will include the name of the publisher, the place of publication and the date.

For this example :  Book title: HTML, XHTML, and CSS Bible Author: Steven M. Schafer Publisher: Wiley Publications, Inc.

If you cannot find the place or date of publication on the title page, refer to the  COPYRIGHT PAGE  for this information. The copyright page is the page behind the title page, usually written in a small font, it carries the copyright notice, edition information, publication information, printing history, cataloging data, and the ISBN number.

For this example : Place of publication: Indianapolis, IN Date of publication: 2010

Article - Academic OneFile Database

In the article view:

Bibliographic information can be found under the article title, at the top of the page. The information provided in this area is  NOT  formatted according to any style.

Citations can also be found at the bottom of the page; in an area titled  SOURCE CITATION . The database does not specify which style is used in creating this citation, so be sure to double check it against the style rules for accuracy.

Article - ProQuest Database

Bibliographic information can be found under the article title, at the top of the page. The information provided in this area is  NOT  formatted according to any style. 

Bibliographic information can also be found at the bottom of the page; in an area titled  INDEXING . (Not all the information provided in this area is necessary for creating citations, refer to the rules of the style being used for what information is needed.)

Other databases have similar formats - look for bibliographic information under the article titles and below the article body, towards the bottom of the page. 

  • << Previous: Plagiarism
  • Next: Research Databases >>
  • Last Updated: Jan 8, 2024 10:20 AM
  • URL: https://pgcc.libguides.com/researchprocess
  • Directories
  • What are citations and why should I use them?
  • When should I use a citation?
  • Why are there so many citation styles?
  • Which citation style should I use?
  • Chicago Notes Style
  • Chicago Author-Date Style
  • AMA Style (medicine)
  • Bluebook (law)
  • Additional Citation Styles
  • Built-in Citation Tools
  • Quick Citation Generators
  • Citation Management Software
  • Start Your Research
  • Research Guides
  • University of Washington Libraries
  • Library Guides
  • UW Libraries
  • Citing Sources

Citing Sources: What are citations and why should I use them?

What is a citation.

Citations are a way of giving credit when certain material in your work came from another source. It also gives your readers the information necessary to find that source again-- it provides an important roadmap to your research process. Whenever you use sources such as books, journals or websites in your research, you must give credit to the original author by citing the source. 

Why do researchers cite?

Scholarship is a conversation  and scholars use citations not only to  give credit  to original creators and thinkers, but also to  add strength and authority  to their own work.  By citing their sources, scholars are  placing their work in a specific context  to show where they “fit” within the larger conversation.  Citations are also a great way to  leave a trail  intended to help others who may want to explore the conversation or use the sources in their own work.

In short, citations

(1) give credit

(2) add strength and authority to your work

(3) place your work in a specific context

(4) leave a trail for other scholars

"Good citations should reveal your sources, not conceal them. They should honeslty reflect the research you conducted." (Lipson 4)

Lipson, Charles. "Why Cite?"  Cite Right: A Quick Guide to Citation Styles--MLA, APA, Chicago, the Sciences, Professions, and More . Chicago: U of Chicago, 2006. Print.

What does a citation look like?

Different subject disciplines call for citation information to be written in very specific order, capitalization, and punctuation. There are therefore many different style formats. Three popular citation formats are MLA Style (for humanities articles) and APA or Chicago (for social sciences articles).

MLA style (print journal article):  

Whisenant, Warren A. "How Women Have Fared as Interscholastic Athletic Administrators Since the Passage of Title IX." Sex Roles Vol. 49.3 (2003): 179-182.

APA style (print journal article):

Whisenant, W. A. (2003) How Women Have Fared as Interscholastic Athletic Administrators Since the Passage of Title IX. Sex Roles , 49 (3), 179-182.

Chicago style (print journal article):

Whisenant, Warren A. "How Women Have Fared as Interscholastic Athletic Administrators Since the Passage of Title IX." Sex Roles 49, no. 3 (2003): 179-182.

No matter which style you use, all citations require the same basic information:

  • Author or Creator
  • Container (e.g., Journal or magazine, website, edited book)
  • Date of creation or publication
  • Publisher 

You are most likely to have easy access to all of your citation information when you find it in the first place. Take note of this information up front, and it will be much easier to cite it effectively later.

  • << Previous: Basics of Citing
  • Next: When should I use a citation? >>
  • Last Updated: Oct 24, 2023 3:46 PM
  • URL: https://guides.lib.uw.edu/research/citations

what is the bibliography in a research paper

authorsafterdark.org

research paper writing tips from the top experts

what is the bibliography in a research paper

How to Write a Bibliography for a Research Paper: Guide & Samples

A bibliography is undoubtedly one of the most essential parts of every research paper. A research paper without a bibliography equals poorly conducted research. As such, the importance of a bibliography in a research paper cannot be overemphasized.

To produce well-constructed research, you must consult other authors and use their materials to support your argument. This is the only way to give these authors their credit; by listing them out in your bibliography.

Acknowledging the sources used in writing the topic is an effective approach to avoiding plagiarism in your research. It is also a great way to provide and inform other scholars with sources they may not be aware of. This article seeks to provide the format of bibliography and how to write a good bibliography for your research paper.

What Is a Bibliography in a Research Paper?

Simply put, a bibliography is a list of works used in writing a research paper. Every research paper must contain a list of sources the author used in preparing the research paper. Your source can range from books to scholarly papers, speeches, private records, interviews, letters, websites, and other sources.

Your research paper was not created in a vacuum; you must have consulted other authors or sourced to create enriched content. Therefore, at the end of your research, you must attach a list of all the sources used.

How to Write Bibliography for Research Paper

Writing a bibliography isn’t so hard; all you need is a list of sources used and a format for documenting them. To make things easier for you, prepare a draft comprising all of the sources you must have used. Ensure you include the book’s full title, the author, place of publication, publication date, and publisher. All sources must be listed out alphabetically using the authors’ names.

It is also important to note that there are different formats for writing a bibliography. As such, you must decide which format to employ in your research paper. You can format your work in Modern Language Association (MLA) format, American Psychological Association (APA), or Chicago Manual of Style (CMS). Regardless of which you choose, ensure that it is done correctly.

How to Format a Bibliography

Once you have an idea of how to format a bibliography, most of the work is completed. However, when writing it you need to pay close attention to each format and its content to avoid mixing them up. For printed sources, the bibliography of a research paper should provide the following:

  • Author’s name
  • Title of publication (provide the title of the article if it’s a journal, magazine, or encyclopedia)
  • Date of publication
  • Place of Publication of a book
  • The publishing house of a book
  • Volume number of magazine or encyclopedia
  • The page numbers

To document sources from a website, you need to document the following:

  • The author or editor’s name if there’s any
  • Title of the page
  • The organization of the webpage
  • The URL of the website
  • The date the information was gotten from the website

The common formats for writing a bibliography in a research paper are MLA and APA style. A bibliography is known as “Works Cited” in MLA and on the other hand, it is called “References List” in APA. Though both formats contain similar information about the sources used, there are still slight differences in formatting style. Here’s  what MLA and APA bibliography styles should look like.

How to Write a Bibliography APA

Your research paper’s bibliography must be attached at the end of it with the tag “References” at the center. In addition to this, you need to pay attention to the basics of APA style, such as capitalization, abbreviation, punctuation, underlining or italics, hanging indentation, and others. Here’s a research paper bibliography example in APA style:

Author’s last name, first initial. (Publication date). Book title . City of Publication: Publishing company.

  • For encyclopedia:

Author’s last name, first initial. (Date). Title of Article. Title of Encyclopedia (Volumes, pages). City of Publication: Publishing Company.

  • For magazine and newspaper articles:

Author’s last name, first initial. (Publication Date). Article title. Periodical title, Volume number (issue number if there’s any), page numbers.

How to Write a Bibliography MLA

Bibliography in MLA format is called “Works cited” and must be arranged in alphabetical order according to the author’s name. Here’s the required format for MLA style for varying sources:

“Author’s last name, first name. Book title. Publication City: Publishing company, publication date.”

  • Encyclopedia & dictionary:

“Author’s last name, first name. “Title of Article.” Encyclopedia’s Title. Date.”

  • Magazine & newspaper:

“Author’s last name, first name. “Article title.” Periodical title volume Date: inclusive pages.”

what is the bibliography in a research paper

Leave a Reply Cancel reply

Your email address will not be published. Required fields are marked *

Help | Advanced Search

Computer Science > Computer Vision and Pattern Recognition

Title: sora: a review on background, technology, limitations, and opportunities of large vision models.

Abstract: Sora is a text-to-video generative AI model, released by OpenAI in February 2024. The model is trained to generate videos of realistic or imaginative scenes from text instructions and show potential in simulating the physical world. Based on public technical reports and reverse engineering, this paper presents a comprehensive review of the model's background, related technologies, applications, remaining challenges, and future directions of text-to-video AI models. We first trace Sora's development and investigate the underlying technologies used to build this "world simulator". Then, we describe in detail the applications and potential impact of Sora in multiple industries ranging from film-making and education to marketing. We discuss the main challenges and limitations that need to be addressed to widely deploy Sora, such as ensuring safe and unbiased video generation. Lastly, we discuss the future development of Sora and video generation models in general, and how advancements in the field could enable new ways of human-AI interaction, boosting productivity and creativity of video generation.

Submission history

Access paper:.

  • Download PDF
  • HTML (experimental)
  • Other Formats

References & Citations

  • Google Scholar
  • Semantic Scholar

BibTeX formatted citation

BibSonomy logo

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

  • Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

This paper is in the following e-collection/theme issue:

Published on 5.3.2024 in Vol 26 (2024)

The Digital Therapeutics Real-World Evidence Framework: An Approach for Guiding Evidence-Based Digital Therapeutics Design, Development, Testing, and Monitoring

Authors of this article:

Author Orcid Image

  • Meelim Kim 1, 2, 3, 4 , PhD   ; 
  • Kevin Patrick 1, 3 , MS, MD   ; 
  • Camille Nebeker 1, 3, 4 , PhD   ; 
  • Job Godino 1, 3, 5 , PhD   ; 
  • Spencer Stein 6 , MPH   ; 
  • Predrag Klasnja 7 , PhD   ; 
  • Olga Perski 1, 8 , PhD   ; 
  • Clare Viglione 1 , MPH   ; 
  • Aaron Coleman 9 , BA   ; 
  • Eric Hekler 1, 3, 4 , PhD  

1 Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, La Jolla, CA, United States

2 Department of Preventive Medicine, Yonsei University College of Medicine, Seoul, Republic of Korea

3 The Qualcomm Institute, University of California San Diego, La Jolla, CA, United States

4 The Design Lab, University of California San Diego, La Jolla, CA, United States

5 Laura Rodriguez Research Institute, Family Health Centers of San Diego, San Diego, CA, United States

6 Spiral Health Inc, San Diego, CA, United States

7 School of Information, University of Michigan, Ann Arbor, MI, United States

8 Faculty of Social Sciences, Tampere University, Tampere, Finland

9 Small Steps Labs LLC dba Fitabase Inc, San Diego, CA, United States

Corresponding Author:

Eric Hekler, PhD

Herbert Wertheim School of Public Health and Human Longevity Science

University of California San Diego

9500 Gilman Dr

La Jolla, CA, 92093

United States

Phone: 1 858 429 9370

Email: [email protected]

Digital therapeutics (DTx) are a promising way to provide safe, effective, accessible, sustainable, scalable, and equitable approaches to advance individual and population health. However, developing and deploying DTx is inherently complex in that DTx includes multiple interacting components, such as tools to support activities like medication adherence, health behavior goal-setting or self-monitoring, and algorithms that adapt the provision of these according to individual needs that may change over time. While myriad frameworks exist for different phases of DTx development, no single framework exists to guide evidence production for DTx across its full life cycle, from initial DTx development to long-term use. To fill this gap, we propose the DTx real-world evidence (RWE) framework as a pragmatic, iterative, milestone-driven approach for developing DTx. The DTx RWE framework is derived from the 4-phase development model used for behavioral interventions, but it includes key adaptations that are specific to the unique characteristics of DTx. To ensure the highest level of fidelity to the needs of users, the framework also incorporates real-world data (RWD) across the entire life cycle of DTx development and use. The DTx RWE framework is intended for any group interested in developing and deploying DTx in real-world contexts, including those in industry, health care, public health, and academia. Moreover, entities that fund research that supports the development of DTx and agencies that regulate DTx might find the DTx RWE framework useful as they endeavor to improve how DTxcan advance individual and population health.

Introduction

Digital therapeutics (DTx) are health software tools designed to prevent, treat, or alleviate a disease, disorder, condition, or injury by delivering interventions that have demonstrable positive therapeutic effects on individual health and produce real-world outcomes [ 1 - 3 ]. DTx are often complex interventions [ 4 ] as they include multiple components, such as goal-setting or problem-solving elements, and algorithms that adapt the provision of support to each person’s changing needs. Common goals for DTx include improving medication adherence or regular use of medical devices (eg, glucometers), facilitating behavior change, such as improving diet, physical activity, or sleep, or improving mental health, such as care for depression, anxiety, or stress. DTx can also supplement other care, such as additional support in between clinic visits. The aspiration is that DTx can improve a patient’s health outcomes, reduce the burden on health care professionals, and increase access to and usability of interventions [ 1 , 5 ], by providing safe, effective, and equitable support for individual and population health [ 6 ]. While myriad frameworks exist for DTx development, to date, no single unifying framework guides DTx evidence production and regulatory decision-making [ 7 - 11 ]. By evidence production, we mean the use of scientific methods and processes to produce meaningful data about interventions, such as DTx, both qualitative and quantitative. By regulatory decision-making, we mean the set of oversight activities governing bodies such as the US Food and Drug Administration (FDA) engage in to ensure the products or services in a targeted sector (eg, the pharmaceutical industry) are safe, effective, and aligned with individual and societal needs. We believe that a framework that streamlines linkages between evidence production and regulatory decision-making for DTx will accelerate the development, adoption, and impact of DTx.

For a comprehensive framework to be successful, it must address 2 overarching issues that distinguish DTx from other therapeutics commonly used in health care. The first issue is that DTx are large pieces of software and thus benefit from decades of experience in how software is developed, used, and improved over time. The second issue relates to the relatively new regulatory environment for DTx, which has unique demands likely to evolve further as the field of DTx advances. With respect to the first issue, the basis of DTx in software renders them as dynamic entities that benefit from, and indeed require, periodic upgrades and regular maintenance to ensure they fit with ever-evolving user needs and technological changes. DTx needs to be interoperable with the constantly changing landscape of other software solutions used within health care and requires high levels of software sophistication based on enterprise-grade code embedded within a robust system architecture that supports security, privacy, and ongoing maintenance. Research-grade code is often of good enough quality to enable a novel digital health tool to be tested in small studies and efficacy trials, such as the activities commonly done by academics when engaging in frameworks like the Obesity-Related Behavioral Intervention Trials (ORBIT) [ 12 ], but it is rarely of sufficient quality to be sustainably deployed in real-world contexts. Thus, to be successful, a new framework must be able to guide appropriate evidence production that matches the inherent dynamic and often context-dependent nature of DTx.

The second overarching issue a new framework must address relates to regulatory issues. To date, regulation of DTx typically follows one of three approaches: (1) providing relatively limited guidance on evidence production, biasing toward the trustworthiness of DTx companies, as used in the FDA Precertification (Pre-Cert) Program [ 13 ]; (2) using emerging standards relevant to real-world data (RWD) and real-world evidence (RWE), such as reliance on data quality standards, use of RWD to efficiently run simulated clinical trials [ 14 - 17 ], and open science practices [ 18 , 19 ]; or (3) simply following variations of the 4-phase model [ 12 ] originally created for pharmaceuticals [ 20 - 22 ]. Payors are not providing adequate reimbursement systems, likely in part because of these issues, causing some DTx companies to declare bankruptcy [ 23 ]. We contend that a new framework that incorporates elements of these approaches may be helpful to multiple stakeholders.

Our Approach

Based on this background, we had 2 primary objectives for our proposed DTx RWE framework (hereafter called the “Framework”): (1) to create for users a decision-focused flowchart of key steps to develop DTx, with clear go/no-go milestones needed to move between phases of DTx RWE production that maps to the needs of regulatory decision-making (a point we return to in the discussion); and (2) to provide guidance on how to use RWD to develop this RWE. The 4-phase model is adapted from the National Institutes of Health (NIH)–supported ORBIT [ 12 ] model for behavioral intervention development, with phases focused on design, development, testing, and monitoring. We also considered it important that the Framework provide guidance on RWD use and evidence production in accordance with safe, timely, effective, efficient, equitable, and patient-centered (STEEEP) targets [ 24 ], as well as accessibility [ 25 ], sustainability [ 26 ], and scalability [ 27 ].

To accomplish this, we synthesized frameworks and best practice methods from relevant fields including, but not limited to, behavioral medicine, psychology, public health, medicine, human-centered design, human-computer interaction, bioinformatics, agile software development, computer engineering, health equity research, and community-based participatory research. We drew upon a range of different scoping reviews of frameworks for DTx evidence production (eg, Torous et al [ 28 ] and Lagan et al [ 29 ]). However, rather than using a scoping review process or other formalized expert consensus approach, first and foremost, we were guided by the issues summarized above because we consider them critical to the development of successful DTx yet underemphasized to date.

While we drew from many sources, the ORBIT model, an NIH-recognized approach for behavioral intervention development that uses 4 phases analogous to pharmaceutical development, was a foundational source. However, for our purposes, the ORBIT model had important limitations. It is set up to be broad and accommodate the development and testing of a wide range of different types of behavioral interventions. This domain-specificity can make it challenging for those who may have limited familiarity with behavioral sciences to know how to use it for their specific needs. Additionally, the ORBIT model focuses on evidence production in support of the design of phase III efficacy trials. This is well-matched to studying novel interventions or the efficacy of technologies in ideal conditions, but it is very different from our goal of optimizing the development and sustainable deployment of DTx over time.

We also drew from expert consensus recommendations, including those from the World Health Organization (WHO) [ 30 ] and consensus statements from relevant workshops hosted by the NIH [ 31 ] within the United States, when creating the Framework. When necessary, the authors used first-hand knowledge based on their participation in expert consensus statements in related fields [ 7 , 32 , 33 ]; experience with both the research methods and community practices delineated in the Framework [ 8 , 32 , 34 ]; experience teaching graduate-level methods courses on topics covered in the Framework ; and as innovators engaged in the development, use, and evaluation of novel methods explicitly created for digital health evidence production [ 35 - 37 ].

In addition, we drew on the FDA Pre-Cert Program, which was created to assess the credibility and readiness of a group to engage in DTx evidence production. The FDA Pre-Cert Program begins with an excellence appraisal, which aims to establish credibility by demonstrating the company’s readiness through evaluating organizational excellence and a culture of quality [ 13 ]. Following that, the product goes through a streamlined review process to ensure a reasonable level of safety and effectiveness assurance, which leads to a decision on whether commercial distribution is approved. Once the product is on the market, they are asked to provide RWE based on RWD with a limited list of clinical trial designs in a specified period of time. The Pre-Cert Program ensures that companies have high standards of organizational excellence, that they carry out real-world monitoring of the software as it is used, and, critically, provide a mechanism that could be used to allow DTx groups to be reimbursed in some fashion while RWE production occurs.

The DTx RWE Framework

The Framework ( Figure 1 ) is centered on a flowchart with 4 phases analogous to the ORBIT model but adapted for DTx RWE production (phase I: design; phase II: develop; phase III: test; and phase IV: monitor). Phase I activities correspond to the “double diamond” approach [ 38 ] used in human-centered design and related methods where the problem and solution specifications are delineated. Phase II activities are drawn primarily from ORBIT [ 12 ], its extensions [ 39 ], and the Multiphase Optimization Strategy (MOST) [ 40 , 41 ]. Phase III activities are based on insights from pragmatic clinical trial best practices, including Pragmatic Explanatory Continuum Indicator Summary-2 (PRECIS-2) [ 42 ], reach effectiveness, adoption, implementation, and maintenance (RE-AIM) [ 43 , 44 ], and recommendations from an NIH-recognized expert panel on comparator selection for behavioral interventions [ 45 ]. Phase IV activities are drawn from implementation science [ 46 , 47 ], and emerging recommendations on RWE use for postmarketing surveillance from the WHO [ 48 ]. The approach to RWD in the Framework draws on recent recommendations for the use of RWD for pharmaceuticals [ 48 - 51 ] and on recommendations on open science best practices, which are integrated into each phase, with additional suggested recommendations [ 52 ] summarized in the Discussion section.

The Framework has 4 phases as described in detail below. Sufficient resources need to be provided at both the start of this process and as it goes through the development life cycle if it is to be successful. Moreover, analogous to what we outlined above for the FDA Pre-Cert process, we recommend that groups undertaking this process commit to the following, either on their own or through one or more partnerships, as illustrated in the Framework use case provided in Multimedia Appendix 1 . With this, there are 3 critical roles that must be present:

  • Designate a DTx implementor with the capability to provide ongoing, sustained deployment of the DTx. Examples could be industry, medical centers, or public health departments with proven software development and management capabilities.
  • Designate a community-serving organization that is working with and serving a target population that can provide RWD. These could be hospitals and clinics, federally qualified health centers, community clinics, or public health departments.
  • Designate a DTx evaluator with expertise in the relevant methods and approaches recommended throughout the Framework, both in terms of the flowchart of research activities and the use of RWD.

The responsibility of key stakeholders across all phases of the Framework is to intentionally consider the relevant ethical, legal, and social implications of the DTx pipeline. Including a consultant on the team who is well versed in thinking about, for example, participant characteristics and enrollment, data management (eg, collection, storage, analysis, and sharing), and related issues of bias and privacy are important throughout the process in the Framework. Engaging with an ethics review board, like a research ethics committee or institutional review board, can also be useful at various points in the process.

what is the bibliography in a research paper

Phase I: Design

The goal of phase I is to design the DTx product. The two activities work iteratively together: (1) problem-specification and (2) solution-specification, in accordance with the well-recognized “double diamond” used in human-centered design [ 38 ]. Problem specification includes delineating real-world needs, constraints, assets and approaches to support future sustainability, both financial and ecological [ 53 , 54 ]. Solution specification focuses on iteratively creating a DTx through agile development and prototyping from low-fidelity (eg, paper concepts and storyboards) to high-fidelity (fully functional code) prototypes, with this iterative work being situated within ethical practices and exploring the potential for reusing components or functionalities from available DTx where possible, in line with open science practices [ 55 ].

Intervention plausibility claims are the key focus of phases I and II. We are explicitly using the word “plausible” instead of “feasible” given emerging nomenclature recommendations. By “intervention plausibility,” we are referring to context-dependent probabilistic claims regarding the interaction between the DTx and targeted populations and settings, such as acceptability, demand, capacity to be adapted to a local context, etc. (a full list of plausibility targets is available at [ 56 ]; note: it was labeled feasibility in this paper given that it was published before emerging nomenclature recommendations). Progress can be determined in phase I through the specification of benchmarks that conform with the specific, measurable, actionable, realistic, and timely (SMART) goal concept [ 57 ] but are applied to establishing benchmarks. The transition from phase I to II is justifiable when benchmarks relevant to, at a minimum, safety, and effectiveness are defined and a minimal viable product (MVP) DTx has been produced that meets basic ethical, functional, and usability requirements.

Question 1a: Have Needs, Assets, Constraints, and Sustainability Plans in the Target Population of Users Been Satisfactorily Defined?

This work specifies the target population and setting and answers other key questions relevant to DTx [ 7 ]. Foundational to this is the RWD from a community-serving organization to identify unmet needs, current assets (eg, standard practices and billable activities related to the targeted DTx), and constraints (eg, likely number of billable sessions and the scope of real-world needs, if scaled). Beyond RWD, a mixed methods approach is recommended for this stage. This includes formative research such as ethnographic studies, focus groups, and interviews; community-based participatory and community-driven methods; literature reviews, such as reviewing previous intervention and epidemiological studies; and market research and analysis. Determination of whether these have been satisfactorily defined can be achieved by reaching a consensus to determine if there is overarching agreement among stakeholders, which involves a decision being made when no one objects [ 58 ].

Question 1b: Have an MVP and Corresponding Benchmarks Been Created?

The focus is to iteratively build the DTx and finalize benchmarks. In terms of DTx creation, movement to phase II is justified when a group has produced evidence showing that an MVP is functioning according to minimal usability and accessibility requirements and meets a threshold of being plausible as a tool to enable phase II activities. This could be demonstrated through DTx that are free of “bugs” and meet basic usability requirements (eg, good System Usability Score [ 59 ]). Ideally, benchmarks are established in relation to STEEEP targets [ 24 ] as well as accessibility [ 25 ], sustainability [ 26 ], and scalability [ 27 ]. To guide future implementation, data could be gathered about critical implementation issues [ 56 ]. Minimal benchmarks are needed for safety and effectiveness.

When setting benchmarks, the team should balance what is meaningful relative to current best practices with what is plausible to achieve for the target population and setting [ 60 ]. These benchmarks should, ideally, be based on RWD and establish a threshold that defines if the proposed DTx could plausibly produce desired effects safely relative to current practices. Examples of benchmarks include “a decrease of 3% in hemoglobin A 1c among 60% of our DTx users after 6 months of intervention” for effectiveness and “less than 10% of our DTx users experienced nonserious adverse events associated with digital treatment after a month of intervention” for safety. With a functioning MVP and benchmarks defined, the team has produced the requisite information to transition to phase II. Key approaches here include agile or lean development practices, prototype testing and development, rapid prototype testing, and qualitative methods [ 61 ]. See Multimedia Appendix 1 for an example.

Phase II: Develop

The goal of phase II is to guide DTx development and optimization. There are two types of activities: (1) proof-of-concept trials and (2) optimization trials. Movement from phase II to phase III requires RWE—produced either from a proof-of-concept trial or optimization trial—that demonstrates it is plausible that the DTx can produce clinically meaningful effects in the targeted population and setting while meeting requisite implementation requirements. Given that ORBIT is our primary starting point, it is important to flag that we are rearranging where these trials take place. Specifically, in ORBIT, optimization trials occur late in phase I (ie, phase Ib). In contrast, proof-of-concept trials occur in phase IIa in ORBIT. In the authors’ opinion, the approach used in ORBIT creates 2 issues. First, it places many important activities as phase I. Second, it implicitly signals that phase I and II activities are subservient to phase III activities (a point we return to when discussing phase III of the Framework). In the Framework , we unpack activities in phase I of ORBIT to spread them across phases I and II and provide more explicit labels of the key purposes for each phase, namely design and development. With this, our intent was to provide clearer guidance on how one progresses between phases and also to connote the unique value of each phase without any need for one to be subservient to another; instead, the phase is selected based on the type of evidence production needed.

Within this phase, RWD should be used to support targeted recruitment and selection of study participants with a particular eye toward accounting for health equity in defining a target population (eg, targeting a population that explicitly underuses the current standard of care). In addition, RWD can be used to monitor for unintended consequences, both positive and negative, of the use of the DTx within the development trials. RWD could also be used, particularly with deployed DTx, for conducting data-driven algorithm development [ 62 , 63 ].

Question 2a: Do Any Elements of the DTx Need to Be Improved or Tested?

The proof-of-concept trial focuses primarily on intervention plausibility testing about if the overall package is producing meaningful results, as defined through the benchmarks established in phase I. Optimization trials produce evidence to improve elements of the DTx. For example, optimization trials can be used to support the evidence-based selection of DTx components (factorial trials) [ 64 - 68 ], refinement of components, particularly those used across time (microrandomized trials) [ 35 , 69 , 70 ], and refinement of adaptation algorithms to match the provision of support to context, individual differences, and timing (microrandomized trials [ 35 , 71 ], and sequential multiple assignment randomized trials [ 72 - 75 ]). Note also that optimization trials could also be conducted that are explicitly used to support algorithm development (eg, system identification experiments [ 76 - 78 ] or more data-driven algorithm development from RWD [ 79 , 80 ]). Like phase I, proof-of-concept and optimization trials can be used iteratively. Determining if optimization is needed is based on whether any element of a DTx needs to be improved. If no elements of the DTx package need to be refined, then a proof-of-concept trial is appropriate. If some element of the DTx package needs to be tested or improved, then an optimization trial is needed.

Question 2b: Has a Meaningful Benchmark Been Attained in the Intended Population and Setting?

This is a fundamental question for the proof-of-concept trial, an emerging approach in behavioral trials that tests a DTx in a small group (eg, 10-20) in relation to a benchmark (eg, 70% of the 10 patients shift from hypertensive to systolic blood pressure <120 mm Hg). This approach is used because of 2 known issues with small sample trials within formative work. First, small samples render the use of frequentist inferential statistics problematic [ 60 , 81 , 82 ]. Second, humans have confirmation bias, which refers to the tendency for individuals to seek out or interpret evidence in ways that align with previous beliefs, expectations, or hopes [ 83 ]. A proof-of-concept trial overcomes these challenges without the need for running larger trials, using a clearly specified a priori benchmark that can be tested using descriptive statistics. Its use of benchmarks enables resource-efficient studies with clear go/no go decision-making that reduces the risk of falling prey to confirmatory bias. Quasi-experimental, single-case, or within-participant designs in which the participants serve as their own controls are appropriate design options for a proof-of-concept trial. Further, mixed methods, where both qualitative and quantitative data are relevant to the goals of evidence production and real-world intervention implementation plausibility [ 56 ] targets, should be used. Proof-of-concept trials provide clear go/no go milestones established a priori, thus reducing the risk of continuing when not justified, which is common with more traditional piloting study approaches [ 39 ]. If benchmarks are met, the group can either shift to an optimization trial or transition to phase III. If the benchmarks are not met, then the team should consider returning to phase I or focusing on DTx optimization.

Question 2c: Have the Optimization Criteria Relevant to the Optimization Trial Been Met?

Optimization is a concept drawn from engineering that emphasizes data-driven improvement for a DTx [ 41 , 84 ]. Optimization supports any problem arising with the DTx, such as the DTx costing too much, not being sufficiently adhered to, being difficult to implement in real-world contexts, being inaccessible to the target population, etc. Like a benchmark, the key logic here is to specify clear optimization criteria, meaning a definition of success that can be tested using an appropriately selected optimization trial. Specification of optimization criteria is a central focus of MOST [ 41 , 84 ]. The goal is to create optimization criteria that are measurable and, ideally, account for real-world constraints. For example, one could establish optimization criteria that a DTx includes only intervention components with demonstrated effectiveness, that the DTx can be deployed for under US $50 per client, or that total interactions per week with the DTx stay below 30 minutes. These criteria can be translated into clear go/no go criteria that can be assessed in an optimization trial. If the optimization criteria are met, then this can often justify movement to phase III. Plausible optimization trials for this phase could include but are not limited to: A/B testing (as used in the technology industry for improving usability) [ 85 - 87 ], factorial trials as used in MOST [ 64 , 65 , 67 , 68 ], sequential multiple assigned randomized trials [ 72 - 75 ], microrandomized trials [ 35 , 36 , 69 - 71 ], system identification experiments [ 76 - 78 ], studies explicitly designed to support algorithm development [ 79 , 80 ], and control optimization trials [ 37 , 88 ]. Nahum-Shani et al [ 40 ] provide guidance on when to use common optimization trial designs. See Multimedia Appendix 1 for an example.

Phase III: Test

The goal of phase III is to test if the DTx produces meaningful improvements relative to a comparator in real-world contexts. There may be two types of activities: (1) feasibility or pilot studies; or (2) an effectiveness trial. As in phase II, we have shifted phase labeling from the original ORBIT model while still honoring the types of evidence production ORBIT generally advocates for. Specifically, in the ORBIT model, feasibility or pilot studies are conducted in phase IIb. In ORBIT, phase III is reserved purely for an efficacy trial to test if the intervention impacts health outcomes. We relabeled each phase intentionally in the Framework to allow each phase of work to be conducted and produce insights that are valuable alone, with no phase treated as subservient to other phases. Thus, we sought to have development phase activities stand alone in terms of their unique value for evidence production. We recommend this shift in thinking to clearly flag the critical, independent importance of each phase of work, particularly phase II development, which could feasibly be used in perpetuity alone as a rigorous approach to continuous quality improvement. While speculative, we contend that ORBIT and related evidence production models that implicitly or explicitly treat earlier phases as subservient to phase III trials, particularly efficacy trials in ideal conditions, send a message that privileges one type of evidence at the expense of other evidence. This is problematic, as the evidence from the other phases is particularly important for fostering real-world implementation and health equity. Further, privileging one type of evidence over others establishes the risk of a mono-method bias within scientific knowledge. This creates issues with fostering trustworthy scientific knowledge [ 89 ], reducing confidence in any consensus statements from overemphasizing one particular type of evidence, and might slow the pace of learning and progress [ 90 ]. This is particularly true when evidence production privileges tests occurring in ideal conditions, which is a valuable focus for novel interventions. When groups are developing novel interventions, they should use the ORBIT model, given its emphasis on ensuring appropriate evidence is produced to complete a high-quality efficacy trial. With phase III of the Framework, our focus is on testing if a DTx package produces meaningful results in real-world contexts to foster evidence production with high ecological validity. Thus, in the Framework we bias toward pragmatism through a focus on effectiveness trials, inclusion of benchmarks added to a modified CONSORT (Consolidated Standards of Reporting Trials) diagram guided by RE-AIM [ 91 ] to justify generalization claims, use of decision-oriented comparator selection, and emerging best practices on power calculations [ 39 ]. Moreover, the use of hybrid clinical trials, which incorporate the focus on testing both effectiveness and implementation outcomes [ 92 , 93 ] can be another option to be used in phase III testing of the Framework.

Even within the context of RWE, we recognize the classic tension between pragmatism and explanatory knowledge, as illustrated in PRECIS-2 [ 42 ] (eg, recruitment, eligibility, and setting). With context-invariant interventions, such as vaccines, explanatory knowledge tends to be highly valuable for producing both robust internal validity and generalizable knowledge. Further, with highly novel interventions, explanatory knowledge is valuable to determine if a signal is present in ideal conditions for the novel intervention. For evidence production to guide regulation of use in the real-world context of DTx, we suggest a bias toward pragmaticism over explanatory knowledge to support ecologically valid knowledge as an approach to increase the likelihood of generalizable knowledge. Recognizing that, just like in traditional trials, this tension must be balanced in each trial conducted based on the goals of the work and what is already known. RWD should be used to support recruitment, with a particular eye toward accounting for health equity in recruiting truly representative samples. Confidence in any claims of representation and, thus, generalizability, from the trial can be supported by clearly defining meaningful while also achievable benchmarks for reach and adoption, which can be measured and reported in a modified CONSORT diagram [ 91 ]. Furthermore, once a DTx is widely deployed, RWD could be used to run simulated clinical trials to test effectiveness in real-world contexts [ 14 - 17 ].

Question 3a: Is Evidence Available to Show an Effectiveness Trial Can Be Conducted?

This is a key question for feasibility or pilot studies, which are used to pave the way for a future effectiveness trial. A feasibility study examines whether and how a proposed or planned effectiveness trial can be done, but without a requirement for resemblance to the future trial [ 39 ]. Note, we are explicitly using the word “feasibility” only to refer to attributes of a targeted future fully powered trial, in alignment with 2010 CONSORT recommendations on clinical trial nomenclature [ 94 , 95 ]. The word “feasibility” is sometimes used in reference to issues about the intervention, such as if it would be acceptable, have sufficient demand, or could be integrated into real-world contexts [ 56 ]. To avoid confusion, we refer to these targets as intervention plausibility. Emerging recommendations from scientific groups, such as the 2010 CONSORT recommendations and others more relevant to digital health [ 96 , 97 ], suggest that the term “feasibility” be used to describe the probability that a particular type of study, such as a phase III clinical trial, can be conducted with sufficient rigor and fidelity by an investigative team in a given context. Based on this, we will honor this emerging consensus and only use the term “feasibility” to refer to context-dependent probabilistic claims in relation to the likelihood that a targeted study can be conducted in a particular setting by a particular group with sufficient fidelity to allow conclusions to be drawn from it. Similarly, we use emerging naming conventions and reserve the term “pilot” to refer to a specific type of feasibility trial that implements the exact eventual full protocol of a fully powered trial, but with fewer participants. The goal of a pilot study, thus, is to gather information about the likelihood that if a full trial were conducted, the data quality would be sufficient to enable trustworthy inferences [ 39 ]. If the evidence is available to show the feasibility of an effectiveness trial, the team can proceed directly to the effectiveness trial. If this evidence is not available, then the team should consider conducting a feasibility or pilot study before effectiveness trials to increase the likelihood that, if a trial is run, it will be conducted with sufficient fidelity to provide sufficient quality evidence to guide decisions.

Question 3b: Is the DTx Producing Meaningful Effects Compared to a Decision-Supporting Comparator?

The complexities of DTx establish higher evidentiary standards for generalization claims. By generalization, we specifically focus on transportability, meaning the degree to which insights gleaned from a given study sample are relevant to a stated population and setting. Recent studies of health behaviors, cognitive processes, and emotions, all factors that may influence the effectiveness of a DTx, show that behaviors, cognitions, and emotions have multiple contextual influences, can differ widely from person to person, and fluctuate over time within individuals [ 50 , 98 - 103 ]. Thus, research studies intended to produce generalizable knowledge about the effectiveness of DTx, including where, for whom, and when a given DTx is useful, need to take these factors into account. Given these complexities with regard to DTx evidence production, transportability is difficult to establish. We suggest, first, the use of a modified CONSORT diagram that integrates insights from the RE-AIM framework, and second, benchmarks be established relevant to the percentage of plausible settings and participants enrolled and completing the study. This modified CONSORT diagram provides an approach to quantifying the degree to which a sample may or may not be representative of a stated population and setting. If, for example, there is a large disparity between an eligible population and the number of patients who are enrolled, then transportability claims would be questionable. As with other phases in this Framework, it is suggested that achievable benchmarks related to the modified CONSORT diagram be specified a priori (eg, 80% of eligible clinics will take part, 80% of eligible staff will take part, and 50% of eligible participants will be enrolled). As before, these benchmarks need to balance the need to be ambitious while also being achievable based on what is known. Only if the benchmarks are met can transportability claims be justified. These benchmarks on the modified CONSORT diagram should be derived from RWD.

As this is an effectiveness trial, comparator selection should support real-world decision-making. For example, if the clinical or community partner has a current standard of care and they are considering replacing it with the DTx, the standard of care should be the comparator. Alternatively, if the DTx would fulfill a new area of need, a stepped wedge trial [ 104 , 105 ], in which the DTx is released in a phased fashion across clinics, could be considered. For detailed guidance on comparator selection, see NIH expert panel recommendations [ 45 ]. For DTx testing, options include but are not limited to between-person RCTs [ 106 , 107 ], including remote RCTs [ 108 , 109 ], cluster RCTs [ 110 , 111 ], stepped wedge trial [ 104 , 105 ], and, when sufficient RWD is available, the use of simulated clinical trials [ 14 - 17 ] could be considered.

We also recommend the use of best-practice recommendations for power calculations that specifically do not rely upon underpowered studies to infer effect sizes [ 60 , 112 , 113 ]. Instead, what is recommended is to establish 2 effect size estimates, a threshold of clinical significance, and a plausible effect size that could be observed in the trial. The threshold of clinical significance is the smallest effect size of interest [ 114 ] that would influence clinical decision-making based on an explicit qualitative determination of a noticeable difference. This threshold of clinical significance should be informed by RWD and can be translated from the benchmarks for effectiveness defined in phase I. The plausible effect size is the most likely effect size to be observed if the trial were conducted. This plausible effect size can be informed, in part, from the results of the proof-of-concept trial, particularly if the benchmark that was met is well matched to the threshold of clinical significance. That said, given the unreliability of small sample sizes, RWD and effect sizes from previous trials most like the proposed study should be used to establish the plausible effect size. If the plausible effect size is at or above the threshold of clinical significance, then a trial is warranted. If the plausible effect size is below the threshold of clinical significance, then the trial should not be conducted, as the results gathered would not be sufficient to make a convincing argument to change clinical practice.

If benchmarks set to the modified CONSORT diagram are not met or there is no clinically significant difference observed between the DTx and comparator, then returning to phase I or phase II activities is appropriate. If benchmark and clinically meaningful differences are observed, then results can be submitted to regulators for official review. If the trial was done ethically and responsibly and the results are positive, then regulating bodies can certify the DTx and allow the DTx to market to the population and setting that was studied within the phase III trial. See Multimedia Appendix 1 for an illustrative example.

Phase IV: Monitor

The goal of phase IV is to monitor the use of the DTx within the real world, enable DTx implementers to improve the DTx with additional RWD collected from their clinical or community partners and support the expansion of the target market for the DTx through stepwise additional assessments. This is analogous to traditional non-DTx phase IV activities, including RWD use with pharmaceuticals [ 48 ].

Question 4: Are There Diminishing Positive Effects Over Time in Real-World Use? Are There Any DTx Elements to Improve? Is There a Broader Target Market?

As described earlier and elsewhere [ 7 , 32 , 115 ], continuous improvements in the DTx are not only desired but required for a DTx. For example, user interface expectations of technologies and the use of application programming interfaces drive the evolution of technology. A web application designed and tested in the 1990s [ 116 , 117 ], if it was not continually updated to meet changing user expectations and remain up-to-date with related application programming interfaces, would, at best, be perceived as “old” and, at worst, would not work. Thus, prespecified quality control methods for these updates would need to be used, and this could be one area where regulatory guidelines could be helpful. For example, there is active discussion about when the accumulative changes to a DTx warrant running another clinical trial [ 118 ].

Given this, any notion of a “definitive” clinical trial, a concept traditionally used, is inappropriate for DTx. Pragmatic ways of gleaning insights about when DTx is meeting expectations across the STEEEP and related criteria listed earlier are critical to monitor over time. To support this, the implementation of science practices, particularly strategies for ongoing monitoring, thoughtful adaptation, and guidance on rigorous continuous quality improvement, can be gleaned from the dynamic sustainability framework [ 46 ]. Through ongoing monitoring, issues of potential diminishing benefit (labeled voltage drop) can be observed and used to inspire a response [ 22 ]. For example, if effectiveness levels go below some predefined threshold, regulators could provide DTx with a time-limited window for continued marketing while also requiring the DTx company to reestablish a partnership with a clinical or community partner and reengage with earlier stages of the process. With this potential risk looming, it could establish an incentive for the DTx company to engage in continuous improvement, guided by the other 2 questions, and to maintain mutually beneficial partnerships.

These recommendations conform with recommendations from the WHO for monitoring and evaluating digital health interventions [ 119 ]. According to the WHO, the 4 major components of digital health monitoring (ie, functionality, stability, fidelity, and quality) should guide ongoing monitoring, with these questions mapping onto our proposed questions:

  • (Quality) Is the content and the delivery of the intervention of high enough quality to yield intended outcomes?
  • (Quality) How well and consistently is the intervention delivered?
  • (Functionality) Does the system operate as intended?
  • (Stability) Does the system consistently operate as intended?
  • (Fidelity) Do the realities of field implementation alter the functionality and stability of the system, changing the intervention from that which was intended?

RWE for postmarket surveillance is being explored, and opportunities and pitfalls that are also relevant to DTx are being articulated and should be considered regarding DTx regulation [ 48 ]. Monitoring could require benchmarks to be set for all key targets of evidence production (eg, effectiveness, safety, and equity) as one pathway for cultivating more rigor in monitoring efforts and reducing the risk of confirmation bias during phase IV.

If the DTx implementer believes their DTx can support a more diverse market share than what was approved in phase III, RWD collected in phase IV may help the DTx implementer accelerate this expansion. For example, monitoring could be used to identify plausible new populations, settings, or areas for improvement of the DTx, particularly if done with other community-serving organization partners who may have providers prescribing the DTx for “off-label” uses. One plausible way to improve evidence production during phase IV monitoring would be to focus evidence production more on testing and improving the elements of a DTx (eg, intervention components and adaptation algorithms) instead of the DTx package. A more detailed rationale for this is described elsewhere [ 7 , 32 , 47 , 82 , 120 ]. A second opportunity would be to link activities and efforts with ongoing behavioral ontology efforts to foster better knowledge comparison across various DTx [ 121 , 122 ]. With that said, standards for ongoing monitoring of RWD and RWE are rapidly evolving; thus, this is a critical area for continued work. See Multimedia Appendix 1 for an illustrative example.

The Framework provides guidance to groups seeking to sustainably deploy DTx for use in real-world contexts and may be helpful to regulatory and funding entities as they provide support and oversight of DTx. We acknowledge that the Framework has not been rigorously vetted and that additional work is needed to establish its value. This includes determining whether the use of the Framework has greater or lesser utility in specific domains of DTx applications, such as those used in mental health, behavioral health, or as an adjunct to pharmacological and other interventions for chronic diseases like cancer, musculoskeletal disorders, and cardiovascular disease. We know of no specific reasons why such differences should exist, but as published reports of DTx research emerge in the future, these distinctions might become evident.

Another issue pertains to how the Framework can assist with evaluating DTx that are already in the field, including digital wellness tools that do not meet the definition of a DTx. The longer a DTx has been sustainably deployed at scale, the more likely it is that simulated clinical trial methods could be used to study DTx along STEEEP criteria. With this, efficiencies could be further advanced for evidence production through simulated clinical trials through RWD. Future work would benefit from continued focus on the refined development of simulated clinical trial best practices to improve the pace and resource efficiency of learning.

Regarding digital health wellness tools, these tools often build on foundational behavior change techniques, such as self-monitoring and goal setting that have decades of evidence to guide their design and implementation. When situations like this exist, the burden of proof should be to justify why current evidence is not sufficient already. The most likely gaps in research for these may relate to insufficient evidence for their effectiveness in a broad range of users or settings, so a targeted adaptation of the Framework to fill in these gaps might be the best approach. For example, future work could explore ways to adapt the Framework for use with community-based organizations and community-serving well-being institutions such as the YMCA or Jewish Family Services, along with corporate wellness and related wellness programs that are not implemented by or in partnership with the health care system.

With these limitations recognized, we expand below on the need for three areas of future work related to the Framework: (1) how using RWD advances health equity; (2) cultivating trustworthy partnerships that foster the use of Framework, as a secondary pathway to advance health equity; and (3) suggesting next steps with regard to regulation and funding.

Advancing Health Equity Through RWD

In our view, increased sophistication on the effective use of RWD can become a critical tool to overcome some of the major challenges currently faced in health care, including identifying and addressing health disparities to advance health equity for all, and to foster more targeted and resource-efficient evidence production. RWD provides the information needed to specify unmet needs in general as well as those for individuals, communities, and populations where current practices are not producing desired results. This can help focus resource expenditures and efforts to reduce health disparities.

Future work could advance the use of RWD to drive the development of evidence-based solutions that serve communities most in need. Clinically meaningful benchmarks based on RWD provide an approach for guiding DTx development, both for individual DTxs and for DTxs at large. These would create pressure for DTx not simply to replicate existing standards of care but to improve upon them. Indeed, RWD can be used to establish benchmarks across the various evidence production targets, such as effectiveness, safety, and equity, to provide the foundational data needed to measure, monitor, and, thus, drive equitable progress in individual and population health.

Cultivating Trustworthy Partnerships

As presented in the section above introducing the Framework, we recommend a tripartite approach to its use, comprising an entity committed to sustaining the DTx, a community-serving organization from which the RWD comes, and an entity with appropriate expertise in RWE evaluation efforts. While these conditions can be met within well-resourced settings such as academic medical centers, we suggest that there are opportunities for implementing the Framework through partnerships among groups that may historically not have worked as closely together, such as industry partners working with federally qualified health centers and supported with an academic partner, as illustrated in Multimedia Appendix 1 .

The trustworthiness of all actors involved must be acknowledged as a foundational starting point for any approach to evidence production [ 123 , 124 ]. This includes not merely thinking that trust can be achieved with effective communication but that, at its core, trust involves acknowledging and centering ethics, inclusion, and equity as central guiding principles in the work [ 89 , 123 ]. To do this, we propose the use of best practices in cultivating and maintaining partnerships that have already been delineated like community-based participatory research [ 125 , 126 ], patient-led innovation [ 127 , 128 ], community-driven design [ 129 ], community psychology practices [ 130 , 131 ], and ethical digital health research practices [ 33 , 34 , 132 , 133 ]. Incorporation of approaches to determining corporate trustworthiness that was formatively tested in the FDA Pre-Cert program can be used, including excellence appraisal, and streamlined review elements (eg, real-world performance plan and review determination information) [ 13 ]. The Digital Health Checklist [ 133 ] might also be helpful to guide ethical practices for evidence production relevant to DTx pertaining to issues such as accessibility, privacy, data management, balancing risks, and benefits, all grounded in fundamental ethical principles including respect for persons, beneficence, justice, and respect for law and the public interest.

Regulatory and Funding Issues

We encourage regulators and funders of DTx to explore whether the Framework can help guide their efforts. The principles embodied in the Framework could be used to establish generalized regulatory expectations for DTx. Clarifying these expectations could help get multiple DTx developers and purchases “on the same page” with respect to achieving and maintaining appropriate standards of quality throughout the life cycle of DTx use. Similarly, funders of DTx research and development, such as the NIH, the Agency for Healthcare Research and Quality, the Patient-Centered Outcomes Research Institute, and the Health Resources and Services Administration, could encourage applicants to use the Framework, and if they do, then demonstrate how they propose to achieve the benchmarks that it includes.

The Framework is intended to improve evidence production and sustainable deployment of DTx in real-world contexts. The Framework provides guidance on how to design, develop, test, and monitor DTx, both in the early stages of their development and over time as they are used in real-world contexts. Our hope is that the Framework can help address issues commonly seen with DTx, including low DTx uptake, long-term sustainability, and insufficient attention to health disparities. Overall, there is considerable opportunity to improve individual and population health equitably through DTx, and we hope the Framework can contribute to this end.

Acknowledgments

We want to thank Dr Kenneth Freedland for providing expert consultation and review of earlier versions of this manuscript. This study is supported by the National Library of Medicine (R01LM013107) and the National Cancer Institute (R01CA244777) of the National Institute of Health. CN was supported by the Patient-Centered Outcomes Research Institute (PCORI; award ME-2020C3-21310) and the Altman Clinical and Translational Research Institute (ACTRI) at the University of California, San Diego. The ACTRI is funded from awards issued by the National Center for Advancing Translational Sciences (NIH UL1TR001442). PK is supported by the National Cancer Institute (1U01CA229445) of the National Institute of Health. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or PCORI.

Authors' Contributions

MK, KP, CN, and EH contributed to conceptualizing, drafting, reviewing, and finalizing the paper. All authors contributed to critical reviews and revisions of the paper and have given approval for the paper to be submitted for publication. The authors read and approved the final manuscript.

Conflicts of Interest

SS has an ownership interest in Spiral Health, Inc. AC has an ownership interest in Fitabase, Inc. OP acts as an unpaid scientific advisor for the Smoke Free app. All other authors declare no competing interests.

The hypothetical example of DTx RWE Framework.

  • Understanding DTx: a new category of medicine. Digital Therapeutics Alliance. 2022. URL: https://dtxalliance.org/understanding-dtx/ [accessed 2024-02-02]
  • Digital therapeutics get a brand new definition. Healthcare Brew. 2023. URL: https://tinyurl.com/y4uyr9ww [accessed 2024-02-02]
  • ISO/TR 11147:2023(en) health informatics—personalized digital health—digital therapeutics health software systems. International Organization for Standardization (ISO). 2023. URL: https://tinyurl.com/bddx8vtx [accessed 2024-02-02]
  • Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, et al. Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337:a1655. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Patel NA, Butte AJ. Characteristics and challenges of the clinical pipeline of digital therapeutics. NPJ Digit Med. 2020;3(1):159. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Dang A, Arora D, Rane P. Role of digital therapeutics and the changing future of healthcare. J Family Med Prim Care. 2020;9(5):2207-2213. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Murray E, Hekler EB, Andersson G, Collins LM, Doherty A, Hollis C, et al. Evaluating digital health interventions: key questions and approaches. Am J Prev Med. 2016;51(5):843-851. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res. 2017;19(6):e232. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Voorheis P, Zhao A, Kuluski K, Pham Q, Scott T, Sztur P, et al. Integrating behavioral science and design thinking to develop mobile health interventions: systematic scoping review. JMIR Mhealth Uhealth. 2022;10(3):e35799. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Torous J. Evaluating apps: making informed decisions with available real-world data. Digital health interventions from wellness to therapeutics: development and dissemination. 2022. Presented at: National Institutes of Health, Digital Health Intervention Workshop; July 13, 2022, 2022; NA.
  • Miao BY, Arneson D, Wang M, Butte AJ. Open challenges in developing digital therapeutics in the United States. PLOS Digit Health. 2022;1(1):e0000008. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Czajkowski SM, Powell LH, Adler N, Naar-King S, Reynolds KD, Hunter CM, et al. From ideas to efficacy: the ORBIT model for developing behavioral treatments for chronic diseases. Health Psychol. 2015;34(10):971-982. [ CrossRef ] [ Medline ]
  • The Software Precertification (Pre-Cert) pilot program: tailored total product lifecycle approaches and key findings. U.S. Food & Drug Administration. 2022. URL: https://tinyurl.com/ycwsjh8u [accessed 2024-02-02]
  • Chen Z, Zhang H, Guo Y, George TJ, Prosperi M, Hogan WR, et al. Exploring the feasibility of using real-world data from a large clinical data research network to simulate clinical trials of Alzheimer's disease. NPJ Digit Med. 2021;4(1):84. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Swift B, Jain L, White C, Chandrasekaran V, Bhandari A, Hughes DA, et al. Innovation at the intersection of clinical trials and real-world data science to advance patient care. Clin Transl Sci. 2018;11(5):450-460. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Müller P, Chandra NK, Sarkar A. Bayesian approaches to include real-world data in clinical studies. Philos Trans A Math Phys Eng Sci. 2023;381(2247):20220158. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Wang C, Rosner GL. A bayesian nonparametric causal inference model for synthesizing randomized clinical trial and real-world evidence. Stat Med. 2019;38(14):2573-2588. [ CrossRef ] [ Medline ]
  • Kokol P. Agile software development in healthcare: a synthetic scoping review. Appl Sci. 2022;12(19):9462. [ FREE Full text ] [ CrossRef ]
  • Van Velthoven MH, Smith J, Wells G, Brindley D. Digital health app development standards: a systematic review protocol. BMJ Open. 2018;8(8):e022969. [ FREE Full text ] [ CrossRef ]
  • Basic principles on use of registries in approval applications. Pharmaceuticals and Medical Devices Agency. URL: https://www.pmda.go.jp/files/000240810.pdf [accessed 2024-02-02]
  • Guideline on the application of real world evidence of medical device. Ministry of Food and Drug Safety (MFDS). 2019. URL: https://tinyurl.com/mr3zpj85 [accessed 2024-02-02]
  • Framework for FDA's real-world evidence program. U.S. Food & Drug Administration. 2018. URL: https://www.fda.gov/media/120060/download [accessed 2024-02-02]
  • Pear therapeutics assets sold for $6M at auction after bankruptcy. MobiHealthNews. 2023. URL: https://tinyurl.com/4v7b87kk [accessed 2024-02-02]
  • Six domains of healthcare quality. Agency for Healthcare Research and Quality. 2022. URL: https://www.ahrq.gov/talkingquality/measures/six-domains.html [accessed 2024-02-02]
  • Gulliford M, Figueroa-Munoz J, Morgan M, Hughes D, Gibson B, Beech R, et al. What does 'access to health care' mean? J Health Serv Res Policy. 2002;7(3):186-188. [ CrossRef ] [ Medline ]
  • McCool J, Dobson R, Muinga N, Paton C, Pagliari C, Agawal S, et al. Factors influencing the sustainability of digital health interventions in low-resource settings: lessons from five countries. J Glob Health. 2020;10(2):020396. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Milat A, Lee K, Conte K, Grunseit A, Wolfenden L, van Nassau F, et al. Intervention scalability assessment tool: a decision support tool for health policy makers and implementers. Health Res Policy Syst. 2020;18(1):1. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Torous JB, Chan SR, Gipson SYMT, Kim JW, Nguyen TQ, Luo J, et al. A hierarchical framework for evaluation and informed decision making regarding smartphone apps for clinical care. Psychiatr Serv. 2018;69(5):498-500. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Lagan S, Sandler L, Torous J. Evaluating evaluation frameworks: a scoping review of frameworks for assessing health apps. BMJ Open. 2021;11(3):e047001. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Consensus statement: role of policy-makers and health care leaders in implementation of the Global Patient Safety Action Plan 2021–2030. World Health Organization. 2022. URL: https://www.who.int/publications/i/item/WHO-UHL-IHS-PSF-2022.3 [accessed 2024-02-02]
  • NIH consensus statement. National Institutes of Health (NIH). 2000. URL: https://tinyurl.com/4txypc3d [accessed 2024-02-02]
  • Patrick K, Hekler EB, Estrin D, Mohr DC, Riper H, Crane D, et al. The pace of technologic change: implications for digital health behavior intervention research. Am J Prev Med. 2016;51(5):816-824. [ CrossRef ] [ Medline ]
  • Nebeker C, Torous J, Ellis RJB. Building the case for actionable ethics in digital health research supported by artificial intelligence. BMC Med. 2019;17(1):137. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Nebeker C, Ellis RJB, Torous J. Development of a decision-making checklist tool to support technology selection in digital health research. Transl Behav Med. 2020;10(4):1004-1015. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Klasnja P, Hekler EB, Shiffman S, Boruvka A, Almirall D, Tewari A, et al. Microrandomized trials: an experimental design for developing just-in-time adaptive interventions. Health Psychol. 2015;34S:1220-1228. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Klasnja P, Smith S, Seewald NJ, Lee A, Hall K, Luers B, et al. Efficacy of contextually tailored suggestions for physical activity: a micro-randomized optimization trial of HeartSteps. Ann Behav Med. 2019;53(6):573-582. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Hekler EB, Rivera DE, Martin CA, Phatak SS, Freigoun MT, Korinek E, et al. Tutorial for using control systems engineering to optimize adaptive mobile health interventions. J Med Internet Res. 2018;20(6):e214. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Framework for innovation: Design Council's evolved double diamond. Design Council. 2019. URL: https://tinyurl.com/bdhmmyuw [accessed 2024-02-02]
  • Powell LH, Freedland KE, Kaufmann PG. Behavioral Clinical Trials for Chronic Diseases: Scientific Foundations. Cham, Switzerland. Springer International Publishing; 2021.
  • Nahum-Shani I, Dziak JJ, Wetter DW. MCMTC: a pragmatic framework for selecting an experimental design to inform the development of digital interventions. Front Digit Health. 2022;4:798025. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Collins LM. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: The Multiphase Optimization Strategy (MOST). Cham, Switzerland. Springer International Publishing; 2018.
  • Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. 2015;350:h2147. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Glasgow RE, Harden SM, Gaglio B, Rabin B, Smith ML, Porter GC, et al. RE-AIM planning and evaluation framework: adapting to new science and practice with a 20-year review. Front Public Health. 2019;7:64. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Holtrop JS, Estabrooks PA, Gaglio B, Harden SM, Kessler RS, King DK, et al. Understanding and applying the RE-AIM framework: clarifications and resources. J Clin Transl Sci. 2021;5(1):e126. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Freedland KE, King AC, Ambrosius WT, Mayo-Wilson E, Mohr DC, Czajkowski SM, et al. The selection of comparators for randomized controlled trials of health-related behavioral interventions: recommendations of an NIH expert panel. J Clin Epidemiol. 2019;110:74-81. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci. 2013;8:117. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, et al. Agile science: creating useful products for behavior change in the real world. Transl Behav Med. 2016;6(2):317-328. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Beaulieu-Jones BK, Finlayson SG, Yuan W, Altman RB, Kohane IS, Prasad V, et al. Examining the use of real-world evidence in the regulatory process. Clin Pharmacol Ther. 2020;107(4):843-852. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Garrison LP, Neumann PJ, Erickson P, Marshall D, Mullins CD. Using real-world data for coverage and payment decisions: the ISPOR Real-World Data Task Force report. Value Health. 2007;10(5):326-335. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Hekler E, Tiro JA, Hunter CM, Nebeker C. Precision health: the role of the social and behavioral sciences in advancing the vision. Ann Behav Med. 2020;54(11):805-826. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Vaithinathan AG, Asokan V. Public health and precision medicine share a goal. J Evid Based Med. 2017;10(2):76-80. [ CrossRef ] [ Medline ]
  • Burns L, Roux NL, Kalesnik-Orszulak R, Christian J, Hukkelhoven M, Rockhold F, et al. Real-world evidence for regulatory decision-making: guidance from around the world. Clin Ther. 2022;44(3):420-437. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chevance G, Hekler EB, Efoui-Hess M, Godino J, Golaszewski N, Gualtieri L, et al. Digital health at the age of the anthropocene. Lancet Digit Health. 2020;2(6):e290-e291. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chevance G, Fresán U, Hekler E, Edmondson D, Lloyd SJ, Ballester J, et al. Thinking health-related behaviors in a climate change context: a narrative review. Ann Behav Med. 2023;57(3):193-204. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kwasnicka D, Keller J, Perski O, Potthoff S, Hoor GAT, Ainsworth B, et al. White paper: open digital health—accelerating transparent and scalable health promotion and treatment. Health Psychol Rev. 2022;16(4):475-491. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Bowen DJ, Kreuter M, Spring B, Cofta-Woerpel L, Linnan L, Weiner D, et al. How we design feasibility studies. Am J Prev Med. 2009;36(5):452-457. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • SMART goals: a how to guide. University of California. 2016. URL: https://tinyurl.com/mvy8cxzt [accessed 2024-02-02]
  • Rau T. The difference between whole-group consensus and dynamic governance/sociocracy. Sociocracy For All. 2023. URL: https://tinyurl.com/ysy47skf [accessed 2024-02-02]
  • System Usability Scale (SUS). usability.gov. URL: https://tinyurl.com/22pkunfy [accessed 2024-02-02]
  • Freedland KE. Pilot trials in health-related behavioral intervention research: problems, solutions, and recommendations. Health Psychol. 2020;39(10):851-862. [ CrossRef ] [ Medline ]
  • Design thinking 101. Nielsen Norman Group. 2016. URL: https://www.nngroup.com/articles/design-thinking/ [accessed 2024-02-02]
  • Martín CA, Rivera DE, Hekler EB, Riley WT, Buman MP, Adams MA, et al. Development of a control-oriented model of social cognitive theory for optimized mHealth behavioral interventions. IEEE Trans Control Syst Technol. 2020;28(2):331-346. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chevance G, Golaszewski NM, Baretta D, Hekler EB, Larsen BA, Patrick K, et al. Modelling multiple health behavior change with network analyses: results from a one-year study conducted among overweight and obese adults. J Behav Med. 2020;43(2):254-261. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Spring B, Pfammatter AF, Marchese SH, Stump T, Pellegrini C, McFadden HG, et al. A factorial experiment to optimize remotely delivered behavioral treatment for obesity: results of the Opt-IN study. Obesity (Silver Spring). 2020;28(9):1652-1662. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Thomas JG, Goldstein CM, Bond DS, Lillis J, Hekler EB, Emerson JA, et al. Evaluation of intervention components to maximize outcomes of behavioral obesity treatment delivered online: a factorial experiment following the multiphase optimization strategy framework. Contemp Clin Trials. 2021;100:106217. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Lee H, Choi EH, Shin JU, Kim TG, Oh J, Shin B, et al. The impact of intervention design on user engagement in digital therapeutics research: factorial experiment With a mixed methods study. JMIR Form Res. 2024;8:e51225. [ CrossRef ] [ Medline ]
  • Cipriani A, Barbui C. What is a factorial trial? Epidemiol Psychiatr Sci. 2013;22(3):213-215. [ FREE Full text ] [ CrossRef ]
  • Kip H, Da Silva MC, Bouman YHA, van Gemert-Pijnen LJEWC, Kelders SM. A self-control training app to increase self-control and reduce aggression—a full factorial design. Internet Interv. 2021;25:100392. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Seewald NJ, Smith SN, Lee AJ, Klasnja P, Murphy SA. Practical considerations for data collection and management in mobile health micro-randomized trials. Stat Biosci. 2019;11(2):355-370. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Bell L, Garnett C, Qian T, Perski O, Potts HWW, Williamson E. Notifications to improve engagement with an alcohol reduction app: protocol for a micro-randomized trial. JMIR Res Protoc. 2020;9(8):e18690. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Battalio SL, Conroy DE, Dempsey W, Liao P, Menictas M, Murphy S, et al. Sense2Stop: a micro-randomized trial using wearable sensors to optimize a just-in-time-adaptive stress management intervention for smoking relapse prevention. Contemp Clin Trials. 2021;109:106534. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Czyz EK, King CA, Prouty D, Micol VJ, Walton M, Nahum-Shani I. Adaptive intervention for prevention of adolescent suicidal behavior after hospitalization: a pilot sequential multiple assignment randomized trial. J Child Psychol Psychiatry. 2021;62(8):1019-1031. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • De Barros Gonze B, Da Costa Padovani R, Do Socorro Simoes M, Lauria V, Proença NL, Sperandio EF, et al. Use of a smartphone app to increase physical activity levels in insufficiently active adults: feasibility Sequential Multiple Assignment Randomized Trial (SMART). JMIR Res Protoc. 2020;9(10):e14322. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Tamura RN, Krischer JP, Pagnoux C, Micheletti R, Grayson PC, Chen YF, et al. A small n sequential multiple assignment randomized trial design for use in rare disease research. Contemp Clin Trials. 2016;46:48-51. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Lu X, Nahum-Shani I, Kasari C, Lynch KG, Oslin DW, Pelham WE, et al. Comparing dynamic treatment regimes using repeated-measures outcomes: modeling considerations in SMART studies. Stat Med. 2016;35(10):1595-1615. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Freigoun MT, Martín CA, Magann AB, Rivera DE, Phatak SS, Korinek EV, et al. System identification of just walk: a behavioral mHealth intervention for promoting physical activity. 2017 Presented at: 2017 American Control Conference (ACC); May 24-26, 2017, 2017;116-121; Seattle, WA, USA. [ CrossRef ]
  • dos Santos PL, Freigoun MT, Martin CA, Rivera DE, Hekler EB, Romano RA, et al. System identification of just walk: using matchable-observable linear parametrizations. IEEE Trans Control Syst Technol. 2020;28(1):264-275. [ FREE Full text ] [ CrossRef ]
  • Hojjatinia S, Daly ER, Hnat T, Hossain SM, Kumar S, Lagoa CM, et al. Dynamic models of stress-smoking responses based on high-frequency sensor data. NPJ Digit Med. 2021;4(1):162. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Chevance G, Baretta D, Heino M, Perski O, Olthof M, Klasnja P, et al. Characterizing and predicting person-specific, day-to-day, fluctuations in walking behavior. PLoS One. 2021;16(5):e0251659. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Goldstein SP, Thomas JG, Foster GD, Turner-McGrievy G, Butryn ML, Herbert JD, et al. Refining an algorithm-powered just-in-time adaptive weight control intervention: a randomized controlled trial evaluating model performance and behavioral outcomes. Health Informatics J. 2020;26(4):2315-2331. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Ioannidis JPA. Why most published research findings are false. PLoS Med. 2005;2(8):e124. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Freedland KE. Progress in health-related behavioral intervention research: making it, measuring it, and meaning it. Health Psychol. 2022;41(1):1-12. [ CrossRef ] [ Medline ]
  • Nickerson RS. Confirmation bias: a ubiquitous phenomenon in many guises. Rev Gen Psychol. 1998;2(2):175-220. [ CrossRef ]
  • Collins LM, Kugler KC. Optimization of Behavioral, Biobehavioral, and Biomedical Interventions: Advanced Topics. Cham, Switzerland. Springer International Publishing; 2018.
  • Austrian J, Mendoza F, Szerencsy A, Fenelon L, Horwitz LI, Jones S, et al. Applying A/B testing to clinical decision support: rapid randomized controlled trials. J Med Internet Res. 2021;23(4):e16651. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Miller HN, Plante TB, Gleason KT, Charleston J, Mitchell CM, Miller ER, et al. A/B design testing of a clinical trial recruitment website: a pilot study to enhance the enrollment of older adults. Contemp Clin Trials. 2021;111:106598. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Senderey AB, Kornitzer T, Lawrence G, Zysman H, Hallak Y, Ariely D, et al. It's how you say it: systematic A/B testing of digital messaging cut hospital no-show rates. PLoS One. 2020;15(6):e0234817. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Rivera DE, Pew MD, Collins LM. Using engineering control principles to inform the design of adaptive interventions: a conceptual introduction. Drug Alcohol Depend. 2007;88(Suppl 2):S31-S40. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Hekler E, Anderson CAM, Cooper LA. Is it time to restructure the National Institutes of Health? Am J Public Health. 2022;112(7):965-968. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kwasnicka D, Hoor GAT, Hekler E, Hagger MS, Kok G. Proposing a new approach to funding behavioural interventions using iterative methods. Psychol Health. 2021;36(7):787-791. [ CrossRef ] [ Medline ]
  • Glasgow RE, Huebschmann AG, Brownson RC. Expanding the CONSORT figure: increasing transparency in reporting on external validity. Am J Prev Med. 2018;55(3):422-430. [ CrossRef ] [ Medline ]
  • Wolfenden L, Williams CM, Wiggers J, Nathan N, Yoong SL. Improving the translation of health promotion interventions using effectiveness-implementation hybrid designs in program evaluations. Health Promot J Austr. 2016;27(3):204-207. [ CrossRef ] [ Medline ]
  • Ullman AJ, Beidas RS, Bonafide CP. Methodological progress note: hybrid effectiveness-implementation clinical trials. J Hosp Med. 2022;17(11):912-916. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010;340:c869. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Montgomery P, Grant S, Mayo-Wilson E, Macdonald G, Michie S, Hopewell S, et al. Reporting randomised trials of social and psychological interventions: the CONSORT-SPI 2018 extension. Trials. 2018;19(1):407. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Teresi JA, Yu X, Stewart AL, Hays RD. Guidelines for designing and evaluating feasibility pilot studies. Med Care. 2022;60(1):95-103. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Pearson N, Naylor PJ, Ashe MC, Fernandez M, Yoong SL, Wolfenden L. Guidance for conducting feasibility and pilot studies for implementation trials. Pilot Feasibility Stud. 2020;6(1):167. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Heino MTJ, Knittle K, Noone C, Hasselman F, Hankonen N. Studying behaviour change mechanisms under complexity. Behav Sci (Basel). 2021;11(5):77. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Hekler EB, Klasnja P, Chevance G, Golaszewski NM, Lewis D, Sim I. Why we need a small data paradigm. BMC Med. 2019;17(1):133. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Regal RE, Billi JE, Glazer HM. Phenothiazine-induced cholestatic jaundice. Clin Pharm. 1987;6(10):787-794. [ Medline ]
  • Komatsu T. Analysis of a characteristic alteration of glycogen in the liver of alloxan diabetic rat by fasting. Nagoya J Med Sci. 1977;39(3-4):59-67. [ Medline ]
  • Korinek EV, Phatak SS, Martin CA, Freigoun MT, Rivera DE, Adams MA, et al. Adaptive step goals and rewards: a longitudinal growth model of daily steps for a smartphone-based walking intervention. J Behav Med. 2018;41(1):74-86. [ CrossRef ] [ Medline ]
  • Nahum-Shani I, Hekler EB, Spruijt-Metz D. Building health behavior models to guide the development of just-in-time adaptive interventions: a pragmatic framework. Health Psychol. 2015;34S:1209-1219. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Palermo TM, de la Vega R, Murray C, Law E, Zhou C. A digital health psychological intervention (WebMAP mobile) for children and adolescents with chronic pain: results of a hybrid effectiveness-implementation stepped-wedge cluster randomized trial. Pain. 2020;161(12):2763-2774. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Copas AJ, Lewis JJ, Thompson JA, Davey C, Baio G, Hargreaves JR. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation approaches. Trials. 2015;16:352. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kim M, Kim Y, Go Y, Lee S, Na M, Lee Y, et al. Multidimensional cognitive behavioral therapy for obesity applied by psychologists using a digital platform: open-label randomized controlled trial. JMIR Mhealth Uhealth. 2020;8(4):e14817. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kollins SH, DeLoss DJ, Cañadas E, Lutz J, Findling RL, Keefe RSE, et al. A novel digital intervention for actively reducing severity of paediatric ADHD (STARS-ADHD): a randomised controlled trial. Lancet Digit Health. 2020;2(4):e168-e178. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Luderer H, Chiodo L, Wilson A, Brezing C, Martinez S, Xiong X, et al. Patient engagement with a game-based digital therapeutic for the treatment of opioid use disorder: protocol for a randomized controlled open-label, decentralized trial. JMIR Res Protoc. 2022;11(1):e32759. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kaizer AM, Wild J, Lindsell CJ, Rice TW, Self WH, Brown S, et al. Trial of Early Antiviral Therapies during Non-hospitalized Outpatient Window (TREAT NOW) for COVID-19: a summary of the protocol and analysis plan for a decentralized randomized controlled trial. Trials. 2022;23(1):273. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Fitzsimmons-Craft EE, Taylor CB, Graham AK, Sadeh-Sharvit S, Balantekin KN, Eichen DM, et al. Effectiveness of a digital cognitive behavior therapy-guided self-help intervention for eating disorders in college women: a cluster randomized clinical trial. JAMA Netw Open. 2020;3(8):e2015633. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Pallejà-Millán M, Rey-Reñones C, Uriarte MLB, Granado-Font E, Basora J, Flores-Mateo G, et al. Evaluation of the Tobbstop mobile app for smoking cessation: cluster randomized controlled clinical trial. JMIR Mhealth Uhealth. 2020;8(6):e15951. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Kraemer HC, Kupfer DJ. Size of treatment effects and their importance to clinical research and practice. Biol Psychiatry. 2006;59(11):990-996. [ CrossRef ] [ Medline ]
  • Kraemer HC, Mintz J, Noda A, Tinklenberg J, Yesavage JA. Caution regarding the use of pilot studies to guide power calculations for study proposals. Arch Gen Psychiatry. 2006;63(5):484-489. [ CrossRef ] [ Medline ]
  • Anvari F, Lakens D. Using anchor-based methods to determine the smallest effect size of interest. J Exp Soc Psychol. 2021;96:104159. [ FREE Full text ] [ CrossRef ]
  • Wang XQ, Mao SP. Comparison of morphology, pathogenicity and drug response among three isolates of schistosoma japonicum in the mainland of China. Ann Parasitol Hum Comp. 1989;64(2):110-119. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • 14 iconic websites that show off classic 90s web design. Webflow. 2022. URL: https://webflow.com/blog/90s-website-design [accessed 2024-02-02]
  • Beckman BL. What Apple, Google, and Amazon's websites looked like in 1999. Mashable. 2020. URL: https://mashable.com/article/90s-web-design [accessed 2024-02-02]
  • Perski O. Scientific and ethical challenges to defining what constitutes 'proportionate evidence' for the regulation and accreditation of applications to treat addiction. Addiction. 2021;116(12):3285-3287. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. World Helath Organization. 2016. URL: https://www.who.int/publications/i/item/9789241511766 [accessed 2024-02-02]
  • Klasnja P, Hekler EB, Korinek EV, Harlow J, Mishra SR. Toward usable evidence: optimizing knowledge accumulation in HCI research on health behavior change. Proc SIGCHI Conf Hum Factor Comput Syst. 2017;2017:3071-3082. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Larsen KR, Michie S, Hekler EB, Gibson B, Spruijt-Metz D, Ahern D, et al. Behavior change interventions: the potential of ontologies for advancing science and practice. J Behav Med. 2017;40(1):6-22. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Larsen KI, Hekler EB, Paul MJ, Gibson BS. Improving usability of social and behavioral sciences‘ evidence: a call to action for a national infrastructure project for mining our knowledge. Commun Assoc Inf Syst. 2020;46:1-17. [ FREE Full text ] [ CrossRef ]
  • Reardon J, Lee SSJ, Goering S, Fullerton SM, Cho MK, Panofsky A, et al. Trustworthiness matters: building equitable and ethical science. Cell. 2023;186(5):894-898. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Espie CA, Torous J, Brennan TA. Digital therapeutics should be regulated with gold-standard evidence. Health Affairs Forefront. 2022. URL: https://www.healthaffairs.org/do/10.1377/forefront.20220223.739329/ [accessed 2024-02-02]
  • Mullins CD, Abdulhalim AM, Lavallee DC. Continuous patient engagement in comparative effectiveness research. JAMA. 2012;307(15):1587-1588. [ CrossRef ] [ Medline ]
  • Minkler M, Wallerstein N. Community-Based Participatory Research for Health: From Process to Outcomes, 2nd Edition. San Francisco, CA. Wiley; 2011.
  • Birnbaum F, Lewis D, Rosen RK, Ranney ML. Patient engagement and the design of digital health. Acad Emerg Med. 2015;22(6):754-756. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Petersen C, Austin RR, Backonja U, Campos H, Chung AE, Hekler EB, et al. Citizen science to further precision medicine: from vision to implementation. JAMIA Open. 2020;3(1):2-8. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Wilson BB. Resilience for All: Striving for Equity Through Community-Driven Design. Washington, DC. Island Press; 2018.
  • Jason L, Glantsman O, O'Brien JF, Ramian KN. Introduction to Community Psychology: Becoming an Agent of Change. Montreal, Quebec, Canada. Rebus Community; 2019.
  • Whittaker A. Research Skills for Social Work, 2nd Edition. London. Sage Publications; 2012.
  • Ellis RB, Wright J, Miller LS, Jake-Schoffman D, Hekler EB, Goldstein CM, et al. Lessons learned: beta-testing the digital health checklist for researchers prompts a call to action by behavioral scientists. J Med Internet Res. 2021;23(12):e25414. [ FREE Full text ] [ CrossRef ] [ Medline ]
  • Nebeker C, Gholami M, Kareem D, Kim E. Applying a digital health checklist and readability tools to improve informed consent for digital health research. Front Digit Health. 2021;3:690901. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by T Leung; submitted 21.05.23; peer-reviewed by R Barak Ventura, D Boeldt; comments to author 06.12.23; revised version received 13.01.24; accepted 29.01.24; published 05.03.24.

©Meelim Kim, Kevin Patrick, Camille Nebeker, Job Godino, Spencer Stein, Predrag Klasnja, Olga Perski, Clare Viglione, Aaron Coleman, Eric Hekler. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 05.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

  • Knowledge Base
  • Citing sources

How to Cite Sources | Citation Generator & Quick Guide

Citing your sources is essential in  academic writing . Whenever you quote or paraphrase a source (such as a book, article, or webpage), you have to include a  citation crediting the original author.

Failing to properly cite your sources counts as plagiarism , since you’re presenting someone else’s ideas as if they were your own.

The most commonly used citation styles are APA and MLA. The free Scribbr Citation Generator is the quickest way to cite sources in these styles. Simply enter the URL, DOI, or title, and we’ll generate an accurate, correctly formatted citation.

Generate accurate citations with Scribbr

Instantly correct all language mistakes in your text.

Upload your document to correct all your mistakes in minutes

upload-your-document-ai-proofreader

Table of contents

When do you need to cite sources, which citation style should you use, in-text citations, reference lists and bibliographies.

Scribbr Citation Generator

Other useful citation tools

Citation examples and full guides, frequently asked questions about citing sources.

Citations are required in all types of academic texts. They are needed for several reasons:

  • To avoid plagiarism by indicating when you’re taking information from another source
  • To give proper credit to the author of that source
  • To allow the reader to consult your sources for themselves

A citation is needed whenever you integrate a source into your writing. This usually means quoting or paraphrasing:

  • To quote a source , copy a short piece of text word for word and put it inside quotation marks .
  • To paraphrase a source , put the text into your own words. It’s important that the paraphrase is not too close to the original wording. You can use the paraphrasing tool if you don’t want to do this manually.

Citations are needed whether you quote or paraphrase, and whatever type of source you use. As well as citing scholarly sources like books and journal articles, don’t forget to include citations for any other sources you use for ideas, examples, or evidence. That includes websites, YouTube videos , and lectures .

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

what is the bibliography in a research paper

Try for free

Usually, your institution (or the journal you’re submitting to) will require you to follow a specific citation style, so check your guidelines or ask your instructor.

In some cases, you may have to choose a citation style for yourself. Make sure to pick one style and use it consistently:

  • APA Style is widely used in the social sciences and beyond.
  • MLA style is common in the humanities.
  • Chicago notes and bibliography , common in the humanities
  • Chicago author-date , used in the (social) sciences
  • There are many other citation styles for different disciplines.

If in doubt, check with your instructor or read other papers from your field of study to see what style they follow.

In most styles, your citations consist of:

  • Brief in-text citations at the relevant points in the text
  • A reference list or bibliography containing full information on all the sources you’ve cited

In-text citations most commonly take the form of parenthetical citations featuring the last name of the source’s author and its year of publication (aka author-date citations).

An alternative to this type of in-text citation is the system used in numerical citation styles , where a number is inserted into the text, corresponding to an entry in a numbered reference list.

There are also note citation styles , where you place your citations in either footnotes or endnotes . Since they’re not embedded in the text itself, these citations can provide more detail and sometimes aren’t accompanied by a full reference list or bibliography.

A reference list (aka “Bibliography” or “Works Cited,” depending on the style) is where you provide full information on each of the sources you’ve cited in the text. It appears at the end of your paper, usually with a hanging indent applied to each entry.

The information included in reference entries is broadly similar, whatever citation style you’re using. For each source, you’ll typically include the:

  • Author name
  • Publication date
  • Container (e.g., the book an essay was published in, the journal an article appeared in)
  • Location (e.g., a URL or DOI , or sometimes a physical location)

The exact information included varies depending on the source type and the citation style. The order in which the information appears, and how you format it (e.g., capitalization, use of italics) also varies.

Most commonly, the entries in your reference list are alphabetized by author name. This allows the reader to easily find the relevant entry based on the author name in your in-text citation.

APA-reference-list

In numerical citation styles, the entries in your reference list are numbered, usually based on the order in which you cite them. The reader finds the right entry based on the number that appears in the text.

Vancouver reference list example

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

  • Missing commas and periods
  • Incorrect usage of “et al.”
  • Ampersands (&) in narrative citations
  • Missing reference entries

what is the bibliography in a research paper

Because each style has many small differences regarding things like italicization, capitalization , and punctuation , it can be difficult to get every detail right. Using a citation generator can save you a lot of time and effort.

Scribbr offers citation generators for both APA and MLA style. Both are quick, easy to use, and 100% free, with no ads and no registration required.

Just input a URL or DOI or add the source details manually, and the generator will automatically produce an in-text citation and reference entry in the correct format. You can save your reference list as you go and download it when you’re done, and even add annotations for an annotated bibliography .

Once you’ve prepared your citations, you might still be unsure if they’re correct and if you’ve used them appropriately in your text. This is where Scribbr’s other citation tools and services may come in handy:

Plagiarism Checker

Citation Checker

Citation Editing

Plagiarism means passing off someone else’s words or ideas as your own. It’s a serious offense in academia. Universities use plagiarism checking software to scan your paper and identify any similarities to other texts.

When you’re dealing with a lot of sources, it’s easy to make mistakes that could constitute accidental plagiarism. For example, you might forget to add a citation after a quote, or paraphrase a source in a way that’s too close to the original text.

Using a plagiarism checker yourself before you submit your work can help you spot these mistakes before they get you in trouble. Based on the results, you can add any missing citations and rephrase your text where necessary.

Try out the Scribbr Plagiarism Checker for free, or check out our detailed comparison of the best plagiarism checkers available online.

Scribbr Plagiarism Checker

Scribbr’s Citation Checker is a unique AI-powered tool that automatically detects stylistic errors and inconsistencies in your in-text citations. It also suggests a correction for every mistake.

Currently available for APA Style, this is the fastest and easiest way to make sure you’ve formatted your citations correctly. You can try out the tool for free below.

If you need extra help with your reference list, we also offer a more in-depth Citation Editing Service.

Our experts cross-check your in-text citations and reference entries, make sure you’ve included the correct information for each source, and improve the formatting of your reference page.

If you want to handle your citations yourself, Scribbr’s free Knowledge Base provides clear, accurate guidance on every aspect of citation. You can see citation examples for a variety of common source types below:

And you can check out our comprehensive guides to the most popular citation styles:

At college level, you must properly cite your sources in all essays , research papers , and other academic texts (except exams and in-class exercises).

Add a citation whenever you quote , paraphrase , or summarize information or ideas from a source. You should also give full source details in a bibliography or reference list at the end of your text.

The exact format of your citations depends on which citation style you are instructed to use. The most common styles are APA , MLA , and Chicago .

The abbreviation “ et al. ” (Latin for “and others”) is used to shorten citations of sources with multiple authors.

“Et al.” is used in APA in-text citations of sources with 3+ authors, e.g. (Smith et al., 2019). It is not used in APA reference entries .

Use “et al.” for 3+ authors in MLA in-text citations and Works Cited entries.

Use “et al.” for 4+ authors in a Chicago in-text citation , and for 10+ authors in a Chicago bibliography entry.

The Scribbr Citation Generator is developed using the open-source Citation Style Language (CSL) project and Frank Bennett’s citeproc-js . It’s the same technology used by dozens of other popular citation tools, including Mendeley and Zotero.

You can find all the citation styles and locales used in the Scribbr Citation Generator in our publicly accessible repository on Github .

APA format is widely used by professionals, researchers, and students in the social and behavioral sciences, including fields like education, psychology, and business.

Be sure to check the guidelines of your university or the journal you want to be published in to double-check which style you should be using.

MLA Style  is the second most used citation style (after APA ). It is mainly used by students and researchers in humanities fields such as literature, languages, and philosophy.

Is this article helpful?

Other students also liked.

  • Citation Styles Guide | Examples for All Major Styles
  • APA vs. MLA | The Key Differences in Format & Citation
  • The Basics of In-Text Citation | APA & MLA Examples

More interesting articles

  • Citation examples for common sources types
  • Et Al. | Meaning & Use in APA, MLA & Chicago
  • Hanging Indent | Word & Google Docs Instructions
  • How to Cite a Book | APA, MLA, & Chicago Examples
  • How to Cite a Journal Article | APA, MLA, & Chicago Examples
  • How to Cite a Lecture | APA, MLA & Chicago Examples
  • How to Cite a Newspaper Article | MLA, APA & Chicago
  • How to Cite a Website | MLA, APA & Chicago Examples
  • How to Cite a Wikipedia Article | APA, MLA & Chicago
  • How to Cite a YouTube Video | MLA, APA & Chicago
  • How to Cite an Image | Photographs, Figures, Diagrams
  • How to Cite an Interview | APA, MLA & Chicago Style
  • Parenthetical Citation | APA, MLA & Chicago Examples
  • What Are Endnotes? | Guide with Examples
  • What Are Footnotes? | Guide with Word Instructions
  • What Does Ibid. Mean? | Definition & Examples
  • What is a DOI? | Finding and Using Digital Object Identifiers
  • What Is an Annotated Bibliography? | Examples & Format

What is your plagiarism score?

U.S. flag

An official website of the United States government

Here’s how you know

Official websites use .gov A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS A lock ( Lock A locked padlock ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

https://www.nist.gov/publications/2023-nist-center-neutron-research-accomplishments-and-opportunities

2023 NIST Center for Neutron Research Accomplishments and Opportunities

Download paper, additional citation formats.

  • Google Scholar

IMAGES

  1. How to write an annotated bibliography step-by-step with examples

    what is the bibliography in a research paper

  2. Annotated Bibliography Journal Example Apa

    what is the bibliography in a research paper

  3. Common Research Paper Bibliography Formats

    what is the bibliography in a research paper

  4. Follow Our Turabian Style Annotated Bibliography Sample

    what is the bibliography in a research paper

  5. Annotated Bibliography Sample in Word and Pdf formats

    what is the bibliography in a research paper

  6. Annotated Bibliography Cover Page

    what is the bibliography in a research paper

VIDEO

  1. How to write References, Citations & Bibliography in Thesis/Dissertation

  2. What is Citation?

  3. How to write bibliography for a school project or Research paper

  4. After the Annotated Bib

  5. bibliography 😊😊

  6. How to Write a Bibliography for a Research Paper?

COMMENTS

  1. How to Write a Bibliography, With Examples

    What is the purpose of a bibliography? A bibliography is the list of sources a work's author used to create the work. It accompanies just about every type of academic writing, like essays, research papers, and reports.You might also find a brief, less formal bibliography at the end of a journalistic piece, presentation, or video when the author feels it's necessary to cite their sources.

  2. How to Write a Bibliography for a Research Paper

    Bibliography Entry for a Book. A bibliography entry for a book begins with the author's name, which is written in this order: last name, comma, first name, period. After the author's name comes the title of the book. If you are handwriting your bibliography, underline each title. If you are working on a computer, put the book title in ...

  3. Bibliography

    For bibliography entries, you list the sources alphabetically by last name, so you will list the last name of the author or creator first in each entry. You should single-space within a bibliography entry and double-space between them. When an entry goes longer than one line, use a hanging indent of .5 inches for subsequent lines.

  4. How to Write an APA Format Bibliography

    An APA format bibliography is an alphabetical listing of all sources that might be used to write an academic paper, essay, article, or research paper—particularly work that is covering psychology or psychology-related topics. APA format is the official style of the American Psychological Association (APA). This format is used by many ...

  5. Common Research Paper Bibliography Formats

    Research papers take a lot of different steps to format your outline and discuss where your sources came from. Every paper ends with a citation page or bibliography. ... Now that you understand the styles to use, it's time to create your research paper citation page. This can be a piece of cake if you follow the correct steps. Step 1: Create ...

  6. Bibliography: Definition and Examples

    A bibliography is a list of works (such as books and articles) written on a particular subject or by a particular author. Adjective: bibliographic. Also known as a list of works cited, a bibliography may appear at the end of a book, report, online presentation, or research paper. Students are taught that a bibliography, along with correctly ...

  7. What is a Bibliography?

    A bibliography is a list of all of the sources you have used in the process of researching your work. In general, a bibliography should include: the authors' names. the titles of the works. the names and locations of the companies that published your copies of the sources. the dates your copies were published.

  8. Writing a Bibliography

    A bibliography is a detailed list of all the sources consulted and cited in a research paper or project. The bibliography structure always includes citing the author's name, the title of the work ...

  9. How to Write a Bibliography (MLA, APA Examples)

    An annotated bibliography should include a reference list of any sources you use in writing a research paper. Any printed sources from which you use a text citation, including books, websites, newspaper articles, journal articles, academic writing, online sources (such as PDFs), and magazines should be included in a reference list.

  10. What Is a Bibliography?

    A bibliography is a list of books, scholarly articles, speeches, private records, diaries, interviews, laws, letters, websites, and other sources you use when researching a topic and writing a paper. The bibliography appears at the end. The main purpose of a bibliography entry is to give credit to authors whose work you've consulted in your ...

  11. LibGuides: Bibliography and Historical Research: Introduction

    All bibliographies are organized differently, but the best include indexes that help you pinpoint the most relevant entries. A smart researcher will also use the index to obtain an overview of the entire source base: the index as a whole presents a broad outline of the available sources--the extent of available sources, as well as the the strengths and weaknesses of the source base.

  12. How to Write a Bibliography in APA and MLA styles With Examples

    Examples of Bibliography Format Examples of Bibliography Formats. There are standards for documenting sources of information in research papers. Even though different journals may use a slightly different format for the bibliography, they all contain the same basic information.

  13. How to Write a Bibliography for a Research Paper: Guidelines

    Step 1. Develop an initial reference page. While accumulating data for research papers, establishing an initial bibliography can be advantageous. It simplifies the final stages of your work and aids in the organization of your ideas. When composing an initial draft, ensure to compile details: Author (s) and editor (s);

  14. How to Write a Research Paper: Annotated Bibliography

    Annotated bibliographies can be part of a larger research project, or can be a stand-alone report in itself. Annotation versus abstracts . An abstract is a paragraph at the beginning of the paper that discusses the main point of the original work. They typically do not include evaluation comments. Annotations can either be descriptive or ...

  15. LibGuides: Research Process: Bibliographic Information

    A bibliography is a list of works on a subject or by an author that were used or consulted to write a research paper, book or article. It can also be referred to as a list of works cited. It is usually found at the end of a book, article or research paper. Gathering Information. Regardless of what citation style is being used, there are key ...

  16. Research Paper Format

    Formatting an APA paper. The main guidelines for formatting a paper in APA Style are as follows: Use a standard font like 12 pt Times New Roman or 11 pt Arial. Set 1 inch page margins. Apply double line spacing. If submitting for publication, insert a APA running head on every page. Indent every new paragraph ½ inch.

  17. Citing Sources: What are citations and why should I use them?

    Different subject disciplines call for citation information to be written in very specific order, capitalization, and punctuation. There are therefore many different style formats. Three popular citation formats are MLA Style (for humanities articles) and APA or Chicago (for social sciences articles). MLA style (print journal article):

  18. APA Formatting and Citation (7th Ed.)

    Throughout your paper, you need to apply the following APA format guidelines: Set page margins to 1 inch on all sides. Double-space all text, including headings. Indent the first line of every paragraph 0.5 inches. Use an accessible font (e.g., Times New Roman 12pt., Arial 11pt., or Georgia 11pt.).

  19. Citations, References and Bibliography in Research Papers [Beginner's

    Reference in research papers: A reference is a detailed description of the source of information that you want to give credit to via a citation. The references in research papers are usually in the form of a list at the end of the paper. The essential difference between citations and references is that citations lead a reader to the source of ...

  20. Citation Styles Guide

    There are three main approaches: Parenthetical citations: You include identifying details of the source in parentheses in the text—usually the author's last name and the publication date, plus a page number if relevant ( author-date ). Sometimes the publication date is omitted ( author-page ). Numerical citations: You include a number in ...

  21. Guide & Samples for Writing a Bibliography of a Research Paper

    What Is a Bibliography in a Research Paper? Simply put, a bibliography is a list of works used in writing a research paper. Every research paper must contain a list of sources the author used in preparing the research paper. Your source can range from books to scholarly papers, speeches, private records, interviews, letters, websites, and other ...

  22. Author-date citation system

    For a work with one or two authors, include the author name(s) in every citation. For a work with three or more authors, include the name of only the first author plus "et al." in every citation (even the first citation). The following table shows the basic in-text citation styles:

  23. Sora: A Review on Background, Technology, Limitations, and

    Sora is a text-to-video generative AI model, released by OpenAI in February 2024. The model is trained to generate videos of realistic or imaginative scenes from text instructions and show potential in simulating the physical world. Based on public technical reports and reverse engineering, this paper presents a comprehensive review of the model's background, related technologies, applications ...

  24. What Is an Annotated Bibliography?

    An annotated bibliography is a list of source references that includes a short descriptive text (an annotation) for each source. It may be assigned as part of the research process for a paper, or as an individual assignment to gather and read relevant sources on a topic. Scribbr's free Citation Generator allows you to easily create and manage ...

  25. Journal of Medical Internet Research

    Digital therapeutics (DTx) are a promising way to provide safe, effective, accessible, sustainable, scalable, and equitable approaches to advance individual and population health. However, developing and deploying DTx is inherently complex in that DTx includes multiple interacting components, such as tools to support activities like medication adherence, health behavior goal-setting or self ...

  26. How to Cite Sources

    At college level, you must properly cite your sources in all essays, research papers, and other academic texts (except exams and in-class exercises). Add a citation whenever you quote, paraphrase, or summarize information or ideas from a source. You should also give full source details in a bibliography or reference list at the end of your text.

  27. 2023 NIST Center for Neutron Research Accomplishments and Opportunities

    A locked padlock) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.