Assignments

Reflections

The reflection assignment is submitted after each lecture, and should be ~1 paragraph with a maximum of 250 words in the body text. You are not required to have references, but feel free to include them if they support your arguments. References will not count towards the word limit.

Sample Reflection
Student Name: Bruce Banner PhD
Reflection Assignment no.: 1/10
Word Count: 168
Topic: Why Machine Learning in Healthcare

For decades there have been efforts to incorporate computer models into the practice of medicine but those efforts were limited by lack of digital data and computational resources. Now digital medical data has matured at a time when computational advancements and machine learning techniques power many aspects of our society. Similar to other use cases like finance and commerce, learning associations automatically from medical data, including unstructured data like images and text, is also possible with machine learning and deep learning models. A key insight is that it is possible for machine learning to achieve this from the data directly and without specific rules-based programming enabling rapid development of high performing applications across the healthcare landscape. However, biases can be introduced with these approaches that can result in useless or even harmful solutions and highlights the importance of interdisciplinary teams and that medical domain experts in particular understand the concepts and principles of machine learning approaches in order to safely develop, evaluate, and use these tools in healthcare

Scientific Paper Review


Preparation
As preparation for the scientific review assignment, you should first read the following two papers (accessible through Stanford network or Stanford Library). The paper by Liu et al. prepares you to read literature that uses Machine Learning for Healthcare and the paper by Budovec helps structure your scientific paper review.

Liu, Y., Chen, P.-H. C., Krause, J. & Peng, L. How to Read Articles That Use Machine Learning: Users’ Guides to the Medical Literature. JAMA 322, 1806 (2019). https://jamanetwork.com/journals/jama/article-abstract/2754798
Budovec JJ, Kahn CE Jr. Evidence-based radiology: a primer in reading scientific articles. AJR Am J Roentgenol. 2010;195(1):W1-W4. doi:10.2214/AJR.10.4696 https://www.ajronline.org/doi/pdf/10.2214/AJR.10.4696

Selecting a Clinical Machine Learning Paper
For your scientific paper review, you will select a paper of your own choice; perhaps in your own clinical field. You are free to select a paper that is published 2016 or later and is relevant, i.e. is centered around usage of Machine Learning for a Clinical Healthcare problem. You may only select papers published in peer-reviewed journals (not ArXiv or MedRxiv). The paper should be a research article (not review, editorial etc.)

We recommend looking in the following journals: Nature / Nature Medicine, JAMA / JAMA Network, New England Journal of Medicine, Radiology, Lancet / Lancet Digital Health, npj Digital Health

You can also use Doctor Penguin to find papers: http://doctorpenguin.com/index

Writting the Assignment
Selected paper: Indicate which paper you selected with a Nature style reference, and a link.
(10%) Summary
Write a short summary of the study in your own words (do not copy the abstract). The summary should be an objective summary of the research. Be sure to address the following items:
  • What was the study objective (1-2 sentences)
  • What was the study setup? In particular emphasize how AI was incorporated into the research and what kind of AI was used (4-6 sentences)
  • What were the main results (2-3 sentences)
  • What is the authors’ main conclusion (1 sentence)
(90%) Critical Appraisal
Critical appraisal is the systematic evaluation of scientific research papers. This section should identify strengths and weaknesses of a paper, while specifically addressing the below items.

We advise you to structure your appraisal around the usual chronology of scientific papers. The below outline is adapted from the paper by Budovec et al. with modifications that are relevant to AI papers. The bullet points below are meant to guide the content of each section; do not write your review in question/answer or bullet-point style:

  • (10%) Introduction
  • Does the study address a clearly focused research question, and how is the question clinically relevant?
  • What are the general issues surrounding the authors’ question?
  • How does the authors’ specific question fit into what is already known about the subject?
  • Do the authors build a logical case and context for their hypothesis?
  • (30%) Methods
  • How were the subjects/data selected.
  • Could the subject/data-selection have introduced any bias?
  • Was the usage and/or development of Artificial Intelligence scientifically sound?
  • Was the AI evaluated in an appropriate selection of patients? For example, was the AI tested in patients in whom it would be routinely used in clinical practice?
  • Was there a reference standard (ground truth), and if yes was the reference standard scientifically valid.
  • Was the AI validated on a second, independent dataset / group of patients?
  • (20%) Results
  • Do the results follow from the investigators methods? Is it clear where the results came from?
  • Are all the relevant results presented and clear?
  • Could the results have been from chance?
  • Are there any results or analysis missing that you would have liked to see?
  • (30%) Disucssion
  • Are the results of this study important? What clinical implications can the results have?
  • Are the results generalizable; i.e. applicable to any patient or population?
  • What conclusions did the authors draw from the data? Would you draw the same conclusions?
  • Do the authors acknowledge limitations of the study? Are there additional limitations that should be included?