https://web.stanford.edu/~mgens/2022-09-10T00:00:00-07:002022-09-10T00:00:00-07:002022-09-10T00:00:00-07:00Michael Gensheimertag:web.stanford.edu,2022-09-10:/~mgens/trial-results-change-over-time.html

<p>I recently read the long-term follow-up results from the NLST lung cancer screening trial, which got me thinking about how trial results can change over time as the initial benefits of an intervention wear off. To explore this idea, I created a simple simulation to show how the efficacy signal …</p>

<p>I recently read the long-term follow-up results from the NLST lung cancer screening trial, which got me thinking about how trial results can change over time as the initial benefits of an intervention wear off. To explore this idea, I created a simple simulation to show how the efficacy signal can get diluted as more years of data are added. In the original NLST study, screening with chest CTs showed clear benefits for lung cancer survival after 3 years, but after 12 years of follow-up, the difference in survival between the screened and unscreened groups was no longer statistically significant. My simulation used a toy example with a blood sugar-lowering medication to demonstrate how this can happen.</p> <p>See the full details on Google Colab: <a href="https://colab.research.google.com/drive/1sXvXmcJtGp_T2JgSY1ckiq7xk_xGFdgt?usp=sharing">link</a></p>2020-06-30T00:00:00-07:002020-06-30T00:00:00-07:00Michael Gensheimertag:web.stanford.edu,2020-06-30:/~mgens/change-in-patient-population-can-affect-auc.html

<p>I made a demo showing how changes in a patient population can affect the performance of a biomarker or classifier, specifically measured by AUC (Area Under the Receiver Operating Characteristic Curve). Using a simulated dataset, I first showed that hemoglobin A1c had an AUC of 0.77 for predicting foot …</p>

<p>I made a demo showing how changes in a patient population can affect the performance of a biomarker or classifier, specifically measured by AUC (Area Under the Receiver Operating Characteristic Curve). Using a simulated dataset, I first showed that hemoglobin A1c had an AUC of 0.77 for predicting foot ulcers in a population of diabetic patients. But when I added 1,000 non-diabetic patients who didn’t develop ulcers, the AUC jumped to 0.93. This illustrates how a more diverse or broader patient population can artificially inflate performance metrics, highlighting the need for careful evaluation of AUC when comparing models across different datasets.</p> <p>See the full details on Google Colab: <a href="https://colab.research.google.com/drive/1akAk76Fi3wyOoXptHQks8owb7rQMvBvQ?usp=sharing">link</a></p>