Course Calendar (Winter 2024)

The planned course schedule is provided below. Deadlines for homework assignments and project deliverables will be updated soon.

Week 1

Jan 8

BackgroundCourse Overview & Introduction to Foundation Models

Lecture Material:

• Course Overview
• Applications of FMs in healthcare

Jan 10

BackgroundSelf-Supervised Learning for Computer Vision

Lecture Material:

• Computer Vision

Suggested Reading:

• A Simple Framework for Contrastive Learning of Visual Representations
• Masked Autoencoders Are Scalable Vision Learners

Coding Exercise:

• Autoencoder Demo

Week 2

Jan 17

BackgroundIntroduction to Large Language Models

Lecture Material:

• Intro to LLMs

Suggested Reading:

• InstructGPT

Week 3

Jan 22

BackgroundCurrent Status of Large Language Models

Lecture Material:

• Health LLMs

Suggested Reading:

• Med-PaLM
• PLIP

Jan 24

BackgroundVision-Language Models

Lecture Material:

• Intro to VLMs
• Vision-Language Methods

Suggested Reading:

• Learning Transferable Visual Models From Natural Language Supervision
• Reproducible scaling laws for contrastive language-image learning
• Contrastive Learning of Medical Visual Representations from Paired Images and Text

Additional Resources:

• OpenCLIP

Week 4

Jan 29

Clinical NeedsClinical Needs for Foundation Models in Healthcare

Lecture Material:

• Clinical Needs, Part 1
• Clinical Needs, Part 2

Suggested Reading:

• LLMs are few-shot clinical clinical information extractors
• Automated coronary calcium scoring
• LLMs in ophthalmology applications

Jan 31

Clinical NeedsDatasets for Building Foundation Models for Healthcare

Lecture Material:

• Datasets

Suggested Reading:

• LAION-5B
• MIMIC-IV
• SKM-TEA

Week 5

Feb 5

ApplicationsGenerative LLMs for Healthcare Applications

Lecture Material:

• LLM Frontiers

Suggested Reading:

• FrugalGPT
• LLM Negotiation

Feb 7

ApplicationsLLMs for Healthcare in Industry

Lecture Material:

• Applications in Industry

Suggested Reading:

• Retrieval-Based CXR Generation
• Improving Report Generation Systems
• Evaluating CXR interpretations

Week 6

Feb 12

New MethodsLightweight Fine-Tuning of LLMs

Lecture Material:

• LLM Finetuning

Suggested Reading:

• Clinical Text Summarization
• LoRA: Low-Rank Adaptation of LLMs

Feb 14

ApplicationsGenerative Vision-Language Models in Health

Lecture Material:

• Generative VLMs

Suggested Reading:

• Latent Diffusion Models
• RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

Feb 16

Homework #1 Due

Week 7

Feb 21

New Methods Guest Lecture (Michael Poli and Eric Nguyen): State Space Models for Healthcare

Suggested Reading:

• HyenaDNA

Week 8

Feb 26

ApplicationsGenerative AI for Drug Discovery

Lecture Material:

• AI for Drug Discovery

Suggested Reading:

• Multi-Objective GFlowNets

Feb 28

Applications Retrieval-Augmented Generation

Lecture Material:

• Retrieval-Augmented Generation

Suggested Reading:

• MuRAG: Multimodal Retrieval-Augmented Generator
• REALM: Retrieval-Augmented Language Model Pre-Training

Week 9

Mar 4

Opportunities Safety, Ethics, and Regulation

Lecture Material:

• LLM Safety

Mar 6

Opportunities Guest Lecture (Jason Fries): Foundation Models for EHR Data

Lecture Material:

• EHR Foundation Models

Week 10

Mar 11

Opportunities Deployment Considerations

Homework #2 Due

Lecture Material:

• Advanced Topics

Mar 13

Project Presentations DueFinal presentations

Week 11

Mar 18

Project Reports Due