Course Calendar

The planned course schedule is provided below.

Week 1

Jan 05

BackgroundCourse Introduction & Overview of Foundation Models

Lecture Material:

• Course Overview and Introduction
• Introduction (Pt. 2)

Recording:

• Link

Jan 07

BackgroundIntroduction to LLMs

Lecture Material:

• Intro to LLMs and agents

Suggested Reading:

• Monitoring AI-Modified Content at Scale
• Scaling Instruction-Finetuned Language Models
• Training Language Models to Follow Instructions
• Using ChatGPT to Facilitate Truly Informed Medical Consent

Recording:

• Link

Week 2

Jan 12

BackgroundAgentic Systems

Lecture Material:

• AI Agents for Accelerating Research and Discovery

Recording:

• Link

Jan 14

BackgroundEvaluation of LLMs and Agents, MedArena (Eric Wu)

Lecture Material:

• MedArena.ai: A Clinician-Centric Platform for Expert Preferences on Medical Queries

Recording:

• Link

Week 3

Jan 19

No Class - MLK Day

Jan 21

BackgroundEvaluation of LLMs (Pt. 2)

Lecture Material:

• Evaluation of LLMs

Suggested Reading:

• The Leaderboard Illusion
• MedVAL: Toward Expert-Level Medical Text Validation with Language Models
• A Survey on LLM-as-a-Judge

Recording:

• Link

Week 4

Jan 26

BackgroundRetrieval Augmented Generation, Mixture of Experts

Lecture Material:

• Improving LLM performance

Suggested Reading:

• Retrieval-Augmented Generation (RAG) in Healthcare: A Comprehensive Review

Recording:

• Link

Jan 28

BackgroundInference Scaling and Reasoning

Lecture Material:

• Improving LLMs: PEFT, Alignment, Inference Scaling and Reasoning

Suggested Reading:

• Review of Inference-Time Scaling Strategies: Reasoning, Search and RAG

Recording:

• Link

Week 5

Feb 02

BackgroundVision-Language Models (Maya Varma)

Lecture Material:

• Introduction to Vision-Language Models

Suggested Reading:

• Learning Transferable Visual Models From Natural Language Supervision
• Sigmoid Loss for Language Image Pre-Training
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings

Recording:

• Link

Feb 04

BackgroundMechanistic Interpretability of VLMs (Robbie Holland)

Lecture Material:

• Mechanistic Interpretability of Vision-Language Foundation Models

Recording:

• Link

Week 6

Feb 09

BackgroundGenerative VLMs for Vision

Lecture Material:

• Generative Vision-Language Models

Suggested Reading:

• High-Resolution Image Synthesis with Latent Diffusion Models
• MedVAE: Efficient Automated Interpretation of Medical Images with Large-Scale Generalizable Autoencoders
• A vision–language foundation model for the generation of realistic chest X-ray images

Recording:

• Link

Feb 11

BackgroundGenerative VLMs for Text

Lecture Material:

• VLMs for Text

Suggested Reading:

• Visual Instruction Tuning
• xGen-MM (BLIP-3): A Family of Open Large Multimodal Models
• A Vision-Language Foundation Model to Enhance Efficiency of Chest X-ray Interpretation

Recording:

• Link

Week 7

Feb 16

No Class - Presidents’ Day

Feb 18

BackgroundFoundation Models for Segmentation

Lecture Material:

• VLMs for Text (Pt. 2)
• Segmentation Models

Recording:

• Link

Week 8

Feb 23

ApplicationsRegulation and Deployment: EchoNet Case Study

Recording:

• Link

Feb 25

ApplicationsSleepFM: Multimodal Foundation Model of Sleep

Week 9

Mar 02

BackgroundAdvanced Architectures

Mar 04

BackgroundEMR Foundation Models (Jason Fries)

Week 10

Mar 09

OpportunitiesEthics of Foundation Models (Roxana Daneshjou)

Mar 11

Project Presentations DueFinal Presentations