Suggested textbook sections are listed by topic, in the order of the lecture plan below. The readings are organized by topic rather than lecture date because the schedule is tentative.
Inference; Hypothesis Testing; MAP and ML
Gallager, §§8.1–8.2
Gaussian Hypothesis Testing; Error Probability
Gallager, §§8.2.2–8.2.3 (Gaussian detection), §8.4 (error curves)
Minimum-Distance Decoding
Related reading: Gallager, §8.5 (multiple-hypothesis detection); see lecture material for minimum-distance decoding
Detection with Multiple Observations
Gallager, §8.2.3 (vector observations)
Machine Learning; Hypothesis Classes and Loss Functions; Empirical Risk Minimization
UML, §§2.1–2.3
Generalization Gap
UML, §2.3.1
Weak LLN
Gallager, §1.7.1
Markov's Inequality; Chebyshev's Inequality
Gallager, §§1.6.1–1.6.2; Vershynin, §1.2
CLT
Gallager, §1.7.3; Vershynin, §1.3
Hoeffding's Inequality
Vershynin, §2.2; UML, Appendix B.4
Generalization Bounds; Uniform Convergence
UML, §2.3.1 and §§4.1–4.2
Approximation Error and Excess Risk
UML, §5.2 (error decomposition; excess risk is termed estimation error)
Random Vectors; Covariance Matrices; Linear Transforms of Random Vectors
Related reading: Vershynin, §3.2 (covariance and linear transformations); Gallager, §§3.3–3.4 (Gaussian examples and covariance)
Gaussian Random Vectors; Gaussian Densities and Degeneracy; Affine Transformations of Gaussian Vectors; Independence of Gaussian Components
Gallager, §§3.3–3.4; Vershynin, §3.3.2
Eigenvalues and Eigenvectors; Spectral Decomposition
Gallager, §3.4.1; Vershynin, §3.2
Principal Component Analysis; Dimensionality Reduction
UML, §23.1; Vershynin, §3.2.1 (PCA background)
Expectation, Variance, and Covariance; Correlation
Related reading: Gallager, §1.5 (expectation and probability review)
Conditional Expectation; Law of Iterated Expectations
Related reading: Gallager, §1.5.2 (conditional expectation)
Conditional Variance; Law of Total Variance
Lecture material
MMSE Estimation
Gallager, §10.1.1 (squared-error estimation)
Properties of the MMSE Estimator; Inner Product Spaces of Random Variables; Orthogonal Projection; Orthogonality Principle
Gallager, §10.1.1 (squared-error estimation), §10.6 (geometry and orthogonality)
Linear MMSE Estimation
Gallager, §10.3
Vector Linear MMSE Estimation
Gallager, §10.3 (linear MMSE estimation); related reading: §10.4 (vector signal estimation)
Error Covariance Matrices
Related reading: Gallager, §10.4 (vector signal estimation); see lecture material
Gaussian Conditioning
Gallager, §3.5
Gaussian Estimation
Gallager, §10.2
Innovations
Related reading: Gallager, §§10.2.1, 10.4.3 (iterative estimation); see lecture material for innovations
State-Space Models
Related reading: Gallager, §10.2.2 and §10.4.4 (Kalman filter models)
Kalman Filtering; State Prediction and Filtering; Kalman Gain; Error Covariance Recursions
Gallager, §10.2.2 (scalar), §10.4.4 (vector)
Random Processes; IID Processes
Gallager, §1.4 and the opening of §3.6 (random processes); related reading: §1.4.1 (Bernoulli processes as an IID example)
Random Walks
Gallager, §9.1 (random walks)
Markov Processes
Related reading: Gallager, §4.1 (finite-state Markov chains), §7.1 (countable-state Markov processes)
Processes with Independent Increments
Related reading: Gallager, §2.2 (Poisson-process example); see lecture material for the general definition
Counting Processes
Related reading: Gallager, §2.1.1 (arrival processes)
Poisson Limit Theorem
Related reading: Gallager, §2.2.5 (Poisson limits of Bernoulli processes); see lecture material for the binomial-to-Poisson limit
Poisson Processes
Gallager, §2.2
Mean and Autocorrelation Functions
Related reading: Gallager, the opening of §3.6 and §3.6.1 (means and covariance functions); see lecture material for autocorrelation functions
Stationarity
Gallager, §3.6.1
Gaussian Processes
Gallager, the opening of §3.6
Wide-Sense Stationarity (WSS)
Gallager, §3.6.1
Power Spectral Density
Gallager, §§3.6.6–3.6.7
Cross-Correlation
Lecture material
Cross-Spectral Density
Lecture material
Linear Systems with Random Process Inputs
Gallager, §§3.6.6–3.6.7
White Noise
Gallager, §3.6.8 (white Gaussian noise)
Linear MMSE Estimation of Random Processes
Related reading: Gallager, §10.3 (finite-vector estimation)
Wiener Filtering; Spectral Factorization; Causal Wiener Filtering
Lecture material
Lookahead; Linear Prediction; Prediction in Practice
Lecture material
Lecture topics, pacing, and teaching assignments are tentative and subject to change. Lectures meet on Tuesdays and Thursdays. Lecture numbers include the midterms and Democracy Day. University holidays and recess dates follow the Stanford academic calendar.
Lecture 1 (September 22): Inference; Hypothesis Testing; MAP and ML — Professor Ayfer Özgür
Lecture 2 (September 24): Gaussian Hypothesis Testing; Error Probability; Minimum-Distance Decoding; Detection with Multiple Observations — Professor Ayfer Özgür
Lecture 3 (September 29): Machine Learning; Hypothesis Classes and Loss Functions; Empirical Risk Minimization; Generalization Gap; Weak LLN — Professor Ayfer Özgür
Lecture 4 (October 1): Markov's Inequality; Chebyshev's Inequality; Weak LLN; CLT; Hoeffding's Inequality; Generalization Bounds; Uniform Convergence — Professor Ayfer Özgür
Lecture 5 (October 6): Uniform Convergence; Generalization Bounds; Approximation Error and Excess Risk; Random Vectors; Covariance Matrices; Linear Transforms of Random Vectors — Professor Ayfer Özgür
Lecture 6 (October 8): Gaussian Random Vectors; Gaussian Densities and Degeneracy; Affine Transformations of Gaussian Vectors; Independence of Gaussian Components — Professor Ayfer Özgür
Lecture 7 (October 13): Covariance Matrices; Eigenvalues and Eigenvectors; Spectral Decomposition; Principal Component Analysis; Dimensionality Reduction — Professor Ayfer Özgür
Lecture 8 (October 15): Midterm I (in class during class time; covers the first three weeks)
Lecture 9 (October 20): Expectation, Variance, and Covariance; Correlation; Conditional Expectation; Law of Iterated Expectations; Conditional Variance; Law of Total Variance; MMSE Estimation — Professor Tsachy Weissman
Lecture 10 (October 22): Properties of the MMSE Estimator; Inner Product Spaces of Random Variables; Orthogonal Projection; Orthogonality Principle; Linear MMSE Estimation — Professor Tsachy Weissman
Lecture 11 (October 27): Vector Linear MMSE Estimation — Professor Tsachy Weissman
Lecture 12 (October 29): Vector Linear MMSE Estimation; Error Covariance Matrices; Gaussian Conditioning; Gaussian Estimation; Innovations; State-Space Models — Professor Tsachy Weissman
Lecture 13 (November 3): Democracy Day (no classes)
Lecture 14 (November 5): Midterm II (in class during class time; covers Lectures 1–12, with emphasis on Lectures 7–12)
Lecture 15 (November 10): Kalman Filtering; State Prediction and Filtering; Innovations; Kalman Gain; Error Covariance Recursions — Professor Tsachy Weissman
Lecture 16 (November 12): Random Processes; IID Processes; Random Walks; Markov Processes; Processes with Independent Increments; Counting Processes — Professor Tsachy Weissman
Lecture 17 (November 17): Poisson Limit Theorem; Poisson Processes; Mean and Autocorrelation Functions; Stationarity — Professor Tsachy Weissman
Lecture 18 (November 19): Gaussian Processes; Wide-Sense Stationarity (WSS); Power Spectral Density — Professor Tsachy Weissman
Lecture 19 (December 1): Cross-Correlation; Cross-Spectral Density; Linear Systems with Random Process Inputs; White Noise; Linear MMSE Estimation of Random Processes; Wiener Filtering — Professor Tsachy Weissman
Further Topics (Schedule to Be Determined): Spectral Factorization; Causal Wiener Filtering; Lookahead; Linear Prediction
Lecture 20 (December 3): Course Wrap-Up and Prediction in Practice — Professor Ayfer Özgür