dstork@stanford.edu (496-5720)
Office: Packard 253
Office hour: Friday 4:00-5:00pm and by appointment
Chuck Lam
chucklam@stanford.edu
Office: Packard 107
Office hour: Monday 2:00-4:00pm
Tuesdays and Thursdays, 4:15 to 5:30pm, Gates B12.
Statistical learning and pattern classification covers the theory and heuristics of the most important and successful techniques in pattern classification and clustering, such as maximum-likelihood, Bayesian and Parzen window estimation, k-nearest-neighbor algorithm, Perceptron and multi-layer neural networks, hidden Markov models, Bayesian networks, and decision trees. The course also covers the foundations from statistics and computational learning theory, such as the bias-variance trade-off, theory of regularization as well as resampling techniques such as cross-validation, boosting and arcing. There are no pre-requisites, but multivariable calculus, linear algebra and basic probability theory will be used frequently. Students should have basic programming experience; there will be a short tutorial on MATLAB and sample code for the weekly programming assignments.
Pattern Classification (2nd ed) by Richard O. Duda, Peter E. Hart and David G. Stork (Wiley, 2001). This is available from Wiley or Amazon.com or of course the Stanford bookstore. Be sure that you have the third printing or later; you can check this at the bottom of the page facing the dedication page at the front. You will see 10 9 8 7 6 5 4 3 (for the third printing) or 10 9 8 7 6 5 4 3 2 (for the second printing).
There is MATLAB software and manual available online, as will be discussed.
Explain and explore the machine learning techniques that can be applied to problems in which random or statistical variation plays a role.
This is a 200-level course, aimed at advanced undergraduates and beginning graduate students.
Pre-requisites:
There are no formal pre-requisites, but high-school calculus as well as linear algebra and basic probability will be used regularly throughout the course. There will be an optional review session of this mathematics. Students should be proficient in basic programming in a language but all homeworks will be in MATLAB; no previous experience with MATLAB is required for this course. There will be an introduction to MATLAB and much software available online.
- Homeworks: 60%
- Final exam: 40%
TBD
A set of the figures from the book, powerpoint slides, and other material can be retrieved free of charge here.
Homeworks are due on Tuesdays before class. You may work together, but each student must write and hand in a separate homework set that represents his or her understanding of the work. In particular, it is not acceptable to hand in two or more copies of a single "master" solution, even if that master solution represents the work of more than one student. Likewise, while students can discuss issues related to software, all code must be written individually, and students may not copy or share code or hand in the same code or copies of the outputs. We will try our best to hand back graded homeworks on the following Thursday (that's why we have a stringent late policy); after that, the homeworks will be kept in a bin outside the TA's office.
Each student may hand in at most one homework late (by less than 24 hours) in the course. Otherwise a homework assignment will not be accepted. If you have a good reason why you cannot turn in your homework on time, please contact me as soon as possible. Don't assume that merely because you notified me that it is fine to hand in a homework late; you must receive confirmation ahead of time in order to avoid a penalty.
The final exam will be in class and closed book at a time to be scheduled by Stanford. I will put one homework problem on the test essentially verbatim, so be sure to study the homework solutions in preparation for the test.
All reading and homework problems are from the text, unless otherwise noted.
|
|
|
|
|
|
Tuesday: 4/1 4:15-5:30pm Gates B12 |
Chapter 1 (all) | Introduction, overview of the problems of statistical approaches to machine learning and pattern recognition; Bayes rule, discriminant functions. | |
|
Thursday: 4/3 4:15-5:30pm Gates B12 |
Chapter 2: Sects. 1--6 | Multi-dimensional Gaussians, error probabilities and bounds. | |
|
Friday: 4/4 4-5pm: Packard 253 |
Optional: Appendix A, Sections1, 2, 4, 5, 7 | Faculty office hour | |
|
Monday: 4/7 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 4/8 4:15-5:30pm Gates B12 |
Chapter 2: Sects. 7--12. | Chapter 2: 2, 8, 12 and MATLAB | Signal detection theory and ROC curves, missing features, Bayes belief nets. |
|
Thursday: 4/10 4:15-5:30pm Gates B12 |
Chapter 3: Sects. 1--5, 8, 9 |
Maximum-likelihood and Bayesian estimation. Issues in dimensionality and training data; component analysis, principal components, expectation-maximization (EM) algorithm. |
|
|
Friday: 4/11 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 4/14 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 4/15 4:15-5:30pm TCSeq201 Note change! |
Special lecture: Physics Colloquium |
Chapter 2: 30, 37, 41 Click here and solve this version of the problem. Chapter 2 Computer Exercise: 2 Chapter 3: 4 |
Special lecture: "Did the great masters 'cheat' using optics?" Physics Colloquium |
|
Thursday: 4/17 4:15-5:30pm Gates B12 |
Chapter 3: Sect. 9-10 | Principal component analysis, Expectation-Maximization; Hidden Markov Models (evaluation & decoding) | |
|
Friday: 4/18 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 4/21 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 4/22 4:15-5:30pm Gates B12 |
Chapter 4: Sects. 1--6 |
Chapter 3: 5, 8, 17, 34, Chapter 3 Computer Exercise: 3 |
Hidden Markov Models (learning); Parzen windows, k-nearest-neighbor algorithm |
|
Thursday: 4/24 4:15-5:30pm Gates B12 |
Chapter 5: Sects. 1--5 | Linear discriminants, Perceptron algorithm, relaxation procedures | |
|
Friday: 4/25 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 4/28 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 4/29 4:15-5:30pm Gates B12 |
Chapter 5: Sects. 6,7, 11 | Note: because of my mixup in postings, I've postponed the HW due date until Thursday. | Nearest-neighbor error limits; linear discriminants, three-layer neural networks, introduction to backpropagation |
|
Thursday: 5/1 4:15-5:30pm Gates B12 |
Chapter 6: Sects. 1--6 |
Chapter 4: 4, 7, 14, 21, 25 Chapter 4 Computer Exercise: 4 |
Backpropagation and practical tricks for enhancing training and classification |
|
Friday: 5/2 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 5/5 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 5/6 4:15-5:30pm Gates B12 |
Chapter 6: Sects. 8--11 |
Chapter 3 Computer Exercise: 13. You may use the MATLAB code provided in the course for this exercise. Note: change the sample 4 for ω1 from AD to ADB. Chapter 5: 5, 10 Chapter 5 Computer Exercise: 2 Chapter 6: 6 |
Second-order methods in three-layer networks, introduction to stochastic methods |
|
Thursday: 5/8 4:15-5:30pm Gates B12 |
Chapter 7: Sects. 1--3 | Boltzmann learning | |
|
Friday: 5/9 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 5/12 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 5/13 4:15-5:30pm Gates B12 |
Chapter 7: Sects. 5--6 Chapter 8: Sects. 1--3 |
Chapter 6: 9, 12, 23 Chapter 6 Computer Exercises: 4 and 9 |
Conclusion on stochastic methods; introduction to non-metric methods, tree-based classifiers and pruning |
|
Thursday: 5/15 4:15-5:30pm |
Chapter 8: Sects. 4--7 | Tree-based classifiers, string matching, | |
|
Friday: 5/16 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 5/19 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 5/20 4:15-5:30pm Gates B12 |
Chapter 9: Sects. 1--3 |
Chapter 7: 2, 6 Chapter 8: 6, 8, 15, 23 Chapter 8 Computer exercise: 3 |
Parsing, grammatical methods; introduction to theory of learning, bias-variance, no free lunch theorems |
|
Thursday: 5/22 4:15-5:30pm Gates B12 |
Chapter 9: Sects. 4--7 | Resampling, boosting, combining classifiers | |
|
Friday: 5/23 12:00 noon, Redwood Neurosciences Institute, 1010 el Camino Real (above Kepler's Books and Cafe Borrone), Menlo Park directions and further information |
Optional lecture by David G. Stork, "The Open Mind Initiative: Large-scale knowledge acquisition from non-experts via the web" | The Open Mind Initiative | |
|
Friday: 5/23 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 5/26 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 5/27 4:15-5:30pm Gates B12 |
Chapter 10: Sects. 1--4 |
Chapter 9: 9, 33, 39, 45 Chapter 9 Computer exercise: 4 |
Introduction to clustering, maximum likelihood, k-means clustering |
|
Thursday: 5/29 4:15-5:30pm Gates B12 |
Chapter 10: Sects. 5--8 | Self-organizing feature maps, Multidimensional scaling | |
|
Friday: 5/30 4-5pm Packard 253 |
Faculty office hour | ||
|
Monday: 6/2 2-4pm Packard 107 |
TA office hour | ||
|
Tuesday: 6/3 4:15-5:30pm Gates B12 |
Chapter 10: Sects. 9--13 |
Chapter 10: 7, 19, 23, 38 Chapter 10 Computer exercise: 3 |
Feature clustering for dimensionality reduction; course summary and exam review |
|
Thursday: 6/5 4:15-5:30pm Gates B12 |
Optional open question and answer period | ||
|
Tuesday: 6/10 12:15-3:15pm Gates B12 |
Exam formula sheet | Final Exam: Closed book except for exam formula sheet, which will be distributed with the exam. You may use a (numerical) calculator but no laptop or other computer or notes. |
