EECS 545: Machine Learning.

University of Michigan, Fall 2009

Instructor: Clayton Scott
Classroom: 1690 CSE
Time: MW 10:30-12
Office: 4433 EECS
Email:
Office hours: Monday 2-4 PM

GSI: Gowtham Bellala
Uniqname: gowtham
Office hours: TBA.

Final Projects from Fall 2007

Required text: None.

Recommended texts:

• Duda, Hart, and Stork, Pattern Classification, Wiley, 2001
• Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer, 2001
• Bishop, Pattern Recognition and Machine Learning, Springer, 2006
• Sutton and Barto, Reinforcement Learning: An Introduction, MIT Press, 1998

Additional references

• Tan, Steinback, and Kumar, Introduction to Data Mining, Addison-Wesley, 2005.
• Scholkopf and Smola, Learning with Kernels, MIT Press, 2002
• Mardia, Kent, and Bibby, Multivariate Analysis, Academic Press, 1979 (good for PCA, MDS, and factor analysis).
• Rasmussen and Williams, Gaussian Processes for Machine Learning, MIT Press, 2006.
• Boyd and Vandenberghe, Convex Optimization, Cambridge University Press, 2004
• MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003

Machine learning bibliography

Prerequisites: (the current formal prerequisite is currently listed as EECS 492, Artificial Intelligence, but this is inaccurate)

• Probability: random variables, densities and mass functions, expectation, independence, conditional distributions, Bayes rule, maximum likelihood estimation, the multivariate normal distribution
• Linear algebra: rank, nullity, linear independence, inner products, orthogonality, positive (semi) definite matrices, eigenvalue decompositions, least squares, pseudo-inverses, projections.

Topics roughly in order (# in parens = estimated # of lectures):

• (4) Linear methods: Least squares regression, principal component analysis (PCA), linear discriminant analysis (LDA), logistic regression, max-margin linear classifiers
• (4) Kernel methods: smoothing kernels, kernel density estimation, kernel ridge regression, support vector machines
• (1) Model selection and error estimation: cross-validation, bootstrap
• (1) Boosting
• (2) Clustering: K-means; the EM algorithm for Gaussian mixture models
• (3) Reinforcement learning: Markov decision processes, policy evaluation, optimal control, function approximation and RL
• ------ The exam will cover everything above --------
• (2) Trees: classification and regression trees, hierarchical clustering
• (2) Ensemble methods: bagging, random forests
• (2) Nonlinear dimensionality reduction and Euclidean embedding: multi-dimensional scaling, Isomap, graph drawing, kernel PCA
• (2) Spectral embedding and clustering
• (1) Feature selection and the curse of dimensionality
• (1) Gaussian processes for regression
• Additional topics as time permits (suggestions welcomed): Generalized linear models, learning theory, online learning, neural networks, etc.

Lecture notes:

• Statistical machine learning
• Least squares linear regression
• Principal component analysis
• The Bayes classifier
• Linear discriminant analysis
• Logistic regression
• Naive Bayes
• Separating hyperplanes
• Kernel density estimation
• Locally linear regression
• Regularization
• Inner product kernels and the kernel trick
• Kernel ridge regression
• Constrained optimization
• Support vector machines
• K-means clustering
• The EM algorithm for Gaussian mixture models

Grading:
Homework: 35%
Exam: 10%, Thursday Nov. 12, 6-9 PM, location TBA
Participation and attendance: 5%
Final project: 50%

Homeworks:
Homework will be assigned every one or two weeks. The assignments will be much smaller and easier toward the end of the course, when you are working on your project.

Computer programming
Most or all assignments will involve some computer programming. MATLAB will serve as the official programming language of the course. You are free to use another language, such as R, but I will sometimes provide you with fragments of code, or suggested commands, in MATLAB.

Group work:
Group work will take place on two levels. You will work on homeworks in small groups of 2, and the final project in large groups of 3 or 4. I will help you find groups as needed.

Exam: Time and location TBD.
Collaboration of any form will not be allowed. Allowed materials will be specified in advance of the exam.

Final Project:
There will be a final, open-ended group project. The project must explore a methodology or application (and preferably both) not covered in the lectures. The work must not simply reproduce the results of a paper, but explore some new aspect of a problem. I will assist groups in selecting a topic as much as necessary. The project will be judged based on clarity, thoroughness, and originality. The project will be graded based on the following components:

• Project proposal (10%): Due date TBA. 2 pages max. Each group must meet with me in advance of the proposal deadline to discuss the project.
• Progress report (20%): Due Nov. 25 in class. 5 pages max.
• Final report (35%): Due Thursday, Dec. 17, at noon. 10 to 12 pages.
• Poster presentation (35%): Saturday, Dec 19, 1-5 PM, Duderstadt center room 1180.

Collaboration:
Each group will turn in one product representative of the group. Solutions to homework problems from outside sources may not be used.

Honor Code
All undergraduate and graduate students are expected to abide by the College of Engineering Honor Code as stated in the Student Handbook and the Honor Code Pamphlet.

Students with Disabilities
Any student with a documented disability needing academic adjustments or accommodations is requested to speak with me during the first two weeks of class. All discussions will remain confidential.