Lecture 10 | Machine Learning (Stanford)

Lecture by Professor Andrew Ng for Machine Learning (CS 229) in the Stanford Computer Science department. Professor Ng continues his lecture on learning theory by discussing VC dimension and model selection.

This course provides a broad introduction to machine learning and statistical pattern recognition. Topics include supervised learning, unsupervised learning, learning theory, reinforcement learning and adaptive control. Recent applications of machine learning, such as to robotic control, data mining, autonomous navigation, bioinformatics, speech recognition, and text and web data processing are also discussed.

Complete Playlist for the Course:
http://www.youtube.com/view_play_list?p=A89DCFA6ADACE599

CS 229 Course Website:
http://www.stanford.edu/class/cs229/

Stanford University:
http://www.stanford.edu/

Stanford University Channel on YouTube:
http://www.youtube.com/stanford

Category:

Education

Tags:

• science
• math
• engineering
• computer
• technology
• robotics
• learning
• algorithm
• theory
• VC
• dimension
• model
• selection
• hypothesis
• class
• floating
• point
• number
• Vapnik
• Chervonenkis

[nghiaho12]

The first 40 min or so is about theoretical bounds for the number of training data required. The more interesting and practical stuff is after that, where he presents practical issues on applying classifiers to real world problems. Here is my quick summary:

40:10 - Model selection, avoiding under/over fitting, Hold out cross validation

44:40 - K-fold cross validation

53:25 - Rule of thumb for number of training set vs number of features, for logistic regression

54:30 - Feature selection

[VancouverData]

i love how everyone (in the comments) disappeared after the support vector lecture :)

[nghiaho12]

@darfunkelidas I was thinking of logistic regression and decision trees. Both are blazingly fast compared to SVM, to train and predict, and great for quick evaluation when you are frequently changing features :) Libsvm recommends you do cross-validation to find good parameters (C and gamma for Gaussian), which for me was painfully slow to watch. My data was in the 10,000s with dimensions around 12 maybe, don't have the dataset on my computer so can't give definitiive numbers.

[darfunkelidas]

@nghiaho12 I know, VC theory is very complex! SVM's are usually fast (depending on your speed standards!) but I guess you took time optimizing the parameters for the kernel and error penalty, etc? What was the size of your dataset? If its a large dataset then I guess it's "notmal" that its picking up lots of SV.

[nghiaho12]

@darfunkelidas I actually got a bit lost in the theoretical stuff, kinda dozed off so can't help you there :) As a side comment. I have used SVM briefly and have found them to be very slow in training and classification, using libsvm. It seems to pick way too many support vectors (in the 100s and 1000s) in general (at least for my problems), which slows down classification considerably. This issue wasn't discussed in the 2011 online Machine Learning course, which he should.

[darfunkelidas]

@nghiaho12 I don't get one thing. He sayd VC dimension bounds are lose so he doesn't use them for model selection. But then he says that SVMs are the best machine learning algorithm, when the very thing that is supposed to make them better is the fact that they are based on he idea of minimizing those theoretical bounds. Isn't that kind of contradictory?

[rahulsevakula31]

These lectures are excellent. I appreciate the clarity of his concepts.