Bias and Variance - Debugging Learning Algorithms

Update: I completed the course on 12/29/2013.)

I have been taking the Machine Learning class on Coursera taught by Andrew Ng. This class is amazing! Thank you Patrick for telling me about this course. I'm adding Machine Learning to the list of things that should be taught in high school. I now really know what a neural network is. Getting back on topic, when building a learning algorithm you can run into problems where the algorithm doesn't work very well. There are quite a few things you can do to determine how you can modify the algorithm to get more optimal output.

We generalize the two problems with an algorithm as bias (aka underfitting) and variance (aka overfitting).

1. Get more training examples -> fixes high variance
2. Try smaller set of features -> fixes high variance
3. Try getting additional features -> fixes high bias
4. Try adding polynomial features -> fixes high bias
5. Try decreasing lambda -> fixes high bias
6. Try increasing lambda -> fixes high variance