Our decision making process can be divided into the key options:

Training Examples

• Getting more training examples: Fixes high variance

Features

• Trying smaller sets of features: Fixes high variance
• Adding features: Fixes high bias
• Adding polynomial features: Fixes high bias

Lambda λ

• Decreasing λ: Fixes high bias
• Increasing λ: Fixes high variance.

Diagnosing Neural Networks

• A neural network with fewer parameters is prone to underfitting. It is also computationally cheaper.
• A large neural network with more parameters is prone to overfitting. It is also computationally expensive. In this case you can use regularization (increase λ) to address the overfitting.

Note: Using a single hidden layer is a good starting default. You can train your neural network on a number of hidden layers using your cross validation set. You can then select the one that performs best.

Model Complexity Effects

• Lower-order polynomials (low model complexity) have: high bias and low variance. In this case, the model fits poorly consistently.