Bias-Variance Tradeoff

Ajeet Acharya

March 22, 2024

To discuss what Bias-Variance trade off is, First Let’s see what Bias and Variance are:

Bias

Bias is something that describes the performance of a Machine Learning Model by comparing it’s performance with the training set. It is described in terms of training set prediction error. There are two states of bias:

• High Bias: High bias is when the model has high training error and is unable to capture the patterns and important features within the training set. It is often called as underfitting.
• Low Bias: Low bias is when a machine learning model effectively captures the patterns and relation between features and has low training error.

Variance

Variance is concerned with the performance of a machine learning model on testing set.Variance is described in terms of test set prediction error. There are two states of variance:

• High Variance: When a Machine Learning model is unable to fit properly on new and unseen dataset but fits very well on training set it is said to have High Variance. The model having high variance is over sensitive to data in the training set i.e. when the data in training set is slightly changed, the output given by the model will differ significantly. High variance often leads to overfitting in our model.
• Low Variance: A model with low variance is not over sensitive to data and when it is fed with new and unseen related data, the model is able to produce good output with good prediction.

Bias-Variance Trade off

Credit: Google Images

The given figure shows the plot between model complexity and error. When we keep on increasing the model complexity i.e. more parameters, the bias will increase and the variance will decrease. When the model complexity is low(simple model) then there is high bias. On On increasing the model complexity bias decreases but variance increases. The zone where model complexity is low has high variance and low bias (Underfitting) and the zone where the complexity is high has low bias and high variance (Overfitting). We need to find a perfect and optimal solution where both bias and variance are low which also gives us minimal error. This is called as bias-variance trade off. We have to trade off low bias for obtaining low variance or trade off low variance for obtaining low bias based on the type of problem we are trying to solve.