What is the significance of degree of freedom?
I have seen that it is used at variety of places and have different meaning in all context.
I am confused about its application.
Please explain the need of it and how it impact the model output.
What is degree of freedom?
Definition: The degrees of freedom is the values in the final calculation of a statistic that are free to vary.
Example: If you know that mean of 4 numbers is 20, and two of those numbers are 40, 50. This means that if you can find out one of the remaining numbers (say x), you can tell with certainty that the last number (say y) will be
y = 20 - ((40 + 50 + x) / 3)
i.e. you have one degree of freedom
In Machine learning theory, the independent variables on which the target depends on are called the degrees of freedom.
Need of degree of freedom
It is important to know which variables does the model depend on. This can then be used for feature selection.
Impact of degrees of freedom on Machine learning model
There’s a bias-variance tradeoff between setting the degrees of freedom. The more variables you have, more complex your model is. But it tends to overfit. So there must be a balance in selecting the variables.