How important understanding the evaluation metric in data science competitions?

kaggle
evaluation_metric

#1

Dear all,

Should one understand the evalution metric given as part of any data science competition in order to perform better in the competitions? How much important it is? Does understanding it will reduce the time to achieve good accuracy?

Thank you.

Regards,
Karthikeyan P


#2

@karthe1

Very good question! I would recommend understanding the evaluation metric in any competition as early as possible. Why? Because, if you are competing in any competition, ranks usually go out for a few points. In these scenarios, you can improve your score by understanding how the metric works.

For example, if a competition uses RMSE as a parameter to judge the competition, you can be very significantly affected by the outliers. So, you should be extra careful to make sure you are not making gross assumptions about outliers. Similarly, there are a few metrics, which get affected by balancing your data, but others which remain unaffected. This would directly impact the way you build your models.

As per my experience, you get get into top 30% of a competition by applying generic techniques and not having a deep understanding of evaluation metric. In order to move above that, you would need to understand how it works.

@srk and @tavish_srivastava an add more perspective on this

Hope this helps!

Kunal


#3

I second @kunal that it is very important to understand the evaluation metric to get a better rank in the competition. Understanding the nuances of the metric and accordingly correcting the methodology is very important.

As mentioned by Kunal, RMSE is affected by outliers and so removing them is important when the competition metric is RMSE. One more example is that, when the competition metric is RMSLE (Root Mean Square Logarithmic Error), using the log(response variable) for training the models generally gives better results than training directly on the response variable. This is because RMLSE mmetric gives more importance to smaller values as well but most of the learning algorithms we use, optimize for the mean square error and therefore gives more importance to higher values than smaller values. Therefore by using log(response variable) we are scaling down the range of response variable to get a better result.

Hope this helps!

Thanks,
Sudalai @ SRK


#4

Thank you @kunal sir,
Thanks a ton @SRK sudalai