Plausible error in RMSE calculation by server

big_mart_sales

#1

I just want to point out that there is something wrong with the way the RMSE is being calculated. Initially when I made my model I forgot to impute my test set.Hence a lot of NAs were present in my prediction.However to my surprise the RMSE was still calculated by the server. I then corrected my mistake and ran my model (in which the predictions DID NOT have any NAs) . My RMSE score was around 1148 . I then tried to find out what was wrong. What I found was that the RMSE value kept decreasing as I kept on deliberately introducing NAs into my predictions. In the extreme case i set all my predictions as NA except the last one and got a score of 155 . I would suggest everyone else taking part to try this out as well (It does not work if all predictions are set to NA), but even if you introduce 1000 NAs your score will improve drastically.

I just wanted to bring this to everyone’s attention. My score is not 155 but its much closer to 1148


#2

@Sunil- Have look to the problem


#3

There is a problem with the RMSE calculation on the server.

I submitted the first solution using OLS, then using RandomForest, but my score and rank remain the same.

Just to check, I then entered 1500 for ALL the predicted values and submitted. Still same rank.

Please check.