I participated in datahack Knocktober 2016 and ended with public score 0.82(rank 14). However, my private score dropped to 0.73(rank 51). This was a kind of shock to me. I didn’t expect this much difference. My model is a simple ensemble of 1 random forest and 7 XGB models. I started practicing ML from few months. I would like to hear best ways from experts to cross check this kind of overfitting to public leader board. Thanks in advance.
A solid cross-validation on your end is what you need. Most often the Leaderboard (LB) is misleading.
In case where time factor is involved, it is best that you keep a hold out set and check your model on that to know the performance.
In most competitions having time factor, the drop in score is always there but most of the seasoned data-scientists keep that in control.
I also experienced a large drop-off in scoring between the public and private datasets. This was my first hackathon to participate in and I too was surprised by the drop. I cross-validated my model with a 5 folds and saw relatively stable accuracy and precision metrics between them.
Can you elaborate on what you mean by time factor and how that impacts the dropping of the score between the public and private leaderboards? I don’t follow what you are saying.