@shuvayan Wouldn’t that defeat the purpose? You would be using the test set to calibrate the model, which will then be used to predict test set values again. I think it would be better to split the training set into a smaller train and test/CV set. To get a variety of data, one way is to split the train set into smaller bits, keeping one of them as a test set, and merge the rest as the new training set. Repeat this with keeping another of smaller sets as test/CV, and so on. The exact proportions of the smaller sets will, of course, depend on the original size of the training set. If you can measure the AUC of these different models, you would get an idea of its efficacy by taking the average AUC, for example.
This something I learnt while participating in a EdX/Kaggle contest. Please let me know if I’m mistaken about this process.