When we are doing Hyperparameter tuning in developing a model, what is the best way of tuning either using randomsearch or gridsearch? how to set the hyperparameters, if any other methods available?
Picking up random values is a quick way to select some values for your hyperparameters, but the results may not be very accurate using randomsearch. If you use gridsearch, you are likely to get better values for you hyperparameters, but it is slower process. So, if you are not worried about the computation cost, go for gridsearch, otherwise randomsearch can also be a good option.
Hyperparameters vary from model to model. Random forest will have different set of hyperparameters whereas XGBoost will have different set. Refer the articles given below to understand how tuning can be done for different models:
Lets take the case of a Random Forest Model. Taking the most basic parameters,we can tune for number of trees, max number of nodes, minimum number of elements in leaf node and variables to consider.
All the parameters mentioned are inherently related to the size of the data at hand.
For instance, if the data set has 20000 records, minimum number of elements in the leaf nodes can start from 1000 (5%).
Max nodes can vary from 4-7 depending upon how you want to grow the tree.More number of levels will mean less number of records in leaf nodes and vice versa.
Number of trees: They ca vary between 50 to 250 trees
Number of variable used to create tree: Standard for regression and classification problems. For regression it is sqrt of total variables and for classification it is 1/3 of total variables.
The point of the whole explanation is that the range of values over which hyper tuning is done is around standard values
I’m not sure if my method is the best or not but FWIW this is my approach.
Supposing you have some parameter say ZZ I will use a gridsearchCV using say zz = 1, 6 and 10 then see which gives the best results. If the answer lies at 6 I will run again using 6, 9 and 11 then if the answer lies at 6 again I will go again using 6, 7 and 8 And so on. It can be tedious but I find this constantly narrows the field. You can also do the same thing using 5 variables at a time depending on how quickly you can run it etc