The metric "Accuracy" was not in the result set. ROC will be used instead. # Parameter Tuning #Caret


#1

Here I m trying to do parameter tuning in xgboost using caret pacakge.I m getting a warning msg like this. Pls tell me what should i do so that i can remove this warning and i can get the accuracy of the model for the paramters selected.

ERROR :

In train.default(x = data.matrix(train), y = a, trControl = xgb_trcontrol_1, :
The metric “Accuracy” was not in the result set. ROC will be used instead.

Code :

Dataset : BNP paribas Kaggle Comp

library(caret)

train <- read.csv(“train.csv”)
test <- read.csv(“test.csv”)

train[is.na(train)] <- -1
test[is.na(test)] <- -1

train$target <- as.factor(train$target)

set up the cross-validated hyper-parameter search

xgb_grid_1 = expand.grid(
nrounds = 140,
eta = 0.2,
max_depth = c(6,8,10),
gamma = 0,
colsample_bytree = 0.8,

subsample = 0.8,

min_child_weight = c(1,6,2))

scale_pos_weight = 1)

pack the training control parameters

xgb_trcontrol_1 = trainControl(
method = “cv”,
number = 5,
verboseIter = TRUE,
returnData = FALSE,
returnResamp = “all”, # save losses across all models
classProbs = TRUE, # set to TRUE for AUC to be computed
summaryFunction = twoClassSummary,
allowParallel = TRUE
)

a<- make.names(train$target,unique=F)

train the model for each parameter combination in the grid,

using CV to evaluate

xgb_train_1 = train(
x = data.matrix(train),
y = a,
trControl = xgb_trcontrol_1,
tuneGrid = xgb_grid_1,
method = “xgbTree”
)

Result :

xgb_train_1

Aggregating results
Selecting tuning parameters
Fitting nrounds = 140, max_depth = 6, eta = 0.2, gamma = 0, colsample_bytree = 0.8, min_child_weight = 1 on full training set

Resampling results across tuning parameters:

max_depth min_child_weight ROC Sens Spec ROC SD Sens SD Spec SD
6 1 1 1 1 0 0 0
6 2 1 1 1 0 0 0
6 6 1 1 1 0 0 0
8 1 1 1 1 0 0 0
8 2 1 1 1 0 0 0
8 6 1 1 1 0 0 0
10 1 1 1 1 0 0 0
10 2 1 1 1 0 0 0
10 6 1 1 1 0 0 0

Everything is Zero :frowning:
I dont know were i m getting wrong. Pls help me out with the code

@aayushmnit @Lesaffrea @shuvayan @Aarshay @Manish

Thanks,
Rohit


#2

Hi @Rohit_Nair,

I guess your AUC is 1 because you have kept target variable in your training set too, try dropping it off and then run the code.

Hope this helps.

Regards,
Aayush


#3

Thanks alot @aayushmnit for spotting the error… Bang on target :stuck_out_tongue:

One more doubt.

When i try to do parameter tuning in xgboost using caret why does it not let me tune paramters like subsample,alpha ,lambda?

It gives an error :

Error in train.default(x = data.matrix(train), y = a, trControl = xgb_trcontrol_1, :
The tuning parameter grid should have columns nrounds, max_depth, eta, gamma, colsample_bytree, min_child_weight

Code :
xgb_grid_1 = expand.grid(
nrounds = 140,
eta = 0.2,
max_depth = 6,
gamma = 0,
colsample_bytree = 0.7,
subsample = 0.8,
min_child_weight = 1,
alpha =c(0.01,1,100,0.00001))

When i remove subsample.alpha the code runs fine.

Whats the issue can here?

@Lesaffrea @aayushmnit @Aarshay @shuvayan

Thanks,
Rohit


#4

Hi @Rohit_Nair,

It may be related to caret’s functionality , that’s how the package is designed it allows only certain parameters to be tuned. Although I don’t use caret these days so can’t be sure about it.

Regards,
Aayush


#5

hi @aayushmnit

Thanks for the rply… If u dont use caret then can u plz tell what do u use for parameter tuning in R for xgboost ? Or rather how do u do parameter tuning in xgboost ?

Thanks,
Rohit


#6

Hi @Rohit_Nair,

I use simple xgboost package of R, and to optimize u can read this post by binga.

Regards,
Aayush


#7

Hi @aayushmnit,

Thanks for the link. Very nice informative tips shared by binga :slight_smile:

Do u do gridsearch kindoff parameter tuning like the one I have done in caret ? or do u tune one parameter at a time ?

Can u share the code in R for xgboost you use for paramter tuning.

Thanks,
Rohit


#8

Hi @Rohit_Nair,

I do one at a time manual tuning, as suggested by post in Binga. I have no automated code for doing the same.

Regards,
Aayush