In-time and Out-of-time validations

crossvalidation
python

#1

Hi…

What is in-time and out-of-time validations ? Are these validation techniques specific to different situations. What are the possible ways to perform in-time and out-of-time validations ? Explanation with example codes in R/Python will be helpful.
Thanks.


#2

Hi @shan4224

Let us take an example

Our training data looks like this

Our test data looks like this

Now, when we use the training model on the test data set, this is what happens

data1 <- data.frame(x=seq(from=1,to=100),y=c(seq(from=1,to=30),seq(from=60,to=90),seq(from=1,to=39)))
row.names(data1) <- NULL

# Now let us take the first 60 entries as our training set
# and take the last 40 entries as our test set
train <- data1[1:60,]
test <- data1[61:100,]

# Let us choose the simplest Linear Regression Model
model <- lm(y~x,data=train)

test$predictedY <- (test$x * model$coefficients[[1]]) + model$coefficients[[2]]

TEST 1

the RED Line is the actual result, the BLUE line is the predicted result

TEST 2
To resolve this, let us say we include 20 more entries from the test data into the train data

train <- data1[1:80,]
test <- data1[81:100,]

A bit better

Now let us exclude the initial 20 30 points from the training data set, then we might get something different. So you can see how the results are changing. It is because we chose the 80:100 points of the dataset as a test data set. This choice of a set of later data points as a testset is called out of time validation and helps reduce bias in comparison to the model when we include all available data as training data and use the model on production data

Let me know if this helped

Regards,
Anant


#3

Hi…

Thanks for the explanation. Its informative.
Just a query, how does In-time and Out-of-time validation differs.

Thanks and Regards,
Shan


#4

Hi @shan4224,

I will explain this with a small example.

Suppose you have built a predictive model for the events of Feb 2016 considering a historical data window. if you are trying to test the model consistency against the events of Mar 2016, then it would be considered Out-of-time validation.

Hope this helps.


#5

:slight_smile::clap: