Pre Post Analysis using A/B Testing

descriptivestats

#1

Hi, I am working as a business analyst. As a part of my job, I have to do pre/post analysis, a lot of times.
For ex: on one page of the website, there will be some changes and then we have to see what is the impact of that change? We try to do a simple pre/post, where we look at page performance and overall performance of the website but more often than not, we do not get any conclusive answer. Does anyone know how to do pre/post analysis in the best possible way?


#2

Perform hypothesis testing. If your performance is measured as page load time or something that you are going to take multiple samples and figure out a mean, perform the hypothesis testing to see if that mean value is same for pre and post changes. Easiest way to extrapolate.

And can you explain how you do the simple pre/post test. Most often your statistical approach will be in line with your simple test you do by intuition or measuring some value. It just gives to solid mathematical evidence over intuition in most cases.


#3

Hypothesis testing is good. But there is a problem here:

Thanks for the response.

  1. Sometimes business will go down because of seasonality from pre to post
    or normal fluctuations. For example, lets say my revenue was 1 million pre
    change and post change it became, 90 K. This does not necessarily mean that
    the feature which was launched is bad. It might just be a normal cyclical
    thing for business.

How should we solve for this?


#4

If its seasonal, you need to do the same testing against a similar seasonal data. For example Q1 is the 1M revenue per change cycle and Q2 is 90k Reveneue cycle.

You test your hypothesis for Q1 against Q1 historic data value(say 1M)
You test your hypothesis for Q2 against Q2 historic data value(say 90K)

This can be done when you know the seasonality


#5

You can perform a regression analysis using, among the predictors, indicator variables for the known seasonality you mention (month, semesters, etc.) and test the equality of the associated coefficients. In addition, you need another variable indicating where the changes in the webpage were done. Testing this last coefficient gives you the answer if changes in the webpage were significative.

Cheers.