Hi, I’m new and would be very grateful for confirmation or critic on my thoughts along with the answers to five basic queries below.
I’m starting a new role as Digital Campaign Executive and want to hit the ground running. I’m no statistician but I am familiar with basic campaign tests stats using excel but I’m a bit rusty.
This first post is about my usual practice of attempting to ensure that the randomly generated campaign test split groups (eg, Test & control populations) are not skewed for some identifiable ‘key’ variable/s. For example, age (where across the business it is established that younger targets are more likely to respond than older ones. Therefore a larger proportion of ‘younger’ targets in the test split than in the control split may skew the result.
In a past role this checking process was a simple T-test comparing the means (age in this example) of the two test splits. On occasion the means differed significantly and the null hypothesis, that both means were equal, was rejected and the splits re-run and these new splits re-tested. Once the null hypothesis could not be rejected the campaign random test cell population was accepted and the campaign set up continued through to final mailing/broadcast…
Is this process overkill or a sensible way to limit occasional skew in test cells
that random splits may introduce?
If Gender was a key variable how do I test the gender splits in a similar way? Gender is nominal data and T-tests need to use ordinal data, right?
T-tests are usually used for small sample sizes. In this case, 100% of the identified target audience (a subset of the database) is being selected and split into two smaller test & control groups. Therefore:
a- is the T-test still appropriate and
b- at what volume is it no longer a small sample size, 30+ in each split?
c- should the z-score be used if each split is GT 30 or is T-test still ok?
For a multivariate test, where there are more than just a control and a single test split, (i.e. more than two spits), is it best to use ANOVA to compare the means?
Finally, should I be aware of or make any other considerations?