How can you choose variations for several content experiments?

Tag: google-analytics , content-experiments Author: sheepxdm Date: 2013-04-07

I'm using the Javascript API for Google Analytics Content Experiments and it seems like you have two choices:

  1. Run one experiment at a time and have Google assign the variation, which you can get using chooseVariation(), or
  2. Run several experiments at a time and assign the variation yourself using setChosenVariation()

(1) lets Google control the number of users assigned to each variation, which I need, but I have several experiments. This seems pretty basic. What am I missing?

Are you trying to achieve anything in particular ? Or just curious about what is possible using the GA Content Experiment JS API ?
@nt_1 My boss has a dozen experiments he wants to run, many simultaneously. It'd be better if we could control those experiments from GA.

Best Answer

I don't think you're missing anything. My understanding is that the GA Experiments API allows you to allocate a visitor to any version of a single A/B/N test with the Choose Variation command :

cxApi.chooseVariation();

However, this function is only available if you specify the experiment ID when loading the api.js on the page.

If what you want to do is to run several A/B tests on one page at the same time, this is really multi-variate testing and unfortunately this is not available 'out-of-the-box' in GA Content Experiments (it used to be available with Google Website Optimizer). You can still 'fake' it using your own bucketing, with code like :

<!-- 1. Load the Content Experiments JavaScript Client -->
<script src="//www.google-analytics.com/cx/api.js"></script>

<!-- 2. See if the visitor has seen an experiment already -->
<script>
// loop through all active experiments server-side. 
cxApi.getChosenVariation(
 $experimentId
 );

<!-- 3. Bucket the visitor if he isn't bucketed already -->
// loop through your live experiments 
  cxApi.setChosenVariation(
    $chosenVariation,             // The index of the variation shown to the visitor
    $experimentId                 // The id of the experiment the user has been exposed to
  );
</script>

You will need to decide whether you want a visitor to only view one experiment at a time or enter multiple experiments concurrently.

The first option is probably more reliable but means you will have to split your visitor in quite a lot of buckets, meaning your tests will take a long time before returning a result.

If you use the second option, you need to be careful : if your experiments are not independent, you won't be able to trust the test results from Google Analytics, as the statistical tests used in the Content Experiments assume independence. Even if your experiments are independent, you'll need to have an equal split among all variation combinations.

I hope that helps !

comments:

Thank you, it's exactly what I am doing, but I felt foolish because it seemed like I was re-inventing the wheel. We have been doing a crude multi-variate testing: assigning equal splits at random, and only testing (apparently) unrelated parts of the UI.

Other Answer1

when running many simultaneously experiments the problem is that if you have visitors from one experiment that interact with elements of another experiment running on your site then it is difficult to account for these interactive effects. This will skew your data and lead to some faulty conclusions.if you’re absolutely confident in how to analyze multiple concurrent experiments, it’s okay be safe and run one experiment at a time.

comments:

Not looking for an explanation of why I shouldn't want to do what I in fact want to do, but thanks.

Other Answer2

I solved this using a node.js script.

out-of-the-box Google makes you choose between: - load cxApi patched only for 1 experiment - load cxApi plain, and loose the ability to call chooseVariation() function, making you implement your own choosing algorithm (which is a shame, as google's multi-armed bandit idea is really smart).

So I did some reverse engineering on how google cxApi js is patched for 1 experiment, it turns it out it just bundles the experiment info (the variations weights) in the middle of the file, I experimented touching the bundle manually adding multiple experiments, and now I can call cxApi.chooseVariation(experimentId) for each experiment, and works nicely!

Weights change every 12hs, so I automated it, I created a node.js little app that will download cxApi from google, and then repeatedly download it for each experiment you specify, and patch a cxApi with all your experiments on it.

Voila!, multiple experiments on any page.

Here's the repo: https://github.com/benjamine/mutagen, fork at will

this node.js app will patch cxApi for multiple experiments, and will also pack (and minify) your variations