What is the statistical significance ?
Z-score is a statistical measurement allowing to determine the validity of the results of a variation and if a variation is more or less efficient than the reference. For further information about z-score, you can read the Standard score article on Wikipedia.
For example, if the statistical significance of your test is 90%, it will mean that the variation has 90% chances to beat the original, but there is also a 10% risk to be wrong.
Let’s take the example below:
- Variation A – reference: 142,000 unique visitors tested, 3.52% conversion rate (5,000 sales)
- Variation B – tested: 216,000 unique visitors tested, 3.64% conversion rate (1,850 sales)
In this example, variation B is more efficient than variation A, but is your test significant enough? By calculating the statistical significance, we notice that variation B is more efficient than variation A and that significance is 96%: your A/B test is a success! You can now use variation B as reference.
Checking statistical significance
If you are using Google Analytics or Kameleoon as reporting tool, Kameleoon will calculate automatically the significance of your test.
In Google Analytics, you will only see the conversion rate for each goal, but you cannot know easily which variation is the most efficient.
Kameleoon calculates automatically the z-score, allowing you to check if your results are significant and if your variation is more or less efficient than its reference.
You can see these measurement in your personal space. To do this, you must be logged in on your personal space.
Once you are logged in, click on the "All tests" button.
Then, click on your test to see its statistical significance.
The results page allow you to see the performance of your tests. For each variation, you will find:
- The improvement rate
- The trust rate, or statistical significance
- The conversion rate
- The number of conversions
- The number of converted visits
- The amount of visits
You can also find the trust rate on the left of the page. In the example below, the trust rate is 99%.
Evolution and reliability
On the results page, click on the following icon to display the evolution curve of the trust rate:
This curve allows you to display the evolution of the trust rate for each of your variations, on the goal and the date range selected.
Once you have launched your test, your results might vary. This way, your trust rate – which is the probability for a variation to beat the original – will vary a lot at the beginning of your test. This means that the data cannot be used yet and that the test should continue running to gather more data.
When the curve is flat, it means the results of your test are stable and you can use them safely.
In our example, both curves are stabilized at almost 100%. It means that, for the selected foal, both variations are likely to beat the original.
The blue curve was stable before the orange one, which means that the data collected for this variation were reliable sooner.