How to gather learnings from your marketing tests & make data-backed decisions?

Testing is crucial for the growth of any business. It has been proven over & over again that testing is correlated with exponential growth.

However, most companies that have acknowledged that, still run into a lot of problems when it comes to testing & often end up frustrated, because they don’t see the growth that they had expected… 

The reason for this? Their testing process & methodology is flawed which inevitably sets all their tests up for failure. 

The reason why you want to run tests for your business is not because of testing’s sake, but to know more about your customer & thoroughly understand what they want to engage with.

Yes, testing is important, but you first need to make sure that you have implemented a process that lets you gather learnings from your tests, so you can make data-backed decisions for your company and project, because this is what ultimately distinguishes a great from a good company: data-backed decision-making.

Common mistakes when it comes to testing… and actually gathering learnings from it

At first glance, starting to test more within your business, project or campaign might sound like the simplest thing to do, right?
Just launch a variant to your control and see how it performs and if it outperforms your control variant, then you will just go with that variant from then on… well, it’s not that easy.

This is the best possible scenario, but still what have you then learnt from this test? Why is your customer more likely to engage with your variant then with your control? And why hasn’t the control performed better? What could be the reason for this?

As you can see there are a lot of questions left unanswered and although you might be off with a slightly better performing landing page or Facebook ad creative, you don’t know how you can drill down on that & build off of your successes. 

So, let’s talk about the most common mistakes when it comes to testing: 

  1. Not laying out your tests
    People tend to think that the most important part in running a test is actually running it and launching the control. Well, although this obviously is a crucial component of running your test, it is not the most important.

    The most important part of a test is probably the preparation. Meaning, when you run a test you always do that for a specific reason. As I said, you never run a test just for testing’s sake rather than to answer a question that you have in regards to your consumer behaviour.

    Meaning, you don’t just want to launch random tests rather than form questions, or also known as hypotheses, to why something might not be performing.

    For example: “The average watch time of the video ad that I’m running on Facebook right now is 4 seconds. However, we actually have our most important value prop at second 8. So, the majority of our viewers will not get to watch this really important part of our ad with this creative. So: How will our creative perform if we create a variant of this creative where we actually convey this important feature BEFORE second 4?

    With this example, you have the first step into the direction and towards making a more data-backed decision for your creative.

  2. Not having statistical significance

    With every test it is important to reach statistical significance. If you want to know more about statistical significance, then here is the link to a complete blog article about it, however, the general principle of statistical significance is explained pretty easily with an example:


Scenario 1: You spent $10.000 on creative A and that resulted in 10.000 clicks at a CTR of 2.5% and 250 purchases which lead to an overall Return-On-Ad-Spend (ROAS) of 2.5x. 

Scenario 2: You spend $100 on creative B, which generated 200 clicks with a CTR of 5% and 5 sales, resulting in a 5x Return-On-Ad-Spend (ROAS).
Although the performance metrics for “Creative B” look a lot better, the sample size is a lot smaller, meaning the probability that those metrics will decrease significantly & be more volatile is a lot higher.

With that in mind you always want to put your testing results into perspective to the particular sample size, but keep in mind that you can never have a 100% statistical significance, since you can never be 100% sure that “Variant A” will outperform “Variant B” forever. 

  1. Not structuring your tests properly
    The thought process when launching a test is always the same: “Let’s see if this performs better than my current control & if it doesn’t then we do this, that & that..”

    However, there are also a lot of tests that will not quite turn out as you had imagined which is amazing, because those tests (if done correctly) are the tests that reveal the most new learnings about your consumer behavior that you haven’t noticed before.

    It doesn’t matter if a test fails or wins, in both scenarios you have learned something about your creative, audience and strategy. The only scenario where you lose either way is when you don’t gather your learning from a test to then implement them going forward.

    So, you want to make sure that you are prepared for when your tests fail & what you want to do afterwards then to make sure to leverage that “failed” test or to implement the learning that you can even gather from your “failed” tests.

How to make data-backed decision & why proper hypothesis setting will be your best friend

This testing cycle is pretty much how the whole life from a good marketer or business owner looks like: Analyse your data → form hypothesis based off of this data set that could have caused your data to “behave” like that → Prioritize your hypothesis or “problems” & putting them into perspective with the possible impact they have on your current situation or operations → run tests to see how your consumer behave & then do the whole cycle all over again. This essentially is how you make data-backed decisions…

However, what step of those four is the most important one?

Obviously, the cycle does not work if one step is just neglected, so to a certain extent all steps are equally important, but the second step: forming your hypothesis is particularly important in my opinion.

This step puts your data into perspective & makes certain assumptions based on this data that could be the root factor for why your marketing campaign, website or email automation performs like this. 

Additionally to that you also form the cornerstone for the coming steps of your testing cycle, since this will also determine the prioritization of this hypothesis and ultimately what you will do to test this and the metrics that you will take into consideration.

So, make sure that you form your hypothesis correctly and accurately that gets everyone aligned. It describes a problem, a proposed solution and predicts an outcome

An example of a hypothesis would be as follows: “We have a below average CTR on our creative A, when we emphasise our CTA-button and the benefit at the end of our video, then we will see an increase in CTR.”

It’s easier to express your theory in an action/consequence format.

I’ve inserted a link to understand more about creative testing and what you should look out for Here.

Conclusion

Testing is important for the growth of your eCommerce business. Obviously, every test is better than no test. You should always be testing, however, I’ve noticed that companies that realize the importance of testing, quickly become overwhelmed with running too many tests. 

This normally leads to two scenarios, either they run too many tests to actually implement the learnings that they can gather out of those which ends in inefficiencies, or they run too many tests without actually building off of their “winners” or even identify those which then ends in them not seeing the value in testing, since they haven’t seen any growth from it. 

Both scenarios you rather want to avoid when it comes to testing, therefore, I always encourage everyone and also every account that we manage at VictoryMedia to start conservatively and plan out your testings properly and rather run one test less than one test too many, so you can gather your learnings properly. 


Testing sometimes is similar to an exponential equation, by stacking up your small wins from time to time in the beginning this will lead to enormous wins further down the line.

Leave a Reply

Your email address will not be published. Required fields are marked *