If you think a “test” is going to help you understand your customers of your business, think again. A test can validate a decision but cannot think for you. Most “tech” companies A/B test because employees are afraid to think for themselves and take a risk, luckily it wasn’t always that way.
A good example of A/B testing:
In the 1700s, a British ship’s captain observed the lack of scurvy
among sailors serving on the naval ships of Mediterranean
countries, where citrus fruit was part of their rations. He then
gave half his crew limes (the Treatment group) while the other
half (the Control group) continued with their regular diet. Despite
much grumbling among the crew in the Treatment group, the
experiment was a success, showing that consuming limes
prevented scurvy. While the captain did not realize that scurvy is
a consequence of vitamin C deficiency, and that limes are rich in
vitamin C, the intervention worked. British sailors eventually
were compelled to consume citrus fruit regularly, a practice that
gave rise to the still-popular label limeys.
In this case, A/B testing was used to validate a hunch (or validate a hypothesis). This is the right application of the tactic and leverages data to support an insight that was created through another type of knowing (in this case, observation and inductive reasoning).
A common misapplication of A/B testing is for people to use A/B testing as a source of knowledge, rather than a validation of a hypothesis. This manifests itself commonly with people saying “we’ll test into it” or “let the data speak for itself”. This will not achieve the outcome necessary, and ultimately people are abnegating responsibility for making a decision. This is very tempting because it allows a stakeholder to look into the results of a test, apply their pre-existing bias or motive, and say that the data justifies it post hoc.
To use A/B testing successfully in your organization try the following 3 tactics:
- Consider the customer experience – Similar to the Sailors who had to get scurvy to prove the existence of vitamin C, come up with two good customer experiences that can still validate the hypothesis. How would you feel if you were in the hold out group?
- Create a “perpetual” hold out group – Don’t run a single test. Set up your infrastructure (processes and technology) to provide a continuous understanding of the business and, this is key, use it to confirm if you were correct (if you weren’t, you need to change direction – post coming soon!).
- Don’t test and see – Use a hold out group to validate your hypothesis, rather than creating a hypothesis to explain the data. If you try something without having an expected outcome you’ll end up writing fiction rather than fact.
Please Think Responsibly 