The more things change the more they stay the same. Data can support our thinking, enrich our decision making and knowledge and help us understand something we did not understand before but it cannot tell us what to do. David Hume described this problem nearly 400 years ago, but not only has it not gone away, in today’s “data obsessed” world it is hindering our decision making more than ever.
Companies and individuals often get mislead by available data and jump to conclusions without first understanding context due to a cognitive bias known as the “availability heuristic”. This convenient mental shortcut that allows us to easily draw conclusions on what comes to mind quickly (or what is easily available to us) even if they are not necessarily the correct ones.
For example:
Back during World War II, the RAF lost a lot of planes to German anti-aircraft fire. So they decided to armor them up. But where to put the armor? The obvious answer was to look at planes that returned from missions, count up all the bullet holes in various places, and then put extra armor in the areas that attracted the most fire.
Obvious but wrong. As Hungarian-born mathematician Abraham Wald explained at the time, if a plane makes it back safely even though it has, say, a bunch of bullet holes in its wings, it means that bullet holes in the wings aren’t very dangerous. What you really want to do is armor up the areas that, on average, don’t have any bullet holes. Why? Because planes with bullet holes in those places never made it back. That’s why you don’t see any bullet holes there on the ones that do return.
In this case, a decision on what to be done was made given the data at hand which, while valuable, did not produce the right conclusion right away because the available data was useless without understanding the conditions that the data represented. Without the right understanding of the right conditions, you can’t form the right hypothesis and draw the right conclusion.
A more recent example of the same phenomena was in 2016 when an offshore transcription company ran an experiment to determine if training associates on specific industries increased throughput and quality. Associates were trained on key parts of each industry and the audio to be transcribed was routed to the trained teams. The conclusion drawn from the results of the experiment was that the industry training made a difference however when the company tried to scale this out it proved ineffective. Why? The data only showed that getting the same type of audio over and over again improved throughput and quality not the training. The key, it was later discovered, was the repetition. The conclusion the management made by the desire for a certain outcome (prove industry level training was a worthwhile investment) and the human tendency to use data to support “my conclusion” rather than understanding how the is-ought problem influences our thinking.
So how do you determine what business decision needs to be made based on the data you have at hand? Here are three tactics:
- Interrogate data sets to determine exactly what they are measuring and document assumptions that may be implicit in the data – Ask the questions that everyone thinks are obvious such as: “Are we sure that a customer’s clicking pattern on a website is the right way to analyze how interested they are in a product?” Often times others are thinking the same thing but don’t want to speak up.
- Be clear about what data does and does not show – If you define your assumptions with declarative statements like: “We are using a customer’s website browsing activity as a proxy for interest in a particular product” you can set a standard against with you can be right or wrong.
- Consider other methods to validate your results – People are unpredictable and you can enhance your business strategy by looking at other ways of understanding behavior. If things go well, you have user testimonials and if they don’t, you have a narrative to understand what to do better next time.
Please Think Responsibly 