(This is a business case study. It will be used to guide discussions during the session: “How do you discover and share new truths about your business?” at the Vendo Partner Conference in Barcelona on Wednesday, September 6th.)
Charles wanted the test to show a 5% lift in conversion. He really, really wanted 5%. The promise of a 5% lift was how he and his partners had decided to invest so much in their new tours. If the goal was just 1% they wouldn’t have bothered to even try. But 5%? That was something to shoot for. You could call it a Big Hairy Audacious Goal (BHAG).
His heart rate increased to 140 BPM as he waited for the results to load on his screen. His activity monitor thought he was exercising. It started counting up the minutes of elevated heart rate towards his goal of 45 minutes of exercise for the day. It looked to anyone passing by his desk that he was simply sitting and looking at a screen. But there was much more going on. There was an enormous amount of activity.
Charles had spent the last two years realizing the new truths that defined his business today. “We have been running assumptions about our business for 20 years looking at conversions but we were not really clear if they were true. In the old days it mattered less, but now…” Charles recalled.
The first step was learning how to measure. He focused on measuring revenue.
His team was full of ideas for how to improve revenue. What he needed to do was be scientific about the development of ideas (which ones to do?) and measuring their impact (how do they grow revenue?). He was looking for change – not maintenance.
Charles laid out his process, “For example, we focus on a tour to increase conversions. Everyone comes with in ideas. Then we decide what to measure. Is it conversion? Is it price point? We go with conversion for argument’s sake. Now we have to guess/project what changing a button or bigger font will do. We estimated 5% as a BHAG. We need to have something to shoot for.”
Charles continued, “Then we have to determine how to measure it. Could be A/B. We track. We run the numbers. Now we could easily pick what we want to see – or desire to see – but instead we get an analyst on it to make sure the conclusion is scientifically valid. Let’s say we get 4.3%. Now we wanted 5% and 4.3% is good. Now we have to be very precise how we got there. We need to know where it comes from. We need to get better at predicting the future. Rinse and repeat. Why predict? Because that helps us to decide what we invest in or don’t invest in.”
The next question for Charles was how to share the new truths he learns within his team.
There was the way Charles would like to see it work: Data goes into a knowledge base. Easy!
Then there is his experience with how it actually works, “People are lazy and don’t want to put the numbers to paper for public view. Why? Because they will have to share their own inconvenient truth. And that is that the actual test did not deliver the results we expected or hoped for. The real truth was different,” Charles said.
Charles had learned from experience that, “Sharing results is really hard for people to get behind. But this is our only way forward. Because raising revenue is so hard these days. If you do not know what you are doing then you could cause more harm than good. And right now we need f**king good and that comes from science. Without that we got luck. Numbers don’t tell you what to do but they certainly help!!”
They’d gotten him to 4.3%. Now Charles is going back to his team to see what else they can do to gain new knowledge, share it and grow.
Questions for discussion
- How do you discover and share new truths about your business?
- Your business is changing all the time. How do you find answers to new questions?
- How do you share the knowledge you gain (and keep it up to date) in your organization?