Screen Shot 2015-09-04 at 10.03.50

(This is a business case study. It will be used to guide discussions during the session: “Testing” at the Vendo Partner Conference in Barcelona on Wednesday, September 14th.)


It wasn’t working. And Joey was losing money by the minute. He felt the like his screen was zooming in and out at the same time, like a 1950’s horror movie effect.

The reaction in his body was immediate. He felt his chest tighten. Blood was draining from his fingertips. He wasn’t sure what to do with his fingers anyway. What to type?

The deal was done and it would go on for another month. He did the deal to multiply his profits. But, watching the figures roll in, it was turning into losses everywhere. A Red Sea of negative numbers.

Joey called Laurent, his business partner. It was early there, or here, Joey wasn’t sure. He was in a hotel room in some city at some conference. Typical. Laurent was in Burbank with the team. “You seeing this?” asked Joey. “Yes,” Laurent replied on speaker, “Joey, I think it’s time to launch it. We don’t have a choice.”

The “It” was a testing tool that could, potentially, maybe turn those red numbers black. It wasn’t an A/B test. That was useless for the complexity of ads, devices, clickstreams and products that Joey and Laurent had to optimize. It was a machine learning tool. It would learn how to make money and make millions of small adjustments on its own to optimize.

Part of the new tool was based on Scott’s famous article on Google’s machine learning. It took advantage of the three requirements for AI: cheap processing, big data and advanced algorithms. What they had built was the third pillar, the algorithms. They already had the other two.

It was an act of faith to turn it on and let it decide which experiences to give each user. But something had changed. They didn’t know what and didn’t have time to find out. Would it work? Almost anything would be better than what was happening right now. Anything but losing even more money.

“Ok – What the hell. Let’s do it,” Joey said. It would be a long day of watching figures. Would they start to turn around?

He pushed his chair back from the hotel room’s desk and reflected:

What kind of testing am I doing?

  • It’s machine learning. It’s different from any kind of testing I’ve done before. I’m not even sure of why the machine chooses X rather than Y. I just have to look at results to know that it is working.
  • Specifically I’m…
    • Testing to verify
      • What for : Compliance, Quality, Competition
      • Why is that complex : carrier connection, changing environment, stealthy behavior
      • Which tools : proxy networks, crawling tools (geoedge, RiskIQ)
      • Do and don’t
    • Testing to decide
      • What for : choose the best conversion funnel : banner, landing page, registration page, prices, upsell recommendation
      • Why is that complex : contextual change, evolving performances, expensive traffic
      • Which tool : embedded tool, dedicated tool (PerfectBanner, optimizely, VWO, ..) Machine learning algorithm
      • Do and Don’t  

What is working?

  • Machine learning is producing results so you see tech leaders like Google, Facebook, Amazon, Apple and Elon Musk betting the farm on it.

What is confusing me rather than bringing clarity?

  • I’d like to be able to say, “This is working and this is why.” Machine learning doesn’t offer that window into its results because it is working at such a micro level, albeit on a grand scale.

What do I plan on testing in the future?

  • I plan on bringing this testing to areas that have previously be “dumb” or ruled by hunches. We can challenge all of the conventional wisdom.