Extreme Testing With Mark Harnett

In the last San Francisco Agile Marketing Meetup Mark gave a presentation, “Extreme testing: Massive conversion rate improvements without going bankrupt” . Mark gave an example of testing different prices $19.95, $14.95, and $9.95, and showed that there’s a significant decrease in conversion rates as you raise the price. But, in order to choose the most profitable price you need to look at cohorts, cancellation rates and the lifetime value of each test cell. So although the $9.95 price point brought in less revenue in the first month, it  turned out to have the highest Lifetime Value.

Mark Harnett studied Systems Design Engineering at Waterloo, Ontario, Canada, then he moved to Silicon Valley 18 years ago, and first landed roles in Quality Assurance, then he became a product manager, and evolved into running marketing. One of the first companies Mark worked at Metareward applied very rigorous statistics to every marketing effort, there was no ego in any marketing decisions, and all about doing tests.

Mark eventually moved to IMVU, a lean start up, and saw a very rigorous agile process being applied. Mark was an early proponent of agile marketing techniques. Out of the experience at IMVU Mark came up with the idea of extreme tests, learn and move on quickly, and only do a test if you can make a significant jump, if you are going to run a test it’s better to run an extreme test because it takes the same effort and time as an incremental test. Mark said, “Expect failure, occasionally it works, but you may have to go through a lot of tests to get one that works. With extreme testing because of the cost of testing, it’s better to go to extreme testing, similar failure rates, but occasionally you have great improvement rate.”

Mark talked about one of the extreme tests he ran at IMVU. “We were testing, people buying virtual credits, virtual clothes, and those credits are bought with real dollars. When we gave occasional discounts, there was a 50% jump in revenue, so you had to be careful about how much liquidity into the market. The extreme test he devised was to try a 50% off sale. We had to mitigate the risks of the test and only run the test for one day. I got buy in. We did 300% improvement in our daily revenue, revenue was in the low 10’s of thousands, and because of the test we were in the 100’s of thousands of dollars. Those sales should be run at sparse intervals, that’s an example of an extreme test.”

Mark made the tie in to agile and extreme testing and how agile is another aspect of continuous improvement, a way of working that teaches you faster, you make sure the failures are small. In terms of extreme testing, you need to pick an exercise that you can go through, but you need to apply to your own situation, think how you can apply what would make extreme situation, and if the test goes wrong make sure it’s manageable.

Mark described why he believes the concept of extreme testing resonates and brings value to marketers, “The other thing I like about the extreme testing moniker, it’s okay to fail, come up with risk mitigation. Test, in a limited risk way. That’s the whole concept behind agile. At the meet up in San Francisco I challenged the audience to come up with their own extreme testing, a handful of people came up with their own tests. Big tests, not incremental. Pushing them outside of their comfort zone.”