Too often we get caught up in tactical discussions early on about specific ideas for improvements to client’s website. Ideas are great but in order to be successful it’s crucial to take a step back and view the bigger picture. Here’s three fundamental things anyone managing a website needs to understand before they start split testing.
1. Make data driven decisions
“Enlightened trial and error outperforms the planning of flawless execution.” – David Kelly, founder of Ideo
As marketers, it’s a given that you measure the success of planned activities against core strategic metrics. But many marketers are still relying on their instincts when it comes to making changes to websites rather than testing and evaluating this activity to understand the impact. It’s certainly an extremely inefficient approach, wasting resources and potentially costing you sales.
By understanding data and carrying out controlled experiments before investing time and money into creative/development, you can determine what impact your idea will have on those all-important core metrics. It will allow you to decide where to invest development resource, how to improve the experience for users, how to build valuable features while culling those that don’t present value and lead to feature bloat. Without this processes in place it’s entirely possible that you’re implementing changes which aren’t driving significant improvements or worse, actually damaging your efforts to reach key metrics.
Tip: Test, measure and analyse the impact of your onsite activities against core metrics allowing you to prioritise activities that drive significant performance improvement and align with your core marketing strategy.
2. Understand the importance of controlled testing
“One accurate measurement is worth more than a thousand expert opinions.” – Admiral Grace Hopper
Working with websites and online experiences allows for a fairly unique ability to run controlled tests quickly, without long release cycles or hardware launches.
Putting the ground work in to configure your testing tools and educating your team about designing robust experiments is extremely important. Trustworthy results can have a significant impact on product development and improved performance, however, distorted or invalid results can be extremely damaging and eat away at confidence in your testing programme.
Tip: Investing up-front to make sure that your tool is well configured and train your team to understand experiment design best practice, to build confidence and provide some protection from those who might challenge your results.
3. Realise we are all poor at assessing the value of ideas
“The best way to have a good idea is to have lots of ideas.” – Linus Pauling or Thomas A. Edison depending on where you look.
There seems to be a general consensus that somewhere between 67–90% of ideas tested by companies such as Google, Bing, Netflix, etc. do not drive the improvements that they had anticipated. This will largely depend on the scale at which you are testing, the source of your insights and a number of other variables, but it’s important to recognise, that even your most confident predictions can suffer at the hands of testing. The up-side here is that testing can drive out emotional and political decisions that are harmful to performance.
Tip: Embrace innovation and give optimisation some time to prove itself. Pushing for results too quickly can lead to risk averse teams that miss out on some of the less obvious opportunities.
If you want to read more, this is an interesting paper from the team at Bing. >> http://www.exp-platform.com/Documents/2013%20controlledExperimentsAtScale.pdf