My experience in user research has taught me a lot about people. It’s taught me how to step into the shoes of the user. It’s taught me how to gather valuable and valid insights through selecting the right research method and asking the right questions. It’s taught me how to come up with solutions to website issues. However, something it hasn’t taught me is how well those solutions actually fair in the real world, with real users, on live websites.
One of my main frustrations with conducting user research is without a view to test solutions in a live environment we (as researchers) don’t get to see the impact of the recommendation on the conversion rate and other key metrics. Worse still is when solutions are implemented without testing, which can cause the ‘before & after’ effect, where an increase in conversion rate is attributed to a specific website change when actually external factors were at play, such as seasonality or competitors marketing activity.
Here are three ways in which testing hypotheses derived from user research helps us become better strategists.
1. Best practice isn’t always best
If research participants experience problems within research sessions as a result of a website not conforming to best practice, we may assume that recommending the website rectifies this by adhering to the ‘rules’ would be the most effective solution. However, without conducting testing it is impossible to judge if what is considered “best” for the majority of websites is really the most effective solution for a specific website. This encourages us to think outside the box and consider alternatives that may be more appropriate to the specific website and its customers, fuelling innovation and originality through methods such as collaborative sketching. It also prevents us from getting complacent by assuming that the solution for x is always y.
2. There’s always a chance you’ll be proved wrong
As a strategist it’s important not to be subjective towards your own ideas or solutions and consider the opinions of other team members and the experience they bring to the table. With that in mind it is inevitable that if you come up with an especially good idea, it’s only human to become passionate about that idea. After all, this passion stems from our belief that, from our previous experiences in UX, this solution will be best for the users and as a result, will increase conversion rates. However while your idea might be good it might not be the most effective solution available. By standing by your ideas through A/B or multivariate tests you always have to be prepared to be proved wrong. It’s the best way to remove any subjectivity or personal attachment to solutions. Clearly, the best solution for any given UX problem is never set in stone. There are always multiple potential solutions that all have their own set of pros and cons. So until you test with real users through conversion optimisation, you can never really know which solution is most effective.
3. Continued learnings
There is more to learn from an unsuccessful test than from not testing at all. Continuously making recommendations with no idea of their impact means you have no real idea as to whether or not the recommendations you are making are actually effective. You may always recommend the same thing for the same issue, never really knowing if it was an effective solution. Although every website is different and what applies for one, doesn’t necessarily apply for another, optimisation testing allows us to build up a library of insights that can be used to help improve the solutions we suggest in the future. Even if we know that something didn’t achieve great results for one website, having that experience and knowledge allows us to assess how appropriate a similar solution would be for another website.
Each of these points correlate with some of the key stages involved in undertaking an optimisation programme; such as hypotheses and concept development and learning from test outcomes. They also relate directly to some of the fundamental benefits of conducting CRO for a business, such as making decisions based on real life data. However, from a strategist’s perspective, being involved in CRO allows us to learn and refine by getting feedback on our solutions from actual website visitors.