Content testing is a way to optimize the performance of your website by testing different elements and observing which ones perform the best for the outcomes you value. Performance is measured against measures such as the effect on conversion rates, basket size or changes in the volume of customer service calls.
A simple form of content testing is A/B testing, where the performance of one version of a site is compared to another one to see which performs best. In a typical scenario, the current site (version A) might have a red ‘Go to checkout’ button, whilst the other (version B) might have a green button.
Multivariate testing (MVT) is a process by which more than one component of a website is tested in a live environment to maximize effectiveness. Multivariate testing allows for several components of a site to be tested at once.
A typical program of MVT will explore different content for many elements across one or more pages to identify the combination of changes that yield the best outcome.
As it isn’t uncommon for a multivariate test program to encompass several dozen variations, technology is usually vital to keep on top of not only the delivery of different versions of the page to users but also the data and reporting to better understand the effects of the test.
You’ll also see MVT and A/B testing called ‘split’ or ‘bucket’ testing. MVT is a critical part of good user experience as it aims to optimize your site for the outcomes you desire by making the user’s experience the best it can possibly be to achieve that outcome.
Business Case
The obvious case for testing is that it can improve desirable outcomes, such as raising conversion rates or reducing abandoned baskets. Companies who have used the process often see other benefits emerge including having concrete reasons for making decisive design changes rather than being driven by unsubstantiated opinions or the preferences of personalities within the business, making it easier to make sound business decisions backed up by fact.
Other advantages include gaining insight into customer segments based on their on-site behaviors, reduced customer service inquiries as users find what they need online and better allocation of resources based on more confident decision-making.
Digging into analytics data during the MVT process can provide you with valuable information about your customers and how they navigate your online presence.
You should be aware that the results of your testing may not be as you expected, and you may need to shed some preconceived ideas about web design or customer preferences. Some companies discover that only a small proportion of their test elements have any noticeable effect on outcomes.
Subtle changes in color or changes to copy may not register with your user base because these aren’t significant page elements or they aren’t reading it in the first place. You should also be wary of jumping to premature conclusions about the changes implemented by your testing.
Sometimes the testing period may be too short (or the user numbers too low) to be able to draw valid conclusions. Don’t make permanent changes to your site before enough data has come in from MVT testing to be certain of what you are doing and the effect this is having on consumer behavior.
Barriers to testing
Barriers to testing include a lack of budget, internal resource and the inability of IT resource or infrastructure to meet the demands of a multivariate testing program. Other barriers can include a lack of internal buy-in, particularly at senior management level. Generally, it’s the case that companies understand how critical good user experience is to their business goals, but that doesn’t mean they use MVT.
Sometimes managers prefer to trust their own instincts or see MVT as an extra burden which isn’t worth the trouble or focus on aspects such as customer acquisition rather than conversion.
Considerations
Audience understanding of web design conventions
Some web design conventions are pretty standard the world over. Partly this will be the result of global giants popularizing design conventions, such as Amazon bringing tab navigation to a wide audience.
Conventions such as tab navigation (based on the old paper-based library index cards), the magnifying glass icon used to denote search and the shopping trolley for online purchases are delivered to audiences who may never have used the ‘old’ technology on which they are based.
RELATED: Understanding the Increasing Decline of Skeuomorphism
Don’t assume your audience will understand what view as standard design conventions, especially in emerging markets where you could be facing an audience which is relatively new to the internet. It’s valuable to look at what popular local sites are doing in that region to understand the conventions the audience may already be exposed to and expect to see on other sites.
Culture
A lifetime of immersion in a culture informs norms or patterns of thinking and behavior, values, heroes, rituals and symbols that affect how a user interacts with a website. Cultural effects like these will have their influence on how elements such as colors, icons, navigation controls and other visual cues are perceived on your website.
Color is one aspect to be sensitive to. Color meanings are often more specific and defined to a Japanese or Middle Eastern audience than to a western one, for example. Your choice of color may have more significance than you expect in these territories.
In China, a black border around a person’s picture would be understood to mean they are deceased and black and white photos also remind people of those used on graves. In Chinese stock market reports red is used to symbolize “up” and green “down” – the opposite of the Western norm.
In many Western markets, the color can red symbolize love, passion or danger. But in China, red is a simple of good fortune and is the main color used during Chinese New Year and Weddings.
Many cultures have different colors which they associate with death, for example, green is associated with the environment and nature in the West but is known as a “forbidden color” in Indonesia, and is best avoided in web design.
Different audiences will tolerate ‘busy’ web design better. Audiences in South Korea or China are more accustomed to seeing a lot on screen, whilst a Scandinavian audience is more used to a minimalist design without as many distractions on the page. This will inform how tolerant they are of ‘clutter’ on your web page and how quickly they can cut through it to reach their goal.
Anthropologists sometimes describe cultures in Africa and Asia as ‘high-context’, meaning they tend to use symbols more, and expect people to interpret meaning from fewer words. By-contrast, ‘low-context’ cultures in North America and Europe, tend to use more text and spell out their messages explicitly.
You may find Chinese, Japanese and Korean users expect to see a greater use of images, videos and sidebars compared to Northern European audiences used to more text-heavy designs. While these aren’t hard and fast rules, they are often reflected in website preferences and ones you might want to incorporate into your testing.
Finally, many international websites often adapt the social context of imagery they use on their international sites to convey different messages about their brand. McDonald’s brightly colored Indian site features a father and son rushing through a crowded supermarket with the slogan “I’m Lovin’ It”. The Swiss McDonald’s site features a woman listening to music alone, in muted colors, with the same tagline to convey a very different feel.
On their Dutch website, global electronics giant Philips chose to show a northern landscape with soft colors and the presence of a middle-aged man pushing a bike in a park with a relaxed smile on his face: a message of tranquility and a sense of well-being.
By contrast, the Japanese version features a young Asian man wearing a white shirt and tie, holding an electric razor in a pose which communicates urban dynamism, determination and tight schedules. Part of this is a question of how you wish to brand yourself locally, but testing a variety of images using MVT may be valuable to see which performs best for different outcomes.
Trust
If you’re a foreign brand trying to establish yourself in a local market, trust is an important issue and multivariate testing should help you explore how to reassure site users about your brand.
Different concerns dominate in different territories. In China, mobile payments are commonly used to pay for eCommerce purchases so be aware consumers may mistrust or simply lack familiarity with unfamiliar payment systems you may be offering.
Alipay, a third-party mobile and online payment platform, is one of China’s most popular online payment methods.
A number of food scares means that foreign brands enjoy higher confidence than local Chinese ones, and consumers use apps to alert them about the latest food safety concerns. If you’re selling food in a culture with these concerns, you need to think carefully about how you communicate your trustworthiness.
Other cultures will have their own concerns and will need reassurance on other matters. Depending on how familiar your offering is in the area, you’ll need to pre-empt questions about your product or service and these may vary according to the market you’re operating in and your relative position in the market.
There have been several studies looking at how the use of human photos contributes to increasing trust on a website. Medalia Art, sellers of Brazilian and Caribbean art, experimented with replacing images of the art itself with the artist’s own photo as part of a testing program to increase conversion rates.
They found an astonishing 95% rise in conversion rates. However, other studies have shown negative effects associated with the use of human images. MVT may help you understand which approach is best for your audience.
You’ll also want to consider how local your team looks when deciding whether to feature their photos or not. A team of ‘local-looking’ people may help build trust as it shows you are established in that region. But if you are actively selling yourself as a foreign brand, it may be better to prove this by showing you have non-locals on the team.
For some American brands, their American nature is seen as inspirational in some cultures, so they need to deliver on this promise. Be very sensitive about these issues – Microsoft was widely ridiculed for crudely editing a black person out of a photo on their Polish site whilst using the same unedited photo in other target markets.
Choice
Many eCommerce retailers have found uplift in examining how consumers shop using MVT by offering different ways of searching products and the number of product choices offered – for instance the range of colors.
Online kilt sellers buyakilt.com found that adding filters to allow visitors to shop by pattern and other product variations led to a significant rise in revenue, conversions and shopping cart visits.
But Adyen, a company offering global online payment systems, reports that in some global markets adding more payment methods to a site actually reduces conversion rates. The kind of choice they offered was perhaps the wrong one.
One interesting study by Socialtriggers found that offering customers between 4 and 6 choices led to optimum purchasing. Their view was that customers wanted to browse, but offering too much choice (more than 6 options) was counterproductive.
Online customers wanted choice without information overload because making decisions was mentally taxing. Restricting the range offered led to more purchases in their study. It may be worth experimenting with the number of choices you offer in different geographies.
Consumers in more developed markets may be better accustomed to choice, and those accustomed to ‘busy’ web design may be better able to process information than those used to more minimalist pages. You’ll need to identify the optimum level of choice for your market.
Website elements
Common website elements such as page header, navigation order and call-to-action buttons can make a significant contribution to performance.
Online fashion retailer, ASOS found that by changing the call-to-action button for new customers on its sign-in page from ‘Create your account’ to simply ‘Continue’, basket abandonment rates reduced by 50%.
In an award-winning MVT test, Hyundai found massive improvement in outcome from increasing button and image size on a page built for booking test drives.
Elements you should consider testing include: navigation labeling, appearance and ordering, headlines and sub-headlines, testimonials, call to action button size, appearance and text, link placement and copy, images, social proof and media mentions, awards, pricing structures, free trial length and delivery cost.
A Burberry product page. Image alignment, call-to-action button colors and products description details are just a few website elements that could be tested as part of your MVT strategy.
Most companies using MVT will focus especially on user experience on their live website, in particular on landing pages, but you should also consider testing mobile- or tablet-optimized sites and apps.
On a practical level, audiences whose written language reads left-to-right will scan the page differently from a right-to-left audience. This means your navigational tools and site layout needs to be different for an Arabic-speaking audience than a German-speaking one.
You’ll also find it best to avoid using complex scripts or Adobe Flash-based elements in emerging markets where high-speed internet access isn’t as established. And remember that device use varies across different markets, so your testing should reflect which devices your audience predominantly use to access your site.
Localizing features
Ginger Software sells a product to improve English by correcting errors. The company investigated whether a British or American accent was best to maximize conversions on their demo video.
Their finding was that the British accent was better at maximizing conversions in most markets, with the exception of the UK itself and India. In these 2 markets, an American accent worked better. This is such a counter-intuitive result that it emphasizes the importance of thinking outside the box when testing internationally.
Charles Tyrwhitt, a premium shirt retailer active across the UK, German and US markets, saw a 30% rise in conversions from an international multivariate testing investigation.
The company examined 64 test variants to determine aspects such as the optimal navigation order and what the best color for it was. A particularly striking finding was an uplift achieved by changing the product image on the right rather than the left-hand side.
Charles Tyrwhitt worked with technology provider Maxymiser to put a multivariate testing solution in place as part of the website’s relaunch in 2011.
Charles Tyrwhitt found it was much better to carry out tests across all regions they operated in rather than just one of the countries they operated in determining practice for the rest. It was found that there was a pronounced difference in user behavior across the different markets. The difference was so significant that category pages were actually scrapped on the German site after MVT testing.
Advise
Have a clear goal in mind
Although multivariate testing involves testing multiple elements, you should only really have one goal in mind. Examples might include an improvement in the conversion rate for first-time visitors to your site. Establish a benchmark for what that measure is over a suitable period, set yourself an improvement goal (e.g. 4% uplift) and continue MVT testing over a period of weeks until it’s achieved.
Allow time
Depending on the traffic to your site, you’ll need to allow a sufficient period to pass before you can really trust the data from your test program. The bigger the volume of users exposed to the changed features, and the longer the period of time this happens over, the more reliable the data and the more durable the results.
Test the big stuff first
Make the biggest changes to your site first, on the most visible pages, then later in the test program make room for the smaller changes. You’ll find the results come in faster and result from smaller changes will be more visible that way. It’s encouraging to see the impact of bigger changes early on in your MVT cycle, so do the big ones first.
Be tenacious
Remember, many of the tests you try won’t have any significant effect and it will all take time. Stick with the testing until you see positive results from the changes you are testing and don’t lose heart.
Use a combined approach
To really take user experience seriously, it makes sense to combine your MVT program with activities such as laboratory testing, where you observe a person navigating through your site and completing certain tasks; remote testing using video or other recording tools; heat mapping; reviews of your site; and other methods including user surveys.
Using a variety of methods helps you combine an in-depth understanding of the user journey based on observing and surveying individuals, with a wider and more quantitative understanding of the traffic around your site.
Think outside the box
Forgetting what you think you know is a big part of MVT testing. Don’t be afraid to go beyond your pre-conceived notions of what you think you know about web design and try some new ideas. You may find a subscription to BEHAVE gives you inspiration by providing hundreds of examples of testing done by a wide range of companies
Assign appropriate resources
Committing to a program of multivariate testing is a significant step as it necessitates a time investment over a fairly long period. This will be especially true if you are embarking on a multi-national testing program across numerous local sites.
Many companies chose to partner with a user experience agency experienced in international projects. If choosing this route, make sure you still have the resource to give them the support they need across all territories you operate in.
Whilst external partners (such as an experienced MVT agency) can support you, the onus is still on your internal capability to provide direction to them on matters such as the goals of the testing program and what elements you are to investigate.
Your partners should be able to support the production of a testing plan and what’s to be measured. You’ll need to co-operate with your agency at all stages to ensure technical access. And finally, you’ll need to be a bridge between their findings and your business as you seek to convey results to key decision makers who can implement the findings as permanent improvements to your site.
Some agencies allow you a choice of a managed service, which gives a stronger role to your external resource by bringing them into the process of scoping and designing the tests to be undertaken, or self-service, which puts greater onus on your internal resource and can have hidden costs and pitfalls.
If you’re working across different markets and multiple sites, you may be facing an inequality of resource, skills and experience across your different markets. It may be the case that an external resource can help you overcome this.
Get the buy-in you need
Not only do you need to resource your testing program properly, you’ll need buy-in from colleagues to support the changes you want to make both in order to implement the testing and later, the permanent changes your testing is desired to inspire.
A lot of stories circulate around the web design and user experience industry about CEOs who insist on having the website designed in their mother’s favorite color, or based on a dream they once had; MVT is supposed to move sites away from personal preferences or whims and towards a more rigorous design process.
But you’ll still inevitably run up against mindsets that put other priorities above optimization. Sometimes these are arbitrary design preferences, rigid adherence to brand guidelines or just resistance to change. This can especially be an issue when working across multiple locations with stakeholders approaching the website from a variety of different positions and motivations.
The best advice is always to remain data-focused and emphasize the impact on ROI of the changes you want to effect. It’s hard to disagree with cost-focused decisions based on strong evidence. If your website isn’t transactional, you’ll need to find other metrics to measure. These may include increased engagement, loyalty or expansion of the subscriber mailing list.
Know your business
You’ll need to have some existing understanding of your audience segments and user behavior which will inform the tests you design. Be prepared to adapt your understanding of your audience as the results of testing emerge: results about their behavior may surprise you and challenge what you thought you knew about them.
Don’t see it as a one-off, isolated activity
Multivariate testing should be seen as an ongoing activity. It’s especially important when a new site is in development or at points of significant change, but MVT should be pursued through the lifetime of a site to keep on top of a changing audience. As your users become more sophisticated or a new audience emerges, their needs may change. MVT helps you stay on top of this.
Don’t make it obvious
It’s important that your users aren’t aware they are experiencing testing and it could be extremely damaging to your brand and relationship with them if they become conscious of testing activities.
Online dating site eHarmony uses a complex matching system to deliver what it believes are compatible matches to subscribers. Understanding user behavior is a big part of this, and eHarmony varies its matching algorithm from country to country based on factors including user behavior onsite as well as their self-completed profiles and preferences.
One subscriber was immensely irritated by very obvious activities to test which content she reacted to best. She received a large number of very differently branded email templates with different subject lines, firing messaging at her to see which she responded to.
Her perception of eHarmony’s brand was negatively affected by this experience. eHarmony’s mistake was that they sent different content to the same individual rather than testing different individuals with different content.
Key takeaways
Try it now
Content testing can offer huge benefits to organisations working across different geographies, in particular, better audience understanding and improved outcomes from the web presence specific to the audience in each geography.
Set clear goals
Before you start an MVT program, try and have clear goals in mind about what you are trying to achieve.
Get buy-in
It’s important to obtain the support you need from colleagues ahead of an MVT programme. You’ll need their practical support during the programme of testing as well as to implement the changes that result.
Assign appropriate resources
Committing to a multivariate testing program requires significant resource and to get the best from it, it’s important to have resources in place. This includes external resource such as an agency as well as the internal resource to support them.
Expect to have your understanding challenged
Results from testing may surprise you and challenge what you thought you knew about user behavior, web design and your audience!