“Today, A/B testing is thriving — it’s been a huge improvement from non-A/B testing,” said George Khachatryan, CEO of AI company OfferFit, in a recent webinar. “At the same time, the people performing these tests every day recognize that’s it’s a lot more difficult than it may seem.”
Designing A/B tests, determining samples sizes, and deploying them takes up a lot of time and resources, and analyzing the findings requires high levels of precision. All in all, the manual tasks required by A/B testing can place a heavy burden on marketers.
“When you’re running a full experimentation program, it’s never enough to run one A/B test,” Khachatryan said. “When you run one, you gain valuable insights, and inevitably want to gain more. So you end up running more.”
He added, “Those who are doing this hands-on realize it just becomes a rapid explosion in the number of tests they need – it becomes infeasible very early on this exponential curve.”
Marketers need a solution that allows them to test a growing number of campaign variables while simultaneously giving them enough time to analyze the data. Fortunately, A/B testing is evolving.
Expanding the power of A/B testing
In the webinar, Khachatryan highlighted the “multi-armed bandit problem” that’s affecting modern A/B testing. In the traditional version of this scenario, a person at a casino must determine which slot machines (the “one-armed bandits” that steal your money) are going to have the best payouts, then figure out which order will be optimal. With A/B testing, the variables are the multi-armed bandits, and the marketer must discover which are most effective so they can allocate more resources to the areas performing well.
“You can think of [a multi-armed bandit] like a smart A/B test,” he said. “It’ll navigate the exploration-exploitation tradeoff — it’ll start randomly pulling those ten handles, but as it goes, it’ll dynamically reallocate resources so that if something looks bad it’ll stop pulling.”
He added, “These multi-arm bandits are designed to experiment just the right amount so you’re learning but also taking advantage of what you’ve already learned.”
While these multi-armed, or A/B, models have served marketers well over the years, there’s a new iteration of the framework that is more accurate and effective. According to Khachatryan, these are “contextual bandits.”
“It does what a multi-armed bandit does, but it takes into account different contexts,” he said. “So, if you have two different customers, with different characteristics, it’ll know to pull different levers.”
Contextual bandit frameworks are essentially automated experimentation and personalization at scale. It’s a model that can completely automate the process, and it’s what every marketer should be moving toward to improve campaign effectiveness at scale.
Get the daily newsletter digital marketers rely on.
Automated experimentation is the future
Many of the tech giants have already adopted contextual bandit frameworks, but marketers should note that this technology is still incredibly new. Brands should allocate enough time and resources to make the transition process easier, because, according to Khachatryan, it’s the “future of experimentation.”
“In the past, manual A/B testing works with one mention at a time,” he said. “With these contextual bandits, you can set it up to simlutantous test multiple dimensions.”
Whether it’s testing email subject line efficacy, call-to-action click-through rates, or optimal article posting times, marketers have a lot of experimental data to keep up with. Automated testing solutions can make these processes more manageable by decreasing the time spent on manual tasks, replacing them with continuous automated experimentation.
“You can think of this as the next iteration of experimentation, or test-and-learn programs,” Khachatryan said. “When a marketer sets up this system of continuous automated experimentation, it creates this interplay where you can see what happens, gain insights, and then use those insights to get new ideas.”
He added, “So you still have the agile test-and-learn cycle, but it’s accelerated.”
Time will tell how quickly marketers adopt these automated experimentation technologies. But, with the high level of marketing technology replacements that took place over the past year, there’s a good chance more brands will sign on sooner rather than later.
Watch this webinar presentation at Digital Marketing Depot.
Marketing automation: A snapshot
What they are. For today’s marketers, automation platforms are often the center of the marketing stack. They aren’t shiny new technologies, but rather dependable stalwarts that marketers can rely upon to help them stand out in a crowded inbox and on the web amidst a deluge of content.
How they’ve changed. To help marketers win the attention battle, marketing automation vendors have expanded from dependence on static email campaigns to offering dynamic content deployment for email, landing pages, mobile and social. They’ve also incorporated features that rely on machine learning and artificial intelligence for functions such as lead scoring, in addition to investing in the user interface and scalability.
Why we care. The growing popularity of account-based marketing has also been a force influencing vendors’ roadmaps, as marketers seek to serve the buying group in a holistic manner — speaking to all of its members and their different priorities. And, ideally, these tools let marketers send buyer information through their tight integrations with CRMs, giving the sales team a leg up when it comes to closing the deal.
About The Author
Corey Patterson is an Editor for MarTech and Search Engine Land. With a background in SEO, content marketing, and journalism, he covers SEO and PPC to help marketers improve their campaigns.
