Advertising: is it art or science? Surely most of us can agree that a healthy dose of both is necessary for powerful advertising.
As direct-response advertisers, testing is a huge part of our daily ad operations; we’re continually running creatives against each other, optimizing campaign bids and budgets, and analyzing down-funnel results between audiences. A lot of the time, well-structured campaigns can answer all the questions we have. Sometimes, though, we have to put our lab coats on, develop a solid hypothesis, isolate our variables and get our A/B testing on.
In this post, we’re sharing some of our experiences with Facebook’s newish testing feature, creative split testing, along with some advice on when and when NOT to leverage this feature.
First, a bit about the Facebook split testing feature.
Facebook Ads Split Testing 101:
Creative split testing (Facebook’s term for ad A/B testing) has evolved from an API-only feature (which we leveraged through our Facebook tech partner Smartly.io), into the self-serve tool it is today.
Launched in November 2017, Facebook advertisers can now A/B test any of the following variables:
Of course, all the standard A/B testing best practices apply:
For an in-depth look at how to set up Facebook ads for split testing, Adespresso has you covered.
When You SHOULD use Creative Split Testing
The value of creative split testing should be apparent; it helps advertisers determine the scientifically best performing ads, audiences, or bidding strategies. It does, however, require deliberate planning and dedicated budget, thus often lowering overall efficiency.
We’re constantly weighing the benefits of A/B testing with those costs, and have outlined some common scenarios in which you SHOULD utilize A/B tests, and others in which you SHOULDN’T.
Here are a few scenarios in which you should.
#1: Testing imagery to guide future creative direction
The golden rule of running split tests is quite simple–use them to test measurable hypotheses. As an advertising agency that strives to maintain campaign structure hygiene, we take this rule a step further. We run A/B tests against hypotheses that are meaningful to our overall strategy.
When set up correctly, A/B testing can provide black-and-white learnings to inform your creative strategy going forward.
Take this example...
We ran a split test that tested an existing high-performing ad against one with the same messaging, concept and copy but a different design element.
Details: Our isolated variable in this test was the design concept; all ad copy, in-image copy, CTAs, and overall concepts were the same. By monitoring cost differences among our top performing audience segments, we wanted to determine which path to take creative concepts going forward.
Results: After running the test for a full week to reach statistical significance, we were able to determine a winner based on lower CPAs. We then utilized that learning to guide the rest of our creative asset production, helping us lower overall costs on average 60%.
#2: Testing messaging to inform company-wide strategy
Similar to our previous example, split-testing can help inform messaging across channels, or even company-wide.
We ran a split test for an early-stage transportation company to determine which first-touch messaging resonated best with their core demographic.
Details: While we could have run typical ad sets, we knew that in helping our client determine future positioning, having a high level of confidence was important. Running a controlled A/B test also gave us insight into lifetime value, which we’ll discuss later in this post.
Results: In this case, our messaging variations didn’t have a major effect on the CTR. Although statistically relevant, our winning messaging had a less than 3% better CTR. It’s important to remember, however, that even those findings are valuable, prompting us to ask us other questions–do we need to focus on other areas like the imagery? Calls-to-action? Value-prop communication? Should we target different audiences with different messaging?
#3: Testing creative concept variations for specific personas
Split testing can also be useful when figuring out what creative resonates with new or hard-to-crack audiences. Employing the “ax first, sandpaper later” mentality, you can utilize a split test to determine which tactics resonate better, and make incremental optimizations afterward.
Take for example...
For one client, we wanted to run a split test to see if persona-specific creative would perform better than generic creative among a new demographic group.
Details: This particular test utilized existing successful ads against new ads to test our hypothesis that persona-specific creative would do better.
Results: These findings were significant in that this was an entirely untouched user segment for us, and gave conclusive guidelines on how to approach this audience in the future. The costs for the losing concept were 270% higher.
#4: Determining long-tail value of messaging or user groups
Although more time- and resource-intensive, you can also use split testing to create pseudo cohorts to monitor over time after your initial test has completed. Here are some ideas...
We implemented this tactic for a subscription consumer goods startup to determine what upfront ad messaging produced the best return on ad spend after 28 days.
Details: For this specific test, we wanted to monitor ROAS of groups with and without an offer.
Results: After running our test long enough to get statistically relevant cost results, we continued to monitor our two groups. We determined that after 28 days, although conversion rates were virtually identical, one group did indeed have 27% higher ROAS.
Proud to work with today's most innovative companies.
Stay in Touch