Autocomplete
What is Autocomplete?
Autocomplete is the "set and forget" feature of AB Split Test. Once your test has collected enough data, AB Split Test takes over automatically.
When a winner is found it:
- Hides the losing variations so all visitors see the winning version
- Sends you an email notification
- Changes the test status to Complete
The losing variations are not deleted. They stay available in the test editor in case you want to iterate or run a follow-up test.
Why does it matter?
Without Autocomplete, your test keeps splitting traffic between a winning and a losing variation until you log in and manually complete it. Every day you wait is a day some of your visitors are still seeing the worse version.
Autocomplete closes that gap as soon as the data is reliable enough to act on.
How does Autocomplete decide when to act?
AB Split Test checks all three of the following conditions before declaring a winner:
1. A clear winner at 95% confidence or above. AB Split Test uses a Bayesian statistics engine to continuously calculate the likelihood that each variation is the true winner. When one variation reaches 95% confidence, the data is reliable enough to act on.
2. The test has run for at least one week. A full week captures a complete cycle of visitor behavior across different days and times. A test that only runs over a weekend may produce results that reflect that particular slice of behavior rather than a real pattern.
3. Each variation has received at least 50 visits. Added in v2.5.1. Declaring a winner on a handful of visits is not statistically meaningful no matter what the confidence level shows. The 50 visit minimum prevents false positives on low traffic tests.
What is 95% confidence?
We use Bayesian statistics to measure the likelihood of success for each variation.
As we collect data from your visitors, the Bayesian method continuously updates and refines its predictions. When we say we have 95% confidence, it means that based on the data collected, we are 95% certain that one version will perform better than the other. This allows us to make reliable decisions about which version is more likely to achieve the desired outcome.
A bit more technical
The Bayesian formula, also known as Bayes' theorem, is expressed as:
P(A|B) = [P(B|A) * P(A)] / P(B)
In this formula:
- P(A|B) is the posterior probability: the updated probability of event A occurring given that event B has occurred.
- P(B|A) is the likelihood: the probability of event B occurring given that event A has occurred.
- P(A) is the prior probability: the initial probability of event A occurring.
- P(B) is the evidence: the total probability of event B occurring.
For most users the important thing to know is that the plugin uses a robust, statistically sound method to analyze results and provide reliable insights. You do not need to understand the formula for the plugin to work correctly.
The Underpowered badge
If your test has not yet met the visit threshold, you will see an Underpowered badge on the results page. This is not an error. It means the test needs more data before a winner can be called.
If the badge has been showing for several weeks, check the following:
Your conversion goal fires rarely. A purchase or form submission goal on a low traffic page can take a long time to reach 50 visits per variation. Consider switching to a higher-funnel goal like scroll depth or time active, with the purchase as a subgoal.
Your traffic allocation is limited. If you set the test to show to only a percentage of visitors, it will take proportionally longer. Check the traffic percentage in your test settings.
Your page has low traffic. This is normal. Low traffic sites need more calendar time. The test is still valid, it just needs patience.
What about Multi-Armed Bandit?
Autocomplete is designed to run a test, find a winner, and stop. If you want to keep optimizing continuously without declaring a final winner, the Multi-Armed Bandit mode progressively shifts traffic toward whichever variation is performing better while keeping a small exploratory allocation on the others.
To enable it, go to Settings > Advanced Settings > Enable Dynamic Traffic Optimization, then set your test to use Dynamic Multi-Armed Bandit in the Winning Mode settings.
What happens when Autocomplete fires?
You receive an email notification to your WordPress admin email address. The test results page will show the winning variation highlighted and the test status changes to Complete.
You can view the full results at any time from All AB Tests. The winning variation continues to show to all visitors until you edit or delete the test.
TL:DR
Set up your test, define a goal, and let it run. Autocomplete handles the rest. When you have a winner with enough data behind it, you get an email and all visitors start seeing the better version automatically.
Last updated: May 2026
2 Comments
Leave a Comment
You must be logged in to post a comment.
How can I see which one did win in the autocomplete?
You should recieve an email to the website admin. Plus the test results page will declare a winner