Variation Check: catch a broken variation in minutes

The worst way to learn a variation was broken is to come back in three days and find a flat result. Variation Check screenshots every variation a few minutes after you publish a test and tells you whether it is live, whether it is actually different from the original, and whether it is intact.

How to set it up

It is on by default for paid plans. Settings → Testing → AI Assist holds the two boxes:

• Variation Check – the check itself.
• CRO Assist feedback – a few suggestions on the change you made, not just whether it works.

Publish a test and the check runs about three minutes later, once your page caches have caught up. The result appears as a card on the test’s results tab.

What it flags

Problems first, then observations:

• Variation broken or original broken – blank, erroring or missing its content.
• Login, maintenance or 404 page – the screenshot never reached your page.
• Looks identical – under 0.2% of pixels changed and the page text matches. Usually a selector that no longer matches.
• Change is off screen – the pages differ in text but not in the picture, so your change is below the fold or behind a click.
• A popup covers the variation – and only the variation.
• Layout problems – overlapping, cut-off or misplaced elements.

With CRO Assist feedback on you also get tips on the change itself: whether a visitor would notice it in five seconds, whether the variation mixes unrelated changes, which headline is clearer, which call to action stands out, trust signals added or removed.

Gotchas

• Three variations at most are checked, one at a time.
• On-page tests wait for traffic. Their variation names only exist once visits record them, so the check looks each hour for a day. No visits in that day and it is dropped quietly.
• Republishing re-runs it and replaces the stored result.
• Clients never see it. Shared public reports leave the card out.
• Turning AI off turns this off. It needs the hosted AI service.
• Free plans see an upgrade link in place of the boxes.
• “Different page” on a full page test is normal – that is the point of the test.

For developers

The settings are variation_check_enabled and variation_check_cro_enabled in the settings registry, so they are available over REST, MCP and WP-CLI. abst_variation_check_endpoint filters the service URL.

Related: AI Features Overview · Not Seeing Test Variations