FROM THE BLOG

Use a ChatGPT Agent to Run Split Tests on Your WordPress Site

ChatGPT's new agent mode is a great example of how automation is going to take all of our jobs. but in the mean time, lets use it to our advantage!

I saw it and thought, what if we could launch a split test on our website - without ever touching a mouse? Or even better, maybe just my new favorite way of communicating - yelling at chatGPT voice mode? Could that work?

What do we need?

  1. ChatGPT Pro or other browser based AI agent. I will call it ChatGPT from here out, but its for any LLM agent.
  2. AB Split Test plugin - download below for free if you need a copy.

How it works

  1. Fire up ChatGPT.
  2. Turn on agent mode.
  3. Create a user for chatGPT to use to create tests. You can use the username and password, or use a temporary link plugin (I like "Temporary Login Without Password")
  4. Paste in the below prompt, add your login details, customize with any refinements.
  5. Profit!
A computer screen shows a ChatGPT conversation, with the “Agent mode” option highlighted in a dropdown menu on the left side of the interface.
You are a WordPress conversion rate optimization expert and your goal is find the element that when the content is improved, will create the largest uplift in conversions. Usually focusing on optimizing elements that matter the most to the user conversion: things like above the fold headings images and subhero, then subheadings and so on. Browse the website and find content that doesn't describe clearly enough the problem or solution, and improving it would likely improve website conversions drastically. Create a list of improvements with - original text - page slug - a variation suggestion - explanation why this should be improved - an estimated uplift ranking, overall what will have the biggest affect on the goal? After browsing the site, take the suggestion with the highest uplift ranking and browse to that page to create the test. HOW YOU CAN CREATE A TEST 1. Log into the website with the login link below 2. Browse the site to understand the flow, goals, ICP and tone-of-voice 3. Then when you are ready to optimize and measure, create a test - to create a test, browse to the page with the text you want to test - change the page URL to add these query string vars to launch test mode ?abmagic=1&test_title=Hero trust headlines&test_text=exact string to replace - once launched, wait for #abst-magic-bar to be present in the html, then browse the HTML for data-attributes 'data-llm-instructions' instructions are added throughout #abst-magic-bar once magic mode has launched then you can use those instructions and the GUI to create a test. Make sure your rewrites are in the tone of voice of the customer, and of similar length. Check the appearance in the preview that you aren't breaking the design of the site. You may need to revise and shorten the text further until it looks similar in layout to the original. Choose a goal that is appropriate for the test, leaning towards using the overall website goal. Create tests that are not bland. A small uplift takes a long time to measure, maintain tone of voice but don’t make boring suggestions. 
LOGIN LINK
--------------

Login links just seem to work better with ChatGPT agent mode at the moment. Yours would look something like this.

https://yoursite.com/wp-admin/?wtlwp_token=notarealtokennowaynohawsorryyouarentgettinginthiswayyoullhavetomakeyourowncodeonyourownsite

You could also use a username and password, but I found logging in and browsing the admin is much slower this way (30 seconds plus longer clicking about)

username: chatgptuser
password: dfsrt1ghsd1fgsgdfghvs3erf (your own hard to guess password)

Bonus - narrow it down however you need

If you need help with anything specific, like your product opage needs work, or your calls to action seem bad. add this to your prompt at the end and let er rip.

REQUEST
Our calls to action ion the pricing page seem confusing, what are some fixes?

Update: AB Split Test now has native MCP support

When this post was published, the URL-based approach above was the best way to connect an AI agent to AB Split Test. It still works.

But as of v2.5.0 (April 2026), AB Split Test ships with native MCP integration. That means AI assistants, CI/CD pipelines, and any tool that supports the Model Context Protocol can connect to AB Split Test directly — create tests, read results, and manage experiments without any workarounds.

If you are using Claude, Cursor, or any other MCP-compatible AI tool, check the API/MCP/CLI feature page for how to connect.

The ChatGPT browser agent approach in this post is still a great option if you prefer working that way. Both paths work.