Reference
Experiments (A/B tests)
Overview
Your agent can run StoreFront Experiments: split live visitors between versions of content and measure which one does better on an objective, such as revenue per session. It builds the variations, lets you preview each one, starts the experiment with your approval, reads the results, and ends it.
There are two kinds:
- A page experiment shows each visitor one of 2 to 5 variations of part of a page, at one address. Use it for copy, offers, heroes and calls to action. The CLI can start a page experiment saved in a page body.
- A URL experiment splits visitors between separate pages at one router address. Use it for a different layout or template. Ads and links must point at the router address.
Example requests
- "Set up a homepage experiment: keep today's hero as the control and add a subscription-led variation. Objective: revenue per session, 50/50 split. Show me each variation in a preview, then start it."
- "How is the hero experiment doing? Is it time to act?"
- "End the checkout-copy experiment with the winning variation."
Commands
| Command | What it does |
|---|---|
sf experiments objectives | List the objectives an experiment can optimize |
recipe instantiate builtin:experiment-ab | Build a correct two-variation page experiment to start from |
sf preview open --variation <n> | Preview one variation; a preview never counts as a session |
sf experiments start-page, start-url | Start a page experiment, or a split between existing pages |
sf experiments list, get | Read results, with an assessment of whether to act |
sf experiments pause, resume | Stop or restart sending new visitors to one variation |
sf experiments end | End with a winner, or without one |
Reading results
sf experiments get returns an assessment, and the agent skill tells the agent to act on it rather than guess:
collecting: too few days or sessions so far. Do not act.ready: past the minimum with a significant result. End it with the leader if the leader beats the control.inconclusive: past the minimum without a significant result. Keep running or end without a winner.
When a page experiment ends with a winner, UltraCart writes the winning content into the page on its next scheduled run, and every visitor gets the winner in the meantime.
What it cannot do
- Change a running experiment's structure, settings or weights. Content inside a variation can still change, and one variation can be paused. For anything else, end it and start a new one.
- Start an experiment that lives in a theme container. The CLI starts page-body experiments and URL experiments.
What needs approval
- Starting an experiment (
start-page,start-url). - Ending with a winner, which cannot be undone, with a fresh authenticator code each time.
- Ending without a winner.
Every experiment change needs --live and publish permission. See Changes that need approval.
Related documentation
- Experiments and split tests: how experiments work in the admin
- Analytics and conversion tracking: measuring results in the warehouse
Was this page helpful?