Skip to main content
Reference

Experiments (A/B tests)

Overview​

Your agent can run StoreFront Experiments: split live visitors between versions of content and measure which one does better on an objective, such as revenue per session. It builds the variations, lets you preview each one, starts the experiment with your approval, reads the results, and ends it.

There are two kinds:

  • A page experiment shows each visitor one of 2 to 5 variations of part of a page, at one address. Use it for copy, offers, heroes and calls to action. The CLI can start a page experiment saved in a page body.
  • A URL experiment splits visitors between separate pages at one router address. Use it for a different layout or template. Ads and links must point at the router address.

Example requests​

  • "Set up a homepage experiment: keep today's hero as the control and add a subscription-led variation. Objective: revenue per session, 50/50 split. Show me each variation in a preview, then start it."
  • "How is the hero experiment doing? Is it time to act?"
  • "End the checkout-copy experiment with the winning variation."

Commands​

CommandWhat it does
sf experiments objectivesList the objectives an experiment can optimize
recipe instantiate builtin:experiment-abBuild a correct two-variation page experiment to start from
sf preview open --variation <n>Preview one variation; a preview never counts as a session
sf experiments start-page, start-urlStart a page experiment, or a split between existing pages
sf experiments list, getRead results, with an assessment of whether to act
sf experiments pause, resumeStop or restart sending new visitors to one variation
sf experiments endEnd with a winner, or without one

Reading results​

sf experiments get returns an assessment, and the agent skill tells the agent to act on it rather than guess:

  • collecting: too few days or sessions so far. Do not act.
  • ready: past the minimum with a significant result. End it with the leader if the leader beats the control.
  • inconclusive: past the minimum without a significant result. Keep running or end without a winner.

When a page experiment ends with a winner, UltraCart writes the winning content into the page on its next scheduled run, and every visitor gets the winner in the meantime.

What it cannot do​

  • Change a running experiment's structure, settings or weights. Content inside a variation can still change, and one variation can be paused. For anything else, end it and start a new one.
  • Start an experiment that lives in a theme container. The CLI starts page-body experiments and URL experiments.

What needs approval​

  • Starting an experiment (start-page, start-url).
  • Ending with a winner, which cannot be undone, with a fresh authenticator code each time.
  • Ending without a winner.

Every experiment change needs --live and publish permission. See Changes that need approval.

Was this page helpful?