Skip to content

Prompt A/B Test Designer

Plan a prompt A/B test: sample size per variant from a two-proportion z-test with Bonferroni correction, cost, duration and a Markdown plan.

Prompt A/B Test Designer

2 variants3.1K samples$46.95
A~26 tokens
B~44 tokens
Diff viewvs
You are a support assistant for Acme.
-Answer the customer's question.
+Answer the customer's question using the help-center article below.
+Cite the article title in your reply.
Keep the reply under 120 words.

Updated . Provided as is. Check the output before you rely on it in production.

How to use Prompt A/B Test Designer

  1. 1

    Define prompt variants

    Add two or more prompt variants using the variant editor. Each variant has a name and the full prompt text. The diff view highlights differences between variants.

  2. 2

    Configure experiment parameters

    Set the significance level (default 0.95), minimum detectable effect (default 5%), and baseline success rate. These determine the required sample size.

  3. 3

    Review sample size and cost

    The calculator shows the required samples per variant, total samples, and estimated experiment cost based on your selected model and average tokens per evaluation.

  4. 4

    Export the experiment plan

    Click Export Plan to generate a complete experiment document with hypothesis, variants, sample size rationale, metrics, timeline, and budget in markdown format.

Questions and answers

What is Prompt A/B Test Designer?
A prompt A/B test runs two or more prompt variants on the same task and compares success rates to tell a real difference from noise. This designer computes the samples needed per variant at 80% power from your baseline rate, minimum effect and significance level, estimates cost and duration, and exports a plan.
How is sample size calculated?
Using two-proportion z-test power analysis with configurable significance level, minimum detectable effect, and baseline success rate. Bonferroni correction is applied for experiments with more than two variants.
Does it send my prompts to a server?
No. All calculations and prompt comparison happen entirely in your browser.
Can I export the experiment plan?
Yes. Export as a markdown document with hypothesis, variants, sample size rationale, success metrics, expected duration, and budget — ready for team review or documentation.
For AI agents: how to call this tool

Machine-readable contract, endpoints and examples. Humans can ignore this section.

Best Path For Builders

Browser workflow

Runs instantly in the browser with private local processing and copy/export-ready output.

Browser Workflow

This tool is optimized for instant in-browser execution with local data handling. Run it here and copy/export the output directly.

/prompt-ab-test-designer/

For automation planning, fetch the canonical contract at /api/tool/prompt-ab-test-designer.json.