Elysia Tools
Navigation
Workflow Playbook
Plan an A/B experiment, size its evidence, select the right test, and review effect size and uncertainty before treating a result as actionable.
Hubs
This workflow is for product, growth, marketing, research, and operations teams that need a defensible A/B readout. It helps define a metric, size evidence, select a test, and review the result. It does not allocate traffic, validate telemetry, monitor guardrails, or decide what the business should ship.
Record the baseline, minimum useful effect, alpha or confidence level, target power, traffic, and stopping rule before the final readout. Match the test to the data: two independent conversion rates use a two-proportion Z test; matched measurements use a paired test; independent continuous groups with unequal variances use Welch.
Read absolute effect, relative lift, confidence interval, p value, sample size, and warnings together. Compare the interval with the practical threshold and check data quality and guardrails. Statistical significance describes compatibility with a null model; it does not establish size, safety, profitability, or business success.
Workflow playbook
Use effect-size-calculator for the expected difference, sample-size-calculator or margin-of-error-calculator for precision, and statistical-power-calculator for a one-sample sensitivity check. For a true two-arm design, use ab-test-significance-calculator with baseline, lift, alpha, power, traffic, and split to estimate users and days.
Use two-proportion-z-test-calculator for independent conversion rates and one-proportion-z-test-calculator for one rate against a benchmark. Use z-test-calculator for a known population standard deviation or single proportion, one-sample-t-test-calculator for one group, paired-t-test-calculator for matched data, two-sample-t-test-calculator only with justified pooled variance, and welch-t-test-calculator for unequal variances.
Enter frozen control and treatment counts or summaries. Use ab-test-significance-calculator for differences, lift, intervals, p values, sample estimates, and warnings; use confidence-interval-calculator and p-value-calculator for independent checks. Do not change the primary metric or stopping rule after seeing the result.
Compare absolute and relative effect and its interval with the practical threshold. Recheck sample quality, exposure rules, multiple comparisons, sequential-looking warnings, and external guardrails, then record the decision and reason. Significance can miss the business bar, while an inconclusive result is not proof of no effect.
Choose two-proportion-z-test-calculator for two independent binary outcomes. Choose welch-t-test-calculator for independent continuous measurements when equal variances are not defensible; choose paired-t-test-calculator only for genuinely matched before-and-after observations.
Use the result only when the interval and effect size meet the predeclared practical threshold and guardrails pass. A small p value is evidence against a null model, not proof that the change is valuable or safe to ship.