Measurement
How Fulcrum measures the number behind your guarantee.
A controlled comparison, not a running total.
The holdout lift: measured against a randomized control group, calculated once at the end of your 12-month term.
The method
One comparison. One number.
When Fulcrum reports a guaranteed value, it isn't a count of clicks and orders - it's the measured difference in revenue per user between shoppers who get Fulcrum's personalization and a small, randomized group who don't. We call this the holdout lift, and it's the only number your guarantee is ever based on.
The mechanism
A randomized holdout, running for your whole contract.
-
Split
Your traffic is divided at random. Most shoppers get Fulcrum's personalization; a fixed 10% holdout never sees it. The split stays locked for the life of your contract.
-
Measure
Both groups are tracked continuously across your whole site: conversion rate, average order value, and revenue per user, collected from day one onward.
-
Compare
At the end of the 12-month term we calculate the difference in revenue per user between the two groups. That difference is the holdout lift.
Timing
Why we wait for the full 12 months.
Statistical confidence comes from sample size. A result you can put a fee behind cannot be drawn from a handful of days' worth of data, so we don't publish interim snapshots or call the test early. We calculate the lift exactly once, at the end, using everything collected along the way.
The 45-day pilot is a separate window from this 12-month guarantee term - see how the pilot works .
Significance
Why we use the lower bound, not the raw number.
A measured lift is a range, not a single point. The data supports a spread of plausible values for the true difference, so instead of reporting whichever figure looks best we calculate the conservative lower bound of that range, at a 95% confidence level. If the lower bound is above zero, the result counts as significant - and that bound is what a guarantee is built from, never the raw top of the range.
Scale
What this means for higher- and lower-traffic sites.
Higher traffic fills both sides of the split faster, which narrows the range around the measured lift, so the guarantee can carry a tighter margin. Lower traffic widens the range, so the same calculation yields a more conservative number. Every account uses the exact same method; what differs is how tight the range is, not the formula.
No separate minimum floor. No fallback metric. Every qualifying client gets a guaranteed number calculated the same way - just with a wider or narrower margin depending on traffic.
Common questions
Questions about the method.
Is this based on clicks and page views?
No. Engagement data like clicks and page views informs how we tune your algorithms internally, but it never factors into your guaranteed figure. Only the holdout comparison does.
Why don't you show me results sooner?
We can show you directional trending in conversion rate, AOV, and revenue per user throughout your contract - but the guaranteed figure itself is only calculated once, at the end of the 12-month term, because that's when the sample is large enough to be confident in it.
What if my traffic is on the lower end?
You still get a guaranteed number, calculated the same way as every other client. It will simply be more conservative, reflecting the wider confidence interval that comes with a smaller sample.
How is this different from a standard A/B test?
The mechanics are similar - a randomized control group and a statistical significance threshold - but the holdout runs continuously for your full contract term rather than a fixed test window, and the result is used to calculate a dollar-denominated guarantee, not just a directional read.
Put a controlled measurement behind your fee.
See what a holdout-tested guarantee could show for your funnel.
Start Your Free Pilot