One live decision. One honest test.
Bring one live decision. We find the mechanism that actually applies, grade the evidence honestly, and run a test that could prove the idea wrong. An independent evidence reviewer checks the work twice: before the test and after it. (We are appointing the reviewer now; no Review is sold until one is signed.)
How a Decision Review works
Intake — Seven questions and two numbers.
Seven questions about one live decision, plus two numbers: monthly users at that step and the date the decision is due. A person reads every intake and replies within 48 hours.
Refusal screen: fake reviews, false urgency, implied official status are declined.
About six to eight weeks from intake to readout (our estimate; it depends on when you run the test).
See prices and detailsWhat the decision memo contains
Written before anything is tested, and checked by the evidence reviewer.
Who is deciding (customer, employee or investor), at what moment, and what counts as success.
One mechanism, at most two, each with its evidence badge and its invariant: one sentence on what always moves.
The famous names that look similar, and why they do not apply.
Whether this is already standard practice in your industry, and what is known about it, before anything new is proposed.
The honest variant to build, and the deceptive variant to avoid, named by its pattern in the CCPA's dark-pattern guidelines.
The test design (the pre-registration fields below), and an open-gaps box: what is disputed, missing or unsure.
Pre-registration fields
Fixed and timestamped before any data.
| Field | Pre-registered specification |
|---|---|
| Hypothesis | Which mechanism, in one sentence. |
| Variants | The control and the honest variant. |
| Primary metric | One number you already track. |
| Detectable effect | The smallest effect your traffic can detect, and the sample needed. |
| Duration & stopping rule | When it ends; no early stops from peeking. |
| Harm rule | What stops the test early. |
| Kill rule | The result that means the idea is dead. |
| Analysis plan | How the result will be read, written before any data. |
| Analyst | Who runs the numbers; the evidence reviewer checks them. |
| Disclosure tier | Named, Anonymized (default), or Private. |
Timestamped through OSF Registries (which allows an embargo) or a dated PDF sent to both sides.
Test types
Chosen by your traffic and what can be split.
A/B test
Use when: Traffic can detect a realistic effect.
Example: Two versions of a repayment reminder.
Staggered rollout with a holdout
Use when: Users can't be split cleanly, but a change can be phased by city or cohort.
Example: A new checkout step rolled out city by city.
Observation
Use when: Low traffic, or behaviour in the physical world.
Example: Counting how far walkers carry a wrapper.
Priors memo
Use when: No test is possible or worth running.
Example: Decide on the best evidence, and say so.
Prices
Starting prices, before GST, set by the first pilots. Later, once three results are published: an annual plan, and Reviews for cohorts and in-house teams.
Pilot Review
The full Review, for the first five clients only. Half when we accept your intake, half when we deliver the decision memo.
Decision Review
After the five pilots, set by what they show. Anonymized result in the Registry, wording approved by you.
Private result
Nothing published: the outcome counts only in the Registry totals.
Priors Memo
For teams with too little traffic for a fair test: the best available evidence, clearly graded and checked by the evidence reviewer. No test.
Disclosure tiers
You choose before the test. Anonymized is the default. Nothing about you is published without your written approval of the exact wording.
Your company is named, only with your written consent. You approve the exact Registry wording. Same price as Anonymized.
Sector, city tier, mechanism and result are published; never your company name. You approve the exact wording, and we use broader categories if a combination could identify you.
Nothing published: the outcome counts only in the Registry totals. About 1.5 times the price.
The agreement in plain words
Scope: one live decision: diagnosis, decision memo, pre-registered test design and readout.
Pre-registration: the hypothesis, metric and kill rule are fixed and timestamped before the test starts. Every pre-registration ends in a recorded outcome, even “abandoned” (with the reason you approve). Named and Anonymized outcomes go in the Registry in wording you approve; Private outcomes count only in the totals.
Disclosure tier: Selected before the test. Anonymized by default.
Payment: 50% when we accept your intake, 50% when we deliver the decision memo. Payment never depends on whether or when you run the test.
Refunds: no refund for a null or killed result: an honest “it didn't work” is part of what you pay for. If we fail to deliver the memo, we refund the part not delivered.
Timeline: the memo is on our side; the test runs on yours. When the readout arrives depends on when you implement the change and how much traffic reaches it.
Not legal advice: this is guidance on a decision, not legal advice. A lawyer reviews the legal note.
Liability cap: set out in the written agreement.
Your Mistake Ledger entry
Before the result, you write what you expect and by when. After the readout, you write what happened and the rule you would add.
Your Mistake Ledger stays strictly with you; only the anonymized “first guess versus finding” is published, and not at all for Private Reviews.
Frequently asked questions
How long does a Decision Review take?
About six to eight weeks from intake to readout is our estimate. The real timeline depends on when you implement the test and how much traffic reaches it.
What if my traffic is too low for an A/B test?
A priors memo, or an observation test. If your traffic cannot detect a realistic effect, we say so rather than run a test that can only produce noise.
Will my company be named?
Only if you choose the Named tier and approve the wording in writing. The default is Anonymized. You can also choose Private.
What if the test fails?
It is recorded as Killed: published in wording you approve, or counted only in the totals if you chose Private. That is useful: a killed idea is one you stop spending on. There is no refund for a killed result.
Is this legal advice?
No. The memo checks each persuasion element against the CCPA's dark-pattern list and defaults to the honest variant, and a lawyer reviews the legal note. It is guidance on a decision, not legal advice.
What decisions will you refuse?
We refuse fake reviews, seeded queues, false urgency, implied official status, clinical claims, and covert experiments on exams, hiring, credit or health. Read our complete refusal list.