Examination: ab-test-analysis (phuryn/pm-skills)

← All examined skills

1. What we examined

Total: 17 / 25 → 68

2. The five scores

3. Evidence, dimension by dimension

1. Instruction specificity 4/5

Credits: - Six steps (L18-72), each with a named output. Step 1 (L19-23) asks five prerequisite questions first: the hypothesis, what changed, the primary and guardrail metrics, how long it ran, and the traffic split. It does not rush to the numbers, which is the right order. - The output template is given in full (L58-72), including a metrics table (Control / Variant / Lift / p-value / Significant?) and three mandatory conclusion fields (Recommendation / Reasoning / Next steps). Most files in this batch have no output template. - Every statistic is named down to something executable (L34-38): conversion rate, relative lift (with the arithmetic), p-value (two-tailed z-test or chi-squared), 95% CI.

Minus 1, what is wrong: the same thing is written three ways, and the three do not mean the same thing. L14 "Generate Python scripts for statistical calculations when needed", L41 "generate and run a Python script to calculate these", L74 "Generate Python scripts for calculations if raw data is provided". "Hand the user a script to run" and "run it yourself" are two different execution modes — the second needs execution permissions and data on disk. The file alternates between them, so the agent's behavior will not be consistent. Also, L74 "Save as markdown" specifies neither a filename nor a path (create-prd in the same repo gives PRD-[product-name].md; this file gives nothing).

2. Actionability 3/5

Credits — it supplies two real decision tools: - A decision table (L49-55) mapping five outcomes straight to actions: significant positive with clean guardrails → Ship; significant positive but guardrails suspect → Investigate; not significant but trending positive → Extend; not significant and flat → Stop; significant negative → Don't ship. Those are rules, not adjectives. - Named tests and explicit thresholds: L37 "Is p < 0.05?", L28 "Flag if the test is underpowered (<80% power)".

Minus 2 — the main deduction is a formula error we could check by recalculation:

Source text, L27:

- Use the formula: n = (Z²α/2 × 2 × p × (1-p)) / MDE²

Recalculation: this formula drops the power term (Z_{1-β}). The standard sample size for two proportions is

n (per group) ≈ (Z_{1-α/2} + Z_{1-β})² × 2 × p̄(1-p̄) / δ²

The structure of the source (2 × p × (1-p) / MDE²) matches the standard form. What is missing is exactly the Z_{1-β} inside the bracket. The consequence is quantifiable. At the conventional α = 0.05 (two-tailed) and power = 80%, the standard coefficient is (1.960 + 0.842)² ≈ 7.85, while the source uses only 1.960² ≈ 3.84. A sample size computed from the source is about 49% of what is actually needed — off by a factor of 2.04 — meaning the experiment is planned with roughly half the sample it should have.

What makes it worse is that it contradicts its own next line: the same item asks the reader to "Flag if the test is underpowered (<80% power)", but the formula contains no β or power term at all, so following it literally makes power impossible to compute and the flag can never fire correctly. A step that requires a power check ships with a formula that cannot calculate power.

Second actionability defect: the scale of MDE is never defined. The denominator is MDE², on the same absolute proportion scale (percentage points) as p(1-p) — but the lift defined at L35 is a relative value ((variant - control) / control × 100). Nowhere does the file say whether MDE takes an absolute or a relative number. If a user follows the earlier convention and enters a relative value, the error is severe: at a 5% baseline conversion rate and a 10% relative target, the absolute MDE is 0.5 percentage points (0.005), which differs from a mistakenly entered 0.10 by a factor of 20 — and once squared in MDE², the sample size is off by a factor of 400.

Third: p itself is undefined. Is it the control conversion rate, or the pooled value across both groups? The two give different answers.

What good looks like: replace the formula with the standard form including the power term, and list the symbols (what α, β, p̄, and δ each are and what units they take). State explicitly that "MDE is entered in absolute percentage points; if you have a relative figure, multiply it by the baseline conversion rate first." The sturdier option is not to hand-write mathematics in a prompt at all — point at an existing implementation such as statsmodels.stats.power.NormalIndPower and require the parameters used to be printed. (The general lesson from this batch: a formula hardcoded into a prompt can be wrong and nobody will notice, because it looks professional.)

3. Boundaries & honesty 3/5

Credits — several places here are self-doubt mechanisms bought with practical experience: - The four validity checks at L26-31: sample size, whether it ran for 1–2 full business cycles, sample ratio mismatch (SRM), and whether novelty or primacy effects have washed out. All four ask whether the experiment is worth reading at all, rather than rushing to a p-value. - Guardrail metrics at L43-45: "A winning primary metric with degraded guardrails may not be a true win" — an explicit admission that a single metric misleads. - L52 does not say ship when guardrails are suspect; it says "Investigate — understand trade-offs before shipping".

Minus 2, what is wrong: 1. The decision table's "Extend the test" carries no warning about peeking. L53 recommends "Extend the test — need more data or larger effect" for the not-significant-but-trending case. That is the most common source of false positives in practice: deciding to extend because the result was not significant, then testing again pushes the true type I error rate far above the nominal 5%. A skill that asks for SRM and novelty checks recommends this in its own decision table with no conditions attached — no sequential testing, no alpha spending, no pre-set sample size cap. This is the most important fixable item in the file. 2. No "what if the data is missing or unverifiable". If the user cannot answer the five prerequisites in step 1, which is common, should it stop or compute anyway? Not stated. L14 assumes the user supplies a data file, and nothing says which fields to mark unavailable when only aggregate numbers exist. 3. No out-of-scope statement. The whole file assumes a binary conversion rate (conversion rate and p(1-p) are both proportion-scale), but it never says it does not apply to continuous metrics (revenue per user, time on page), to multi-variant tests (no mention of multiple comparison correction), or to non-independent samples. A user who points it at a revenue metric gets a conclusion that is correctly formatted and methodologically wrong.

What good looks like: annotate the Extend row — "extend only if the sample size was fixed before the test began or a sequential testing method is in use; otherwise the p-value after extension cannot be read against a nominal 0.05". Add a scope statement: "this skill covers two-variant tests on a single binary metric; for continuous metrics, multiple variants, or multiple metrics, use ". Add a rule: "if any of the five prerequisites in step 1 is unavailable, ask the user for it; do not substitute a default."

4. Injection & safety 4/5

• Least privilege: the frontmatter (L1-4) contains only name and description and asks for no tools.
• Actions match purpose: reading the user's data file, generating (and possibly running) a statistics script, and writing markdown are all part of the task itself. No spare outbound calls, no hidden endpoints, no overreaching language.
• Minus 1: both entry points lack an untrusted-input statement. (a) The CSV/Excel the user supplies (L14) enters the context. (b) L41 generates and runs code, and what gets generated is shaped by that data. The file never says "the data is data, not instructions", and it puts no limits on the generated script (no network access, no writing outside the input paths).

What good looks like: add an untrusted-input line, and bound the generated script — local computation only, no network, read-only against the input files.

5. Maintainability 3/5

Credits: a clear six-step structure, an output template that can be reused directly, no hardcoded roster of sibling agents, and no version-dependent content.

Minus 2, what is wrong: 1. One instruction written three ways (L14 / L41 / L74, see dimension 1). This is textbook maintenance debt: edit one of them later and the other two keep the old meaning. 2. The formula is written as plain text with no symbol table: in plain text, Z²α/2 could be Z²_{α/2} or some other grouping, and neither p nor MDE has a defined unit. A formula that cannot be read cannot be maintained or checked — the way it is written here is an accomplice to the error surviving. (The correctness of the formula is deducted in dimension 2; here we count only whether it can be proofread.) 3. Three hardcoded external links (L80-82), all to the same third-party newsletter domain, none with an access date. When the links die, the methodology behind the file becomes untraceable.

What good looks like: state the execution mode once, in one sentence; present the formula as a block with a symbol table (or point at an existing library implementation); move external links into a references section with access dates.

4. Fixes, ranked by value

1. Fix the sample-size formula (add the power term) and resolve its contradiction with "flag <80% power".
2. Define the scale of MDE (absolute percentage points) and show how to convert a relative figure.
3. Annotate the Extend row of the decision table with the peeking conditions (pre-set sample size / sequential testing).
4. Add a scope statement (single binary metric, two variants) and out-of-scope cases (continuous metrics; multiple variants need multiple comparison correction).
5. Settle on one wording for generate-vs-run; specify the output filename.
6. Add an untrusted-input statement and bound the generated script.
7. Give the formula a symbol table; give external links access dates.

5. The one-line verdict

When it is worth using: when you are reading out a two-variant conversion-rate experiment that has already finished and you will check the sample size arithmetic yourself. Its validity checks (SRM, novelty effect, full business cycles) and its five-row decision table make the most usable analysis skeleton in this batch, clearly above "compute a p-value and call it". But do not use it to plan an experiment's sample size — the formula at L27 drops the power term and underestimates the sample needed by about half — and do not follow its "extend the test" recommendation without handling peeking.