Sample Audit Findings

What we found. A CI check exists to verify that every numeric claim in published content can be found in the underlying database — a numeric honesty gate. On inspection, it had flagged 100% of claims as unverifiable on every content change since it was introduced (38 of 38 on one change, 39 of 39 on the next). The cause: the checker queried hardcoded column names that did not exist in any table, every query raised an error, and the errors were silently swallowed. One of the tables it searched did not exist at all. The gate was advisory, so nothing blocked — it simply produced noise that everyone learned to ignore.

Why it matters. A verifier that silently checks nothing is worse than no verifier: it manufactures the feeling of oversight while making it impossible to distinguish a genuinely fabricated number from the checker's own blind spot. This pattern generalizes — data-quality checks, reconciliation jobs, and alerting rules that error out and default to green are among the most common findings we make.

Recommendation (adopted). Fix the schema references, drop the phantom table, and extend the checker to recompute derived aggregates so ratio- and rollup-style claims become checkable rather than skipped. After the fix, the same content that had scored 0 of 39 verified scored 39 of 39. Add the meta-rule: any check that can fail must fail loudly, and "checker found nothing" must be distinguishable from "checker could not look."

What we found. An early look at a thin slice of the sample suggested one routing source (name withheld) delivered on its quotes less often at the $1M trade size than at $1k, pointing at a large-trade slippage problem. Instead of publishing, the claim was held until each measurement cell met a rating threshold: at least 30 realized samples spanning at least 7 days. With the full sample (916 rows over 9.6 days), the pattern reversed: the source realized 65.8% at $1k, 72.9% at $10k, 87.7% at $100k, and 90.7% at $1M. The $1M cohort was its best; retail-size quotes were the real weakness. That weakness held on every day of the sample rather than tracing to one bad run, no peer source shared it, and every miss was a real failure to deliver — not a benign routing artifact.

Why it matters. The thin-sample conclusion was not just imprecise — it pointed in the opposite direction of the truth, and it would have been published under the system's own name. Cohort comparisons on thin samples are the most dangerous class of data-product claim, because they are usually plausible, directional, and wrong.

Recommendation (adopted). Gate every published cohort comparison on the rating threshold, expose the threshold in the methodology, and treat "not yet rated" as a first-class published state rather than an excuse to ship the early number.

What we found. When the sampler produced its first statistically rated results (112 of 168 measurement cells crossing the threshold), the public leaderboard showed 5 of the 6 sampled sources. The sixth was fully rated in the data layer — 20 rated cells — but never appeared in the public rollup: it was dropped at the boundary between the rating layer and the presentation layer.

Why it matters. Every boundary between layers — raw to derived, derived to published — is a place where records can vanish without any error. Nothing was "wrong" in either layer individually; the loss only shows up when you reconcile counts across the boundary. In client systems this same pattern appears as customers missing from dashboards, transactions missing from revenue reports, and cohorts missing from investor metrics.

Recommendation. Add a reconciliation check at each layer boundary asserting that everything rated upstream is represented downstream (or explicitly excluded, with a reason). Boundary reconciliation is cheap to add and catches a class of bug that no within-layer test ever will.