Why an audit, not a discovery call
Most consulting opens with an open-ended retainer pitch. We open with something bounded: a fixed-price engagement scoped to the one system you already have doubts about — the pricing model, the attribution pipeline, the reconciliation job, the metric your investors see. The scope is named in writing before we start, and the engagement ends on a date, with a deliverable.
The report is the product. It is not a sales document with the substance held back; it is the findings themselves, quantified, with every number tied to a query that can be re-run. If the audit is the only work we ever do together, it should still have been worth it.
What we look for
These are the failure modes we hunt first — because we have hit every one of them building and operating our own live data products:
- Numbers that don't tie back. Can every metric you publish, report to investors, or bill against be re-derived from raw data? Silent drift between the raw layer and the reporting layer is the most common finding we make.
- Conclusions from samples too thin to support them. Cohort comparisons and headline claims made before the data can statistically carry them — these often reverse once the sample fills in.
- Validation that flatters the model. Label leakage, look-ahead bias, backtests without point-in-time discipline. A model that looks great in the notebook and quietly degrades in production usually failed here.
- Monitoring that fails silently. Checks that error out, get skipped, and report green anyway. A broken verifier is worse than no verifier, because it manufactures false confidence.
- Promised vs. realized gaps. Where quoted prices, predicted scores, or advertised accuracy diverge from what the system actually delivers — measured, not assumed.
How the three weeks run
- Week 1 — Trace. Read the code, map the data lineage end to end, and inventory every assumption the system makes. Output: a shared map of how the system actually works, which is often news by itself.
- Week 2 — Test. Quantify. Re-derive the headline metrics from raw data, run leakage and point-in-time checks, measure promised-vs-realized gaps, and probe the failure modes above.
- Week 3 — Report. Written findings with severity ratings and an ROI-ranked roadmap: what to fix, in what order, and what each fix is worth. Delivered with a walkthrough call.
Who it's for
Funded seed to Series A teams with a model or data product in production (or close to it) and no senior data hire yet. If your data system is load-bearing — customers, investors, or your own roadmap depend on its numbers being right — and nobody senior has ever adversarially checked it, this is for you.
After the audit
The roadmap stands on its own; your team can execute it. When clients want us to run point on executing it instead, that is what our fractional Head of Data / ML engagements are for — but the audit carries no obligation in that direction.