Why an audit, not a discovery call
Most consulting opens with an open-ended retainer pitch. We open with something bounded: a fixed-price engagement scoped to the one system you already have doubts about: the pricing model, the attribution pipeline, the reconciliation job, the metric your investors see. The scope is named in writing before we start, and the engagement ends on a date, with a deliverable.
The report is the product. It is not a sales document with the substance held back; it is the findings themselves, quantified, with every number tied to a query that can be re-run. If the audit is the only work we ever do together, it should still have been worth it.
What we look for
These are the failure modes we hunt first, because we have hit every one of them building and operating our own live data products:
- Numbers that don't tie back. Can every metric you publish, report to investors, or bill against be re-derived from raw data? Silent drift between the raw layer and the reporting layer is the most common finding we make.
- Conclusions from samples too thin to support them. Cohort comparisons and headline claims made before the data can statistically carry them. These often reverse once the sample fills in.
- Validation that flatters the model. Label leakage, look-ahead bias, backtests without point-in-time discipline. A model that looks great in the notebook and quietly degrades in production usually failed here.
- Monitoring that fails silently. Checks that error out, get skipped, and report green anyway. A broken verifier is worse than no verifier, because it manufactures false confidence.
- Promised vs. realized gaps. Where quoted prices, predicted scores, or advertised accuracy diverge from what the system actually delivers, measured, not assumed.
How the three weeks run
- Week 1: Trace. Read the code, map the data lineage end to end, and inventory every assumption the system makes. Output: a shared map of how the system actually works, which is often news by itself.
- Week 2: Test. Quantify. Re-derive the headline metrics from raw data, run leakage and point-in-time checks, measure promised-vs-realized gaps, and probe the failure modes above.
- Week 3: Report. Written findings with severity ratings and an ROI-ranked roadmap: what to fix, in what order, and what each fix is worth. Delivered with a walkthrough call.
Who it's for
Funded seed to Series A teams with a model or data product in production (or close to it) and no senior data hire yet. If your data system is load-bearing, if customers, investors, or your own roadmap depend on its numbers being right, and nobody senior has ever adversarially checked it, this is for you.
After the audit
The roadmap stands on its own; your team can execute it. When clients want us to run point on executing it instead, that is what our fractional Head of Data / ML engagements are for, but the audit carries no obligation in that direction.