Why an audit, not a discovery call
Most consulting opens with an open-ended retainer pitch. We open with something bounded: a fixed-price engagement scoped to the one system you already have doubts about: the pricing model, the attribution pipeline, the reconciliation job, the metric your investors see. The scope is named in writing before we start, and the engagement ends on a date, with a deliverable.
The report is the product. It is not a sales document with the substance held back; it is the findings themselves, quantified, with every number tied to a query that can be re-run. If the audit is the only work we ever do together, it should still have been worth it.
What we look for
These are the failure modes we hunt first, because we have hit every one of them building and operating our own live data products:
- Numbers that don't tie back. Can every metric you publish, report to investors, or bill against be re-derived from raw data? Silent drift between the raw layer and the reporting layer is the most common finding we make.
- Conclusions from samples too thin to support them. Cohort comparisons and headline claims made before the data can statistically carry them. These often reverse once the sample fills in.
- Validation that flatters the model. Label leakage, look-ahead bias, backtests without point-in-time discipline. A model that looks great in the notebook and quietly degrades in production usually failed here.
- Monitoring that fails silently. Checks that error out, get skipped, and report green anyway. A broken verifier is worse than no verifier, because it manufactures false confidence.
- Promised vs. realized gaps. Where quoted prices, predicted scores, or advertised accuracy diverge from what the system actually delivers, measured, not assumed.
How the three weeks run
- Week 1: Trace. Read the code, map the data lineage end to end, and inventory every assumption the system makes. Output: a shared map of how the system actually works, which is often news by itself.
- Week 2: Test. Quantify. Re-derive the headline metrics from raw data, run leakage and point-in-time checks, measure promised-vs-realized gaps, and probe the failure modes above.
- Week 3: Report. Written findings with severity ratings and an ROI-ranked roadmap: what to fix, in what order, and what each fix is worth. Delivered with a walkthrough call.
What counts as delivered
The scope is fixed before week one, and so is the bar the work is measured against. Both are written into the scope document you sign, which is published in full, so neither side is arguing about the standard after the report lands.
What makes a finding count
A finding counts toward the engagement only if it passes all four:
- In scope. It concerns the one system named in the scope document.
- Reproducible. You can re-run it yourself, against your own data, and get the same result. Evidence that cannot be re-run is not a finding.
- Consequential. It names what the defect costs you: a number that cannot be re-derived, a claim the data does not support, a failure your monitoring will not catch. Preferences about how we would have built it are not findings.
- Actionable. It carries a severity rating and a specific recommended fix, with its position in the ROI ranking.
How many
The audit commits to five qualified findings. Severity is assigned against a rubric fixed before week one, not negotiated after delivery, so the count cannot be met by padding it with trivia.
A clean system is a real result. If fewer than five qualified findings come back, you choose: a $3,000 rebate, or a fourth week against a second system you name, at no extra cost. That clause exists so that padding the report is never the cheaper option for us.
What each finding carries
- The query, script, or check that produces it, runnable by your team.
- The output of that run, dated.
- The exact input it ran against: table, snapshot, date range.
- What a passing result looks like, so your team can verify the fix without us.
What you can reject a finding on
You may reject a finding in writing within ten business days of delivery, on any of these grounds:
- Out of scope. It concerns a system other than the one named.
- Not reproducible. You run the supplied query against the named input and do not get the reported result.
- Wrong. The system does not behave as described, and you can show it.
- Already documented. It was in your tracker or your docs before the start date, and you can link it.
A rejected finding does not count toward the five, and we replace it at no charge.
That list is exhaustive. A finding cannot be rejected for being unwelcome, for confirming something you had suspected but never written down, or for reaching a conclusion you would rather it did not. The criteria are set before the work starts and do not move while it runs: we cannot loosen them to inflate the count, and they cannot be tightened to shrink it.
Who it's for
Funded seed to Series A teams with a model or data product in production (or close to it) and no senior data hire yet. If your data system is load-bearing, if customers, investors, or your own roadmap depend on its numbers being right, and nobody senior has ever adversarially checked it, this is for you.
After the audit
The roadmap stands on its own; your team can execute it. When clients want us to run point on executing it instead, that is what our fractional Head of Data / ML engagements are for, but the audit carries no obligation in that direction.