Data engineering
A dashboard nobody believes is worse than no dashboard, because people still quote it in meetings.
Trust is the actual deliverable. The moment two reports disagree about revenue, every number in the system becomes a matter of opinion, and the organisation quietly reverts to spreadsheets. Preventing that is less about tooling than about discipline: one definition of each metric, written down and implemented once, with the calculation living in the warehouse rather than being re-derived slightly differently in every dashboard that needs it.
Most pipelines fail quietly rather than loudly, and that is the dangerous part. A source changes a field, a nightly job dies, an API starts silently truncating results — and the dashboard keeps rendering, just with numbers that are now wrong. So the pipeline needs to assert what it expects: row counts within a plausible range, no unexpected nulls in a key column, yesterday's total not moving today. A job that fails loudly is a nuisance; one that succeeds incorrectly is a business decision made on fiction.
The reasonable starting point is smaller than most proposals suggest. A warehouse, a handful of well-understood sources, defined metrics and one report that answers a question somebody is currently answering by hand — that is enough to be useful, and it establishes whether the definitions survive contact with the people who will argue about them. Modelling every source you own before anyone has used a single number is how data projects run for a year and deliver nothing anyone missed.
How we work
- One definition per metric, implemented once in the warehouse. Every report reads it rather than recalculating it.
- Pipelines assert their expectations and fail loudly. Silent success on bad data is the failure mode that costs money.
- Lineage is visible: for any number on any dashboard, you can trace which source and which transformation produced it.
What this includes
Pick what you need and send it over.
Questions
- Can we just query the production database?
- For a while, yes, and plenty of companies should. It stops working when analytical queries start competing with your application for resources, or when you need history that the production schema overwrites — at which point you are not choosing a warehouse for fashion but because the alternative is slowing down the product.