Data Engineering
Pipelines that report failures clearly, restart without creating duplicates, and can be diagnosed by an engineer who did not write them. The build covers ingestion, change data capture, transformation and orchestration.
The problem
We have hundreds of jobs in a scheduler nobody fully understands. When one fails overnight, the whole morning's reporting is wrong and we find out from the business.
Most pipeline failures start outside the transformation logic. A source system changes a column type without notice, a job that was assumed to be idempotent turns out not to be, or a partial load succeeds and no test catches it. The outage itself is the smaller cost. Once a business team has twice acted on a number that was wrong, they start checking figures by hand against the source system, and that habit is slow to reverse.
Our approach
We build pipelines the way application teams build services: version-controlled, tested, observable, and restartable from any point without producing duplicates. Log output gets the same attention. A failure should be traceable to its cause without anyone opening the pipeline code.
- 01
Source contract
We establish what each source guarantees: delivery frequency, schema stability, whether deletes are propagated, and how late-arriving data behaves. Where a guarantee does not exist, we write down the assumption we are making and build a check that catches it being wrong.
- 02
Ingestion and change data capture
We land raw data unmodified, with the load metadata needed to reproduce any historical state. The change data capture method is a deliberate choice, because log-based, query-based and full-refresh replication fail in different ways and the right one depends on how the source handles deletes.
- 03
Transformation
We model transformations in dbt and attach tests to the assumptions that matter: uniqueness, referential integrity, accepted values, freshness and row count deltas. Incremental models are written so that a rerun over an overlapping window produces the same result as a clean run.
- 04
Orchestration and alerting
Dependencies are expressed explicitly and retries use backoff. Every alert names the model, the failing test and the reports affected downstream. Without that, the engineer on call spends the first part of the incident working out what broke and what it fed.
How it fits together
What you get
- Ingestion configuration and change data capture setup, documented per source with its delivery contract
- A dbt project with models, tests, sources and generated documentation
- Orchestration definitions with dependency graph, retry policy and alert routing
- Data quality test suite covering freshness, uniqueness, referential integrity and volume anomalies
- Runbook for the failure modes we encountered during build, with the diagnosis steps for each
- CI pipeline that runs tests against a pull request before it can merge
Technology
Transformation
- dbt Core
- dbt Cloud
- SQL
- Python
Ingestion and CDC
- Fivetran
- Airbyte
- Snowpipe
- Debezium
- Qlik Replicate
- Hevo
Orchestration
- Apache Airflow
- Dagster
- dbt Cloud jobs
- Snowflake Tasks
Platforms
- Snowflake
- Databricks
- BigQuery
- Microsoft Fabric
Engagement
- Duration
- Eight to twenty weeks for a build, or an ongoing arrangement where we hold part of the platform backlog
- Team
- Two to four data engineers with an architect across the design decisions
- Starts with
- We start with a pipeline review. We read the existing jobs and tests, work through the failure modes, and come back with the defects we found and what each one would cost to fix.