AI and Machine Learning
Models and language model applications, put into production and kept there. The work includes retrieval systems, agentic workflows, evaluation harnesses and the monitoring that detects changes in cost, latency and answer quality after release.
The problem
We built something six months ago that impressed the board. It still isn't in production and nobody can tell me exactly why.
Moving from a demonstration to a production release means answering several questions in writing: what happens when retrieval returns nothing relevant, how a wrong answer is caught before a user acts on it, and who is accountable when output quality drifts after a model version changes. Measurement is the second problem. Without an evaluation harness a team cannot tell whether a change improved the system or made it worse, and the changes get made less and less often.
Our approach
We build the evaluation harness before we tune anything. Once quality is scored against a fixed set of cases, a retrieval change, a prompt change and a move to a newer model version can all be compared on the same basis. Those scores also support the decision about whether the system is ready to go live.
- 01
Problem framing
We decide what a correct answer looks like and how a wrong one will be caught. Some problems that arrive framed as machine learning are better solved with a query, and we say so when that is the case.
- 02
Evaluation harness
A fixed case set with expected outcomes is scored automatically, with a human review step for the cases where automatic scoring is not credible. The case set is version-controlled and grows as real failures come in.
- 03
Build
For retrieval systems we work on chunking, embedding, retrieval strategy and reranking, tuned against measured retrieval quality, since a fluent answer is not evidence that the right document was found. For predictive models we build feature pipelines that produce identical values in training and in serving, because that mismatch is a common cause of a model that works in development and fails in production.
- 04
Deployment and monitoring
Models and prompts are versioned, cost and latency are tracked per request, and drift is monitored on inputs and outputs. Rollback is a written procedure. We test the path back to the previous version before go-live.
How it fits together
What you get
- Evaluation harness with a versioned case set and scoring, runnable in CI
- The application or model, deployed, with prompts and model versions under version control
- Feature or retrieval pipeline sharing one code path between training and serving
- Cost and latency instrumentation per request, with budget alerting
- Drift monitoring on inputs and outputs with defined thresholds
- Written assessment of failure modes, including the ones we did not fully solve
Technology
Language model platforms
- Anthropic Claude
- OpenAI
- Amazon Bedrock
- Azure OpenAI
- Snowflake Cortex
Retrieval and orchestration
- pgvector
- Snowflake Cortex Search
- LangGraph
- LlamaIndex
Machine learning
- Python
- scikit-learn
- PyTorch
- MLflow
- SageMaker
- Snowpark
Engagement
- Duration
- Six to sixteen weeks to a production deployment, with a shorter feasibility phase available first
- Team
- A machine learning engineer, a data engineer and a lead who owns the evaluation design
- Starts with
- The first phase is a feasibility review of two to three weeks. It produces the evaluation harness and one working path through the whole system, which is usually enough to show whether the approach holds up.