LLM observability
Know why every AI answer went right or wrong.
Each answer can be traced step by step, and every failure becomes a test that runs before the next release.
- About 3 weeks
- to live on real traffic
- Founder-led
- no handoff to juniors
- Fixed price
- agreed before work starts
| support-triage run | Agent | 4.82s | |
| search help centre | Retrieval | 0.53s | |
| classify intent | Model | 0.76s | |
| crm.lookup_customer | Tool | 2.00s | |
| crm.lookup_customer, retry | Tool | 0.29s | |
| draft reply | Model | 0.89s | |
| eval: grounded answer | Eval | 0.20s |
A customer sends a screenshot of a wrong answer, and nobody can find the call that made it
Prompts change in a pull request nobody tests, and quality slips for weeks before anyone notices.
of data leaders lack full visibility into AI decisions.
Sourcedataiku.com
of data leaders say wrong AI answers caused business issues last year.
Sourcedataiku.com
of IT teams saw AI errors slip through into production.
Sourcehelpnetsecurity.com
When an answer goes wrong, you see why and it cannot come back.
| What changes | Without it | With it |
|---|---|---|
| Finding a bad answer | Days of searching logs. | Its trace, open in a minute. |
| Prompt changes | Edited live, never tested. | Versioned and tested before release. |
| Quality | Known when customers complain. | Scored every day on real traffic. |
What we build
What changes for your teams. We pick the tools to fit the systems you already run.
| support-triage run | Agent | 4.82s | |
| search help centre | Retrieval | 0.53s | |
| classify intent | Model | 0.76s | |
| crm.lookup_customer | Tool | 2.00s | |
| crm.lookup_customer, retry | Tool | 0.29s | |
| draft reply | Model | 0.89s | |
| eval: grounded answer | Eval | 0.20s |
| support-triage run | Agent | 4.82s | |
| search help centre | Retrieval | 0.53s | |
| classify intent | Model | 0.76s | |
| crm.lookup_customer | Tool | 2.00s | |
| crm.lookup_customer, retry | Tool | 0.29s | |
| draft reply | Model | 0.89s | |
| eval: grounded answer | Eval | 0.20s |
Measured against where you started
What changes at a company like Acme Inc
Illustrative figures, not a client result. Your own baseline is taken in the first week.
How others built it
Another company built the same kind of system and wrote publicly about how it went.
- DropboxA practical blueprint for evaluating conversational AI at scale
What they learned
Answer quality stays high when every real failure becomes a test that must pass before the next release.
For each release, engineers reviewed 5 to 10% of the regression suite by hand.
Everything runs in your cloud, and stays yours
You own the config, the tests and the runbooks, so you can swap any part later.
Versioned prompts with a review step.
| support-triage run | Agent | 4.82s | |
| search help centre | Retrieval | 0.53s | |
| classify intent | Model | 0.76s | |
| crm.lookup_customer | Tool | 2.00s | |
| crm.lookup_customer, retry | Tool | 0.29s | |
| draft reply | Model | 0.89s | |
| eval: grounded answer | Eval | 0.20s |
Daily quality scores on a sample of real answers.
A test set built from real failures, run on every release.
A runbook your team owns.
Typical building blocks: Self-hosted Langfuse through OpenTelemetry (OpenLLMetry); versioned prompts; online judge scores; failing traces sent to CI datasets.
Alternatives: Braintrust; Arize Phoenix for internal use only (ELv2 licence).
Live in about 3 weeks
The work happens in your cloud, with the people who will own it.
- Week 1Map your AI features and agree what a good answer looks like.
- Week 2Add tracing, version the prompts and start scoring live answers.
- Week 3Failing traces flow into the test set that gates release, then hand over the runbook.
- After thatRunning on real traffic
Every answer traced, every failure a test.
Before we start
- Access to the repos and the cloud account where the AI features run.
- A named owner for each feature who decides what a good answer is.
- About 2 hours a week from each owner.
Is this the right first build?
It fits when
- A user reports a bad answer.
- A prompt changes and quality drops.
- A team on your side will own it after we leave.
It does not fit when
- You want a hosted tool picked and resold to you.
- You want a strategy deck in place of working software.
- Nobody on your side can own it after we leave.
What’s included and what isn’t
Included
- Tracing for up to 3 AI features.
- Up to 20 prompts moved under version control.
- One daily quality score for each feature.
- A starting test set of up to 100 real cases.
Not included
- Rewriting the prompts or the features themselves.
- Labelling by hand beyond the starting test set.
- Monitoring for services that do not call a model.
About 3 weeks. You get a price range before the call and one fixed number after it. That number is the invoice.
- If the work takes longer than we planned, we absorb the cost.
- You pay 50% at the start and 50% at acceptance. Builds of 6 weeks or more pay 40% at the start, 30% at the midpoint and 30% at acceptance.
- If a build starts within 60 days of your Audit results, the full Audit fee comes off its price.
- We start three new projects a quarter, so your start date depends on when you book.
Questions teams ask before they book
Which tracing tool do you use?
We pick the one that fits the systems you already run. You own the config, the tests and the runbooks, so you can swap any part later. Most often self-hosted Langfuse through OpenTelemetry; Braintrust when a hosted tool fits, and Arize Phoenix for internal use only, because of its ELv2 licence.
Do traces leave our cloud?
No. Tracing runs in your account, so prompts and answers stay with you.
Does tracing slow answers down?
No. Traces are sent in the background, after the answer is on its way.
Who scores the answers?
A model judge scores a daily sample against rules your team writes, and people check the judge against their own scores.
Do we need the LLM gateway first?
No. If you have one, the gateway sends its traces here too.
LLM observability, running in your cloud in about 3 weeks.
Every answer traced, every failure a test. You own the config, the tests and the runbooks.