LLM observability

Know why every AI answer went right or wrong.

Each answer can be traced step by step, and every failure becomes a test that runs before the next release.

About 3 weeks
to live on real traffic
Founder-led
no handoff to juniors
Fixed price
agreed before work starts
Trace 8f21 Support triage, ticket 48213Retried4.82s total
Steps 7 spans0s2.4s4.8s
support-triage runAgent4.82s
search help centreRetrieval0.53s
classify intentModel0.76s
crm.lookup_customerTool2.00s
crm.lookup_customer, retryTool0.29s
draft replyModel0.89s
eval: grounded answerEval0.20s
Timed outcrm.lookup_customer took 2.00s and hit its limit. The retry returned in 0.29s.
Evals this run
Grounded in sources0.94
Follows tone guide0.98
Policy check1.00
Latency under 6s0.80
Recent runs
8f21Support triageRetried
8f20Support triagePassed
8f1fContract reviewPassed
8f1eText to SQLPassed
The problem

A customer sends a screenshot of a wrong answer, and nobody can find the call that made it

Prompts change in a pull request nobody tests, and quality slips for weeks before anyone notices.

95%

of data leaders lack full visibility into AI decisions.

Sourcedataiku.com

59%

of data leaders say wrong AI answers caused business issues last year.

Sourcedataiku.com

16%

of IT teams saw AI errors slip through into production.

Sourcehelpnetsecurity.com

How we fix it

When an answer goes wrong, you see why and it cannot come back.

What changesWithout itWith it
Finding a bad answerDays of searching logs.Its trace, open in a minute.
Prompt changesEdited live, never tested.Versioned and tested before release.
QualityKnown when customers complain.Scored every day on real traffic.
How it works

What we build

What changes for your teams. We pick the tools to fit the systems you already run.

Steps 7 spans0s2.4s4.8s
support-triage runAgent4.82s
search help centreRetrieval0.53s
classify intentModel0.76s
crm.lookup_customerTool2.00s
crm.lookup_customer, retryTool0.29s
draft replyModel0.89s
eval: grounded answerEval0.20s
Timed outcrm.lookup_customer took 2.00s and hit its limit. The retry returned in 0.29s.
What changes

Measured against where you started

Example: Acme Inc

What changes at a company like Acme Inc

Illustrative figures, not a client result. Your own baseline is taken in the first week.

MeasureBeforeAfter
Time to find the call behind a bad answer2 days10 minutes
Answers scored for qualityNoneA daily sample
Reported failures that become testsNoneAll of them
Prompt changes tested before releaseSomeEvery one

How others built it

Another company built the same kind of system and wrote publicly about how it went.

What you keep

Everything runs in your cloud, and stays yours

You own the config, the tests and the runbooks, so you can swap any part later.

Tracing running in your cloud, configured as code in your repo.

Evals this run
Grounded in sources0.94
Follows tone guide0.98
Policy check1.00
Latency under 6s0.80

Versioned prompts with a review step.

Steps 7 spans0s2.4s4.8s
support-triage runAgent4.82s
search help centreRetrieval0.53s
classify intentModel0.76s
crm.lookup_customerTool2.00s
crm.lookup_customer, retryTool0.29s
draft replyModel0.89s
eval: grounded answerEval0.20s
Timed outcrm.lookup_customer took 2.00s and hit its limit. The retry returned in 0.29s.
  • Daily quality scores on a sample of real answers.

  • A test set built from real failures, run on every release.

  • A runbook your team owns.

Typical building blocks: Self-hosted Langfuse through OpenTelemetry (OpenLLMetry); versioned prompts; online judge scores; failing traces sent to CI datasets.

Alternatives: Braintrust; Arize Phoenix for internal use only (ELv2 licence).

How the weeks run

Live in about 3 weeks

The work happens in your cloud, with the people who will own it.

  1. Week 1Map your AI features and agree what a good answer looks like.
  2. Week 2Add tracing, version the prompts and start scoring live answers.
  3. Week 3Failing traces flow into the test set that gates release, then hand over the runbook.
  4. After thatRunning on real traffic

    Every answer traced, every failure a test.

Before we start

  • Access to the repos and the cloud account where the AI features run.
  • A named owner for each feature who decides what a good answer is.
  • About 2 hours a week from each owner.
Fit

Is this the right first build?

It fits when

  • A user reports a bad answer.
  • A prompt changes and quality drops.
  • A team on your side will own it after we leave.

It does not fit when

  • You want a hosted tool picked and resold to you.
  • You want a strategy deck in place of working software.
  • Nobody on your side can own it after we leave.

What’s included and what isn’t

Included

  • Tracing for up to 3 AI features.
  • Up to 20 prompts moved under version control.
  • One daily quality score for each feature.
  • A starting test set of up to 100 real cases.

Not included

  • Rewriting the prompts or the features themselves.
  • Labelling by hand beyond the starting test set.
  • Monitoring for services that do not call a model.
Fixed quoteOne price, agreed before work starts

About 3 weeks. You get a price range before the call and one fixed number after it. That number is the invoice.

  • If the work takes longer than we planned, we absorb the cost.
  • You pay 50% at the start and 50% at acceptance. Builds of 6 weeks or more pay 40% at the start, 30% at the midpoint and 30% at acceptance.
  • If a build starts within 60 days of your Audit results, the full Audit fee comes off its price.
  • We start three new projects a quarter, so your start date depends on when you book.
Book a demoSee starting prices
FAQ

Questions teams ask before they book

Which tracing tool do you use?

We pick the one that fits the systems you already run. You own the config, the tests and the runbooks, so you can swap any part later. Most often self-hosted Langfuse through OpenTelemetry; Braintrust when a hosted tool fits, and Arize Phoenix for internal use only, because of its ELv2 licence.

Do traces leave our cloud?

No. Tracing runs in your account, so prompts and answers stay with you.

Does tracing slow answers down?

No. Traces are sent in the background, after the answer is on its way.

Who scores the answers?

A model judge scores a daily sample against rules your team writes, and people check the judge against their own scores.

Do we need the LLM gateway first?

No. If you have one, the gateway sends its traces here too.

LLM observability, running in your cloud in about 3 weeks.

Every answer traced, every failure a test. You own the config, the tests and the runbooks.