[09] Observability & evals · required/ How you run it

Observability & evals for AI agents

Traces of what it did, and a score for whether that was any good.

6 options tracked · 4 open source · 4 self-hostable · required in every stack

· All 6 options/ Compare
OptionWhat it doesLicenceSelf-host
LangfuseOpen-source tracing, prompt management and evals. Self-hostable in one compose file.Open sourceYes
LangSmithTracing, datasets and evals from the LangChain team.ProprietaryNo
BraintrustEval-first platform — scorers, datasets and a playground for prompt iteration.ProprietaryYes
Arize PhoenixOpenTelemetry-native tracing and evals you can run locally.Open sourceYes
Pydantic LogfireOpenTelemetry observability with first-class Python and Pydantic AI support.Open sourceNo
promptfooLocal eval and red-team harness that runs in CI. No account needed.Open sourceYes
· Head to head/ 15 comparisons
· Common questions/ FAQ

What is the observability & evals layer of an AI agent?

Traces of what it did, and a score for whether that was any good. Every stack needs one — it is not optional.

How many observability & evals options are there?

This registry tracks 6. 4 are open source and 4 can run on your own infrastructure.

Which observability & evals option should I choose?

It depends on constraints rather than preference: whether you must self-host, whether the budget allows a hosted service, and which language your team writes. Describe what you are building and the advisor fills this layer along with the other 9.

· The other 9 layers/ Keep going
· For agents/ This page, machine-readable

Every page here answers to Accept: text/markdown and returns the same content at roughly a tenth the tokens. No separate site, no toggle — same URL.

curl -s -H "Accept: text/markdown" https://newagent.build/layers/observability