The deciding difference is operational: promptfoo can run on infrastructure you control, LangSmith cannot. If that constraint is real for you, it settles the choice before anything else is considered.
Tracing, datasets and evals from the LangChain team.
Local eval and red-team harness that runs in CI. No account needed.
Its red-team mode doubles as your prompt-injection regression suite.
| LangSmith | promptfoo | |
|---|---|---|
| Licence | Proprietary | Open source |
| Pricing | freemium | open source |
| Self-hostable | No | Yes |
| Languages | Python, TypeScript | TypeScript, Python |
| Install | — | npm install -g promptfoo |
| Keys required | LANGSMITH_API_KEY | None |
The deciding difference is operational: promptfoo can run on infrastructure you control, LangSmith cannot. If that constraint is real for you, it settles the choice before anything else is considered. LangSmith: Tracing, datasets and evals from the LangChain team. promptfoo: Local eval and red-team harness that runs in CI. No account needed.
LangSmith is a hosted service only. promptfoo can run on your own infrastructure.
LangSmith is proprietary (freemium). promptfoo is open source (open source).
Every page here answers to Accept: text/markdown and returns the same content at roughly a tenth the tokens. No separate site, no toggle — same URL.
curl -s -H "Accept: text/markdown" https://newagent.build/compare/langsmith-vs-promptfoo