Best AI Agent Observability & Eval Tools (2026)

Tracing, evaluation, guardrail and monitoring tools for seeing what your AI agents do and checking that they work. Below are the top 20 of 25 matching tools in the AgentScout directory, ranked.

How we ranked: Ranked by GitHub stars as recorded on each AgentScout listing (link checks 2026-10-18). Tools without a tracked public repo come after, in alphabetical order. Stars measure developer popularity, not quality or fit for your job, so read each listing before you choose. No paid placements.

1. M

   1. MLflow

   evals & observabilityOpen source + cloud★ 28,418 GitHub stars

   Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.

   Pricing (as stated): Open source + cloud · Mature

2. A

   2. Arize Phoenix

   evals & observabilityOpen source★ 11,828 GitHub stars

   Open-source tracing and evaluation for LLM apps.

   Pricing (as stated): Open source · Mature

3. G

   3. Guardrails AI

   evals & observabilityOpen source + cloud★ 7,529 GitHub stars

   Open-source framework for validating LLM inputs and outputs with configurable guardrails.

   Pricing (as stated): Open source + cloud · Mature

4. A

   4. AI-Infra-Guard

   evals & observabilityOpen source★ 6,887 GitHub stars

   Tencent's AI red-teaming platform that scans agents, skills and MCP servers for risks.

   Pricing (as stated): Open source · Mature

5. A

   5. Agent Governance Toolkit

   evals & observabilityOpen source★ 6,435 GitHub stars

   Microsoft toolkit for policy enforcement, identity and sandboxing for AI agents.

   Pricing (as stated): Open source · Mature

6. F

   6. Failproof AI

   evals & observabilityOpen source★ 5,560 GitHub stars

   Observability and policy enforcement for AI agent harnesses.

   Pricing (as stated): Open source · Mature

7. A

   7. Agenta

   evals & observabilityOpen source + cloud★ 4,840 GitHub stars

   Open-source LLMOps platform for prompt management, evaluation and observability.

   Pricing (as stated): Open source + cloud · Active

8. T

   8. TruLens

   evals & observabilityOpen source★ 3,602 GitHub stars

   Evaluation and tracking for LLM experiments and AI agents.

   Pricing (as stated): Open source · Active

9. S

   9. Snyk Agent Scan

   evals & observabilityOpen source★ 3,142 GitHub stars

   Snyk's scanner for security risks in MCP servers and agent skills.

   Pricing (as stated): Open source · Active

10. O

    10. OpenLIT

    evals & observabilityOpen source★ 2,853 GitHub stars

    Open-source platform to trace, evaluate and guard agents and LLM apps.

    Pricing (as stated): Open source · Active

11. A

    11. Agent Memory Leaderboard

    evals & observabilityOpen source★ 1,110 GitHub stars

    Public leaderboard and open evaluation protocol, run by a group of universities and labs, for comparing long-term memory systems for agents in recurring submission rounds.

    Pricing (as stated): Open source · Active

12. G

    12. Galileo Agent Leaderboard

    evals & observabilityOpen source★ 226 GitHub stars

    Galileo's leaderboard that ranks how well LLMs perform as agents on tool use and multi-step tasks.

    Pricing (as stated): Open source · Active

13. A

    13. AgentOps

    evals & observabilityFreemium

    Monitoring and session replay for AI agents.

    Pricing (as stated): Free tier + paid plans · Active

14. B

    14. Basalt

    evals & observabilityPaid

    Captures customer behavior so your AI agent can learn and improve from real interactions.

    Pricing (as stated): Paid · Active

15. B

    15. Braintrust

    evals & observabilityFreemium

    Evals and observability platform for AI products.

    Pricing (as stated): Free tier + paid plans · Mature

16. C

    16. CoreWeave Forge

    evals & observabilityFreemium

    One product to run, observe, evaluate and post-train agents (W&B, OpenPipe, sandboxes).

    Pricing (as stated): Free tier; Pro from $60/mo · Newly launched

17. H

    17. Hamming AI

    evals & observabilityPaid

    Testing and QA platform for voice and chat agents that generates scenarios and replays real calls.

    Pricing (as stated): Paid · Active

18. H

    18. Helicone

    evals & observabilityOpen source + cloud

    Open-source LLM observability and gateway.

    Pricing (as stated): Open source + paid cloud · Mature

19. K

    19. Kore.ai Autoloop

    evals & observabilityEnterprise

    Optimization engine in Kore.ai Agent Platform Artemis that evaluates agents against business goals, diagnoses failures from traces, applies targeted fixes, and re-verifies; GA Oct 2026.

    Pricing (as stated): Enterprise (part of Kore.ai Agent Platform Artemis) · Generally available

20. L

    20. Langfuse

    evals & observabilityOpen source + cloud

    Open-source LLM observability, tracing and evals.

    Pricing (as stated): Open source + paid cloud · Mature

Browse by category

Related: Monitoring agents in production · Evaluating agents · Langfuse vs LangSmith

More best-of lists: All best lists · Best AI Coding Agents · Best AI Agent Frameworks · Best AI Browser Agents · Best Open-Source MCP Servers · Most-Starred Open-Source AI Agent Projects · Best Self-Hosted AI Agents · Best Free AI Agents (Free Plans) · Best No-Code AI Agent Builders · Best AI Voice Agent Platforms · Best AI Vision Tools for Agents · Best Personal AI Agents · Best AI Agents for Business Teams · Best Crypto AI Agents · Best AI Productivity Agents

Facts come from our directory listings; pricing is as each maker states it and GitHub stars are as recorded on each listing. Check official sites before you choose.