Free AI agent evaluation tools (2026)

AI agent evaluation tools that you can use without paying: open-source projects plus tools with a free plan. Here are the top 25 of 61 from the AgentScout directory.

How we picked and ranked: tools are matched by category, the pricing each maker states and what each tool says it does. They are ranked by GitHub stars as recorded on each listing (link checks 2026-10-18); tools without a tracked repo follow, by listing maturity. No paid placements.

1. M

   1. MLflow

   evals & observabilityOpen source + cloud★ 28,418 GitHub stars

   Open-source platform for tracing, evaluating and managing agents, LLM apps and ML models.

   Pricing (as stated): Open source + cloud · Mature

2. B

   2. Beads

   codingOpen source★ 27,943 GitHub stars

   Issue-tracking memory for coding agents so they can keep track of long-running work.

   Pricing (as stated): Open source · Mature

3. O

   3. Opik

   evals & observabilityOpen source + cloud★ 22,700 GitHub stars

   Comet's open-source platform for tracing, evaluating and monitoring LLM apps, RAG systems and agent workflows, self-hosted or in the cloud.

   Pricing (as stated): Open source + cloud · Mature

4. M

   4. Moli

   browser/computer useOpen source★ 19,349 GitHub stars

   Lightweight Rust headless browser for AI agents that fetches pages, searches and extracts content, usable via CLI, CDP or WebDriver.

   Pricing (as stated): Open source · Mature

5. C

   5. CVAT AI Agents

   visionFreemium★ 16,943 GitHub stars

   CVAT feature that runs your own detection, segmentation or tracking model as an agent on your infrastructure and feeds its auto-annotations back into CVAT tasks.

   Pricing (as stated): Freemium · Active

6. O

   6. Oumi

   modelsOpen source★ 9,400 GitHub stars

   Open platform for fine-tuning, evaluating and deploying open agentic LLMs and VLMs.

   Pricing (as stated): Open source · Mature

7. M

   7. Manifest

   MCP & infraOpen source★ 7,588 GitHub stars

   Open-source LLM gateway that routes each request from agents and apps to the right model across API keys, subscriptions and local models, with spend tracking.

   Pricing (as stated): Open source · Mature

8. G

   8. Guardrails AI

   evals & observabilityOpen source + cloud★ 7,529 GitHub stars

   Open-source framework for validating LLM inputs and outputs with configurable guardrails.

   Pricing (as stated): Open source + cloud · Mature

9. U

   9. Unstract

   no-code buildersOpen source + cloud★ 7,285 GitHub stars

   LLM-powered platform for extracting structured data from documents via APIs and ETL pipelines.

   Pricing (as stated): Open source + cloud · Mature

10. A

    10. AI-Infra-Guard

    evals & observabilityOpen source★ 6,887 GitHub stars

    Tencent's AI red-teaming platform that scans agents, skills and MCP servers for risks.

    Pricing (as stated): Open source · Mature

11. S

    11. Superagent

    evals & observabilityOpen source★ 6,766 GitHub stars

    SDK for AI agent safety that blocks prompt injections and unsafe tool calls, redacts PII and secrets, and runs red-team scenarios.

    Pricing (as stated): Open source · Mature

12. A

    12. Agent Governance Toolkit

    evals & observabilityOpen source★ 6,435 GitHub stars

    Microsoft toolkit for policy enforcement, identity and sandboxing for AI agents.

    Pricing (as stated): Open source · Mature

13. M

    13. Mission Control

    codingOpen source★ 6,358 GitHub stars

    Self-hosted control plane for dispatching tasks to agents, reviewing runs and tracking spend.

    Pricing (as stated): Open source · Mature

14. A

    14. Agents CLI

    frameworksOpen source★ 6,099 GitHub stars

    Google's CLI and skills that teach coding assistants to build, evaluate and deploy agents on Google Cloud and the Gemini platform.

    Pricing (as stated): Open source · Mature

15. S

    15. SWE-bench

    evals & observabilityOpen source★ 6,044 GitHub stars

    Benchmark that tests whether language models and coding agents can resolve real GitHub issues, with multimodal and verified variants.

    Pricing (as stated): Open source · Mature

16. F

    16. Failproof AI

    evals & observabilityOpen source★ 5,560 GitHub stars

    Observability and policy enforcement for AI agent harnesses.

    Pricing (as stated): Open source · Mature

17. C

    17. coucou

    codingOpen source★ 5,468 GitHub stars

    Mac notch and phone companion that monitors your Claude Code, Codex, Cursor and Gemini CLI sessions and lets you approve permissions on the go.

    Pricing (as stated): Open source · Mature

18. A

    18. agentgateway

    MCP & infraOpen source★ 5,373 GitHub stars

    Proxy for AI agents and MCP servers with security and observability.

    Pricing (as stated): Open source · Mature

19. S

    19. Sparrow

    visionOpen source★ 5,225 GitHub stars

    Open-source tool for structured data extraction from documents and images with ML, LLMs and vision LLMs, plus agentic workflows.

    Pricing (as stated): Open source · Mature

20. A

    20. Agenta

    evals & observabilityOpen source + cloud★ 4,840 GitHub stars

    Open-source LLMOps platform for prompt management, evaluation and observability.

    Pricing (as stated): Open source + cloud · Active

21. L

    21. Latitude

    evals & observabilityOpen source + cloud★ 4,723 GitHub stars

    Observability platform for AI agents that captures traces, groups failures into signals and verifies fixes against real traces.

    Pricing (as stated): Open source + cloud · Active

22. O

    22. Observal

    evals & observabilityOpen source★ 4,517 GitHub stars

    Self-hosted registry and insight engine for sharing coding-agent skills, MCP servers and agents across a team.

    Pricing (as stated): Open source · Active

23. C

    23. Cloudflare MCP Servers

    MCP & infraOpen source★ 4,387 GitHub stars

    Cloudflare's collection of MCP servers for Workers, observability, DNS analytics and other Cloudflare services.

    Pricing (as stated): Open source · Active

24. F

    24. fast-agent

    frameworksOpen source★ 3,935 GitHub stars

    Framework to build, run and evaluate agents with strong MCP, ACP and A2A support.

    Pricing (as stated): Open source · Active

25. A

    25. Antigravity SDK

    frameworksOpen source★ 3,768 GitHub stars

    Google's Python SDK for building stateful AI agents powered by Antigravity and Gemini, abstracting the agent loop and runtime.

    Pricing (as stated): Open source · Active

FAQ

What are the best free AI agent evaluation tools?

Ranked by tracked GitHub stars, the top free AI agent evaluation tools in our directory are MLflow, Beads, Opik, Moli, CVAT AI Agents. Stars measure popularity, not fit, so compare a few against your own task.

How many free AI agent evaluation tools does AgentScout list?

61 tools match. This page shows the top 25. Pricing mix on this page: Open source (18), Open source + cloud (6), Freemium (1).

Are these tools really free?

Pricing is what each maker states on their site. Open-source tools are free to self-host but may need a paid model API key or offer paid cloud plans; freemium tools have a free plan with limits. Check each official site.

Where can I see every tool for evaluating agents?

Our task page lists all 104 tools we match to evaluating agents.

Related: All tools for evaluating agents · evals & observability (30) · MCP & infra (8) · coding (5) · vision (5)

By role: For AI engineers · For QA engineers

All popular searches · AI agents by role · Free tools

Best-of lists: Best AI Agent Observability & Eval Tools · All best lists

Facts come from our directory listings; pricing is as each maker states it and can change. Check official sites before you choose.