Eval and benchmark tools for building AI agents

Evaluation tools that score agent outputs, catch regressions and compare prompts and models. Of the 251 tools on our building AI agents page, 8 fit this narrower job. By pricing: 6 open source, 1 source-available, 1 free or freemium. The most-starred open repos here are Agents CLI (6,137 stars), Apache Maka (5,866 stars), Kiln (5,223 stars). Most come from the frameworks (6), evals & observability (2) categories.

Updated October 22, 2026

Agents CLI

NewframeworksOpen source

Google's CLI and skills that teach coding assistants to build, evaluate and deploy agents on Google Cloud and the Gemini platform.

Added Oct 18, 2026

Apache Maka

NewframeworksOpen source

Incubating Apache agent harness and workspace that records everything it does and publishes benchmarked run results.

Added Oct 18, 2026

Kiln

Newevals & observabilitySource-available

Free desktop app and open-source library for building AI systems with evals, RAG, agents, fine-tuning and synthetic data in one workbench.

Added Oct 18, 2026

fast-agent

frameworksOpen source

Framework to build, run and evaluate agents with strong MCP, ACP and A2A support.

Added Oct 16, 2026

EvoAgentX

NewframeworksOpen source

Framework for building, evaluating and automatically evolving LLM agents and agentic workflows.

Added Oct 18, 2026

Unreal Agent

NewframeworksOpen source

Unreal Agent is an async-first agent harness from Unreal Labs, a team researching how to make agentic workflows efficient. The library handles inputs with stable IDs and deduplication, append-only sessions that can be forked, and a coordinator that manages each model turn and translates tool calls into operations. It ships with a cloud agent runner, a terminal UI and benchmark runners, under the MIT license.

Added Oct 22, 2026

tRPC-Agent-Go

NewframeworksOpen source

Go framework for production agent systems with graph workflows, tools, memory, A2A, AG-UI, MCP, evaluation and OpenTelemetry.

Added Oct 18, 2026

W&B Weave

evals & observabilityFreemium

Weights & Biases toolkit for tracing and evaluating agents.

Added Oct 14, 2026

GitHub repos (7)

Repo	GitHub stars	Licence	Last push
Agents CLI ↗	6,137	Apache License 2.0	2026-10-20
Apache Maka ↗	5,866	Apache License 2.0	2026-10-22
Kiln ↗	5,223	MIT for the Python libraries; the desktop app is free under the Kiln AI Desktop EULA	2026-10-21
fast-agent ↗	3,935	Apache License 2.0	2026-10-21
EvoAgentX ↗	3,385	—	2026-10-22
Unreal Agent ↗	2,211	—	—
tRPC-Agent-Go ↗	1,867	—	2026-10-22

7 repos. Stars and licence come from each repo, checked when the listing was last verified.

FAQ

What are the best AI tools for building AI agents focused on evals and benchmarks?

We list 8. The first three by GitHub stars and featured status are Agents CLI: Google's CLI and skills that teach coding assistants to build, evaluate and deploy agents on Google Cloud and the Gemini platform.; Apache Maka: Incubating Apache agent harness and workspace that records everything it does and publishes benchmarked run results.; Kiln: Free desktop app and open-source library for building AI systems with evals, RAG, agents, fine-tuning and synthetic data in one workbench.

Are there free or open-source options?

Yes: 6 open source, 1 source-available, 1 free or freemium. Pricing is as each maker states it.

More from building AI agents

Tools are grouped by what each says it does. Missing one? Submit it free.