Penetration and security testing tools for agent evaluation
Agents that probe apps and code the way an attacker would and report vulnerabilities. Of the 404 tools on our evaluating agents page, 8 fit this narrower job. By pricing: 4 open source, 4 pricing on the maker's site. The most-starred open repos here are AI Beat (1,362 stars), Xalgorix (1,322 stars), Rogue (1,072 stars). Most come from the evals & observability (2), MCP servers (2), security & governance (2) categories.
Updated
AI Beat
An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.
Added
Xalgorix
Xalgorix is an open-source agent project that helps builders handle autonomous AI pentesting agents — real-time reconnaissance, vulnerability detection, and exploitation orchestration.
Added
Rogue
Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.
Added
Bugtraceai
Open-source, self-hosted web pentesting suite with agentic scanning, analyst tools, API testing, model checkpoints and MCP integrations.
Added
Almanax
AI security engineer that finds vulnerabilities in applications and smart contracts and reviews pull requests; acquired by depthfirst (June 2026).
Added
ByteRay AI
AI-augmented binary vulnerability analysis with 38 MCP tools for taint tracing and zero-day hunting
Added
Octane
Agentic security analysis that finds exploitable vulnerabilities in mission-critical code, including DeFi smart contracts.
Added
Sherlock AI
Sherlock's AI auditor for smart-contract vulnerabilities.
Added
GitHub repos (4)
| Repo | GitHub stars | Licence | Last push |
|---|---|---|---|
| 1,362 | — | 2026-10-21 | |
| 1,322 | — | — | |
| 1,072 | — | 2026-05-04 | |
| 375 | — | — |
4 repos. Stars and licence come from each repo, checked when the listing was last verified.
FAQ
What are the best AI tools for evaluating agents focused on penetration and security testing?
We list 8. The first three by GitHub stars and featured status are AI Beat: An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.; Xalgorix: Xalgorix is an open-source agent project that helps builders handle autonomous AI pentesting agents — real-time reconnaissance, vulnerability detection, and exploitation orchestration.; Rogue: Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.
Are there free or open-source options?
Yes: 4 open source, 4 pricing on the maker's site. Pricing is as each maker states it.
More from evaluating agents
Tools are grouped by what each says it does. Missing one? Submit it free.