Privacy choices: we'd like to measure visits anonymously (our own cookie-free counter plus Google Analytics, which sets its own cookies; no ads) and load Beehiiv's newsletter sign-up, which may set its own cookies. Nothing optional runs until you choose. Privacy policy

AgentScout / Tasks / Evaluating agents / Penetration and security testing

Penetration and security testing tools for agent evaluation

Agents that probe apps and code the way an attacker would and report vulnerabilities. Of the 404 tools on our evaluating agents page, 8 fit this narrower job. By pricing: 4 open source, 4 pricing on the maker's site. The most-starred open repos here are AI Beat (1,362 stars), Xalgorix (1,322 stars), Rogue (1,072 stars). Most come from the evals & observability (2), MCP servers (2), security & governance (2) categories.

Updated

GitHub repos (4)

RepoGitHub starsLicenceLast push
AI Beat ↗1,362—2026-10-21
Xalgorix ↗1,322——
Rogue ↗1,072—2026-05-04
Bugtraceai ↗375——

4 repos. Stars and licence come from each repo, checked when the listing was last verified.

FAQ

What are the best AI tools for evaluating agents focused on penetration and security testing?

We list 8. The first three by GitHub stars and featured status are AI Beat: An open-source AI red-teaming suite: PromptBeat attacks models and prompts, and AgentBeat runs black-box security evaluations against HTTP-exposed agents.; Xalgorix: Xalgorix is an open-source agent project that helps builders handle autonomous AI pentesting agents — real-time reconnaissance, vulnerability detection, and exploitation orchestration.; Rogue: Agent evaluator and red-team tool that probes your agent with test conversations and attacks and reports pass or fail against your policies.

Are there free or open-source options?

Yes: 4 open source, 4 pricing on the maker's site. Pricing is as each maker states it.

More from evaluating agents

Tools are grouped by what each says it does. Missing one? Submit it free.