llm-as-a-verifier
LLM-as-a-Verifier is a general-purpose framework that provides fine-grained feedback for any agent without requiring additional training. It achieves SOTA performance across coding, robotics, and medical agentic benchmarks.
LLM-as-a-Verifier is a verification framework that provides fine-grained feedback for AI agents without requiring additional training, accompanied by TurboAgent, a Claude Code plugin implementation. The framework uses probabilistic scoring across multiple evaluation criteria to rank agent trajectories, select the best responses, and track progress in real-time. It achieves state-of-the-art performance on coding, medical, and robotics benchmarks by generating multiple candidate solutions and using an LLM verifier to choose the optimal one. The TurboAgent plugin brings this capability to Claude Code as a drop-in proxy that automatically generates and verifies multiple responses in parallel.
Key Features
Use Cases
- 01Selecting the best code solution from multiple agent-generated candidates on SWE-Bench tasks
- 02Tracking coding agent progress in real-time to detect and abandon failing trajectories early
- 03Improving Claude Code response quality through automatic multi-candidate generation and verification
- 04Verifying robotics agent trajectories using visual frames from camera inputs
- 05Ranking medical diagnosis agent responses on MedAgentBench with domain-specific criteria
- 06Scaling test-time performance on terminal automation tasks without additional model training
Related Plugins
View moreandrej-karpathy-skills
A single CLAUDE.md file to improve Claude Code behavior, derived from Andrej Karpathy's observations on LLM coding pitfalls.
claude-mem
Persistent Context Across Sessions for Every Agent – Captures everything your agent does during sessions, compresses it with AI, and injects relevant context back into future sessions. Works with Claude Code, OpenClaw, Codex, Gemini, Hermes, Copilot, OpenCode + More
Understand-Anything
Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
rtk
CLI proxy that reduces LLM token consumption by 60-90% on common dev commands. Single Rust binary, zero dependencies
llm-as-a-verifier — FAQ
What is LLM-as-a-Verifier?+
LLM-as-a-Verifier is a verification framework that scores AI agent trajectories using fine-grained logprob-based rewards across multiple criteria, enabling best-of-N selection, progress tracking, and test-time scaling without additional training. TurboAgent is its Claude Code plugin implementation that acts as an API proxy.
How do I install the LLM-as-a-Verifier framework?+
Install via pip with 'pip install llm-verifier' or from source with 'pip install -e .' in a cloned repository. The TurboAgent Claude Code plugin is installed separately with 'pip install git+https://github.com/llm-as-a-verifier/TurboAgent'.
What API keys are required?+
You need at least one verifier backend API key: DEEPSEEK_API_KEY, VERTEX_API_KEY for Google models, or an OpenAI-compatible endpoint that returns logprobs. Set these in a .env file or environment variables.
How do I use TurboAgent with Claude Code?+
Start the proxy with 'turbo-agent' (runs on port 8888 by default), then launch Claude Code with 'ANTHROPIC_BASE_URL=http://localhost:8888 claude'. The proxy automatically generates multiple candidates and selects the best one using verification.
Is LLM-as-a-Verifier free to use?+
The framework itself is open-source and free, but you pay for API calls to the verifier model backend (DeepSeek, Google Gemini, or self-hosted models via vLLM).
Which AI clients does TurboAgent work with?+
TurboAgent is specifically designed as a Claude Code plugin that works as an Anthropic API proxy. It sits between Claude Code and the model provider to intercept and enhance requests.
How do I install llm-as-a-verifier?+
Open the source repository on GitHub and follow its README. llm-as-a-verifier is a plugin — MCP Agents Market links you directly to the official repo.
Is llm-as-a-verifier free?+
llm-as-a-verifier is an open-source project hosted on GitHub. Check the repository for its license and any usage requirements.