webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Webclaw is a web content extraction tool built in Rust that turns websites into clean markdown, structured JSON, and LLM-ready text. It offers a CLI, REST API, and MCP server that integrates with Claude Desktop, Claude Code, Cursor, Windsurf, and other MCP-compatible AI tools. Developers can scrape single pages, crawl entire sites, extract structured data, compare page changes over time, and pull brand assets—all with local-first processing that doesn't require API keys for most sites. The tool also ships as an installable skill for MCP agents and provides TypeScript, Python, and Go SDKs for programmatic access.
Key Features
Use Cases
- 01Give AI agents clean web content by integrating the MCP server with Claude, Cursor, or other assistants
- 02Prepare documentation sites, blogs, and help centers for RAG ingestion pipelines
- 03Monitor competitor pricing pages, changelogs, and product documentation for changes
- 04Extract structured data from messy web pages for automation workflows
- 05Crawl and summarize research sources with multi-page citation support
- 06Collect brand intelligence including logos, color palettes, fonts, and social metadata
Related MCP Servers
View moremarkitdown
Python tool for converting files and office documents to Markdown.
firecrawl
The context API to search, scrape, and interact with the web at scale. 🔥
prompts.chat
f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.
langflow
Langflow is a powerful tool for building and deploying AI-powered agents and workflows.
webclaw — FAQ
What is webclaw?+
Webclaw is a Rust-based web scraping and extraction tool that converts websites into clean markdown, JSON, or LLM-ready text. It provides a CLI, MCP server for AI agents, REST API, and SDKs, with most features running locally without API keys.
How do I install webclaw for Claude Desktop or Claude Code?+
Run 'npx create-webclaw' to auto-detect your MCP-compatible clients and configure them automatically. Alternatively, add the MCP server manually by pointing your client config at 'npx -y @webclaw/mcp'.
Which AI clients work with webclaw?+
Webclaw's MCP server works with Claude Code, Claude Desktop, Cursor, Windsurf, OpenCode, Codex CLI, and other MCP-compatible tools. It can also be added as a skill to agents that support the skills.sh format.
Do I need an API key to use webclaw?+
No API key is required for local extraction of most sites. Set WEBCLAW_API_KEY only when you need the hosted API for JavaScript rendering, bot-protected pages, web search, or async crawl jobs.
Is webclaw free and open source?+
Yes, webclaw is open source under the AGPL-3.0 license. The CLI and MCP server are free to use locally. A hosted API at webclaw.io is available for features like JavaScript rendering and web search.
What are the prerequisites for building webclaw from source?+
Building from source requires Rust toolchain and native build tools: pkg-config, OpenSSL, CMake, Clang, and Git. On macOS run 'xcode-select --install'; on Debian/Ubuntu install pkg-config, libssl-dev, cmake, clang, git, and build-essential.
How do I install webclaw?+
Open the source repository on GitHub and follow its README. webclaw is a mcp server — MCP Agents Market links you directly to the official repo.
Is webclaw free?+
webclaw is an open-source project hosted on GitHub. Check the repository for its license and any usage requirements.