High-performance web crawling
## Crawl4AI MCP Server: AI-Optimized Web Crawling The **Crawl4AI MCP Server** integrates the AI-focused web crawler directly into Google Antigravity, enabling AI assistants to extract clean, structured content from websites optimized for LLM consumption. This integration brings intelligent web scraping to your development workflow. ### Why Crawl4AI MCP? - **AI-Optimized Output**: Content extracted and formatted specifically for LLM processing - **JavaScript Rendering**: Handle dynamic, JavaScript-heavy websites - **Content Extraction**: Automatic main content identification and extraction - **Structured Output**: Return data in clean, structured formats - **Media Handling**: Extract and process images and embedded media ### Key Features #### 1. Page Crawling ```python from anthropic import Anthropic client = Anthropic() response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{ "role": "user", "content": "Crawl this documentation page and extract the main content in Markdown" }], tools=[{ "name": "crawl4ai_page", "description": "Crawl web pages" }] ) ``` #### 2. Site Crawling ```python # Crawl multiple pages response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{ "role": "user", "content": "Crawl the documentation site and extract content from all tutorial pages" }], tools=[{"name": "crawl4ai_site", "description": "Multi-page crawling"}] ) ``` #### 3. Structured Extraction ```python # Extract structured data response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{ "role": "user", "content": "Extract product information from this e-commerce page as structured JSON" }], tools=[{"name": "crawl4ai_structured", "description": "Structured extraction"}] ) ``` #### 4. Dynamic Content ```python # Handle JavaScript content response = client.messages.create( model="claude-sonnet-4-20250514", max_tokens=1024, messages=[{ "role": "user", "content": "Crawl this SPA and wait for dynamic content to load before extracting" }], tools=[{"name": "crawl4ai_dynamic", "description": "Dynamic content"}] ) ``` ### Configuration ```json { "mcpServers": { "crawl4ai": { "command": "npx", "args": ["-y", "@anthropic/mcp-crawl4ai"], "env": { "HEADLESS": "true", "TIMEOUT": "30000" } } } } ``` ### Use Cases **Research Automation**: Gather and structure information from multiple web sources. **Content Aggregation**: Extract and organize content for knowledge bases. **Data Collection**: Collect structured data from websites for analysis. **Documentation Ingestion**: Process documentation sites for RAG applications. The Crawl4AI MCP Server brings AI-optimized web crawling directly into your development workflow, enabling intelligent content extraction.
{
"mcpServers": {
"crawl4ai": {
"mcpServers": {
"crawl4ai": {
"args": [
"crawl4ai-mcp-server"
],
"command": "uvx"
}
}
}
}
}