Happy slice of mcp pizzaMCP.pizza

mcp-crawl4ai-rag

MCP.Pizza Chef: coleam00

Point it at a single page, a whole documentation site, or a list of links, and it reads everything, files it away, and lets your assistant search that content by meaning later on. It can pull out code examples on their own, and can optionally map a GitHub project so it flags when an assistant invents a function that does not exist. Setup is a real project: expect Docker or Python, a free Supabase database, and an OpenAI key.

Unmaintained · No commits in 14 months.
Coding
Data
Web/Research

Use This MCP server To

Index a documentation site so my assistant answers from it Ask questions about a site I crawled last week Pull working code examples out of docs I saved Search several crawled sources at once, or just one Catch when my assistant invents a function that does not exist Keep a private copy of docs that keep moving around

README

Crawl4AI RAG MCP Server

Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants

A powerful implementation of the Model Context Protocol (MCP) integrated with Crawl4AI and Supabase for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.

With this MCP server, you can scrape anything and then use that knowledge anywhere for RAG.

The primary goal is to bring this MCP server into Archon as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.

Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!

Overview

This MCP server provides tools that enable AI agents to crawl websites, store content in a vector database (Supabase), and perform RAG over the crawled content. It follows the best practices for building MCP servers based on the Mem0 MCP server template I provided on my channel previously.

The server includes several advanced RAG strategies that can be enabled to enhance retrieval quality:

  • Contextual Embeddings for enriched semantic understanding
  • Hybrid Search combining vector and keyword search
  • Agentic RAG for specialized code example extraction
  • Reranking for improved result relevance using cross-encoder models
  • Knowledge Graph for AI hallucination detection and repository code analysis

See the Configuration section below for details on how to enable and configure these strategies.

Vision

The Crawl4AI RAG MCP server is just the beginning. Here's where we're headed:

  1. Integration with Archon: Building this system directly into Archon to create a comprehensive knowledge engine for AI coding assistants to build better AI agents.

  2. Multiple Embedding Models: Expanding beyond OpenAI to support a variety of embedding models, including the ability to run everything locally with Ollama for complete control and privacy.

  3. Advanced RAG Strategies: Implementing sophisticated retrieval techniques like contextual retrieval, late chunking, and others to move beyond basic "naive lookups" and significantly enhance the power and precision of the RAG system, especially as it integrates with Archon.

  4. Enhanced Chunking Strategy: Implementing a Context 7-inspired chunking approach that focuses on examples and creates distinct, semantically meaningful sections for each chunk, improving retrieval precision.

  5. Performance Optimization: Increasing crawling and indexing speed to make it more realistic to "quickly" index new documentation to then leverage it within the same prompt in an AI coding assistant.

Features

mcp-crawl4ai-rag FAQ

What do I need before I start?
Docker or Python 3.12, a free Supabase database, and an OpenAI key. Neo4j only if you want the code-checking extras.
Can I use this to make my assistant answer from a docs site?
Yes. Crawl the site once, then ask questions and it searches what it stored.
Which apps does it work in?
Claude Desktop, Windsurf, and other assistants that can point at a server you run yourself.
Is it hard to set up?
Yes, this is the developer end of the shelf. You will be in a terminal creating a database and running containers.
Does it cost anything?
The software is free. You pay whatever OpenAI charges for reading your crawled text, plus any hosting.
Can it crawl an entire site in one go?
Yes. Give it a sitemap or a starting page and it follows internal links on its own.
Is it finished?
The author calls it a testbed that will be folded into a larger project, so expect rough edges.