Crw MCP Server

Site discovery and page collection follow distinct paths in crw. The crw_map tool finds URLs using a sitemap or a short crawl, returning addresses rather than page content. For individual pages, crw_scrape produces markdown, HTML, or links. Larger collections use crw_crawl to start a background job, followed by crw_check_crawl_status to check progress and fetch the collected pages. Agents can therefore locate pages before retrieving content, or start a crawl to gather pages across a site.

For structured output, crw_extract accepts a prompt, a JSON schema, or both, and requires an LLM. Extraction runs asynchronously; crw_check_extract_status reports progress and returns results for each URL when the job finishes. The crw_cancel_extract tool requests a stop, with cancellation remaining pending until processing of the current URL settles. Local PDF handling uses crw_parse_file with base64 content to produce markdown. It does not perform OCR, so scanned PDFs yield empty markdown and a warning. The Rust-based project includes a built-in MCP server, supports stdio transport, and offers self-hosting under AGPL-3.0.