Context.dev: Web Scraping & Crawl API for AI Agents

What is Context.dev?

Context.dev is a web scraping API for AI agents. One REST endpoint scrapes any URL to clean markdown or rendered HTML, crawls entire websites, and extracts structured data on demand. Purpose-built for LLM pipelines and agent tooling.

How do I scrape a website with the Context.dev API?

Send a single POST to /v1/scrape with a URL and Context.dev returns clean, LLM-ready markdown (or rendered HTML, including JS-executed content). No headless browser to host, no parsers to maintain, no proxy pool to manage. SDKs are available for TypeScript, Python, Ruby, and Go; the REST API works from any language.

How do I crawl an entire website?

Use the crawl website endpoint to discover and convert every page on a domain to markdown in one call. You set maxPages, maxDepth, and main-content filtering; Context.dev handles the spider, deduplication, and sitemap discovery so you can pipe a whole site into embeddings or a RAG index.

How is Context.dev different from Firecrawl, Jina, or other scraper APIs?

Context.dev is a unified web scraping and crawl API focused on AI agents and structured-data extraction. Beyond markdown and HTML scraping, it ships first-class brand intelligence (typed company data, logos, colors, fonts) and AI-powered custom extraction in the same API. We do not run search or browser-automation surfaces; we focus on being the cleanest data layer for agents.

Can I extract structured data from a webpage with AI?

Yes. The AI Query endpoint takes a URL and a list of datapoints (name, description, type) and returns typed structured data extracted from the live page. Useful for product catalogs, pricing tiers, competitive intel, and any custom schema your agent needs.

Can I use Context.dev to power LLMs and RAG pipelines?

Yes. Scrape any URL to clean markdown, crawl an entire site to feed an embedding index, and ground LLM responses with real-time web context, all from one API. We use this internally and ship it as the same primitive customers like Mintlify, daily.dev, and Propane build on.

What programming languages are supported?

Context.dev provides official SDKs for TypeScript/JavaScript, Python, Ruby, and Go. The REST API can be used with any language that supports HTTP requests.

Is there a free tier?

Yes. Context.dev offers a free tier with generous limits and no credit card required to start scraping, crawling, and extracting.