Crawl Sitemap

Retrieve crawl Sitemap with the Context.dev API. Supply a domain. Sample fields include keywords and languages. Use the result as input for a related analysis or application. The GET request schema also lists includeSubdomains, maxLinks.

Selected price
$0.002 / billing creditPublished pricing
Sellers / implementations
1 / 1
Requests · 30d
—Loading...
Success rate · 30d
—Loading...

Run sends a real API request. Charges follow the published API contract.

Request parameters
Domain to map, e.g. `stripe.com`.
Optional parameters (7)
Include URLs on subdomains.
Maximum number of URLs to return.
Fetch this sitemap instead of discovering sitemaps. Must belong to the domain or a subdomain.
Regex pattern. Only URLs matching this pattern are returned and counted against maxLinks.
Regex pattern. URLs matching this pattern are not returned, even when they match urlRegex.
Filter URLs by a topic or phrase, most relevant first.
`enabled` turns on zero data retention. Returns 403 `ZDR_NOT_ENABLED` unless your organization has ZDR.

Sample response

{
  "domain": "context.dev",
  "key_metadata": {
    "credits_consumed": 1
  },
  "request_id": "63cde093-1036-44f3-97d2-798a471f911f",
  "success": true,
  "urls": [
    {
      "description": "Context.dev is the web data API for AI agents: scrape any URL into LLM-ready Markdown, search the live web, crawl entire sites, and extract structured data with one REST API.",
      "keywords": [
        "web data api",
        "web data api for ai agents",
        "scrape api",
        "crawl website",
        "web crawler",
        "web search api",
        "markdown scraper",
        "extract structured data",
        "ai data extraction",
        "web scraping for ai agents",
        "llm web scraping",
        "rag scraping api",
        "brand api"
      ],
      "language": "en",
      "title": "Context.dev: Web Data API for AI Agents (Scrape, Search, Crawl)",
      "url": "https://www.context.dev"
    }
  ]
}

Response result

Response structure

Inferred from examples
  • domainstring
  • key_metadataobject
    • credits_consumedinteger
  • request_idstring
  • successboolean
  • urlsobject[]
    • descriptionstring
    • keywordsstring[]
    • languagestring
    • titlestring
    • urlstring

Common questions

Do I need a separate key for every provider?

Use your OpenAny API key for standard API access. Read it from OPENANY_API_KEY in your environment and keep it out of prompts, source code and logs.

Are these providers interchangeable?

No. Each offering has its own inputs, pricing and response contract. Selecting a catalog offering does not establish automatic routing or fallback. Confirm the service contract before calling.

How is this API billed?

API-key calls use your shared balance at the selected implementation's published rate. Check the billing unit and request-specific pricing rules before making a paid call. Recorded usage determines the final charge under that contract.

Where do the price and contract come from?

The selected implementation publishes its provider, request contract, revision and billing rules through OpenAny's public API catalog. A revision number identifies a contract version, not a price update date. A price update timestamp is not supplied; do not treat a page date as one.

How do OpenAny Balance and x402 differ?

OpenAny Balance uses an API key and one shared balance. Card, USDC and x402 top-ups, card auto top-up and USDC subscriptions fund that balance; available routes appear on the Credits page. Direct x402 is a separate pay-per-request flow, available only on listings that show it and requiring an authorized payment client.

Does Playground send a paid request?

Yes. After you sign in, Run sends a real API request using an active OpenAny API key. Charges follow the published API contract and use your shared balance. Selecting an implementation, editing inputs and viewing sample responses do not send requests.

Use Crawl Sitemap with your agent

Inspect endpoint contracts and confirm availability and pricing before a paid call.