Skip to main content
POST /v1/web/map lists the URLs on a site without reading any of them. It answers in one call and costs 1 credit, which makes it the quickest way to see what a site has before you crawl it.

When to use it

  • You want to see how big a site is before running a Crawl.
  • You want a list of URLs to filter and then read with a Batch.
  • You want every URL that contains a word, such as every page with pricing in its address.
Map returns addresses only, not content. To read the pages, pass the URLs to Scrape, Batch or Crawl.

Example

Options

string
required
The site to list. Must be http or https.
number
The most URLs to return, from 1 to 5000.
Keep only URLs that contain this word, such as "pricing". Up to 200 characters.
string
How to use the site’s published page list (its sitemap): include it alongside links found on the page, use it only, or skip it.
Also add links found on the starting page, not only the site’s page list.
string
For sites that publish several page lists, only read the lists whose address contains one of these comma-separated words. For example "post" to reach a blog’s articles, or "pages" to reach a shop’s information pages instead of its products.
string
Skip page lists whose address contains any of these comma-separated words. Combine it with sitemapInclude, for example include "post" and exclude "tag,category,author".
string
default:"newest"
newest lists the most recently changed pages first, oldest the least recent, and sitemap keeps the order of the site’s own list. Pages with no date come last.
boolean
default:"false"
Also list URLs on the site’s subdomains.
boolean
default:"false"
Treat URLs that differ only in their query string as one URL.
string
For paging through a large archive from newest to oldest: return only URLs last changed at or before this ISO 8601 time. See below.
number
Skip this many results, then return the next limit. Use it to page through sites whose page lists have no dates.

Response fields

object

Paging through a large site

limit returns the most recently changed pages first, not an even sample of the site. On a site where one kind of page changes all the time, such as a shop’s product pages, a small limit can return only that kind. Use sitemapInclude to reach the rest. To get more than one page of results:
  • When pages have dates, pass the oldest lastmod from the previous response as before, and remove any URL you already have.
  • When they do not, step offset through 0, limit, 2 × limit and so on, until offset + count reaches total. Do not stop just because a page came back short.

Cost

1 credit per call, however many URLs come back. A call that finds no URLs is free.

Next steps

Crawl a whole site

Read the pages you mapped.

Read a list of URLs

Send the URLs you picked to a batch.

Scrape one URL

Read any single URL you found.

API reference

The full schema, with a live playground.