POST /v1/web/map lists the URLs on a site without reading any of them. It answers in one call and costs 1 credit, which makes it the quickest way to see what a site has before you crawl it.
When to use it
- You want to see how big a site is before running a Crawl.
- You want a list of URLs to filter and then read with a Batch.
- You want every URL that contains a word, such as every page with
pricingin its address.
Example
Options
How to use the site’s published page list (its sitemap):
include it alongside links found on the page, use it only, or skip it.For sites that publish several page lists, only read the lists whose address contains one of these comma-separated words. For example
"post" to reach a blog’s articles, or "pages" to reach a shop’s information pages instead of its products.Skip page lists whose address contains any of these comma-separated words. Combine it with
sitemapInclude, for example include "post" and exclude "tag,category,author".newest lists the most recently changed pages first, oldest the least recent, and sitemap keeps the order of the site’s own list. Pages with no date come last.For paging through a large archive from newest to oldest: return only URLs last changed at or before this ISO 8601 time. See below.
Skip this many results, then return the next
limit. Use it to page through sites whose page lists have no dates.Response fields
Paging through a large site
limit returns the most recently changed pages first, not an even sample of the site. On a site where one kind of page changes all the time, such as a shop’s product pages, a small limit can return only that kind. Use sitemapInclude to reach the rest.
To get more than one page of results:
- When pages have dates, pass the oldest
lastmodfrom the previous response asbefore, and remove any URL you already have. - When they do not, step
offsetthrough0,limit,2 × limitand so on, untiloffset + countreachestotal. Do not stop just because a page came back short.
Cost
1 credit per call, however many URLs come back. A call that finds no URLs is free.Next steps
Crawl a whole site
Read the pages you mapped.
Read a list of URLs
Send the URLs you picked to a batch.
Scrape one URL
Read any single URL you found.
API reference
The full schema, with a live playground.