Skip to content

RAG Web Browser

Try for free

Web browser for AI agents and RAG pipelines: search Google, fetch the top result pages, and return clean Markdown or text ready for LLMs. Also accepts a single URL.

🌐 RAG Web Browser

Search Google, open the pages that matter, and hand your model clean Markdown instead of a link it still has to read.

Built for RAG pipelines, agents, and research jobs that need the text of a public page in the dataset before the next step of the workflow runs.

Watch the video tutorial to see how it works:

✨ Features

  • 🔎 query is a Google search or a single public URL. A URL skips search and fetches that page.
  • ⚡ Google results come from fast search mode, which returns a results page in a second or two. Any page fast search mode can't return is loaded in browser mode instead, and the run carries on.
  • 📄 maxResults chooses how many organic results to store, from 1 to 100 (the run keeps every extracted page in memory until it returns). A blocked search page is retried. A Google results page that still has no public links is loaded once in a remote scraping browser. A result URL that is not a public document is tried again on a new session, then in headless Chrome. Once maxResults pages are stored the run ends. If it is still short and a result URL is missing, the run fails and writes nothing. When Google returns no further public pages and every result URL was stored, the shorter clean set is stored.
  • 📝 outputFormats returns markdown, text, html, or any combination. Markdown links are absolute, so they still work after the page is gone.
  • 🧹 removeElementsCssSelector drops the matching nodes before conversion. Navigation, scripts, and the other default chrome stay out of the prompt.
  • 🍪 removeCookieWarnings also strips common consent containers from the HTML before conversion.
  • 📖 htmlTransformer set to readableText keeps the readable article when extraction is confident, and keeps the cleaned HTML when it is not. none only applies the element removal.
  • 🏷 Each handled page can include metadata taken from that document: title, description, author, keywords, language, canonical URL, Open Graph tags, and JSON-LD. A field that the page does not have is left off the item.
  • 🔁 A blocked page is fetched again on a new HTTP session. If HTTP still does not return it, headless Chrome loads that URL through the residential proxy. If the page is still missing, one remote scraping-browser session loads it. Reddit, Facebook, Wikipedia, and X also get one public alternate host for the same document. A login wall, cookie consent wall, browser check, bot-protection block page, empty shell, or page of placeholders is not stored, whichever output formats you asked for; the page goes on to the next session.
  • 🚫 Error pages are not stored: a page that answers HTTP 404 or 410, a server error page, or a page that answers 200 but whose title or main heading says the page was not found (in English, Danish, German, French, Spanish and many other languages). A missing page is not loaded again in other sessions. In a search, the next result takes its place and the status message lists the skipped URLs; for a direct URL the status message says why nothing was saved.
  • 🎬 An image, audio, video, or font URL is not stored. The run fails instead of writing a skip row when that was the only requested target.

🚀 Why teams use this

HTTP first, for every page that already returns. Headless Chrome loads only a result URL that HTTP did not store, using the residential proxy and no saved cookies. Markdown is converted from the HTML the page sent. A JSON response, or a short plain-text one, is stored from HTTP as received (in a code block); no browser opens for it.

It is a Scrapely run with a dataset, a timeout, and retries. requestTimeoutSecs bounds the whole query. serpMaxRetries retries search. maxRequestRetries retries a page. desiredConcurrency caps how many pages are open at once. Set it to 0 and the run picks a small pool from the memory it was given.

The dataset contains only handled public pages. A direct URL that cannot be stored, including one that is an error page, fails the run, writes nothing. A search that stops while later Google pages are still unread fails the same way. A result URL that stays blocked is retried on a new session. Once maxResults pages are stored the run ends, even if an earlier result URL stayed blocked. If the run is still short and a result URL is missing, it fails and writes nothing. When Google has no further public pages and every result URL was stored, the shorter clean set is stored and the run says why.

proxyConfiguration chooses how target pages are fetched. When Scrapely Proxy is on and no group or country is set, each page is first requested through a US datacenter exit; a page that answers it with a block, an error status, a near-empty body or no answer is requested again through the residential group (US), and the browser fallback uses the residential group as before. A country set in proxyConfiguration is used for every page load: HTTP, headless Chrome and remote browser sessions all exit in that country, and the log names the exit country. Google Search runs in fast search mode (US results, English interface). If fast search mode fails for a results page (an error, a timeout, or no results on the first page), the log says Switching to browser mode for this query page and that page runs in browser mode: the platform's remote browser with a US exit when the run has one, otherwise a local headless Chromium behind the residential proxy. In browser mode, if Google keeps answering with its consent or unusual-traffic page, the run fails and says so.

💡 Use cases

  • Give an LLM or agent live web access: search Google and read the top pages as Markdown.
  • Ground chatbot answers in fresh sources with their URLs.
  • Turn a single URL into clean Markdown for retrieval-augmented generation.

🧪 Example input

{
  "htmlTransformer": "none",
  "proxyConfiguration": {
    "useScrapelyProxy": true
  },
  "query": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
}

🧾 Sample output

One item from a real run of the example input, shortened. Long text is cut and lists show their first entries.

{
  "crawl": {
    "httpStatusCode": 200,
    "httpStatusMessage": "OK",
    "loadedAt": "2026-10-12T22:14:51.181Z",
    "uniqueKey": "AQUoPf3PyX",
    "requestStatus": "handled"
  },
  "metadata": {
    "title": "Retrieval-augmented generation - Wikipedia",
    "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "languageCode": "en",
    "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
    "openGraph": [
      {
        "property": "…",
        "content": "…"
      },
      {
        "property": "…",
        "content": "…"
      }
    ],
    "headers": {
      "accept-ranges": "bytes",
      "age": "7110",
      "cache-control": "private, s-maxage=0, max-age=0, must-revalidate, no-transform",
      "content-encoding": "gzip",
      "content-language": "en",
      "content-length": "46696",
      "content-security-policy": "script-src 'unsafe-eval' blob: 'self' meta.wikimedia.org *.wikimedia.org *.wikipedia.org *.wikinews.org *.wiktionary.org *.wikibooks.org *.wikiversity.org *.…",
      "content-type": "text/html; charset=UTF-8",
      "date": "Mon, 12 Oct 2026 20:16:18 GMT",
      "last-modified": "Mon, 12 Oct 2026 18:31:50 GMT",
      "nel": "{ \"report_to\": \"wm_nel\", \"max_age\": 604800, \"failure_fraction\": 0.05, \"success_fraction\": 0.0}",
      "report-to": "{ \"group\": \"wm_nel\", \"max_age\": 604800, \"endpoints\": [{ \"url\": \"https://intake-logging.wikimedia.org/v1/events?stream=w3c.reportingapi.network_error&schema_u…",
      "reporting-endpoints": "csp-report-to-endpoint='/w/api.php?action=cspreport&format=json';",
      "server": "mw-web.eqiad.main-6df4688ffd-gg2zh",
      "server-timing": "cache;desc=\"hit-front\", host;desc=\"cp3073\",co_id;desc=\"1572652044\"",
      "strict-transport-security": "max-age=106384710; includeSubDomains; preload",
      "vary": "Accept-Encoding,X-Subdomain,Cookie,Authorization,User-Agent",
      "x-cache": "cp3073 miss, cp3073 hit/1",
      "x-cache-status": "hit-front",
      "x-client-ip": "78.133.77.167",
      "x-content-type-options": "nosniff",
      "x-request-id": "a56e96a4-4cb2-42cb-8af6-761fc6df3095"
    }
  },
  "query": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "markdown": "Retrieval-augmented generation - Wikipedia\n[Jump to content](#bodyContent)\n[![](https://en.wikipedia.org/static/images/icons/enwiki-25.svg)\n![Wikipedia](http…"
}

📥 Input

query is required. Everything else has a default.

FieldWhat it does
queryGoogle Search keywords, including operators such as site:, or an http/https URL.
maxResultsHow many organic results to fetch. Ignored when query is a URL. Default 5.
outputFormatsAny of text, markdown, html. Default ["markdown"].
requestTimeoutSecsBudget for search and page fetches together. The run returns as soon as maxResults pages are stored. Default 240. A direct URL always tries plain HTTP first with a share of this budget; the browser sessions get the rest.
serpMaxRetriesExtra browser sessions for a Google results page that came back as a consent or unusual-traffic page. Default 2.
proxyConfigurationProxy for the target pages. Default is Scrapely Proxy: US datacenter exits first, the residential group for pages they do not load.
scrapingToolRaw HTTP (raw-http, the default) or Browser. Both use HTTP first. Headless Chrome runs only for URLs HTTP did not store.
removeElementsCssSelectorCSS selector removed before conversion.
htmlTransformernone or readableText. Default none.
desiredConcurrencyParallel page fetches. 0 picks from run memory. Default 5.
maxRequestRetriesExtra attempts for one page. Default 1.
dynamicContentWaitSecsAccepted and clamped. This HTTP build does not sleep for JavaScript.
removeCookieWarningsStrip common consent containers before conversion. Default true.
debugModeWhen true, adds crawl.debug.timeMeasures for the page fetch. Default false.
{
  "query": "retrieval-augmented generation site:en.wikipedia.org",
  "maxResults": 1,
  "outputFormats": ["markdown"],
  "scrapingTool": "raw-http",
  "proxyConfiguration": {
    "useScrapelyProxy": true
  }
}

📤 Output

One dataset item per fetched URL. Empty values are omitted. A real false or 0 would stay; this actor does not invent either one.

Search items include query, searchResult (title, url, resultType, and description when Google sent one), and crawl:

  • crawl.httpStatusCode
  • crawl.httpStatusMessage (OK when the page was handled)
  • crawl.loadedAt
  • crawl.uniqueKey
  • crawl.requestStatus (handled for every stored page)
  • crawl.debug only when debugMode is true

metadata is filled from the response and the document:

  • metadata.url
  • metadata.redirectedUrl when the final URL differs from the requested one
  • metadata.title, metadata.description, metadata.author, metadata.keywords, metadata.languageCode
  • metadata.canonicalUrl
  • metadata.openGraph as {property, content} pairs
  • metadata.jsonLd when a JSON-LD script parses
  • metadata.headers from the response, without cookie headers

markdown, text, and html appear only for the formats you requested, and only when that conversion produced text. html is the cleaned page. When nothing is removed from a page (a removeElementsCssSelector that matches nothing, such as dummy_keep_everything, removeCookieWarnings off and htmlTransformer none), it is the page exactly as received (as rendered when a browser loaded it), its <head> included (up to 4 MB). A direct URL has no searchResult. A media URL has no body.

❓ FAQ

Can I schedule it or use the API? Yes. Save your input as a task and schedule it, or start runs and read the results through the Scrape.ly API, CLI, or from AI agents through the Scrape.ly MCP server.

Related actors

The internet is full of data.
Start owning yours.

Scrape.ly gives you the infrastructure, the tooling, and the team to extract, process, and operationalize web data at any scale. Setup takes minutes.

No credit card. No setup calls.