❤
npm
Sign UpSign In

Search results

29 packages found

Sort by: Default
  • Default
  • Most downloaded this week
  • Most downloaded this month
  • Most dependents
  • Recently published

isbot

🤖/👨‍🦰 Recognise bots/crawlers/spiders using the user agent string.

omrilotan
published version 5.1.28, 18 days ago297 dependents licensed under $Unlicense
4,968,545

@crawlee/utils

A set of shared utilities that can be used by crawlers

apify-service-account
published version 3.13.5, 10 days ago16 dependents licensed under $Apache-2.0
184,857

sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

seantomburke
published version 4.0.2, 14 days ago66 dependents licensed under $MIT
130,759

robot-directives

Parse robot directives within HTML meta and/or HTTP headers.

stevenvachon
published version 0.4.0, 8 years ago5 dependents licensed under $MIT
108,416

sitemapper_mos

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

zijua
published version 3.2.9, 3 years ago0 dependents licensed under $MIT
478

@microlink/ua

A simple redis primitives to incr() and top() user agents

kikobeats
published version 1.2.5, 8 months ago0 dependents licensed under $MIT
352

@qbbsh/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

qbb.sh
published version 3.2.10, a year ago1 dependents licensed under $MIT
154

crawler-js

Opensource Framework Crawler in Node.js

rodrigorizando
published version 2.0.10, 6 years ago0 dependents licensed under $ISC
137

bot-detector

It uses the user-agents.org xml file for detecting bots.

slooker
published version 1.0.10, 10 years ago0 dependents licensed under $ISC
66

robots-txt-component

Lightweight robots.txt parsing component without any external dependencies for Node.js.

ibragim64
published version 0.0.5-dev, 3 years ago0 dependents licensed under $MIT
21

hidemyemail

A jQuery plugin that helps you to hide your email on your page and prevent crawlers to get it!

frenchfreelance
published version 0.1.0, 10 years ago0 dependents licensed under $ISC
22

@javivelasco/isbot

🤖 detect bots/crawlers/spiders via the user agent.

javivelasco
published version 3.3.3, 4 years ago0 dependents licensed under $Unlicense
20

@context-labs/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

samheutmaker
published version 3.2.10, 2 years ago4 dependents licensed under $MIT
15

isbot-bundle

isbot Bundle UMD

auban
published version 1.0.4, 5 years ago0 dependents
18

@wmod/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

wmod
published version 1.1.5, a year ago0 dependents licensed under $MIT
17

adibenmat-sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

adibenmatdev
published version 2.1.19, 4 years ago0 dependents licensed under $MIT
17

@yeskiy/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

yeskiy
published version 5.0.0, a year ago0 dependents licensed under $MIT
15

simplecrawling

Crawler made simple

rafa.cesar
published version 0.0.3, 10 years ago0 dependents
13

@public-square/sitemapper

Parser for XML Sitemaps to be used with Robots.txt and web crawlers

vpsqr-protocol
published version 3.2.5, 3 years ago1 dependents licensed under $MIT
13

planisphere

A straightforward sitemap generator written in TypeScript.

cheap-glitch
published version 1.0.1, 4 years ago0 dependents licensed under $ISC
12