Submitted by:

 Arya Dangol (080BCT015)

 Kajal Chaudhary(080BCT037)
 Laxmi Mahato(080BCT044)
 Nabina Thapa(080BCT047)

SCRAPY
Scrapy is one of the essential tool used in Data Wrangling. It is high level and power full
framework which is used in web scraping (The process of browsing the web to discover and index
URLs or links. As a spider navigating the internet, following links from one page to another) and web
crawling (The process of extracting data from web pages in a structured format, such as a database or
spreadsheet). It allows users to visit diverse pages and allow to collect structured data from
them. Web crawling and scraping with Scrapy are speedy and efficient, hence making it an
essential tool for any developer or data scientist concerned about research, as well as research
itself. It can be used for a wide range of purposes, from data mining to monitoring and
automated testing.

Key Features

1) Web Crawling and Data Extraction

Scrapy facilitates its users to develop spiders, that means, custom scripts (unique pieces of code
written to suit a specific purpose) that define crawling a website and the data to be extracted.
Users start by setting URLs from where the spiders should crawl, together with the data
elements of interest using rules and parsing methods. This rule approach allows for a
standardized data extraction process, which becomes very much useful when used for large-
scale collection of data.

2) Asynchronous Processing

One of the major key features of Scrapy is asynchronous processing. It means scrapy can handle
multiple requests at the same time. This indicates while Scrapy is waiting for a response from
one request, it can send out other requests. Which makes the process much more efficient. It's
mostly useful when scraping multiple pages or websites at once, allowing users to collect data
faster than traditional methods that handle requests one at a time.

3) Data Storage Options

Scrapy supports various data storage options, which makes it easy for users to save their
extracted data in multiple formats, such as:

 JSON

 CSV

 XML

 Databases (e.g., SQLite, MongoDB)

This flexibility to store data in various formats helps users to easily include the data they scrape
into their analysis processes.

Services provided by scrapy :

Scrapy is a highly versatile web scraping tool which is used to extract the data from the web page with
the help of selectors based on XPath. It is an open source application framework for crawling web sites
and extracting structured data like tables and list which can be used for a wide range of useful
applications, like data mining, information processing or historical archival.
Scrapy being one of the completely free data wrangling tools with a web scraping framework built in
Python, providing ease of availability for all its users. Hence, it is quite relatable to us, students as we
can take the advantage of it being completely free and can improve our soft skills and enhance our
coding level. Moreover, it is suitable for projects of any size due to its feature of being fast and scalable.
Also, with the help of scrapy, so we can create our own dataset without relying on pre-existing sources.
The various services provided by scrapy are as follows:

1. Web crawling: Web crawling is the technical term for automatically accessing a website and obtaining
data via a software program. In other words, scrapy is like someone who goes through all the books in a
disorganized library and puts together a card catalog so that anyone who visits the library can quickly
and easily find the information they need.

2. Web scraping: Web scrapping refers to the use of bots to gather data or content from a website. We
can define a “spider”, a python class in scrapy where we can define the logic for scrapping. Futhermore,
it can handle multiple requests asynchronously.

3. Information processing: Information processing refers to the steps that are taken after scraping the
data which are then utilized and transformed for further process. Scrapy provides us the feature of
information processing through its powerful pipeline system.

Why scrapy is important for data wrangling?

Efficiency: Scrapy is designed for high- speed and Scalability, making it ideal for large-scale web scraping
projects. Its asynchronous architecture allows it to handle multiple requests concurrently, significantly
reducing.

Built-in Data Cleaning: Scrapy integrates data-cleaning tools (e.g., remove duplicates, handle missing
values) during extraction.

Flexibility: Scrapy provides a flexible framework for customizing data extraction and processing
pipelines. You can define custom rules, filters, and item pipelines to transform data into the desired
format.
Data Persistence: Scrapy supports various data storage options, including CSV, JSON, and databases.
This allows you to efficiently store and manage the extracted data for further analysis or use. how it is
different from other?

Primary Focus on Web Scraping: While other tools may offer data wrangling capabilities, Scrapy is
specifically designed for web scraping. It excels at extracting data from HTML and XML.

Asynchronous Architecture: Scrapy's asynchronous nature sets it apart from many other tools. This
allows it to handle multiple requests concurrently, significantly improving performance for largescale
projects.

Python-Based: Scrapy leverages the power and flexibility of Python, providing access to a vast
ecosystem of libraries and tools for data manipulation, analysis, and visualization.

How scrapy differs from others?

Scrapy differs significantly from tools like Alteryx APA, Tableau Desktop, Microsoft Power Query,
ParseHub, and others due to its specialized purpose, architecture, and approach. Here's a breakdown of
how Scrapy stands out compared to these data wrangling, integration, and visualization tools:

1. Primary Focus

 Scrapy: A web scraping and crawling framework designed to extract unstructured or semi-
structured data from websites. It handles everything from sending requests to parsing
HTML/XML.

 Other Tools: Most of these tools (e.g., Tableau, Power Query, Alteryx) focus on data
integration, data cleaning, ETL (Extract, Transform, Load) pipelines, or data visualization. They
are designed to work with structured datasets from APIs, databases, spreadsheets, or other
sources.

2. Handling Web Scraping

 Scrapy: Built specifically for web scraping. It includes:

 o Handling HTTP requests and responses.

o Navigating and crawling multiple pages.

o Managing dynamic content and JavaScript-heavy websites (with extensions like Scrapy-
Splash or Selenium).

o Built-in asynchronous capabilities for scalability.

 Other Tools:

o ParseHub specializes in web scraping but is GUI-based and suited for simpler, non-
programmatic workflows.

o Tools like Alteryx APA or Microsoft Power Query are not inherently designed for web
scraping, although they can connect to APIs or external sources for structured data.

3. Programming vs. No-Code/Low-Code

 Scrapy: Requires Python programming knowledge to set up spiders, extract data, and customize
workflows.

 Other Tools:

o ParseHub, Tableau, Astera, Tamr, and Alteryx provide no-code or low-code interfaces,
making them user-friendly for non-developers.

o They emphasize drag-and-drop functionality, pre-built connectors, and GUI-based

transformations.

4. Use Cases

 Scrapy: Best for scenarios where:

o Data is locked in web pages or requires crawling through large numbers of sites.
 o You need precise control over the scraping process.

 Other Tools:

o Designed for structured data integration (from databases, APIs, or Excel files).

o Use cases like data visualization (e.g., Tableau), ETL pipelines (e.g., Alteryx), and
collaborative data cleaning (e.g., Tamr).

5. Asynchronous Processing

 Scrapy: Uses an asynchronous architecture (based on Twisted) for efficient large-scale scraping
and crawling.

 Other Tools: Generally synchronous; processing is sequential or batch-oriented.

6. Customization and Extensibility

 Scrapy: Highly customizable for advanced users through Python scripting. You can:

o Write custom parsers.

o Integrate with external storage (e.g., databases, S3).

o Add middleware for proxies, user agents, or captcha handling.

 Other Tools: Offer pre-built templates or workflows that limit customization unless additional
scripting (e.g., Python in Alteryx) or plugins are used.

7. Data Output

 Scrapy: Outputs raw data in formats like JSON, CSV, or databases for further processing.

 Other Tools:

o Focus on transforming, cleaning, and integrating data into analytics-ready forms.

 o Visualization-focused tools (e.g., Tableau) immediately transform data into charts and
dashboards.

8. Ease of Learning

 Scrapy: Has a steep learning curve; requires programming knowledge to build and maintain
spiders.

 Other Tools: Often cater to business analysts or non-technical users, providing guided interfaces
for intuitive workflows.

9. Target Users

 Scrapy: Best for developers, data engineers, and data scientists working on custom scraping
projects.

 Other Tools:

o ParseHub: Ideal for non-coders looking for web scraping.

o Tableau Desktop, Microsoft Power Query, Alteryx: Targeted at business professionals

and analysts who need to integrate, analyze, or visualize structured datasets.

In summary, Scrapy is the go-to solution when your goal is web scraping and crawling, requiring precise
control over the data extraction process. Tools like ParseHub simplify web scraping for non-coders,
while Tableau, Alteryx, and others are better suited for analyzing, visualizing, or integrating structured
data into business workflows.