pythonscrapy爬虫下载并保存图片_scrapy图片下载完毕再储存资源-CSDN下载

共26个文件

pyc：17个

py：8个

cfg：1个

爬虫

scrapy

python

需积分: 47 185 浏览量 2018-08-17 10:54:50 上传评论 1 收藏 11KB RAR 举报

Python Scrapy是一个强大的爬虫框架，它为开发者提供了构建网络爬虫所需的各种工具和模块，使得数据抓取和处理变得更加高效。在这个项目中，我们利用Scrapy爬取了一个汽车网站上的图片，并按照类别将它们保存到本地。下面，我将详细讲解如何实现这一过程。我们需要安装Scrapy。在命令行中输入以下命令进行安装： ```bash pip install scrapy ``` 接着，创建一个新的Scrapy项目。打开终端，定位到你想要创建项目的目录，然后输入： ```bash scrapy startproject myspider ``` 这会生成一个名为`myspider`的文件夹，包含Scrapy项目的结构。进入项目目录： ```bash cd myspider ``` 接下来，我们需要创建一个爬虫。在终端中输入： ```bash scrapy genspider example example.com ``` 这里我们将爬虫命名为`example`，目标网站设为`example.com`。但实际操作时，你应该替换为你要爬取的汽车图片网站的URL。在`myspider/spiders`目录下，你会看到生成了一个名为`example.py`的文件。打开这个文件，替换默认内容，写入以下代码来定义爬虫： ```python import scrapy class ExampleSpider(scrapy.Spider): name = 'example' start_urls = ['http://yourcarwebsite.com'] # 替换为实际汽车图片网站 def parse(self, response): for car in response.css('div.car-image'): # 根据实际网站结构调整选择器 image_url = car.css('img::attr(src)').get() yield {'image_url': image_url} next_page = response.css('a.next::attr(href)').get() # 获取下一页链接，如果存在的话 if next_page is not None: yield response.follow(next_page, self.parse) ``` 这段代码会抓取每个汽车图片的URL，并准备下载。`parse`方法是Scrapy爬虫默认的起始回调函数，负责处理响应并生成请求。`css`方法用于选择HTML元素，`::attr(src)`用于获取图片的URL。为了保存图片，我们需要添加一个下载处理器。在`myspider/settings.py`文件中，找到`ITEM_PIPELINES`，并添加以下配置： ```python ITEM_PIPELINES = { 'myspider.pipelines.ImagePipeline': 300, } ``` 然后在`myspider/pipelines.py`文件中，添加以下代码以定义`ImagePipeline`： ```python from scrapy.pipelines.images import ImagesPipeline class MySpiderPipeline(ImagesPipeline): def file_path(self, request, response=None, info=None): image_guid = request.url.split('/')[-1] return f'cars/{image_guid}' # 分类保存图片，如：cars/imagename.jpg ``` 现在，我们的爬虫已经准备好抓取和保存图片了。运行爬虫： ```bash scrapy crawl example ``` Scrapy将会自动下载图片并按照`file_path`方法中的规则分类存储在项目目录下的`images`文件夹中。如果你没有设置，可以手动创建一个`images`文件夹。在实际应用中，可能需要处理登录、反爬虫策略、异常处理等问题。这个示例仅作为基础，展示了如何使用Scrapy下载并保存图片。对于更复杂的网页结构和需求，你可能需要调整选择器、解析逻辑以及自定义爬虫行为。记得在进行网络爬虫时要尊重网站的robots.txt文件规定，遵循合法、道德的爬虫实践，不要对网站造成过大负担。

资源详情

资源评论

资源推荐

收起资源包目录

crawlCar.rar （26个子文件）

pcauto

scrapy.cfg 256B

begin.py 126B

pcauto

settings.py 3KB

__init__.pyc 131B

pipelines.py 800B

middlewares.py 2KB

items.pyc 478B

__init__.py 0B

pipelines.pyc 1KB

items.py 360B

__pycache__

__init__.cpython-35.pyc 122B

pipelines.cpython-35.pyc 1KB

items.cpython-37.pyc 399B

settings.cpython-37.pyc 343B

items.cpython-35.pyc 421B

__init__.cpython-37.pyc 131B

settings.cpython-35.pyc 349B

spiders

__init__.pyc 139B

pcauto_spider.pyc 2KB

__init__.py 161B

__pycache__

__init__.cpython-35.pyc 130B

pcauto_spider.cpython-37.pyc 1KB

__init__.cpython-37.pyc 139B

pcauto_spider.cpython-35.pyc 1KB

pcauto_spider.py 1KB

settings.pyc 402B

# -*- coding: utf-8 -*- # Scrapy settings for pcauto project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # http://doc.scrapy.org/en/latest/topics/settings.html # http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html # http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html BOT_NAME = 'pcauto' SPIDER_MODULES = ['pcauto.spiders'] NEWSPIDER_MODULE = 'pcauto.spiders' # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = 'pcauto (+http://www.yourdomain.com)' # Obey robots.txt rules ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See http://scrapy.readthedocs.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', # 'Accept-Language': 'en', #} # Enable or disable spider middlewares # See http://scrapy.readthedocs.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # 'pcauto.middlewares.PcautoSpiderMiddleware': 543, #} # Enable or disable downloader middlewares # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # 'pcauto.middlewares.MyCustomDownloaderMiddleware': 543, #} # Enable or disable extensions # See http://scrapy.readthedocs.org/en/latest/topics/extensions.html #EXTENSIONS = { # 'scrapy.extensions.telnet.TelnetConsole': None, #} # Configure item pipelines # See http://scrapy.readthedocs.org/en/latest/topics/item-pipeline.html ITEM_PIPELINES = { "pcauto.pipelines.DownImagePipeline":1 } IMAGES_STORE = 'E:\pcauto20171221' # Enable and configure the AutoThrottle extension (disabled by default) # See http://doc.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See http://scrapy.readthedocs.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = 'httpcache' #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = 'scrapy.extensions.httpcache.FilesystemCacheStorage'