《精通scrapy网络爬虫》第10章

最新推荐文章于 2025-03-14 22:38:47 发布

Z_sam

最新推荐文章于 2025-03-14 22:38:47 发布

阅读量234

点赞数

CC 4.0 BY-SA版权

分类专栏： scrapy

本文链接：https://blog.csdn.net/u010553139/article/details/104559425

scrapy 专栏收录该内容

10 篇文章

订阅专栏

本文详细介绍使用Scrapy网络爬虫进行网站登录的过程，包括抓取登录表单、构造FormRequest请求并填充登录参数，以及如何处理登录后的重定向，为爬虫开发者提供实战经验。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

请看另外一篇文章《精通Scrapy网络爬虫》第10章第10.1 登录的本质

<div id="web2py_user_form">
<form action="#" enctype="application/x-www-form-urlencoded" method="post">
	<table>
		<tbody>
			<tr id="auth_user_email__row">
				<td class="w2p_fl"><label class="" for="auth_user_email" id="auth_user_email__label">电子邮件: </label></td>
				<td class="w2p_fw"><input class="string" id="auth_user_email" name="email" type="text" value=""></td>
				<td class="w2p_fc"></td>
			</tr>
			<tr id="auth_user_password__row">
				<td class="w2p_fl"><label class="" for="auth_user_password" id="auth_user_password__label">密码: </label></td>
				<td class="w2p_fw"><input class="password" id="auth_user_password" name="password" type="password" value=""></td>
				<td class="w2p_fc"></td>
			</tr>
			<tr id="auth_user_remember_me__row">
				<td class="w2p_fl"><label class="" for="auth_user_remember_me" id="auth_user_remember_me__label">记住我(30 天): </label></td>
				<td class="w2p_fw"><input class="boolean" id="auth_user_remember_me" name="remember_me" type="checkbox" value="on"></td>
				<td class="w2p_fc"></td>
			</tr>
			<tr id="submit_record__row">
				<td class="w2p_fl"></td>
				<td class="w2p_fw"><input type="submit" value="Log In" class="btn"><button class="btn w2p-form-button" onclick="window.location='/places/default/user/register?_next=%2Fplaces%2Fdefault%2Findex';return false">注册</button></td>
				<td class="w2p_fc"></td>
			</tr>
		</tbody>
	</table>
	<div style="display:none;">
		<input name="_next" type="hidden" value="/places/default/index">
		<input name="_formkey" type="hidden" value="9fcc7ce6-be55-4a29-81d1-f5cd72674dce">
		<input name="_formname" type="hidden" value="login">
	</div>
</form>
</div>

C:\Users\Administrator>scrapy shell http://example.webscraping.com/places/default/user/login
>>> sel = response.xpath('//div[@style]/input')
>>> sel
[<Selector xpath='//div[@style]/input' data='<input name="_next" type="hidden" v
alue='>, <Selector xpath='//div[@style]/input' data='<input name="_formkey" type
="hidden" val'>, <Selector xpath='//div[@style]/input' data='<input name="_formn
ame" type="hidden" va'>]
>>> fd = dict(zip(sel.xpath('./@name').extract(),sel.xpath('./@value').extract()
))
>>> fd
{'_next': '/places/default/index', '_formkey': 'e47b0c25-273d-4939-98d2-6d0d2c28
47c2', '_formname': 'login'}
>>> fd['email']='zk_zhang1986@sina.com'
>>> fd['password']='123'
>>> fd
{'_next': '/places/default/index', '_formkey': 'e47b0c25-273d-4939-98d2-6d0d2c28
47c2', '_formname': 'login', 'email': 'zk_zhang1986@sina.com', 'password': '1
23'}
>>> from scrapy.http import FormRequest
>>> request = FormRequest('http://example.webscraping.com/places/default/user/lo
gin',formdata=fd)
>>> fetch(request)
2020-02-28 15:49:18 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (
303) to <GET http://example.webscraping.com/places/default/index> from <POST htt
p://example.webscraping.com/places/default/user/login>
2020-02-28 15:49:19 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://exampl
e.webscraping.com/places/default/index> (referer: None)
>>> fetch('http://example.webscraping.com/places/default/user/profile')
2020-02-28 15:52:41 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://exampl
e.webscraping.com/places/default/user/profile> (referer: None)
>>> view(response)
True
>>>

------------------
# FormRequest  隐藏的<input>信息会自动填入
>>> fd = {'email':'zk_zhang1986@sina.com','password':'123'}
>>> fd
{'email': 'zk_zhang1986@sina.com', 'password': '123'}
>>> from scrapy.http import FormRequest
>>> request = FormRequest('http://example.webscraping.com/places/default/user/lo
gin',formdata=fd)
>>> fetch(request)
2020-02-28 16:26:25 [scrapy.core.engine] DEBUG: Crawled (200) <POST http://examp
le.webscraping.com/places/default/user/login> (referer: None)
>>> view(response)
True