小调电影名称爬取

Bra__u

于 2019-01-22 15:33:13 发布

阅读量474

点赞数 1

CC 4.0 BY-SA版权

分类专栏： python|入门

本文链接：https://blog.csdn.net/qq_38537101/article/details/86595577

该博客介绍了如何使用Python爬虫从网站dy2018.com抓取电影名称及其链接。首先，通过requests和lxml库获取页面HTML，然后通过正则表达式提取电影名称。接着，遍历所有页面，提取每页的电影标题和链接，并保存到CSV文件中。过程中处理了编码问题，以避免出现乱码。当遇到版权问题导致无法获取链接时，程序会进行相应提示并跳过。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

import requests,os
from lxml import etree
from urllib.parse import urljoin
import  re

pattern = re.compile(r"《(.*?)》")
url = 'https://www.dy2018.com/html/gndy/dyzz/index.html'
response = requests.get(url)
html_content = response.text
root = etree.HTML(html_content)

all_pages = root.xpath("//select[@name='select']/option/@value")
f = open('小调网.csv','w',encoding = 'utf8')
for page in all_pages:
    page = urljoin(url,page)
    response = requests.get(page)
    response.encoding = 'gbk'
    html_content = response.text
    root = etree.HTML(html_content)
    mov_name = root.xpath('//table/tr/td/b/a/@title')
    mov_url = root.xpath('//table/tr/td/b/a/@href')

    for mov_name,mov_url in zip(mov_name,mov_url):

       try:
            result = pattern.findall(mov_name)[0]
            href = urljoin(url,mov_url)
            response = requests.get(href)
            response.encoding= 'gbk'
            html_content = response.text
            root = etree.HTML(html_content)