网站首页 > 技术文章正文

通过python爬虫下载电影的网址和电影标题信息到excel表格中

hfteth 2025-05-03 14:18:24 技术文章 21 ℃

本代码演示去电影网址上抓取电影网址和电影名称的数据，并保存到本机到excel表格中

操作步骤如下：

一、去百度上搜索查询要爬去的电影网址：

搜索到的电影网址为：
http://dianying.2345.com/list/guzhuang------.html，即我们今天要通过python进行get请求的网址

二、开始测试爬取网址：

1、在360极速浏览器输入网址：
http://dianying.2345.com/list/guzhuang------.html。

2、按F12查看网址的源代码。

3、按住F5加载网页源代码。

4、找到左下角的Network，点击↖箭头，选择对应的电影，如下图所示：

相当于已经得出了如下结论：

所有的电影网址，标题信息都在标签里。

三、去pycharm里编写源代码

1、同时按alt+insert键，新建一个使用find_all匹配电影信息.py的文件。

2、根据以下3个步骤开始编写爬取网页源代码的代码：

①导入相应的库

②定义函数对网页进行get请求，并解析网址，并保存数据到excel中

③调用函数

代码示例如下：

#1、导入相应的库
import re,requests,random,os #导入正则，网页请求，随机，系统库
from bs4 import BeautifulSoup #从bs4中导入网页源代码解析库
import pandas as pd #导入pandas库，并设置为pd
#2、定义函数对网页进行get请求
def gethtml():
 url="http://dianying.2345.com/list/guzhuang------.html"
 agent1 = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.70 Safari/537.36"} # 设定请求头的值agent1
 agent2 = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"} # 设定请求头的值agent2
 agent3 = {"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.62 Safari/537.36"} # 设定请求头的值agent3
 list1 = [agent1, agent2, agent3] # 设置3个请求头组成的列表，并赋值给列表list1
 agent = random.choice(list1) # 请求头的值agent从列表list1里随机选择一个
 try:
 response = requests.get(url, headers=agent) # 伪装浏览器对网站进行get请求，并将请求结果赋值给response
 response.encoding = response.apparent_encoding # 根据内容解析出网页的编码格式并赋值给response.encoding
 html = response.text # 将网页的源代码赋值给html
 soup=BeautifulSoup(html,"html.parser") #使用html.parser解析器对网页源代码进行解析，并将解析结果赋值给soup
 info = soup.find_all("em",attrs={"class":"emTit"}) #查找所有的当参数class属性值为emTit时的em标签，并将所有的查找结果赋值给info
 #发现所有电影信息都在<em class="emTit"><a title="黄飞鸿1壮志凌云" target="_blank" href="//dianying.2345.com/detail/59747.html" data-ajax83="ys_dy_list_title_59747">黄飞鸿1：壮志凌云</a></em>标签里，
 #故对上述标签信息进行正则匹配
 print(info)
 list1=[] #设置列表list1为空列表，用来接收电影的网址信息
 list2=[] #设置列表list2为空列表，用来接收电影的标题信息
 for movieinfo in info: #在info里遍历所有的电影信息movieinfo
 print(movieinfo) #打印所有的电影信息
 list1.append("http:"+movieinfo.a.get("href")) #将遍历出来的a标签的href属性值前加上http作为一个完整的网址添加到列表list1
 list2.append(movieinfo.a.string) #将遍历出来的a标签的值添加到列表list2
 print("list1:",list1) #打印列表list1
 print("list2:",list2) #打印列表list2
 dict1={"序号":range(1,len(list1)+1),"网址":list1,"电影标题":list2}
 # 上面这行代码设置dict1字典，字典的三个键分别是序号，网址，电影标题，对应的是存储在excel表格里的三个表头，即3个列标名称，字典的三个值分别是序号值，电影网址，电影标题名称
 df=pd.DataFrame(dict1) #将dict1的数据保存为表格，并赋值给df变量
 df.to_excel("E:\IT\PYthon\电影数据.xlsx") #将df的数据保存到E:\IT\PYthon\电影数据.xlsx的excel文件中，若没有这个文件，则自动新建这个文件并保存数据
 except: #假如接收到错误反馈信息时
 print("爬取失败") #打印爬取失败的提示
#3、调用爬取数据的函数
if __name__ == '__main__':
 gethtml() #调用gethtml函数

在pycharm同时按住ctrl+shift+F10运行代码,代码运行结果如下：

[<a data-ajax83="ys_dy_list_title_148874" href="
//dianying.2345.com/detail/148874.html" target="_blank" title="王朝的女人杨贵妃">王朝的女人杨贵妃</a>, <a data-ajax83="ys_dy_list_title_35077" href="
//dianying.2345.com/detail/35077.html" target="_blank" title="龙门飞甲">龙门飞甲</a>, <a data-ajax83="ys_dy_list_title_201821" href="
//dianying.2345.com/detail/201821.html" target="_blank" title="开封降魔记">开封降魔记</a>, <a data-ajax83="ys_dy_list_title_59747" href="
//dianying.2345.com/detail/59747.html" target="_blank" title="黄飞鸿1壮志凌云">黄飞鸿1：壮志凌云</a>, <a data-ajax83="ys_dy_list_title_199282" href="
//dianying.2345.com/detail/199282.html" target="_blank" title="神谕通天">神谕通天</a>, <a data-ajax83="ys_dy_list_title_39273" href="
//dianying.2345.com/detail/39273.html" target="_blank" title="笑傲江湖3东方不败风云再起">笑傲江湖3：东方不败…</a>, <a data-ajax83="ys_dy_list_title_202433" href="
//dianying.2345.com/detail/202433.html" target="_blank" title="封魔传">封魔传</a>, <a data-ajax83="ys_dy_list_title_30550" href="
//dianying.2345.com/detail/30550.html" target="_blank" title="恶虎村">恶虎村</a>, <a data-ajax83="ys_dy_list_title_190993" href="
//dianying.2345.com/detail/190993.html" target="_blank" title="少林与武当">少林与武当</a>, <a data-ajax83="ys_dy_list_title_19317" href="
//dianying.2345.com/detail/19317.html" target="_blank" title="镖行天下前传之决战天涯">镖行天下前传之决战…</a>, <a data-ajax83="ys_dy_list_title_198268" href="
//dianying.2345.com/detail/198268.html" target="_blank" title="大唐狐妖传">大唐狐妖传</a>, <a data-ajax83="ys_dy_list_title_164786" href="
//dianying.2345.com/detail/164786.html" target="_blank" title="轩辕大帝">轩辕大帝</a>, <a data-ajax83="ys_dy_list_title_141359" href="
//dianying.2345.com/detail/141359.html" target="_blank" title="白幽灵传奇之绝命逃亡">白幽灵传奇之绝命逃…</a>, <a data-ajax83="ys_dy_list_title_21034" href="
//dianying.2345.com/detail/21034.html" target="_blank" title="火烧圆明园">火烧圆明园</a>, <a data-ajax83="ys_dy_list_title_190410" href="
//dianying.2345.com/detail/190410.html" target="_blank" title="埋伏1973">埋伏（1973）</a>, <a data-ajax83="ys_dy_list_title_120632" href="
//dianying.2345.com/detail/120632.html" target="_blank" title="绣春刀">绣春刀</a>, <a data-ajax83="ys_dy_list_title_84172" href="
//dianying.2345.com/detail/84172.html" target="_blank" title="止杀令">止杀令</a>, <a data-ajax83="ys_dy_list_title_21898" href="
//dianying.2345.com/detail/21898.html" target="_blank" title="黄飞鸿6西域雄狮">黄飞鸿6：西域雄狮</a>, <a data-ajax83="ys_dy_list_title_200132" href="
//dianying.2345.com/detail/200132.html" target="_blank" title="龙族的反击">龙族的反击</a>, <a data-ajax83="ys_dy_list_title_197808" href="
//dianying.2345.com/detail/197808.html" target="_blank" title="白夜侠">白夜侠</a>, <a data-ajax83="ys_dy_list_title_323" href="
//dianying.2345.com/detail/323.html" target="_blank" title="倚天屠龙记之魔教教主">倚天屠龙记之魔教教…</a>, <a data-ajax83="ys_dy_list_title_116829" href="
//dianying.2345.com/detail/116829.html" target="_blank" title="天将雄师">天将雄师</a>, <a data-ajax83="ys_dy_list_title_50331" href="
//dianying.2345.com/detail/50331.html" target="_blank" title="大罗剑侠">大罗剑侠</a>, <a data-ajax83="ys_dy_list_title_36044" href="
//dianying.2345.com/detail/36044.html" target="_blank" title="杨门女将之军令如山">杨门女将之军令如山…</a>, <a data-ajax83="ys_dy_list_title_165827" href="
//dianying.2345.com/detail/165827.html" target="_blank" title="奇侠">奇侠</a>, <a data-ajax83="ys_dy_list_title_202024" href="
//dianying.2345.com/detail/202024.html" target="_blank" title="等到烟暖雨收">等到烟暖雨收</a>, <a data-ajax83="ys_dy_list_title_31845" href="
//dianying.2345.com/detail/31845.html" target="_blank" title="古墓荒斋">古墓荒斋</a>, <a data-ajax83="ys_dy_list_title_50332" href="
//dianying.2345.com/detail/50332.html" target="_blank" title="黑灵官">黑灵官</a>, <a data-ajax83="ys_dy_list_title_200206" href="
//dianying.2345.com/detail/200206.html" target="_blank" title="铁锅传奇">铁锅传奇</a>, <a data-ajax83="ys_dy_list_title_65349" href="
//dianying.2345.com/detail/65349.html" target="_blank" title="花漾">花漾</a>, <a data-ajax83="ys_dy_list_title_30492" href="
//dianying.2345.com/detail/30492.html" target="_blank" title="鹰王">鹰王</a>, <a data-ajax83="ys_dy_list_title_198270" href="
//dianying.2345.com/detail/198270.html" target="_blank" title="皇上命我来选妃">皇上命我来选妃</a>, <a data-ajax83="ys_dy_list_title_19263" href="
//dianying.2345.com/detail/19263.html" target="_blank" title="镖行天下前传之终极任务">镖行天下前传之终极…</a>, <a data-ajax83="ys_dy_list_title_165826" href="
//dianying.2345.com/detail/165826.html" target="_blank" title="绣春刀修罗战场">绣春刀修罗战场 </a>, <a data-ajax83="ys_dy_list_title_196850" href="
//dianying.2345.com/detail/196850.html" target="_blank" title="维京王者之战">维京:王者之战</a>]

<a data-ajax83="ys_dy_list_title_148874" href="
//dianying.2345.com/detail/148874.html" target="_blank" title="王朝的女人杨贵妃">王朝的女人杨贵妃</a>

<a data-ajax83="ys_dy_list_title_201821" href="
//dianying.2345.com/detail/201821.html" target="_blank" title="开封降魔记">开封降魔记</a>

<a data-ajax83="ys_dy_list_title_190993" href="
//dianying.2345.com/detail/190993.html" target="_blank" title="少林与武当">少林与武当</a>

<a data-ajax83="ys_dy_list_title_19317" href="
//dianying.2345.com/detail/19317.html" target="_blank" title="镖行天下前传之决战天涯">镖行天下前传之决战…</a>

<a data-ajax83="ys_dy_list_title_198268" href="
//dianying.2345.com/detail/198268.html" target="_blank" title="大唐狐妖传">大唐狐妖传</a>

<a data-ajax83="ys_dy_list_title_141359" href="
//dianying.2345.com/detail/141359.html" target="_blank" title="白幽灵传奇之绝命逃亡">白幽灵传奇之绝命逃…</a>

<a data-ajax83="ys_dy_list_title_21034" href="
//dianying.2345.com/detail/21034.html" target="_blank" title="火烧圆明园">火烧圆明园</a>

<a data-ajax83="ys_dy_list_title_200132" href="
//dianying.2345.com/detail/200132.html" target="_blank" title="龙族的反击">龙族的反击</a>

<a data-ajax83="ys_dy_list_title_323" href="
//dianying.2345.com/detail/323.html" target="_blank" title="倚天屠龙记之魔教教主">倚天屠龙记之魔教教…</a>

<a data-ajax83="ys_dy_list_title_36044" href="
//dianying.2345.com/detail/36044.html" target="_blank" title="杨门女将之军令如山">杨门女将之军令如山…</a>

<a data-ajax83="ys_dy_list_title_202024" href="
//dianying.2345.com/detail/202024.html" target="_blank" title="等到烟暖雨收">等到烟暖雨收</a>

<a data-ajax83="ys_dy_list_title_198270" href="
//dianying.2345.com/detail/198270.html" target="_blank" title="皇上命我来选妃">皇上命我来选妃</a>

<a data-ajax83="ys_dy_list_title_19263" href="
//dianying.2345.com/detail/19263.html" target="_blank" title="镖行天下前传之终极任务">镖行天下前传之终极…</a>

<a data-ajax83="ys_dy_list_title_165826" href="
//dianying.2345.com/detail/165826.html" target="_blank" title="绣春刀修罗战场">绣春刀修罗战场 </a>

list1: ['http://dianying.2345.com/detail/148874.html', 'http://dianying.2345.com/detail/35077.html', 'http://dianying.2345.com/detail/201821.html', 'http://dianying.2345.com/detail/59747.html', 'http://dianying.2345.com/detail/199282.html', 'http://dianying.2345.com/detail/39273.html', 'http://dianying.2345.com/detail/202433.html', 'http://dianying.2345.com/detail/30550.html', 'http://dianying.2345.com/detail/190993.html', 'http://dianying.2345.com/detail/19317.html', 'http://dianying.2345.com/detail/198268.html', 'http://dianying.2345.com/detail/164786.html', 'http://dianying.2345.com/detail/141359.html', 'http://dianying.2345.com/detail/21034.html', 'http://dianying.2345.com/detail/190410.html', 'http://dianying.2345.com/detail/120632.html', 'http://dianying.2345.com/detail/84172.html', 'http://dianying.2345.com/detail/21898.html', 'http://dianying.2345.com/detail/200132.html', 'http://dianying.2345.com/detail/197808.html', 'http://dianying.2345.com/detail/323.html', 'http://dianying.2345.com/detail/116829.html', 'http://dianying.2345.com/detail/50331.html', 'http://dianying.2345.com/detail/36044.html', 'http://dianying.2345.com/detail/165827.html', 'http://dianying.2345.com/detail/202024.html', 'http://dianying.2345.com/detail/31845.html', 'http://dianying.2345.com/detail/50332.html', 'http://dianying.2345.com/detail/200206.html', 'http://dianying.2345.com/detail/65349.html', 'http://dianying.2345.com/detail/30492.html', 'http://dianying.2345.com/detail/198270.html', 'http://dianying.2345.com/detail/19263.html', 'http://dianying.2345.com/detail/165826.html', 'http://dianying.2345.com/detail/196850.html']

list2: ['王朝的女人杨贵妃', '龙门飞甲', '开封降魔记', '黄飞鸿1：壮志凌云', '神谕通天', '笑傲江湖3：东方不败…', '封魔传', '恶虎村', '少林与武当', '镖行天下前传之决战…', '大唐狐妖传', '轩辕大帝', '白幽灵传奇之绝命逃…', '火烧圆明园', '埋伏（1973）', '绣春刀', '止杀令', '黄飞鸿6：西域雄狮', '龙族的反击', '白夜侠', '倚天屠龙记之魔教教…', '天将雄师', '大罗剑侠', '杨门女将之军令如山…', '奇侠', '等到烟暖雨收', '古墓荒斋', '黑灵官', '铁锅传奇', '花漾', '鹰王', '皇上命我来选妃', '镖行天下前传之终极…', '绣春刀修罗战场 ', '维京:王者之战']

pycharm代码图片示例如下：

检查网址的电影信息图片

保存到本机的EXCEL表格图片

上一篇：一个Python编写的小说下载器（python写的小说阅读程序）
下一篇： python编程实践:下载文件模块wget的使用

网站首页 > 技术文章 正文

通过python爬虫下载电影的网址和电影标题信息到excel表格中

猜你喜欢

网站首页 > 技术文章正文