Scrapy 配置动态代理IP的实现
应用Scrapy框架，配置动态IP处理反爬。
#settings配置中间件
DOWNLOADER_MIDDLEWARES={
'text.middlewares.TextDownloaderMiddleware':543,
#'text.middlewares.RandomUserAgentMiddleware':544,
#'text.middlewares.CheckUserAgentMiddleware':545,
'text.middlewares.ProxyMiddleware':546,
'text.middlewares.CheckProxyMiddleware':547
}
#settings配置可用动态IP
PROXIES=[
"http://101.231.104.82:80",
"http://39.137.69.6:8080",
"http://39.137.69.10:8080",
"http://39.137.69.7:80",
"http://39.137.77.66:8080",
"http://117.191.11.102:80",
"http://117.191.11.113:8080",
"http://117.191.11.113:80",
"http://120.210.219.103:8080",
"http://120.210.219.104:80",
"http://120.210.219.102:80",
"http://119.41.236.180:8010",
"http://117.191.11.80:8080"
]
#middlewares配置中间件
importrandom
classProxyMiddleware(object):
defprocess_request(self,request,spider):
ip=random.choice(spider.settings.get('PROXIES'))
print('测试IP:',ip)
request.meta['proxy']=ip
classCheckProxyMiddleware(object):
defprocess_response(self,request,response,spider):
print('代理IP:',request.meta['proxy'])
returnresponse
到此这篇关于Scrapy配置动态代理IP的实现的文章就介绍到这了,更多相关Scrapy动态代理IP内容请搜索毛票票以前的文章或继续浏览下面的相关文章希望大家以后多多支持毛票票！
热门推荐
10 医院探望朋友祝福语简短
11 毕业祝福语简短女朋友
12 春节虎年爸妈祝福语简短
13 儿童毕业祝福语 简短6
14 开工仪式横幅祝福语简短
15 孩子日常祝福语大全简短
16 清晨寄语诗句祝福语简短
17 结婚送亲认亲祝福语简短
18 虎年喝酒拜年祝福语简短