網(wǎng)頁爬蟲 - 為什么python模擬登陸 appannie一直返回503 code
問題描述
#-*-encoding:utf-8-*-import requests, xlwt, sysfrom bs4 import BeautifulSoupreload(sys)referer = 'https://www.appannie.com/account/login/?_ref=header'user_agent = (’Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36’)sys.setdefaultencoding(’utf-8’)header = {'User-Agent': user_agent, 'Referer': referer, 'Host': 'www.appannie.com', ’Connection’: ’keep-alive’, ’Accept’: ’application/json, text/plain,*/*’, ’Accept-Encoding’: ’gzip, deflate, sdch’, ’Accept-Language’: ’zh-CN,zh;q=0.8’, ’X-NewRelic-ID’: ’VwcPUFJXGwEBUlJSDgc=’, ’X-Requested-With’: ’XMLHttpRequest’, }def main(): url = ’https://www.appannie.com/account/login/’ # content = requests.get(url,headers = header).content # soup = BeautifulSoup(content,’lxml’) # key = soup.select() s = requests.Session() s.get(url,headers = header) key = s.cookies[’csrftoken’] data = { ’csrfmiddlewaretoken’: key , ’next’: ’/dashboard/home/’ , ’username’:’1195615991@qq.com’ , ’password’:’xxxxx’ } req = s.post(url,data = data) if 2 != req.status_code / 100 :raise Exception('Error while logging in, code: %d' % (req.status_code)) cookies = req.cookies n = ’2017-04-11’ url_1 = ’https://www.appannie.com/apps/google-play/top-chart/?country=US&category=game&device=&date={}’.format(n) req_1 = s.get(url_1,headers = header,cookies = cookies).content #print req_1 soup = BeautifulSoup(req_1,’lxml’) print soup # ids = soup.find_all(’span’) # for id in ids : # name = id.get(’title’) # print nameif __name__ == ’__main__’: main()
問題解答
回答1：兩個關(guān)鍵點(diǎn)：1. headers的user-agent2. csrfmiddlewaretoken參數(shù)
# coding: utf-8import requestsurl = ’https://www.appannie.com/account/login’session = requests.Session()session.headers[’user-agent’] = ’Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36’session.get(url)token = session.cookies.get(’csrftoken’)data = { ’csrfmiddlewaretoken’: token, ’next’:’/dashboard/home/’, ’username’:’XXXX’, ’password’:’XXXX’}r = session.post(url, data)print r.status_code
相關(guān)文章：
1. 這段代碼是獲取百度收錄量的！需要怎么設(shè)置才能獲取百度快照旁邊的網(wǎng)址呢？2. vue.js - vue 打包后 nginx 服務(wù)端API請求跨域問題無法解決。3. angular.js - angular指令中的scope屬性中用&獲取父作用域函數(shù)的問題4. CSS響應(yīng)式導(dǎo)航原理，下圖那個具體怎么做出來的？5. html5 - node.js下載失敗，求大神解釋6. html5 - canvas 跨域問題7. oracle - mysql如何將group by的行數(shù)據(jù)轉(zhuǎn)為列8. javascript - 切換掉當(dāng)前頁面后該頁面的js動畫會暫停？9. mysql - 大部分?jǐn)?shù)據(jù)沒有行溢出的text字段是否需要拆表10. javascript - svg小白，想要在svg里面插入圖片
網(wǎng)公網(wǎng)安備