这是我最后的波纹了——记一个低端爬虫

　　呜呜呜，没想到时隔一年俺居然还有写爬虫的时候，虽然是个简单的需求，但是还是记录一下吧，毕竟以后俺也未必有机会写代码了。

需求

　　由于老板个人主页建设的要求，需要将她发表文献的年限、影响因子、卷号期号页码和doi号等信息整理下来，考虑到这是劳动密集型工作，自然地就想到爬虫了。虽然俺久疏战阵，但是既然需要俺出力，还是义不容辞的。

实现步骤

识别中英文

　　由于到手的Excel是中英文标题混排的，首先就得识别出中英文与否，英文去Pubmed就行，中文就得去知网。这部分俺想不到什么精妙的Trick，去网上偷了个轮子：

def checkEn(strs):
    for _char in strs:
        if '\u4e00' <= _char <= '\u9fd5':
            return True
    return False

　　注：之前都是到\u9fa5，新标准又增加了几个字，所以是\u9fd5了。

Pubmed爬虫

　　祭出老伙计Requests & BeautifulSoup，随手先整出一段代码，望忽略里面辣眼睛的正则和split函数，正所谓能用就行：

def getInfoFromPubmed(strs):
    url = "https://pubmed.ncbi.nlm.nih.gov/?term="+strs
    session = requests.Session()
    response = session.get(url)
    soup = BeautifulSoup(response.text,'lxml')    
    cite = soup.find("span", class_="cit").get_text().strip()
    authors = soup.find_all("a", class_="full-name")
    authors = [item.get_text().strip() for item in authors]
    func = lambda authors,i: authors if i in authors else authors + [i]
    authors_set = reduce(func, [[], ] + authors)
    author = ','.join(authors_set)
    doi = soup.find("span", class_="citation-doi").get_text().strip()
    journal = soup.find("button", class_="journal-actions-trigger trigger").get_text().strip()
    year = cite.split(";")[0]
    vol = cite.split(";")[1].split(":")[0]
    if "(" in vol:
        pat = re.compile('\((.*?)\)')
        no = re.findall(pat,vol)[0]
        vol = vol.split("(")[0]
    else:
        no = None
    page = cite.split(":")[-1].strip(".")
    return author,year,vol,no,page,doi,journal

　　由于Pubmed不能显示影响因子，只能先提取期刊名去别处弄。

获取影响因子

　　实在不知道scholarscope的API，只能另辟蹊径，从Justscience里面爬了。

def getInfoFromJustscience(journal):
    url = "http://sci.justscience.cn/?q="+journal
    session = requests.Session()
    response = session.get(url)
    form = pd.read_html(response.text)[1]
    for index in form.index:
        target = form.loc[index,'期刊缩写']
        if target == journal:
            impact = form.loc[index,'影响因子']
        else:
            impact = None
    return impact

完整代码

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# @Time    : Mon Jul 27 22:44:09 2020
# @Author  : Catkin
# @Website : blog.catkin.moe
import requests,re
import pandas as pd
from functools import reduce
from bs4 import BeautifulSoup

def checkEn(strs):
    for _char in strs:
        if '\u4e00' <= _char <= '\u9fd5':
            return True
    return False

def getInfoFromPubmed(strs):
    url = "https://pubmed.ncbi.nlm.nih.gov/?term="+strs
    session = requests.Session()
    response = session.get(url)
    soup = BeautifulSoup(response.text,'lxml')    
    cite = soup.find("span", class_="cit").get_text().strip()
    authors = soup.find_all("a", class_="full-name")
    authors = [item.get_text().strip() for item in authors]
    func = lambda authors,i: authors if i in authors else authors + [i]
    authors_set = reduce(func, [[], ] + authors)
    author = ','.join(authors_set)
    doi = soup.find("span", class_="citation-doi").get_text().strip()
    journal = soup.find("button", class_="journal-actions-trigger trigger").get_text().strip()
    year = cite.split(";")[0]
    vol = cite.split(";")[1].split(":")[0]
    if "(" in vol:
        pat = re.compile('\((.*?)\)')
        no = re.findall(pat,vol)[0]
        vol = vol.split("(")[0]
    else:
        no = None
    page = cite.split(":")[-1].strip(".")
    return author,year,vol,no,page,doi,journal

def getInfoFromJustscience(journal):
    url = "http://sci.justscience.cn/?q="+journal
    session = requests.Session()
    response = session.get(url)
    form = pd.read_html(response.text)[1]
    for index in form.index:
        target = form.loc[index,'期刊缩写']
        if target == journal:
            impact = form.loc[index,'影响因子']
        else:
            impact = None
    return impact

if __name__ == '__main__':
    data = pd.read_excel("论文成果.xls",index_col=0,header=1)
    for index in data.index:
        title = data.loc[index,"论文名称"]
        status = checkEn(title)
        try:
            if not status:
                author,year,vol,no,page,doi,journal = getInfoFromPubmed(title)
                impact = getInfoFromjustscience(journal)
                data.loc[index,'作者名称'] = author
                data.loc[index,'发表时间'] = year
                data.loc[index,'卷号'] = vol
                data.loc[index,'期号'] = no
                data.loc[index,'页面范围'] = page
                data.loc[index,'DOI码'] = doi
                data.loc[index,'影响因子'] = impact
        except:
            pass   
    data.to_excel("result.xlsx")

后记

　　代码简直不忍直视，深感俺真的是太菜了，匿了匿了。