import re import logging import threading from flask import Flask, jsonify, request import netrc import json from platform import uname from sys import platform from urllib.request import urlopen from habanero import Crossref from waitress import serve if platform in ("linux", "linux2"): SEPARATOR_FILE_PATH = "/" if "microsoft-standard" in uname().release: NETRC_FILE = "/home/mtuszowski/.netrc" else: NETRC_FILE = "/home/pi/.netrc" elif platform == "win32": NETRC_FILE = "C:\\Users\\mtusz\\.netrc" HOST_ADDRESS = "192.168.1.15" PORT_ADDRESS = 5001 class Librarian(object): def __init__(self) -> None: netrc_mod = netrc.netrc(NETRC_FILE) authTokens = netrc_mod.authenticators("crossref") self.cr = Crossref(mailto=authTokens[0]) self.last_search = "" self.limit = 1000 def search_crossref(self, query): self.last_search = self.cr.works(query = query, limit = self.limit) for item in self.last_search['message']['items']: print("=====") #print (item) print(item['DOI']) print(item['title']) print(item['type']) print("=====") if "container-title" in item: print("This a part of larger release") print(item['container-title']) print (self.last_search['message']['total-results']) #add to sql with UUID #sql row - title, doi, author, uuid, last retrieved, is_available with open("results.json", "x", encoding="UTF-8") as new_file: new_file.seek(0) json.dump(self.last_search['message']['items'], new_file, indent=4) def check_if_exists(self,page_url): in_scihub_db = True if not page_url.startswith(("http:", "https:")): raise ValueError("URL must start with 'http:' or 'https:'") # trunk-ignore(bandit/B310) with urlopen(page_url) as response: info = response.info() print(info.get_content_type()) data = response.read() text = data.decode("utf-8") for line in text.splitlines(): if re.match(r".*Unfortunately, Sci-Hub doesn't have the requested document.*", line): return None if in_scihub_db: return text return None def download(self, page_url): if response := self.check_if_exists(page_url): for line in response.splitlines(): if m := re.match(r".*