Compare commits

..

99 Commits

Author SHA1 Message Date
gitea 30d0207368 Bugfix in script deploy_music.sh 2024-04-17 20:26:08 +02:00
gitea d27b477db8 Adding playlists, various bufixes and installation scripts
Adding playlists, various bufixes and installation scripts

Fixing

Rename

KUUUURWA

Install script music bot

fx

Fix ?

fix

Synchro

test

test

Undo

tst

test

tst

Undo changes

test

fx

tst

fx

Test commit of two files priority queue.
2024-04-17 19:41:32 +02:00
gitea ca6896ebcf Fixing the bug in the conjurer_musician.py file.
Fx

Fix

fx

Fix ?

Revert "Fix"

This reverts commit 4e185f7b34dfede04ed36ab9813bd9d1f7fb8b35.

Adding deploy for music bot
2024-04-17 19:41:32 +02:00
gitea 0375b0a892 Liq radio + some comments and nice stuff
fix

fix

Fix ?

fx

Liquidsoap init

test

test2

test

Tst

tst

fix

PLS

tst

tst

test

test

fix

tst

aa

aa# Please enter the commit message for your changes. Lines starting

Ping

fix?

fix

fix

Working version

Upgrade

Fix

fix

metadata

test

fix

fix# Please enter the commit message for your changes. Lines starting

Fix

aa

fix

fix

test

FIx

test

test

test

fix

fix

fix?

aa

fix

test

Final v.00

HLS

Serve
2024-04-17 19:38:53 +02:00
gitea 1caecceb10 Logging + playlist + plans
Fix logger
2024-04-17 19:38:52 +02:00
gitea 5cb51fb296 Logging added to music service 2024-04-10 20:37:45 +02:00
gitea 6589b73329 After squash fixes
After squash fixes
2024-04-10 15:45:14 +02:00
gitea 89003c32c9 Serverside work of search 2024-04-10 15:26:40 +02:00
gitea 17eace3e73 Fix rotation and fix of playlist 2024-04-10 15:26:39 +02:00
gitea 3f2248336d Deploy!
Test 1

As

Fix 2

Fix3
2024-04-10 15:26:39 +02:00
gitea f8664c0fd1 Music search move file server side att 1 2024-04-10 15:26:33 +02:00
gitea 82273df13c This is seriously last fix
description update

last fix ?

Small fix

Kolejny fix

Final fix

FX2

fx
2024-04-10 15:26:29 +02:00
gitea ae0d807714 Pornol dl2 2024-04-10 15:26:28 +02:00
gitea edc0a4a502 Pornol dl 1 2024-04-10 15:26:28 +02:00
gitea 39ecb82395 Coroutine fix 2024-04-10 15:26:18 +02:00
gitea d8f5bd1805 Plan for next release 2024-04-10 15:26:18 +02:00
gitea 3f68c6aaed Test
aaa

aa

Bugfixing et masse

bbb

asas

aas

asas

aa

as

qq

as

as

as

as

as

aaa

asa

nn

aa

sda

88

asa

sasa

adsa

a

as

asdas

Final version before tests

Fix1

aa

aa

fx

dx

:)

aa

Hopefully last fix

aaa

aaa

bbb

bb

dd

aa

sdae

aa

bb

aa

kk

a

Fix

fx

aa

deploy

fx

asas

bb

as

fx

fx

aa

aa

mess

kk

A

fix

aa

aa

Viwe fix

a

test

a

aa

FX
2024-04-10 15:26:18 +02:00
gitea aee53de258 Installer
`12`

sa

asda

123

AA

Test
2024-04-10 15:26:12 +02:00
gitea 20caa5600a name change and othre bugfixes
fix

fax

fx

121

as

aa

fx

f5

fx4

fix

fx1
2024-04-10 15:26:06 +02:00
gitea 4a4f8df06e Work on deployment 2024-04-10 15:26:06 +02:00
gitea 6c9f01971a Bugfixes
asdasdasd

:)

asa

asdasda

aa

:)

asa

aa

:)

:)

sa

as

aa

aa

aa

aaa

as

11

aa

11

xxx

dd

ds

aa

a1

aa

Test

Suppress

Move

Installer

aaa

Serwis

`12`

aa

xd

asasa

xd

aa

asa

aaa

:)

Hejka

a

xd

aa

aaas

555

aaa

asa

aa

11

aaa

aaa

as

sa

async

asda

00

123

123

aa

as

ss

as

a

sa

aa

1

RES

T

Hejko

aa

11

AA

test

Test
2024-04-10 15:26:05 +02:00
gitea 211c45891e Work on deployment
fix

fax

fx

Fin part

fx

121

Temp fix

as

aa

fx

f5

fx4

fix

fx1

Work on deployment
2024-04-10 15:26:00 +02:00
gitea de3d6a8f42 Install options 2024-04-10 15:25:59 +02:00
gitea 9c027fd636 Waitress 2024-04-10 15:25:59 +02:00
gitea 9af6b8e187 Bugfixing
add

Test 1

Test 2

Test3

Test phase 2

FIx 1

Fix 2

Fix 10

Fix 11

Fix 13

Fix 14

Fix 15

Test 1

Test 2

Test 3

Fix 15

Test 7

Fix 18

Test 20

Done

Minor 1
2024-04-10 15:25:59 +02:00
gitea d00ede6c62 Service
FIx
2024-04-10 15:25:59 +02:00
gitea 5b7d7b31b2 Linux conf files 2024-04-10 15:25:59 +02:00
gitea 00172920a9 Waitress
Deploy.shgit
2024-04-10 15:25:59 +02:00
gitea 5e3ffdacb5 Minor fix 2
Minor 1

Done

Test 20

Fix 18

Test 7

Fix 15

Test 3

Test 2

Test 1

Fix 15

Fix 14

Fix 13

Fix 12

Fix 11

Fix 10

Fix 9

Fix 2

FIx 1

Test phase 2

Test 8

Final test

Test 5

Test 4

Test3

Test 2

Test 1

add
2024-04-10 15:25:59 +02:00
gitea 471d4e923b Service
FIx
2024-04-10 15:25:53 +02:00
gitea d11f5d8d24 New files for flask services 2024-04-10 15:25:53 +02:00
gitea cfb26d95a4 Nowy lepszy trunk 2024-04-10 15:25:53 +02:00
gitea 5e14299fef Additional lines of code 2024-04-10 15:25:40 +02:00
gitea 126de9a4a4 Fix 2024-04-10 15:16:22 +02:00
gitea 0d132983fc Fix 2.0 2024-04-10 15:05:43 +02:00
gitea 67cae0e68e Fix 2024-04-10 15:05:43 +02:00
gitea eabd0e8e1e Nope 2024-04-10 15:05:43 +02:00
gitea f7c91e3874 Test chat gpt4 2024-04-10 15:05:43 +02:00
gitea c224b2ccc8 Hejo 2024-04-10 15:05:43 +02:00
gitea 3ca9f2c6f3 -1 2024-04-10 15:04:41 +02:00
gitea c52d55e0a2 +1 2024-04-10 15:04:41 +02:00
gitea e1fa15abb3 Async 2024-04-10 15:04:41 +02:00
gitea 74c76e639d Async fix 2024-04-10 15:04:41 +02:00
gitea 35a65e48e9 And even more async 2024-04-10 15:04:41 +02:00
gitea 5d5c136b41 More async 2024-04-10 15:04:40 +02:00
gitea b9fbc0f910 Async wyszukaj 2024-04-10 15:04:40 +02:00
gitea fd742c8f1c Parametryzacja "muzyki mojego ludu" 2024-04-10 15:04:40 +02:00
gitea 1c240342c5 Logging 2024-04-10 15:04:37 +02:00
gitea 7bd0c0aca4 Fix 2.6 2024-04-10 15:04:37 +02:00
gitea 7c8ad99743 Fix 2 2024-04-10 15:04:37 +02:00
gitea 90896fa9be Fix 1 2024-04-10 15:04:37 +02:00
gitea e3bec6bc11 Playlist based on chat history 2024-04-10 15:04:37 +02:00
gitea aa00045dec Trunk + mod 2024-04-10 15:04:32 +02:00
gitea 06e3671ec2 Linting is done 2024-04-10 15:04:22 +02:00
gitea 0879796cc6 Linting 2024-04-10 15:04:13 +02:00
gitea 9f756b3092 Dodana procedura asynchronicznej generacji iteratora oraz komentarz w
jaki sposob inaczej przerobic fora na asynchroniczny. Przydatne ze
wzgledu na to ze szukajka nadal blokuje heartbeat bota - co skutkuje
warningnami i wyjatkami
2024-04-10 15:04:09 +02:00
gitea 579a3f2a52 Reakcje ladacznicy, poprawka na mention w reset the clock 2024-04-10 15:04:09 +02:00
gitea 5505022dbb Funkcjonalnosc logowania incydentow w Absinth Eldritch Horror
Hammertimespace Continuum
2024-04-10 15:04:05 +02:00
gitea d2e5a34b87 Mala poprawka na rozmiar plikow pamieci 2024-04-10 15:03:55 +02:00
gitea c7696fafdc Fixed playlist gen 2024-04-10 15:03:43 +02:00
gitea 7f3de2ea13 Kurwa bugfix pod windowsa 2024-04-10 15:03:42 +02:00
gitea ec505f050f modified 2024-04-10 14:22:01 +02:00
gitea f55c4ce1a7 Nowy lepszy trunk
Additional lines of code

1

AAA
2024-04-10 14:22:00 +02:00
gitea 432abc73b9 Upgrade gpt model 2024-04-10 14:22:00 +02:00
gitea cdaf2589b3 Library update
fix

1111

tezz
2024-04-10 14:22:00 +02:00
gitea 67ab3b4bba Docstring + bugfixes + random cyclic message
Nowy commit
2024-04-10 14:22:00 +02:00
gitea 71623d9812 Update fabryczka
:)

:)
2024-04-10 14:22:00 +02:00
gitea ce54233908 Fix. Was joking. 2024-04-10 14:22:00 +02:00
gitea b0df66c5da Fix. Last. 2024-04-10 14:22:00 +02:00
gitea fda2733600 Token amount change
Last (hopefully) fix

Fix+1

Fix

Fix

Fix

Test

Fix tokens

Fix
2024-04-10 14:22:00 +02:00
gitea 027050a4dd Token counter 2024-04-10 14:22:00 +02:00
gitea b22b9add15 Gpt fixes
FIx

Fix

Fix

Fix. Another.

!

Fix

Tiktoken

Fix

Fix
2024-04-10 14:22:00 +02:00
gitea 1fd5b8cfac GPT fixes
Fix

Test
2024-04-10 14:22:00 +02:00
gitea 5e3d90c038 Nope. It did not happen
Rollback not working

Nowe
2024-04-10 14:22:00 +02:00
gitea fef53a3587 Test heartbeat fix 2024-04-10 14:22:00 +02:00
gitea be1e338e54 Fixes + upgrade GPT model
Fix 2.0

Fix

Nope
2024-04-10 14:22:00 +02:00
gitea d6dd276a92 Mention conjurer 2024-04-10 14:22:00 +02:00
gitea a0cddaa1f0 update to gpt-4 2024-04-10 14:22:00 +02:00
gitea 38355da861 Test chat gpt4
Hejo

fix

:)
2024-04-10 14:21:59 +02:00
gitea 03c73fc55e Various fixes
Fixes

Fix

fix

Add Rev

Add

Fix with \\b

Lilia

Lunka i small fix
2024-04-10 14:21:59 +02:00
gitea 77089d408b ASync search and moving to fully asynchronous work + responses to config command
Async wyszukaj

More async

And even more async

Async fix

Async

+1

-1
2024-04-10 14:21:59 +02:00
gitea f05ae2b006 Improvements 2024-04-10 14:21:59 +02:00
gitea bca63dd94a Additional logging 2024-04-10 14:21:59 +02:00
gitea 2641965792 Playlist based on chat history
Fix 1

Fix 2

Fix 2.6

Logging
2024-04-10 14:21:59 +02:00
gitea dc9946533e Various fixes
Trunk + mod

FIx
2024-04-10 14:21:59 +02:00
gitea 0a39eba723 Create python-package.yml 2024-04-10 14:21:59 +02:00
gitea 114cfc35bc Create python-app.yml 2024-04-10 14:21:59 +02:00
gitea 6a92ff4956 Small fixes continued. 2024-04-10 14:21:59 +02:00
gitea c9d371d941 Small fixes in the paths 2024-04-10 14:21:59 +02:00
gitea 8aa980ff9c Update settings and history 2024-04-10 14:21:59 +02:00
gitea 063f15e3a6 Another fix 2024-04-10 14:21:59 +02:00
gitea 8e64482623 Fix of metadata loading 2024-04-10 14:21:58 +02:00
gitea f686552337 Kolejna poprawka 2024-04-10 14:21:58 +02:00
gitea f29672684a Wrong encoding in settings.json 2024-04-10 14:21:58 +02:00
gitea c8d47ee118 Linting
Linting
2024-04-10 14:21:58 +02:00
gitea 5173ed2e52 Bugfixy 2024-04-10 14:21:58 +02:00
gitea ddf1c5fdca Added playlist display 2024-04-10 14:21:58 +02:00
gitea 0c9298baa4 Lot of fixes async iterator asynchronous, plans for asynchronouse work
Mala poprawka na rozmiar plikow pamieci

Funkcjonalnosc logowania incydentow w Absinth Eldritch Horror Hammertimespace Continuum

Reakcje ladacznicy, poprawka na mention w reset the clock
2024-04-10 14:21:58 +02:00
gitea d3169786ff Fucking windows bugfix and playlist gen fix
Kurwa bugfix pod windowsa

Fixed playlist gen
2024-04-10 14:21:58 +02:00
14 changed files with 35186 additions and 110 deletions
+95 -58
View File
@@ -1,27 +1,29 @@
import re
import asyncio
import json
import logging
import netrc
import re
import threading
import time
from flask import Flask, jsonify, request
import netrc
import json
import asyncio
from json.decoder import JSONDecodeError
from queue import Queue
from typing import Callable
from urllib.request import urlopen
from uuid import UUID
import requests
import search_bot
from flask import Flask, jsonify, request
from habanero import Crossref
from waitress import serve
from queue import Queue
import search_bot
import requests
from typing import Callable
from uuid import UUID
NETRC_FILE = r"C:\Users\Activcom.pl\.netrc"
HOST_ADDRESS = "192.168.1.192"
PORT_ADDRESS = 5001
MAIN_BOT_ADDRESS = "http://192.168.1.191:5000"
SEND_RESULTS = "/conjurer"
BDSM_UUID_TEST = '96b7f85a-1142-4908-8986-62a2ea25a147'
BDSM_UUID_TEST = "96b7f85a-1142-4908-8986-62a2ea25a147"
ENCODING = "utf-8"
app = Flask(__name__)
@@ -29,6 +31,7 @@ app = Flask(__name__)
librarian_queue = Queue()
librarian_list = []
class Librarian(object):
def __init__(self, app, query, uuid) -> None:
@@ -49,25 +52,25 @@ class Librarian(object):
async def search_crossref(self, query):
self.app.logger.info("STARTED SEARCH")
result = self.cr.works(query = query, limit = 1000)
result = self.cr.works(query=query, limit=1000)
self.search_result_from_cr.update(result)
self.total = result['message']['total-results']
self.fetched += len(result['message']['items'])
self.logger.info (self.total)
self.total = result["message"]["total-results"]
self.fetched += len(result["message"]["items"])
self.logger.info(self.total)
self.logger.info(self.fetched)
while self.total > self.fetched and self.limit > self.fetched:
result = self.cr.works(query = query, limit = 1000)
result = self.cr.works(query=query, limit=1000)
self.search_result_from_cr.update(result)
self.total = result['message']['total-results']
self.fetched += len(result['message']['items'])
self.logger.info (self.total)
self.total = result["message"]["total-results"]
self.fetched += len(result["message"]["items"])
self.logger.info(self.total)
self.logger.info(self.fetched)
time.sleep(1)
self.app.logger.info("CROSSREF DONE")
#TODO: add to sql with UUID
#sql row - title, doi, author, uuid, last retrieved, is_available
# TODO: add to sql with UUID
# sql row - title, doi, author, uuid, last retrieved, is_available
with open("rr_results.json", "r+", encoding="utf-8") as data_file:
# First we load existing data into a dict.
try:
@@ -76,18 +79,34 @@ class Librarian(object):
file_data = {}
summarized_results = []
self.app.logger.info("REFINE: Removing all derived works from the list")
for item in self.search_result_from_cr['message']['items']:
for item in self.search_result_from_cr["message"]["items"]:
container = "NO"
if "container-title" in item:
container = "YES"
else:
summarized_results.append({"DOI" : item['DOI'], "title": item['title'] if 'title' in item else None,"type": item['type'] if 'type' in item else None, "container": container})
result = {self.uuid : {"total_results": self.search_result_from_cr['message']['total-results'], "on_page" : 1, "summary":summarized_results, "results":self.search_result_from_cr['message']['items']}}
summarized_results.append(
{
"DOI": item["DOI"],
"title": item["title"] if "title" in item else None,
"type": item["type"] if "type" in item else None,
"container": container,
}
)
result = {
self.uuid: {
"total_results": self.search_result_from_cr["message"][
"total-results"
],
"on_page": 1,
"summary": summarized_results,
"results": self.search_result_from_cr["message"]["items"],
}
}
if file_data:
file_data.update(result)
else:
file_data = result
self.app.logger.info("REFINE: Dumping to file")
data_file.truncate(0)
data_file.seek(0)
@@ -96,7 +115,9 @@ class Librarian(object):
async def check_dois_in_local_db(self, doi):
self.app.logger.info("REFINE: Running search in the backend app")
coro = asyncio.to_thread(search_bot.search_for_doi, doi, self.live_results, self.app.logger)
coro = asyncio.to_thread(
search_bot.search_for_doi, doi, self.live_results, self.app.logger
)
result = await coro
return result
@@ -109,25 +130,34 @@ class Librarian(object):
data = response.read()
text = data.decode("utf-8")
for line in text.splitlines():
if re.match(r".*Unfortunately, Sci-Hub doesn't have the requested document.*", line):
if re.match(
r".*Unfortunately, Sci-Hub doesn't have the requested document.*",
line,
):
return False
if m := re.match(r".*<embed type=\"application.pdf\"\s*src=\"(.*\.pdf)",line):
if m := re.match(
r".*<embed type=\"application.pdf\"\s*src=\"(.*\.pdf)", line
):
direct_download_link = "https:" + str(m.group(1))
self.app.logger.info (direct_download_link)
self.app.logger.info(direct_download_link)
return True
return False
async def check_if_exists(self, doi, brute_force = False):
async def check_if_exists(self, doi, brute_force=False):
result = {}
result = await self.check_dois_in_local_db(doi)
if brute_force and not result:
for item in doi:
item_link = "https://sci-hub.se/" + item[0]
tmp = {"DOI" : item["DOI"], "exists" : await self.check_if_exists_brute_force(item_link), "data": item}
tmp = {
"DOI": item["DOI"],
"exists": await self.check_if_exists_brute_force(item_link),
"data": item,
}
result.update(tmp)
return result
async def refine_query(self, unrefined_reqult, brute_force = False):
async def refine_query(self, unrefined_reqult, brute_force=False):
temp = []
refined_result = []
for key in unrefined_reqult:
@@ -135,7 +165,7 @@ class Librarian(object):
self.logger.info(key)
for item in unrefined_reqult[self.uuid]["summary"]:
if item["container"] == "NO" and item["title"]:
if item["container"] == "NO" and item["title"]:
refined_result.append(item)
for item in refined_result:
@@ -149,20 +179,21 @@ class Librarian(object):
refined_result = temp
self.hit = len(refined_result)
return refined_result
#TODO: DOłożyć sprawdzenie czy w rafinowanym pliku już nie mamy częściowego wyniku
# TODO: DOłożyć sprawdzenie czy w rafinowanym pliku już nie mamy częściowego wyniku
async def answer_query(self):
#TODO: DOłożyć sprawdzenie czy w rafinowanym pliku już nie mamy częściowego wyniku tutaj bo zanim w ogóle otworzymy nierafinowany
#Search sql for UUID
#SQL SELECT * FROM RESULTS WHERE UUID = uuid
# TODO: DOłożyć sprawdzenie czy w rafinowanym pliku już nie mamy częściowego wyniku tutaj bo zanim w ogóle otworzymy nierafinowany
# Search sql for UUID
# SQL SELECT * FROM RESULTS WHERE UUID = uuid
self.logger.info(f"Search started {self.uuid}")
uuid_found = False
database = None
query = self.query
with open("cr_results.json", "r+", encoding="utf-8") as cr_file:
#tu logika komunikacji z SQL i cała logika związana z wyszukaniem
#ale tymczasowo plik
# tu logika komunikacji z SQL i cała logika związana z wyszukaniem
# ale tymczasowo plik
try:
database = json.load(cr_file)
database = json.load(cr_file)
except JSONDecodeError:
pass
@@ -175,7 +206,7 @@ class Librarian(object):
database.update(answer)
cr_file.truncate(0)
cr_file.seek(0)
json.dump(database,cr_file)
json.dump(database, cr_file)
self.result = await self.refine_query(answer)
return self.result
else:
@@ -183,13 +214,14 @@ class Librarian(object):
database = answer
cr_file.truncate(0)
cr_file.seek(0)
json.dump(database,cr_file)
json.dump(database, cr_file)
self.result = await self.refine_query(answer)
self.app.logger.info("Refined")
self.done = True
return self.result
#============================= FLASK INTERNALS===============================
# ============================= FLASK INTERNALS===============================
def flask_debug():
"""
The `flask_debug` function starts a Flask application in debug mode without using the reloader.
@@ -198,6 +230,7 @@ def flask_debug():
# trunk-ignore(bandit/B201)
app.run(debug=True, use_reloader=False, host=HOST_ADDRESS, port=PORT_ADDRESS)
def waitress_run():
"""
The `waitress_run` function serves the `app` on host "0.0.0.0"
@@ -205,6 +238,7 @@ def waitress_run():
"""
serve(app, host=HOST_ADDRESS, port=PORT_ADDRESS)
class BackgroundTaskSearch(threading.Thread):
def run(self):
loop = asyncio.new_event_loop() # loop = asyncio.get_event_loop()
@@ -221,7 +255,7 @@ class BackgroundTaskSearch(threading.Thread):
self.app.logger.info("Saving to file")
with open("s_results.json", "r+", encoding="utf-8") as s_file:
try:
database = json.load(s_file)
database = json.load(s_file)
except JSONDecodeError:
pass
if database:
@@ -236,17 +270,18 @@ class BackgroundTaskSearch(threading.Thread):
self.app.logger.info(result)
coroutine = asyncio.to_thread(
requests.post,
f"{MAIN_BOT_ADDRESS}{SEND_RESULTS}",
json=result,
timeout=360,
)
requests.post,
f"{MAIN_BOT_ADDRESS}{SEND_RESULTS}",
json=result,
timeout=360,
)
self.app.logger.info("SENT")
_ = await coroutine
self.app.logger.info("SEND CONFIRMED")
time.sleep(1)
#==================================SERVER ROUTES==========================================
# ==================================SERVER ROUTES==========================================
@app.route("/query", methods=["POST"])
async def query_database():
record = json.loads(request.data)
@@ -254,8 +289,8 @@ async def query_database():
app.logger.info(record["query"])
app.logger.info(record["UUID"])
uuid = record["UUID"]
#TODO: TESTING ONLY======================
uuid = UUID('{12345678-1234-5678-1234-567812345678}')
# TODO: TESTING ONLY======================
uuid = UUID("{12345678-1234-5678-1234-567812345678}")
cl = Librarian(app, record["query"], uuid)
librarian_queue.put(cl)
@@ -267,6 +302,7 @@ async def query_database():
)
return return_data
@app.route("/get_partial_result", methods=["POST"])
async def get_partial():
record = json.loads(request.data)
@@ -282,12 +318,13 @@ async def get_partial():
)
return return_data
#=======================================MAIN===================================================
# =======================================MAIN===================================================
if __name__ == "__main__":
logger = logging.getLogger('conjurer_musician')
logger = logging.getLogger("conjurer_musician")
logger.setLevel(logging.DEBUG)
threads = []
#threads.append(threading.Thread(target=waitress_run))
# threads.append(threading.Thread(target=waitress_run))
threads.append(threading.Thread(target=flask_debug))
bgtask = BackgroundTaskSearch()
bgtask.app = app
@@ -296,7 +333,7 @@ if __name__ == "__main__":
for worker in threads:
try:
logger.info(f"App number: {i}")
i+=1
i += 1
worker.start()
except RuntimeError as e:
logger.error("Exploded")
@@ -1,20 +1,22 @@
# This Python file uses the following encoding: utf-8
"""
The provided Python script sets up a Flask web server to manage a list of music files, with
functions for rescanning the music folder, updating the music list, and serving the music list via
API endpoints.
"""
import json
import logging
import re
from pathlib import Path
from platform import uname
import threading
import time
from datetime import datetime
from logging import handlers
from pathlib import Path
from platform import uname
from sys import platform
import logging
from flask import Flask, jsonify, request
from waitress import serve
from flask import Flask, jsonify, redirect, request, send_from_directory
from waitress import serve
HOST_ADDRESS = "192.168.1.15"
PORT_ADDRESS = 5000
@@ -26,24 +28,42 @@ if platform in ("linux", "linux2"):
NETRC_FILE = "/home/mtuszowski/.netrc"
LOGSTORE = "/home/mtuszowski/conjurer/logs/"
ENCODING = "utf-8"
GRAPHICS_PATH = "/home/mtuszowski/conjurer/Conjurer_graphics/"
else:
LOGFILE = "/home/pi/Conjurer/discord_mus_service.log"
MEMORY_FIVE_SIARA = "/home/pi/Conjurer/pamiec.json"
SYSTEM_GPT_SETTINGS = "/home/pi/Conjurer/system_gpt_settings.json"
MEMORY_FIVE_MUZYKA = "/home/pi/Conjurer/pamiec_muzyki.json"
MUSIC_FOLDER = "/home/pi/RetroPie/mp3/"
SETTINGS_FILE = "/home/pi/Conjurer/settings.json"
NETRC_FILE = "/home/pi/.netrc"
LOGSTORE = "/home/pi/RetroPie/logs/"
ACCIDENT_LOG = "/home/pi/Conjurer/accident_log.json"
ENCODING = "utf-8"
GRAPHICS_PATH = "/home/pi/RetroPie/Conjurer_graphics/"
DIR_PATH_SADOX = "/home/pi/RetroPie/Fansadox/"
MUSIC_FOLDER = "/home/pi/RetroPie/mp3/"
music_file_list = []
def create_playlist():
"""
Reads a JSON file containing a playlist and writes the playlist items to a text file.
The JSON file path is '/home/pi/Conjurer/playlist.json',
and the text file path is '/home/pi/Conjurer/all_playlist.playlist'.
Raises:
JSONDecodeError: If the JSON file cannot be decoded.
"""
with open("/home/pi/Conjurer/playlist.json", "r+", encoding="utf-8") as r_file:
with open(
"/home/pi/Conjurer/all_playlist.playlist", "w", encoding="utf-8"
) as w_file:
try:
playlist = json.load(r_file)
for item in playlist:
w_file.write(item)
w_file.write("\n")
except json.JSONDecodeError:
pass
def rescan():
"""
The `rescan` function logs a message, scans for mp3 files in a specified folder,
@@ -55,6 +75,14 @@ def rescan():
if platform == "win32":
temp_music_file = temp_music_file.replace("/", "\\")
music_file_list.append(temp_music_file)
with open("/home/pi/Conjurer/playlist.json", "r+", encoding="utf-8") as s_file:
database = music_file_list
# self.app.logger.info("DUMPING DATA")
s_file.truncate(0)
s_file.seek(0)
json.dump(database, s_file)
create_playlist()
def thread_rescan():
"""
@@ -63,14 +91,31 @@ def thread_rescan():
"""
logging.info("Starting filesystemupdater")
while True:
time.sleep(60*60*24)
time.sleep(60 * 60 * 24)
logging.info("Rescan triggered")
rescan()
app = Flask(__name__)
def wyszukaj(word_list, how_many):
logger = logging.getLogger()
# TODO: Odpalić wyszukiwarki w wątkach i dopiero po wszystkim zsumować wyszukiwanie.
def wyszukaj(word_list, how_many, _logger=None, return_to_bot=True):
"""
Searches for music files based on a list of keywords and returns a list of matching files.
Args:
word_list (list): A list of keywords to search for.
how_many (int): The maximum number of matching files to return.
Returns:
list: A list of matching files, sorted by their search weight.
"""
if _logger is None:
fun_logger = logging.getLogger()
else:
fun_logger = _logger
return_list = []
search_weight = []
@@ -84,7 +129,7 @@ def wyszukaj(word_list, how_many):
skip_start = 1
for word in word_list[skip_start:]:
token_weight = len(word)
logger.info("Słowo kluczowe: %s", word)
fun_logger.info("Słowo kluczowe: %s", word)
itr = 0
for file in music_file_list:
file = file.split(SEPARATOR_FILE_PATH)
@@ -124,6 +169,7 @@ def wyszukaj(word_list, how_many):
else:
skip = 4
matched_times = 1
# TODO: To do wątków albo for asynchroniczny
for itm in all_words[skip:]:
pingu += 1
pattern = ".*" + word + ".*"
@@ -139,26 +185,42 @@ def wyszukaj(word_list, how_many):
matched_times += 1
itr += 1
logger.info("Stworzylem tablice wag zajęło mi to %s", datetime.now() - time_start)
fun_logger.info(
"Stworzylem tablice wag zajęło mi to %s", datetime.now() - time_start
)
time_start = datetime.now()
item_to_search = max_weight(search_weight)
itr = 0
if item_to_search == 0:
return None
return []
not_found = True
return_list = []
if int(how_many) <= 0:
logger.info("Jeden plik do zagrania")
fun_logger.info("Jeden plik do zagrania")
while not_found:
if search_weight[itr][0] == item_to_search:
return_list.append(search_weight[itr])
return return_list
if not return_to_bot:
with open(
"/home/pi/Conjurer/priority_queue.playlist", "r+", encoding="utf-8"
) as s_file:
s_file.write(search_weight[itr][1])
break
itr += 1
else:
logger.info("Wiele plików do zagrania")
fun_logger.info("Wiele plików do zagrania")
search_weight.sort(key=lambda x: x[0], reverse=True)
return_list.extend(search_weight[: int(how_many)])
return return_list
if not return_to_bot:
with open(
"/home/pi/Conjurer/priority_queue.playlist", "r+", encoding="utf-8"
) as s_file:
s_file.truncate(0)
for item in return_list:
s_file.write(item[1] + "\n")
return return_list
def max_weight(lista):
"""
@@ -188,6 +250,47 @@ def remove_characters(string, character):
return string.replace(character, "")
@app.route("/stream", methods=["GET"])
def stream_music():
"""
The function `get_music_list` returns a JSON object containing a list of music files.
:return: A JSON response containing a key "music_file_list" with the value of the variable
`music_file_list`.
"""
return send_from_directory("/tmp/hls", "stream.m3u8")
@app.route('/<string:file_name>')
def stream(file_name):
video_dir = '/tmp/hls'
return send_from_directory(video_dir, file_name)
@app.route("/stream_mp3", methods=["GET"])
def stream_music_mp3():
"""
The function `get_music_list` returns a JSON object containing a list of music files.
:return: A JSON response containing a key "music_file_list" with the value of the variable
`music_file_list`.
"""
return redirect("http://www.example.com", code=302)
@app.route("/clear_pr_pls", methods=["GET"])
def clear_pr_pls():
"""
The function `get_music_list` returns a JSON object containing a list of music files.
:return: A JSON response containing a key "music_file_list" with the value of the variable
`music_file_list`.
"""
with open(
"/home/pi/Conjurer/priority_queue.playlist", "w", encoding="utf-8"
) as s_file:
s_file.truncate(0)
return_data = jsonify(isError=False, message="Success", statusCode=200, data=[])
return return_data, 200
@app.route("/mp3", methods=["GET"])
def get_music_list():
"""
@@ -218,6 +321,7 @@ def update_music_list():
)
return return_data
@app.route("/get_music", methods=["POST"])
def look_for_playlist():
"""
@@ -234,7 +338,31 @@ def look_for_playlist():
app.logger.info(record["lista_slow"])
app.logger.info(record["UUID"])
app.logger.info(record["dlugosc_plejlisty"])
return_data = wyszukaj(record["lista_slow"],record["dlugosc_plejlisty"])
return_data = wyszukaj(record["lista_slow"], record["dlugosc_plejlisty"], app.logger, True)
return_data = (
jsonify(isError=False, message="Success", statusCode=200, data=return_data),
200,
)
return return_data
@app.route("/add_to_priority", methods=["POST"])
def add_to_priority():
"""
The function `update_music_list` receives a POST request with a JSON payload, logs the received
item, adds it to a music file list, and returns a success message along with the updated record.
:return: The function `update_music_list` is returning a tuple containing a JSON response and a
status code. The JSON response includes keys `isError`, `message`, `statusCode`, and `data`,
with values indicating the success of the operation and the data that was
received and added to the `music_file_list`.
The status code returned is 200, indicating a successful response.
"""
record = json.loads(request.data)
app.logger.info(record)
app.logger.info(record["lista_slow"])
app.logger.info(record["UUID"])
app.logger.info(record["dlugosc_plejlisty"])
return_data = wyszukaj(record["lista_slow"], record["dlugosc_plejlisty"], app.logger, False)
return_data = (
jsonify(isError=False, message="Success", statusCode=200, data=return_data),
@@ -251,6 +379,7 @@ def flask_debug():
# trunk-ignore(bandit/B201)
app.run(debug=True, use_reloader=False, host=HOST_ADDRESS, port=PORT_ADDRESS)
def waitress_run():
"""
The `waitress_run` function serves the `app` on host "0.0.0.0"
@@ -258,12 +387,27 @@ def waitress_run():
"""
serve(app, host=HOST_ADDRESS, port=PORT_ADDRESS)
if __name__ == "__main__":
logger = logging.getLogger('conjurer_musician')
logger = logging.getLogger("conjurer_musician")
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
handler = handlers.RotatingFileHandler(
filename=LOGFILE,
encoding=ENCODING,
mode="a",
maxBytes=6 * 1024 * 1024,
backupCount=6,
)
handler.setFormatter(formatter)
logger.addHandler(handler)
rescan()
logger.info("Started")
threads = []
#threads.append(threading.Thread(target=flask_debug))
threads.append(threading.Thread(target=waitress_run))
threads.append(threading.Thread(target=thread_rescan))
@@ -272,5 +416,3 @@ if __name__ == "__main__":
for worker in threads:
worker.join()
#TODO: Inne pliki - sadox, dokumenty naukowe itp.
@@ -0,0 +1,11 @@
[Unit]
Description=Conjurer_musician
After=multi-user.target
[Service]
Type=simple
ExecStart=/home/pi/Conjurer/env/bin/python3 /home/pi/Conjurer/conjurer_musician.py
Restart=on-abort
[Install]
WantedBy=multi-user.target
+34804
View File
File diff suppressed because it is too large Load Diff
+48
View File
@@ -0,0 +1,48 @@
let json.parse credentials = file.contents("/home/pi/Conjurer/icecast_credentials.json")
enable_replaygain_metadata()
s1 = replaygain(playlist(reload_mode="watch", "/home/pi/Conjurer/all_playlist.playlist"))
s2 = replaygain(playlist(reload_mode="watch", "/home/pi/Conjurer/priority_queue.playlist"))
s = random(id="randomizer", weights=[7000, 100], [s2, s1])
interactive.harbor(port = 9999)
f = interactive.float("f", description="Frequency", min=0., max=1000.,unit="Hz", 200.)
g = interactive.float("g", description="Gain", min=0., max=20.,unit="dB", 8.)
b = bass_boost(frequency=f, gain=g, s)
s = add([s, b])
a = interactive.float("main_volume", min=0., max=20., 1.)
s = compress.multiband.interactive(bands=3, s)
s = nrj(normalize (s))
s = crossfade(fade_out=3., fade_in=3., duration=5., s)
s = amplify(a,s)
s = blank.skip(max_blank=2., s)
emergency = single("/home/pi/RetroPie/mp3/Youtube/Dr. Peacock - Trip to Ireland [GvrvQTUbUcA].mp3")
radio = fallback(id="switcher",track_sensitive = false, [s, emergency])
handle_metadata = fun (m) -> begin
print(m["title"])
print(m["artist"])
print(m["filename"])
end
radio.on_metadata(handle_metadata)
interactive.persistent("script.params")
log_to_stdout = true
log_to_file = true
logpath = "/home/pi/Conjurer/radio_log.log"
loglevel = 3
# Enable logging on Standard Output and set logging level.
set("log.stdout",log_to_stdout)
set("log.level",loglevel)
# Enable logging to file. See variable: logpath
set("log.file",log_to_file)
set("log.file.path",logpath)
output.file.hls("/tmp/hls",[("mp3-low", %mp3(bitrate=96)), ("mp3-hi", %mp3(bitrate=160))],radio)
output.icecast(%mp3, host="retropie", port=8000,password=credentials.password, icy_metadata="true", mount="mp3-stream", radio)
+14
View File
@@ -0,0 +1,14 @@
[Unit]
Description=radio_conjurer Liquidsoap daemon
After=network.target
Documentation=http://liquidsoap.fm/
[Service]
User=pi
WorkingDirectory=/home/pi/Conjurer/
ExecStart=/home/pi/.opam/4.13.0/bin/liquidsoap /home/pi/Conjurer/radio_conjurer.liq
Restart=always
[Install]
WantedBy=multi-user.target
Alias=radio_conjurer-liquidsoap.service
+6
View File
@@ -0,0 +1,6 @@
#!/bin/bash
cd ./conjurer
git pull
cd ..
cp ./conjurer/conjurer_musician/conjurer_musician.py ./Conjurer
sudo systemctl restart conjurer_musician.service
+6
View File
@@ -0,0 +1,6 @@
#!/bin/bash
cd ./conjurer
git pull
cd ..
cp ./conjurer/conjurer_musician/radio_conjurer.liq ./Conjurer
sudo systemctl restart radio_service.service
-11
View File
@@ -1,11 +0,0 @@
[Unit]
Description=Conjurer_musician
After=multi-user.target
[Service]
Type=simple
ExecStart=/home/pi/Conjurer-service/conjurer/file_webservice/bin/python3 /home/pi/Conjurer-service/conjurer/file_webservice/conjurer_musician.py
Restart=on-abort
[Install]
WantedBy=multi-user.target
@@ -1,11 +0,0 @@
#!/bin/bash
python3 -m venv /home/pi/Conjurer-service/conjurer/file_webservice
source ./bin/activate
./bin/python3 -m pip install --upgrade pip
./bin/python3 -m pip install -r requirements_musician.txt
deactivate
sudo cp ./conjurer_musician.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start conjurer_musician.service
sudo systemctl enable conjurer_musician.service
+6 -2
View File
@@ -1,5 +1,9 @@
#!/bin/bash
cd /home/pi
mdkir Conjurer
cd Conjurer
python3 -m venv /home/pi/Conjurer/env
cp /home/pi/conjurer/requirements_bot.txt /home/pi/Conjurer/
source ./env/bin/activate
./env/bin/python3 -m pip install --upgrade pip
./env/bin/python3 -m pip install -r requirements_bot.txt
@@ -8,5 +12,5 @@ sed -i '1i\import shutil' ./env/lib/python3.11/site-packages/spotify_dl/youtube.
deactivate
sudo cp ./conjurer.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start conjurer_musician.service
sudo systemctl enable conjurer_musician.service
sudo systemctl start conjurer.service
sudo systemctl enable conjurer.service
+26
View File
@@ -0,0 +1,26 @@
#!/bin/bash
cd /home/pi
mkdir Conjurer
cd Conjurer
python3 -m venv /home/pi/Conjurer/env
source /home/pi/Conjurer/env/bin/activate
cp /home/pi/conjurer/conjurer_musician/requirements_musician.txt /home/pi/Conjurer/
cp /home/pi/conjurer/conjurer_musician/conjurer_musician.py /home/pi/Conjurer/
cp /home/pi/conjurer/conjurer_musician/radio_conjurer.liq /home/pi/Conjurer/
touch /home/pi/Conjurer/discord_mus_service.log
touch /home/pi/Conjurer/all_playlist.txt
touch /home/pi/Conjurer/playlist.json
touch /home/pi/Conjurer/priority_queue.playlist
touch /home/pi/Conjurer/radio_log.log
./env/bin/python3 -m pip install --upgrade pip
./env/bin/python3 -m pip install -r requirements_musician.txt
deactivate
sudo cp /home/pi/conjurer/conjurer_musician/conjurer_musician.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start conjurer_musician.service
sudo systemctl enable conjurer_musician.service
sudo cp /home/pi/conjurer/conjurer_musician/radio_service.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start radio_service.service
sudo systemctl enable radio_service.service