gitea/conjurer
1
0
Fork 0
mirror of https://github.com/migatu/conjurer.git synced 2026-07-21 17:22:09 +00:00
Code Issues Packages Projects Releases Wiki Activity

share: dockerize the file-share pipeline and document it

Browse Source 
The Python half of the short-lived share links lived in the repo; the Apache
config and the cron entries that make it work were hand-placed on the host, so
the feature could not be rebuilt from a checkout. This adds the missing half.

Scripts (defaults unchanged, so existing bare-metal cron keeps working):
* scan_shares.py / revoke_shares.py take their paths from CONJURER_SHARE_*
  instead of hardcoding the Pi layout, and create their parent dirs;
* the revoke TTL is now CONJURER_SHARE_TTL_SECONDS. Its --help claimed "2min"
  while the code used a hardcoded 3600 - the help text now reports the real,
  configured value.

New share service:
* docker/Dockerfile.share - Apache + the two jobs, reusing the same scripts
  rather than forking copies;
* docker/share-vhost.conf.tpl - the previously undocumented Apache half.
  Three settings are load-bearing and commented as such: +FollowSymLinks (the
  shares ARE symlinks), -Indexes (a listing would expose every live token), and
  a deny rule for dotfiles (revoke_shares.py keeps .downloads.json - a map of
  every live token - inside the served directory);
* docker/entrypoint.share.sh - renders the vhost, seeds the index on first run,
  then runs scanner/revoker in sleep loops beside Apache (no cron, so their
  output shows up in docker logs);
* compose + env example, incl. the two volumes that MUST be shared with the
  musician (it creates the links and reads the index).

docs/deployment/FILE_SHARING.md documents the mechanism, both deployment routes
(docker and existing host Apache), the Discord command, why each Apache setting
matters, verification commands, and the known limitations - notably that a link
nobody ever downloads is never revoked, since the TTL starts at first download.

Verified: scanner and revoker exercised end-to-end against temp dirs (index
built; token recorded from a combined-format log line and the symlink unlinked).
Compose file not validated - no docker on this machine.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michal Tuszowski
2026-07-19 22:16:49 +02:00
parent dc449dd74c
commit 33ec1c0e35
9 changed files with 437 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment/DOCKER_PROXMOX.md
+5
View File
@@ -7,6 +7,11 @@ Runbook for running Conjurer as Docker containers across Proxmox VMs:
| **Main bot** | VM-bot | `conjurer-bot` | 5000 (Flask comm) | `docker/Dockerfile.bot` |
| **Librarian** | VM-librarian | `conjurer-librarian` | 5001 | `docker/Dockerfile.librarian` |
| **Musician** | VM-musician | `conjurer-musician` | 5000 (+ radio) | `docker/Dockerfile.musician` |
| **Share** (optional) | VM-musician or its own | `conjurer-share` | 8081 → 80 | `docker/Dockerfile.share` |
The share service publishes short-lived file links over Apache and is documented
separately in [FILE_SHARING.md](FILE_SHARING.md) — it shares two volumes with the
musician, so set it up after the musician is running.
The three talk to each other over HTTP on the Proxmox LAN. Direction of calls:
docs/deployment/FILE_SHARING.md
+197
View File
@@ -0,0 +1,197 @@
# File sharing (short-lived share links)
Publishes a file from the media library behind a random, unguessable URL that
stops working shortly after someone downloads it. Driven from Discord.
Historically only the Python half of this lived in the repo; the Apache config
and the cron entries were placed on the host by hand. This document plus
`docker/Dockerfile.share` close that gap.
## What it actually does
1. A **scanner** walks the media library and writes a JSON index of every path.
2. From Discord you search that index and pick files.
3. **Publishing** creates a `symlink` named after a random token
(`uuid.uuid4().hex`, 32 hex chars) inside the share directory, pointing at the
real file. Nothing is copied.
4. **Apache** serves that symlink at `https://<host>/share/<token>`.
5. A **revoker** tails Apache's access log, records when each token was *first*
downloaded, and unlinks it once the TTL expires.
Two things worth being precise about, because the names suggest otherwise:
- The token is **random, not a hash** of the file. Nothing about the content is
recoverable from the URL, and the same file shared twice gets two URLs.
- The "database" is a **JSON file**, not a SQL database.
(`utils/import_to_mysql.py` is an unrelated, unfinished scratch script.)
## The pieces
| Path | Role | Runs in |
|---|---|---|
| `conjurer_musician/scan_shares.py` | Builds the JSON index from the media root | share container (or cron) |
| `conjurer_musician/revoke_shares.py` | Tails the access log, unlinks expired tokens | share container (or cron) |
| `conjurer_musician/media_search_functions.py` | Search + publish helpers used by the service | musician |
| `conjurer_musician.py` → `/get_share_list`, `/get_share_links` | HTTP API, both behind `X-Conjurer-Api-Key` | musician |
| `file_search_functions.py` | Bot-side HTTP client | bot |
| `file_search_commands.py` | Discord cog: search UI + Accept/Cancel | bot |
| `docker/share-vhost.conf.tpl` | Apache vhost (symlinks, no indexes, dedicated log) | share container |
| `conjurer_musician/search_and_share.py` | Standalone CLI doing steps 1–3 by hand | anywhere (optional) |
Flow:
```
media library ──► scan_shares.py ──► share_scan.json ──┐
│ (reads)
Discord ──► bot cog ──► musician /get_share_list ─────────┘
└─► musician /get_share_links ──► symlink in share dir
│ (serves)
Apache vhost ─────────────────────┘
│ (logs)
└──► share_access.log ──► revoke_shares.py ──► unlink
```
The share directory and the index are **shared state between the musician and
the share service**: the musician creates links and reads the index, the share
service serves links and writes the index. Both must see the same two paths.
## Enabling it — Docker (recommended)
```bash
cp docker/env/share.env.example docker/env/share.env
# edit: CONJURER_SHARE_SERVER_NAME, and the TTL/intervals if you want
docker compose -f docker/compose.share.yaml up -d --build
```
Host layout the compose file expects:
| Host path | Contents |
|---|---|
| `/srv/share/media` | the media library (mounted read-only) |
| `/srv/share/links` | the published symlinks — **also mount into the musician** |
| `/srv/share/db` | `share_scan.json` — **also mount into the musician** |
| `/srv/share/logs` | Apache logs, incl. `share_access.log` |
Then point the musician at the same two paths. In `docker/env/musician.env`:
```ini
CONJURER_SHARE_DB=/srv/share/db/share_scan.json
CONJURER_SHARE_DIR=/var/www/html/share
CONJURER_SHARE_BASE_URL=https://czernobog.pl/share
```
and add the matching mounts to `docker/compose.musician.yaml`:
```yaml
- /srv/share/db:/srv/share/db
- /srv/share/links:/var/www/html/share
```
Finally forward `/share` from your public TLS endpoint to the container's port
(`8081` by default). The container speaks plain HTTP on purpose — TLS for
`czernobog.pl` stays on the reverse proxy you already run.
> **The media library must be mounted at the same absolute path the index
> recorded** (`/mnt/shares` by default). The symlinks are absolute, so if the
> scanner saw `/mnt/shares/x.mkv` and Apache has the library somewhere else, every
> link 404s.
The container does not use cron: the entrypoint runs the scanner and the revoker
in sleep loops next to Apache, so their output lands in `docker logs`.
## Enabling it — existing Apache on the host
If you keep serving from the host's Apache instead:
1. Install the vhost. Take `docker/share-vhost.conf.tpl`, replace
`__SERVER_NAME__`, and drop it into `/etc/apache2/sites-available/`, then
`a2ensite` + `systemctl reload apache2`. The three settings that matter are
explained in the Security section below — do not simplify them away.
2. Schedule both jobs. The scripts default to the original bare-metal paths, so
plain cron entries work with no environment at all:
```cron
*/5 * * * * /usr/bin/python3 /opt/conjurer/conjurer_musician/revoke_shares.py
17 * * * * /usr/bin/python3 /opt/conjurer/conjurer_musician/scan_shares.py
```
Defaults: media root `/mnt/shares`, index `/var/log/share_scan.json`, links
`/var/www/html/share`, log `/var/log/apache2/share_access.log`, TTL 3600s.
Override any of them with the `CONJURER_SHARE_*` variables from
`docker/env/share.env.example`.
3. Point the musician at the same paths via `CONJURER_SHARE_DB`,
`CONJURER_SHARE_DIR` and `CONJURER_SHARE_BASE_URL`.
## Using it
```
$tajna_biblioteka_inkwizycji <1-10> <keywords>
```
Restricted to the **Jarl** and **Thane** roles. It replies with the matches, you
select from the dropdown and press Accept; the bot answers with one URL per file.
(The docstring in the cog still says `!findfiles` — that name is stale.)
## Security
The vhost is not boilerplate; three of its settings are load-bearing:
- **`Options +FollowSymLinks`** — the shares *are* symlinks, so without it every
link 403s. `SymLinksIfOwnerMatch` cannot be substituted: the targets are owned
by whoever owns the media, not by `www-data`.
- **`Options -Indexes`** — a directory listing of the share dir would hand out
every currently live token in one request.
- **Deny dotfiles** — `revoke_shares.py` keeps `.downloads.json` (a map of every
live token to its download time) and `.logpos` *inside the served directory*.
Serving `.downloads.json` would leak the entire active link set.
Also worth knowing:
- Both HTTP endpoints call `_authorize_request()`, so they require
`X-Conjurer-Api-Key` whenever `CONJURER_API_KEY` is set. The bot sends it via
`service_headers()`.
- Anyone with the URL can download the file until it expires — the token is the
only credential.
- Mount the media read-only, as the compose file does. This service never needs
to write to the library.
## Verifying
```bash
# index built?
docker compose -f docker/compose.share.yaml exec conjurer-share \
python3 -c "import json;d=json.load(open('/srv/share/db/share_scan.json'));print(len(d['entries']),'entries')"
# a link resolves? (create one from Discord first)
curl -sI https://czernobog.pl/share/<token> | head -1
# the listing must NOT work, and neither must the state file
curl -sI https://czernobog.pl/share/ | head -1 # expect 403
curl -sI https://czernobog.pl/share/.downloads.json | head -1 # expect 403
# revoker seeing downloads?
docker compose -f docker/compose.share.yaml logs --tail=20 conjurer-share
```
Common failures:
| Symptom | Cause |
|---|---|
| Search finds nothing | index missing/empty — check the scanner ran and `CONJURER_SHARE_SCAN_PATH` is right |
| Every link 404s | media not mounted at the path the index recorded |
| Every link 403s | `FollowSymLinks` missing on the share directory |
| Links never expire | revoker not running, or reading a log the vhost isn't writing |
| Bot gets 401 | `CONJURER_API_KEY` set on the musician but not on the bot |
## Known limitations
- **A link nobody downloads is never revoked.** The TTL starts at the first
download, so unused symlinks accumulate in the share directory forever. Prune
it periodically if that matters.
- **Log rotation confuses the revoker.** It stores a byte offset into the access
log; when the file is rotated the offset points into the new, shorter file and
entries can be skipped. Prefer a long rotation interval, or clear
`.logpos` after rotating.
- **The index is a full rescan**, not incremental — cost grows with library size.