mirror of
https://github.com/migatu/conjurer.git
synced 2026-07-21 09:12:09 +00:00
share: dockerize the file-share pipeline and document it
The Python half of the short-lived share links lived in the repo; the Apache config and the cron entries that make it work were hand-placed on the host, so the feature could not be rebuilt from a checkout. This adds the missing half. Scripts (defaults unchanged, so existing bare-metal cron keeps working): * scan_shares.py / revoke_shares.py take their paths from CONJURER_SHARE_* instead of hardcoding the Pi layout, and create their parent dirs; * the revoke TTL is now CONJURER_SHARE_TTL_SECONDS. Its --help claimed "2min" while the code used a hardcoded 3600 - the help text now reports the real, configured value. New share service: * docker/Dockerfile.share - Apache + the two jobs, reusing the same scripts rather than forking copies; * docker/share-vhost.conf.tpl - the previously undocumented Apache half. Three settings are load-bearing and commented as such: +FollowSymLinks (the shares ARE symlinks), -Indexes (a listing would expose every live token), and a deny rule for dotfiles (revoke_shares.py keeps .downloads.json - a map of every live token - inside the served directory); * docker/entrypoint.share.sh - renders the vhost, seeds the index on first run, then runs scanner/revoker in sleep loops beside Apache (no cron, so their output shows up in docker logs); * compose + env example, incl. the two volumes that MUST be shared with the musician (it creates the links and reads the index). docs/deployment/FILE_SHARING.md documents the mechanism, both deployment routes (docker and existing host Apache), the Discord command, why each Apache setting matters, verification commands, and the known limitations - notably that a link nobody ever downloads is never revoked, since the TTL starts at first download. Verified: scanner and revoker exercised end-to-end against temp dirs (index built; token recorded from a combined-format log line and the symlink unlinked). Compose file not validated - no docker on this machine. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Michal Tuszowski
@@ -0,0 +1,197 @@
|
||||
# File sharing (short-lived share links)
|
||||
|
||||
Publishes a file from the media library behind a random, unguessable URL that
|
||||
stops working shortly after someone downloads it. Driven from Discord.
|
||||
|
||||
Historically only the Python half of this lived in the repo; the Apache config
|
||||
and the cron entries were placed on the host by hand. This document plus
|
||||
`docker/Dockerfile.share` close that gap.
|
||||
|
||||
## What it actually does
|
||||
|
||||
1. A **scanner** walks the media library and writes a JSON index of every path.
|
||||
2. From Discord you search that index and pick files.
|
||||
3. **Publishing** creates a `symlink` named after a random token
|
||||
(`uuid.uuid4().hex`, 32 hex chars) inside the share directory, pointing at the
|
||||
real file. Nothing is copied.
|
||||
4. **Apache** serves that symlink at `https://<host>/share/<token>`.
|
||||
5. A **revoker** tails Apache's access log, records when each token was *first*
|
||||
downloaded, and unlinks it once the TTL expires.
|
||||
|
||||
Two things worth being precise about, because the names suggest otherwise:
|
||||
|
||||
- The token is **random, not a hash** of the file. Nothing about the content is
|
||||
recoverable from the URL, and the same file shared twice gets two URLs.
|
||||
- The "database" is a **JSON file**, not a SQL database.
|
||||
(`utils/import_to_mysql.py` is an unrelated, unfinished scratch script.)
|
||||
|
||||
## The pieces
|
||||
|
||||
| Path | Role | Runs in |
|
||||
|---|---|---|
|
||||
| `conjurer_musician/scan_shares.py` | Builds the JSON index from the media root | share container (or cron) |
|
||||
| `conjurer_musician/revoke_shares.py` | Tails the access log, unlinks expired tokens | share container (or cron) |
|
||||
| `conjurer_musician/media_search_functions.py` | Search + publish helpers used by the service | musician |
|
||||
| `conjurer_musician.py` → `/get_share_list`, `/get_share_links` | HTTP API, both behind `X-Conjurer-Api-Key` | musician |
|
||||
| `file_search_functions.py` | Bot-side HTTP client | bot |
|
||||
| `file_search_commands.py` | Discord cog: search UI + Accept/Cancel | bot |
|
||||
| `docker/share-vhost.conf.tpl` | Apache vhost (symlinks, no indexes, dedicated log) | share container |
|
||||
| `conjurer_musician/search_and_share.py` | Standalone CLI doing steps 1–3 by hand | anywhere (optional) |
|
||||
|
||||
Flow:
|
||||
|
||||
```
|
||||
media library ──► scan_shares.py ──► share_scan.json ──┐
|
||||
│ (reads)
|
||||
Discord ──► bot cog ──► musician /get_share_list ─────────┘
|
||||
└─► musician /get_share_links ──► symlink in share dir
|
||||
│ (serves)
|
||||
Apache vhost ─────────────────────┘
|
||||
│ (logs)
|
||||
└──► share_access.log ──► revoke_shares.py ──► unlink
|
||||
```
|
||||
|
||||
The share directory and the index are **shared state between the musician and
|
||||
the share service**: the musician creates links and reads the index, the share
|
||||
service serves links and writes the index. Both must see the same two paths.
|
||||
|
||||
## Enabling it — Docker (recommended)
|
||||
|
||||
```bash
|
||||
cp docker/env/share.env.example docker/env/share.env
|
||||
# edit: CONJURER_SHARE_SERVER_NAME, and the TTL/intervals if you want
|
||||
docker compose -f docker/compose.share.yaml up -d --build
|
||||
```
|
||||
|
||||
Host layout the compose file expects:
|
||||
|
||||
| Host path | Contents |
|
||||
|---|---|
|
||||
| `/srv/share/media` | the media library (mounted read-only) |
|
||||
| `/srv/share/links` | the published symlinks — **also mount into the musician** |
|
||||
| `/srv/share/db` | `share_scan.json` — **also mount into the musician** |
|
||||
| `/srv/share/logs` | Apache logs, incl. `share_access.log` |
|
||||
|
||||
Then point the musician at the same two paths. In `docker/env/musician.env`:
|
||||
|
||||
```ini
|
||||
CONJURER_SHARE_DB=/srv/share/db/share_scan.json
|
||||
CONJURER_SHARE_DIR=/var/www/html/share
|
||||
CONJURER_SHARE_BASE_URL=https://czernobog.pl/share
|
||||
```
|
||||
|
||||
and add the matching mounts to `docker/compose.musician.yaml`:
|
||||
|
||||
```yaml
|
||||
- /srv/share/db:/srv/share/db
|
||||
- /srv/share/links:/var/www/html/share
|
||||
```
|
||||
|
||||
Finally forward `/share` from your public TLS endpoint to the container's port
|
||||
(`8081` by default). The container speaks plain HTTP on purpose — TLS for
|
||||
`czernobog.pl` stays on the reverse proxy you already run.
|
||||
|
||||
> **The media library must be mounted at the same absolute path the index
|
||||
> recorded** (`/mnt/shares` by default). The symlinks are absolute, so if the
|
||||
> scanner saw `/mnt/shares/x.mkv` and Apache has the library somewhere else, every
|
||||
> link 404s.
|
||||
|
||||
The container does not use cron: the entrypoint runs the scanner and the revoker
|
||||
in sleep loops next to Apache, so their output lands in `docker logs`.
|
||||
|
||||
## Enabling it — existing Apache on the host
|
||||
|
||||
If you keep serving from the host's Apache instead:
|
||||
|
||||
1. Install the vhost. Take `docker/share-vhost.conf.tpl`, replace
|
||||
`__SERVER_NAME__`, and drop it into `/etc/apache2/sites-available/`, then
|
||||
`a2ensite` + `systemctl reload apache2`. The three settings that matter are
|
||||
explained in the Security section below — do not simplify them away.
|
||||
|
||||
2. Schedule both jobs. The scripts default to the original bare-metal paths, so
|
||||
plain cron entries work with no environment at all:
|
||||
|
||||
```cron
|
||||
*/5 * * * * /usr/bin/python3 /opt/conjurer/conjurer_musician/revoke_shares.py
|
||||
17 * * * * /usr/bin/python3 /opt/conjurer/conjurer_musician/scan_shares.py
|
||||
```
|
||||
|
||||
Defaults: media root `/mnt/shares`, index `/var/log/share_scan.json`, links
|
||||
`/var/www/html/share`, log `/var/log/apache2/share_access.log`, TTL 3600s.
|
||||
Override any of them with the `CONJURER_SHARE_*` variables from
|
||||
`docker/env/share.env.example`.
|
||||
|
||||
3. Point the musician at the same paths via `CONJURER_SHARE_DB`,
|
||||
`CONJURER_SHARE_DIR` and `CONJURER_SHARE_BASE_URL`.
|
||||
|
||||
## Using it
|
||||
|
||||
```
|
||||
$tajna_biblioteka_inkwizycji <1-10> <keywords>
|
||||
```
|
||||
|
||||
Restricted to the **Jarl** and **Thane** roles. It replies with the matches, you
|
||||
select from the dropdown and press Accept; the bot answers with one URL per file.
|
||||
(The docstring in the cog still says `!findfiles` — that name is stale.)
|
||||
|
||||
## Security
|
||||
|
||||
The vhost is not boilerplate; three of its settings are load-bearing:
|
||||
|
||||
- **`Options +FollowSymLinks`** — the shares *are* symlinks, so without it every
|
||||
link 403s. `SymLinksIfOwnerMatch` cannot be substituted: the targets are owned
|
||||
by whoever owns the media, not by `www-data`.
|
||||
- **`Options -Indexes`** — a directory listing of the share dir would hand out
|
||||
every currently live token in one request.
|
||||
- **Deny dotfiles** — `revoke_shares.py` keeps `.downloads.json` (a map of every
|
||||
live token to its download time) and `.logpos` *inside the served directory*.
|
||||
Serving `.downloads.json` would leak the entire active link set.
|
||||
|
||||
Also worth knowing:
|
||||
|
||||
- Both HTTP endpoints call `_authorize_request()`, so they require
|
||||
`X-Conjurer-Api-Key` whenever `CONJURER_API_KEY` is set. The bot sends it via
|
||||
`service_headers()`.
|
||||
- Anyone with the URL can download the file until it expires — the token is the
|
||||
only credential.
|
||||
- Mount the media read-only, as the compose file does. This service never needs
|
||||
to write to the library.
|
||||
|
||||
## Verifying
|
||||
|
||||
```bash
|
||||
# index built?
|
||||
docker compose -f docker/compose.share.yaml exec conjurer-share \
|
||||
python3 -c "import json;d=json.load(open('/srv/share/db/share_scan.json'));print(len(d['entries']),'entries')"
|
||||
|
||||
# a link resolves? (create one from Discord first)
|
||||
curl -sI https://czernobog.pl/share/<token> | head -1
|
||||
|
||||
# the listing must NOT work, and neither must the state file
|
||||
curl -sI https://czernobog.pl/share/ | head -1 # expect 403
|
||||
curl -sI https://czernobog.pl/share/.downloads.json | head -1 # expect 403
|
||||
|
||||
# revoker seeing downloads?
|
||||
docker compose -f docker/compose.share.yaml logs --tail=20 conjurer-share
|
||||
```
|
||||
|
||||
Common failures:
|
||||
|
||||
| Symptom | Cause |
|
||||
|---|---|
|
||||
| Search finds nothing | index missing/empty — check the scanner ran and `CONJURER_SHARE_SCAN_PATH` is right |
|
||||
| Every link 404s | media not mounted at the path the index recorded |
|
||||
| Every link 403s | `FollowSymLinks` missing on the share directory |
|
||||
| Links never expire | revoker not running, or reading a log the vhost isn't writing |
|
||||
| Bot gets 401 | `CONJURER_API_KEY` set on the musician but not on the bot |
|
||||
|
||||
## Known limitations
|
||||
|
||||
- **A link nobody downloads is never revoked.** The TTL starts at the first
|
||||
download, so unused symlinks accumulate in the share directory forever. Prune
|
||||
it periodically if that matters.
|
||||
- **Log rotation confuses the revoker.** It stores a byte offset into the access
|
||||
log; when the file is rotated the offset points into the new, shorter file and
|
||||
entries can be skipped. Prefer a long rotation interval, or clear
|
||||
`.logpos` after rotating.
|
||||
- **The index is a full rescan**, not incremental — cost grows with library size.
|
||||
Reference in New Issue
Block a user