conjurer/docs/deployment/FILE_SHARING.md

File sharing (short-lived share links)

Publishes a file from the media library behind a random, unguessable URL that stops working shortly after someone downloads it. Driven from Discord.

Historically only the Python half of this lived in the repo; the Apache config and the cron entries were placed on the host by hand. This document plus docker/Dockerfile.share close that gap.

What it actually does

1. A scanner walks the media library and writes a JSON index of every path.
2. From Discord you search that index and pick files.
3. Publishing creates a symlink named after a random token (uuid.uuid4().hex, 32 hex chars) inside the share directory, pointing at the real file. Nothing is copied.
4. Apache serves that symlink at https://<host>/share/<token>.
5. A revoker tails Apache's access log, records when each token was first downloaded, and unlinks it once the TTL expires.

Two things worth being precise about, because the names suggest otherwise:

• The token is random, not a hash of the file. Nothing about the content is recoverable from the URL, and the same file shared twice gets two URLs.
• The "database" is a JSON file, not a SQL database. (utils/import_to_mysql.py is an unrelated, unfinished scratch script.)

The pieces

Path	Role	Runs in
conjurer_musician/scan_shares.py	Builds the JSON index from the media root	share container (or cron)
conjurer_musician/revoke_shares.py	Tails the access log, unlinks expired tokens	share container (or cron)
conjurer_musician/media_search_functions.py	Search + publish helpers used by the service	musician
conjurer_musician.py → /get_share_list, /get_share_links	HTTP API, both behind X-Conjurer-Api-Key	musician
file_search_functions.py	Bot-side HTTP client	bot
file_search_commands.py	Discord cog: search UI + Accept/Cancel	bot
docker/share-vhost.conf.tpl	Apache vhost (symlinks, no indexes, dedicated log)	share container
conjurer_musician/search_and_share.py	Standalone CLI doing steps 1–3 by hand	anywhere (optional)

Flow:

   media library ──► scan_shares.py ──► share_scan.json ──┐
                                                          │ (reads)
Discord ──► bot cog ──► musician /get_share_list ─────────┘
                    └─► musician /get_share_links ──► symlink in share dir
                                                          │ (serves)
                        Apache vhost ─────────────────────┘
                             │ (logs)
                             └──► share_access.log ──► revoke_shares.py ──► unlink

The share directory and the index are shared state between the musician and the share service: the musician creates links and reads the index, the share service serves links and writes the index. Both must see the same two paths.

Enabling it — Docker (recommended)

cp docker/env/share.env.example docker/env/share.env
# edit: CONJURER_SHARE_SERVER_NAME, and the TTL/intervals if you want
docker compose -f docker/compose.share.yaml up -d --build

Host layout the compose file expects:

Host path	Contents
/srv/share/media	the media library (mounted read-only)
/srv/share/links	the published symlinks — also mount into the musician
/srv/share/db	share_scan.json — also mount into the musician
/srv/share/logs	Apache logs, incl. share_access.log

Then point the musician at the same two paths. In docker/env/musician.env:

CONJURER_SHARE_DB=/srv/share/db/share_scan.json
CONJURER_SHARE_DIR=/var/www/html/share
CONJURER_SHARE_BASE_URL=https://czernobog.pl/share

and add the matching mounts to docker/compose.musician.yaml:

      - /srv/share/db:/srv/share/db
      - /srv/share/links:/var/www/html/share

Finally forward /share from your public TLS endpoint to the container's port (8081 by default). The container speaks plain HTTP on purpose — TLS for czernobog.pl stays on the reverse proxy you already run.

The media library must be mounted at the same absolute path the index recorded (/mnt/shares by default). The symlinks are absolute, so if the scanner saw /mnt/shares/x.mkv and Apache has the library somewhere else, every link 404s.

The container does not use cron: the entrypoint runs the scanner and the revoker in sleep loops next to Apache, so their output lands in docker logs.

Enabling it — existing Apache on the host

If you keep serving from the host's Apache instead:

1. Install the vhost. Take docker/share-vhost.conf.tpl, replace __SERVER_NAME__, and drop it into /etc/apache2/sites-available/, then a2ensite + systemctl reload apache2. The three settings that matter are explained in the Security section below — do not simplify them away.

2. Schedule both jobs. The scripts default to the original bare-metal paths, so plain cron entries work with no environment at all:

   */5 *  * * *  /usr/bin/python3 /opt/conjurer/conjurer_musician/revoke_shares.py
   17  *  * * *  /usr/bin/python3 /opt/conjurer/conjurer_musician/scan_shares.py
   

   Defaults: media root /mnt/shares, index /var/log/share_scan.json, links /var/www/html/share, log /var/log/apache2/share_access.log, TTL 3600s. Override any of them with the CONJURER_SHARE_* variables from docker/env/share.env.example.

3. Point the musician at the same paths via CONJURER_SHARE_DB, CONJURER_SHARE_DIR and CONJURER_SHARE_BASE_URL.

Using it

$tajna_biblioteka_inkwizycji <1-10> <keywords>

Restricted to the Jarl and Thane roles. It replies with the matches, you select from the dropdown and press Accept; the bot answers with one URL per file. (The docstring in the cog still says !findfiles — that name is stale.)

Security

The vhost is not boilerplate; three of its settings are load-bearing:

• Options +FollowSymLinks — the shares are symlinks, so without it every link 403s. SymLinksIfOwnerMatch cannot be substituted: the targets are owned by whoever owns the media, not by www-data.
• Options -Indexes — a directory listing of the share dir would hand out every currently live token in one request.
• Deny dotfiles — revoke_shares.py keeps .downloads.json (a map of every live token to its download time) and .logpos inside the served directory. Serving .downloads.json would leak the entire active link set.

Also worth knowing:

• Both HTTP endpoints call _authorize_request(), so they require X-Conjurer-Api-Key whenever CONJURER_API_KEY is set. The bot sends it via service_headers().
• Anyone with the URL can download the file until it expires — the token is the only credential.
• Mount the media read-only, as the compose file does. This service never needs to write to the library.

Verifying

# index built?
docker compose -f docker/compose.share.yaml exec conjurer-share \
    python3 -c "import json;d=json.load(open('/srv/share/db/share_scan.json'));print(len(d['entries']),'entries')"

# a link resolves? (create one from Discord first)
curl -sI https://czernobog.pl/share/<token> | head -1

# the listing must NOT work, and neither must the state file
curl -sI https://czernobog.pl/share/            | head -1   # expect 403
curl -sI https://czernobog.pl/share/.downloads.json | head -1   # expect 403

# revoker seeing downloads?
docker compose -f docker/compose.share.yaml logs --tail=20 conjurer-share

Common failures:

Symptom	Cause
Search finds nothing	index missing/empty — check the scanner ran and CONJURER_SHARE_SCAN_PATH is right
Every link 404s	media not mounted at the path the index recorded
Every link 403s	FollowSymLinks missing on the share directory
Links never expire	revoker not running, or reading a log the vhost isn't writing
Bot gets 401	CONJURER_API_KEY set on the musician but not on the bot

Known limitations

• A link nobody downloads is never revoked. The TTL starts at the first download, so unused symlinks accumulate in the share directory forever. Prune it periodically if that matters.
• Log rotation confuses the revoker. It stores a byte offset into the access log; when the file is rotated the offset points into the new, shorter file and entries can be skipped. Prefer a long rotation interval, or clear .logpos after rotating.
• The index is a full rescan, not incremental — cost grows with library size.