conjurer/docs/deployment/FILE_SHARING.md

# File sharing (short-lived share links)

Publishes a file from the media library behind a random, unguessable URL that
stops working shortly after someone downloads it. Driven from Discord.

Historically only the Python half of this lived in the repo; the Apache config
and the cron entries were placed on the host by hand. This document plus
`docker/Dockerfile.share` close that gap.

> Standing it up on a fresh node (directories, permissions, reverse proxy,
> verification)? Start with **[SHARE_NODE_SETUP.md](SHARE_NODE_SETUP.md)** — this
> document explains how the feature works and how it is secured.

## What it actually does

1. A **scanner** walks the media library and writes a JSON index of every path.
2. From Discord you search that index and pick files.
3. **Publishing** creates a `symlink` named after a random token
   (`uuid.uuid4().hex`, 32 hex chars) inside the share directory, pointing at the
   real file. Nothing is copied.
4. **Apache** serves that symlink at `https://<host>/share/<token>`.
5. A **revoker** tails Apache's access log, records when each token was *first*
   downloaded, and unlinks it once the TTL expires.

Two things worth being precise about, because the names suggest otherwise:

- The token is **random, not a hash** of the file. Nothing about the content is
  recoverable from the URL, and the same file shared twice gets two URLs.
- The "database" is a **JSON file**, not a SQL database.
  (`utils/import_to_mysql.py` is an unrelated, unfinished scratch script.)

## The pieces

| Path | Role | Runs in |
|---|---|---|
| `conjurer_musician/scan_shares.py` | Builds the JSON index from the media root | share container (or cron) |
| `conjurer_musician/revoke_shares.py` | Tails the access log, unlinks expired tokens | share container (or cron) |
| `conjurer_musician/media_search_functions.py` | Search + publish helpers used by the service | musician |
| `conjurer_musician.py` → `/get_share_list`, `/get_share_links` | HTTP API, both behind `X-Conjurer-Api-Key` | musician |
| `file_search_functions.py` | Bot-side HTTP client | bot |
| `file_search_commands.py` | Discord cog: search UI + Accept/Cancel | bot |
| `docker/share-vhost.conf.tpl` | Apache vhost (symlinks, no indexes, dedicated log) | share container |
| `conjurer_musician/search_and_share.py` | Standalone CLI doing steps 1–3 by hand | anywhere (optional) |

Flow:

```
   media library ──► scan_shares.py ──► share_scan.json ──┐
                                                          │ (reads)
Discord ──► bot cog ──► musician /get_share_list ─────────┘
                    └─► musician /get_share_links ──► symlink in share dir
                                                          │ (serves)
                        Apache vhost ─────────────────────┘
                             │ (logs)
                             └──► share_access.log ──► revoke_shares.py ──► unlink
```

The share directory and the index are **shared state between the musician and
the share service**: the musician creates links and reads the index, the share
service serves links and writes the index. Both must see the same two paths.

## Enabling it — Docker (recommended)

```bash
cp docker/env/share.env.example docker/env/share.env
# edit: CONJURER_SHARE_SERVER_NAME, and the TTL/intervals if you want
docker compose -f docker/compose.share.yaml up -d --build
```

Host layout the compose file expects:

| Host path | Contents |
|---|---|
| `/srv/share/media` | the media library (mounted read-only) |
| `/srv/share/links` | the published symlinks — **also mount into the musician** |
| `/srv/share/db` | `share_scan.json` — **also mount into the musician** |
| `/srv/share/logs` | Apache logs, incl. `share_access.log` |

Then point the musician at the same two paths. In `docker/env/musician.env`:

```ini
CONJURER_SHARE_DB=/srv/share/db/share_scan.json
CONJURER_SHARE_DIR=/var/www/html/share
CONJURER_SHARE_BASE_URL=https://czernobog.pl/share
```

and add the matching mounts to `docker/compose.musician.yaml`:

```yaml
      - /srv/share/db:/srv/share/db
      - /srv/share/links:/var/www/html/share
```

Finally forward `/share` from your public TLS endpoint to the container's port
(`8081` by default). The container speaks plain HTTP on purpose — TLS for
`czernobog.pl` stays on the reverse proxy you already run.

> **The media library must be mounted at the same absolute path the index
> recorded** (`/mnt/shares` by default). The symlinks are absolute, so if the
> scanner saw `/mnt/shares/x.mkv` and Apache has the library somewhere else, every
> link 404s.

The container does not use cron: the entrypoint runs the scanner and the revoker
in sleep loops next to Apache, so their output lands in `docker logs`.

## Enabling it — existing Apache on the host

If you keep serving from the host's Apache instead:

1. Install the vhost. Take `docker/share-vhost.conf.tpl`, replace
   `__SERVER_NAME__`, and drop it into `/etc/apache2/sites-available/`, then
   `a2ensite` + `systemctl reload apache2`. The three settings that matter are
   explained in the Security section below — do not simplify them away.

2. Schedule both jobs. The scripts default to the original bare-metal paths, so
   plain cron entries work with no environment at all:

   ```cron
   */5 *  * * *  /usr/bin/python3 /opt/conjurer/conjurer_musician/revoke_shares.py
   17  *  * * *  /usr/bin/python3 /opt/conjurer/conjurer_musician/scan_shares.py
   ```

   Defaults: media root `/mnt/shares`, index `/var/log/share_scan.json`, links
   `/var/www/html/share`, log `/var/log/apache2/share_access.log`, TTL 3600s.
   Override any of them with the `CONJURER_SHARE_*` variables from
   `docker/env/share.env.example`.

3. Point the musician at the same paths via `CONJURER_SHARE_DB`,
   `CONJURER_SHARE_DIR` and `CONJURER_SHARE_BASE_URL`.

## Using it

```
$tajna_biblioteka_inkwizycji <1-10> <keywords>
```

Restricted to the **Jarl** and **Thane** roles. It replies with the matches, you
select from the dropdown and press Accept; the bot answers with one URL per file.
(The docstring in the cog still says `!findfiles` — that name is stale.)

## Security

The vhost is not boilerplate; three of its settings are load-bearing:

- **`Options +FollowSymLinks`** — the shares *are* symlinks, so without it every
  link 403s. `SymLinksIfOwnerMatch` cannot be substituted: the targets are owned
  by whoever owns the media, not by `www-data`.
- **`Options -Indexes`** — a directory listing of the share dir would hand out
  every currently live token in one request.
- **Deny dotfiles** — `revoke_shares.py` keeps `.downloads.json` (a map of every
  live token to its download time) and `.logpos` *inside the served directory*.
  Serving `.downloads.json` would leak the entire active link set.

Also worth knowing:

- Both HTTP endpoints call `_authorize_request()`, so they require
  `X-Conjurer-Api-Key` whenever `CONJURER_API_KEY` is set. The bot sends it via
  `service_headers()`.
- Anyone with the URL can download the file until it expires — the token is the
  only credential.
- Mount the media read-only, as the compose file does. This service never needs
  to write to the library.

## Verifying

```bash
# index built?
docker compose -f docker/compose.share.yaml exec conjurer-share \
    python3 -c "import json;d=json.load(open('/srv/share/db/share_scan.json'));print(len(d['entries']),'entries')"

# a link resolves? (create one from Discord first)
curl -sI https://czernobog.pl/share/<token> | head -1

# the listing must NOT work, and neither must the state file
curl -sI https://czernobog.pl/share/            | head -1   # expect 403
curl -sI https://czernobog.pl/share/.downloads.json | head -1   # expect 403

# revoker seeing downloads?
docker compose -f docker/compose.share.yaml logs --tail=20 conjurer-share
```

Common failures:

| Symptom | Cause |
|---|---|
| Search finds nothing | index missing/empty — check the scanner ran and `CONJURER_SHARE_SCAN_PATH` is right |
| Every link 404s | media not mounted at the path the index recorded |
| Every link 403s | `FollowSymLinks` missing on the share directory |
| Links never expire | revoker not running, or reading a log the vhost isn't writing |
| Bot gets 401 | `CONJURER_API_KEY` set on the musician but not on the bot |

## Known limitations

- **A link nobody downloads is never revoked.** The TTL starts at the first
  download, so unused symlinks accumulate in the share directory forever. Prune
  it periodically if that matters.
- **Log rotation confuses the revoker.** It stores a byte offset into the access
  log; when the file is rotated the offset points into the new, shorter file and
  entries can be skipped. Prefer a long rotation interval, or clear
  `.logpos` after rotating.
- **The index is a full rescan**, not incremental — cost grows with library size.