- the API + admin UI server and the public web server get ReadTimeout 30s, WriteTimeout 60s and IdleTimeout 120s next to ReadHeaderTimeout - the archive import extends its own read and write deadlines to 30 minutes with http.ResponseController, so a big mbox is not cut - tests: a stalled body and an idle connection are cut on both servers, an import longer than the default timeouts succeeds, fields are pinned - README: document the limits and timeouts
289 lines
16 KiB
Markdown
289 lines
16 KiB
Markdown
# parsimony
|
|
|
|
A modern, single-binary mailing list manager: an evolution of Mailman.
|
|
|
|
- Postfix delivers to parsimony over LMTP; parsimony relays through SMTP.
|
|
- Discussion lists with member-only posting, `List-*` headers, one-click unsubscribe
|
|
and DMARC-safe `From` rewriting.
|
|
- DKIM signing of outgoing mail with one key per list domain, so DMARC holds after the From rewrite.
|
|
- Durable outbound queue: a post is accepted once it is stored, one delivery per recipient,
|
|
and a background worker sends it with retries (1m, 5m, 15m, 1h, 3h, then every 6h, for up to
|
|
3 days). Relay outages and restarts lose nothing.
|
|
- Self-service subscribe and unsubscribe by email with double opt-in:
|
|
send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation.
|
|
- Moderated lists: posts wait in a queue that the API can approve or discard.
|
|
- VERP bounces: every recipient gets a signed return address; 3 hard bounces disable a member.
|
|
- Archive with full-text search (SQLite FTS5) and threading. Each list has an
|
|
`archive_policy`: `private` (default, API only), `public` (web pages) or `off`.
|
|
Public pages hide email addresses.
|
|
- Per-recipient one-click unsubscribe links (RFC 8058) with signed tokens.
|
|
- SQLite storage, one YAML config file.
|
|
- Command-line client for administration and scripts (`parsimony list|member|held|archive|queue`).
|
|
- Admin REST API under `/api/v1` (see `docs/openapi.yaml`).
|
|
- Admin web UI (Vite, React, shadcn/ui) served by the API listener at `/`: lists, settings,
|
|
members, held queue and archive.
|
|
|
|
## Run
|
|
|
|
```
|
|
cp parsimony.example.yaml parsimony.yaml
|
|
PARSIMONY_API_TOKEN=<token> go run ./cmd/parsimony serve -config parsimony.yaml
|
|
```
|
|
|
|
The API is disabled when `PARSIMONY_API_TOKEN` is not set.
|
|
|
|
Three listeners, each configurable:
|
|
|
|
| Listener | Default | Purpose |
|
|
|----------|---------|---------|
|
|
| `lmtp` | `127.0.0.1:2424` | mail from Postfix; **no authentication**, see Security notes |
|
|
| `api` | `127.0.0.1:8080` | admin REST API, bearer token |
|
|
| `web` | `127.0.0.1:8081` | public archive and unsubscribe pages (`""` disables) |
|
|
|
|
Put the `web` listener behind a reverse proxy with TLS and set `base_url` to its public URL:
|
|
the unsubscribe links in every message use it. Keep the `api` listener private.
|
|
|
|
```
|
|
curl -H "Authorization: Bearer $PARSIMONY_API_TOKEN" -d '{"name":"dev","domain":"example.org","title":"Dev"}' localhost:8080/api/v1/lists
|
|
```
|
|
|
|
## Command line
|
|
|
|
The same binary is an administration client. It talks to the REST API, so it works against a local or remote daemon.
|
|
|
|
```
|
|
export PARSIMONY_API_TOKEN=... # the token is never a flag
|
|
export PARSIMONY_URL=http://127.0.0.1:8080 # default; --url overrides
|
|
|
|
parsimony list add dev@example.org --title "Developers"
|
|
parsimony list set dev@example.org --post moderated --archive public
|
|
parsimony member add dev@example.org alice@example.com --name Alice
|
|
parsimony member import dev@example.org subscribers.txt # "email[,name]" per line, "-" reads stdin
|
|
parsimony held ls dev@example.org
|
|
parsimony held approve 12
|
|
parsimony archive ls dev@example.org --q "deploy"
|
|
parsimony archive import dev@example.org dev.mbox # an mbox file or "-" for stdin
|
|
parsimony queue ls
|
|
parsimony --json list ls # JSON for scripts
|
|
parsimony help
|
|
```
|
|
|
|
`<list>` is an address or a numeric id. Destructive commands (`list rm`, `archive rm`) need `--yes`.
|
|
Exit status: 0 success, 1 failure (including lines that failed in `member import`), 2 usage error.
|
|
`parsimony serve -config FILE` runs the daemon; the old `parsimony -config FILE` form still works.
|
|
In the container: `docker exec parsimony /parsimony list ls`.
|
|
|
|
## Migrating from Mailman
|
|
|
|
Both Mailman 2 and Mailman 3 (HyperKitty) can export a list's history as an mbox file.
|
|
|
|
```
|
|
parsimony list add dev@example.org --title "Developers"
|
|
parsimony member import dev@example.org subscribers.txt # "email[,name]" per line
|
|
parsimony archive import dev@example.org dev.mbox
|
|
```
|
|
|
|
The import reads standard mbox (a message starts at a `From addr Tue Jan 2 15:04:05 2024` line, `>From ` escapes are
|
|
restored), decodes encoded headers, charsets and multipart bodies, and stores the plain text. It is safe to repeat: a
|
|
message already in the archive (same `Message-ID`) is skipped. Threads are rebuilt from `In-Reply-To` whatever the order
|
|
of the messages. Messages that cannot be read are listed and counted, and the exit status is 1; the others are kept.
|
|
It works whatever the list's archive policy is (the policy only governs live mail). The admin UI has the same import on
|
|
the Archive tab. Search is accent-insensitive: `caffe` finds "Caffè".
|
|
|
|
## Outgoing mail: DKIM, SPF and DMARC
|
|
|
|
Parsimony rewrites `From` to the list address, so a signature of the original author's domain is of no use and
|
|
receivers that enforce DMARC need a signature of the **list domain**.
|
|
|
|
```
|
|
parsimony dkim keygen lists.example.org --selector mail --out /etc/parsimony/dkim-lists.pem
|
|
```
|
|
|
|
prints the DNS TXT record to publish and the config to add (the private key is written with mode 0600 and the
|
|
command never overwrites an existing key without `--force`):
|
|
|
|
```yaml
|
|
dkim:
|
|
- domain: lists.example.org
|
|
selector: mail
|
|
key_file: /etc/parsimony/dkim-lists.pem
|
|
```
|
|
|
|
Once the record is in DNS, `parsimony dkim check lists.example.org --key FILE --selector mail` looks it up and compares it
|
|
with the key. Both commands work offline of the API and need no token.
|
|
|
|
- A message is signed with the key of the domain in its From address (exact match, else a parent domain), at send time, so
|
|
the per-recipient `List-Unsubscribe` headers are covered. Relaxed/relaxed, SHA-256, no body length limit.
|
|
Signed: From, To, Cc, Subject, Date, Message-ID, In-Reply-To, References, Reply-To, MIME-Version, Content-Type,
|
|
Content-Transfer-Encoding and the `List-*` headers. A relay may add `Received` or rewrite other headers.
|
|
- A list on a domain without a key is sent unsigned. A broken key file stops the daemon at startup. If signing fails at
|
|
send time the error is logged and the message goes out unsigned: not delivering is worse than a missed signature.
|
|
- RSA (2048 bits recommended; PKCS#1 or PKCS#8 PEM) and Ed25519 keys are supported. Some receivers do not verify Ed25519 yet.
|
|
- Also publish SPF for the machine that relays your mail (for example `v=spf1 ip4:203.0.113.10 -all` on the list domain)
|
|
and a DMARC record (start with `_dmarc.lists.example.org TXT "v=DMARC1; p=none; rua=mailto:postmaster@example.org"`,
|
|
then tighten the policy once the reports look clean).
|
|
|
|
## Docker
|
|
|
|
The image runs as an unprivileged user (uid 65532) and works with a read-only root filesystem.
|
|
Config is mounted read-only; the only writable path is the `/data` volume (the database).
|
|
|
|
```
|
|
docker build -t parsimony .
|
|
docker run -d --name parsimony --read-only --cap-drop ALL \
|
|
-v parsimony-data:/data \
|
|
-v $PWD/parsimony.yaml:/etc/parsimony/parsimony.yaml:ro \
|
|
-e PARSIMONY_API_TOKEN=... -p 127.0.0.1:8080:8080 -p 8081:8081 parsimony
|
|
```
|
|
|
|
Use `db: /data/parsimony.db` in the config. Do not publish the LMTP port (2424): Postfix reaches it over a Docker network
|
|
shared with the container, and `lmtp.allow` should list that network's range (see Security notes). A new named volume gets the right ownership by itself;
|
|
for a bind mount, make the host directory writable by uid 65532.
|
|
|
|
## Sending through an external relay
|
|
|
|
By default parsimony hands mail to an SMTP server on the same machine (`relay.addr: 127.0.0.1:25`). It can also send through
|
|
an external relay such as Amazon SES, Mailgun or a company smarthost:
|
|
|
|
```yaml
|
|
relay:
|
|
addr: email-smtp.eu-west-1.amazonaws.com:587
|
|
tls: starttls # none (default), starttls or tls (port 465)
|
|
username: AKIAEXAMPLE
|
|
password_file: /run/secrets/relay-password # the password is never in the config
|
|
helo: lists.example.org # EHLO name; this machine's host name by default
|
|
# ca_file: /etc/parsimony/private-ca.pem # to trust a private CA
|
|
# server_name: smtp.example.com # when the certificate is for another name than the host in addr
|
|
```
|
|
|
|
- **TLS is required, never silently skipped.** With `starttls` a relay that does not offer it makes the send fail; the
|
|
certificate is always verified (TLS 1.2 or later; system roots, or `ca_file`).
|
|
- **Credentials only travel encrypted.** SMTP AUTH PLAIN is refused over a clear connection to a remote relay, at load
|
|
time and again at send time. A local relay (127.0.0.1, `::1`, `localhost`) may be used without TLS.
|
|
- **A wrong password does not lose mail.** A refused login (or a relay that wants authentication) is treated as a temporary
|
|
error: the deliveries stay queued, retry with backoff, and show the reason in the queue (without the password). Fix the
|
|
password and restart; the mail goes out. A missing or empty password file, or an unreadable CA file, stops the daemon at
|
|
startup.
|
|
- With `starttls`, the very first EHLO before the upgrade says `localhost` (the SMTP library offers no way to set it
|
|
earlier); the configured `helo` is used right after, over TLS.
|
|
- Keep SPF and DKIM right for the path your mail takes (for SES, `include:amazonses.com` in SPF), and check that the provider
|
|
accepts the list's per-recipient return addresses (`list+bounce-ID-MAC@domain`) as envelope sender. Bounces come back
|
|
to those addresses, so they reach parsimony only if the domain's MX delivers them to its LMTP.
|
|
|
|
## Delivery and the queue
|
|
|
|
LMTP answers `250` when the post is in the queue, so a down relay never bounces mail back to Postfix.
|
|
It answers `451` only when the database itself fails.
|
|
|
|
- Temporary relay errors (4xx, timeouts, connection failures) are retried with backoff.
|
|
- A permanent rejection of the message by the relay (5xx: policy, size, line length...) is never retried, but the
|
|
delivery is not made to vanish: it is kept as **failed**, with the reason, for a week (queue API, `parsimony queue ls`,
|
|
the Queue page; discard it to clear it). The same happens when a delivery is still failing after 3 days.
|
|
- A reply to `RCPT TO` that says the address does not exist (`5.1.1`, `5.1.2`, `5.1.3`, `5.1.6`, `5.1.10`) drops the
|
|
delivery and counts as a hard bounce for the member (3 disable it). Nothing else counts: not "relay access denied", not a
|
|
refused sender (`5.1.7`, `5.1.8`, or any code answering `MAIL FROM`), not a refusal after `DATA`. Those deliveries are kept
|
|
as failed.
|
|
- A relay that accepted the message is never asked again because `QUIT` went wrong.
|
|
- Delivery is at-least-once: a crash between the relay accepting a message and the queue entry being
|
|
removed can send one message to one recipient twice.
|
|
- `GET /api/v1/queue` and the Queue page show what is pending or failing, and a stuck delivery can be discarded.
|
|
|
|
## Admin UI
|
|
|
|
The UI is a single-page app in `admin/`, built into `internal/adminui/dist` and embedded in the binary.
|
|
It is served on the API listener together with `/api`, so it needs no CORS, and only when
|
|
`PARSIMONY_API_TOKEN` is set. You sign in with that token (kept in `sessionStorage`, gone when the tab closes).
|
|
|
|
```
|
|
make admin # npm ci && npm run build
|
|
make build # go build, embeds the UI
|
|
make test # go test + vitest
|
|
make ci # what CI runs: gofmt, vet, race tests, frontend tests and build
|
|
make e2e # end-to-end with a real Postfix in Docker (about 2 minutes, mostly the image build)
|
|
```
|
|
|
|
Without `make admin` the binary still builds and the UI answers 503 with a hint.
|
|
For frontend work run the daemon, then `cd admin && npm run dev` (Vite proxies `/api` to `127.0.0.1:8080`).
|
|
|
|
## Continuous integration
|
|
|
|
`.forgejo/workflows/ci.yaml` runs on Forgejo Actions (Codeberg) for pushes to `main` and for pull requests:
|
|
a Go job (`make go-ci`: gofmt, `go vet`, tests with the race detector) and an admin UI job (`make admin-ci`:
|
|
`npm ci`, tests, type check, build). Enable Actions in the repository settings and provide a runner with the
|
|
`docker` label (change `runs-on` if yours is named differently). `make ci` does the same locally.
|
|
`.forgejo/workflows/e2e.yaml` runs the Postfix end-to-end test on demand, because it needs a runner that can run Docker.
|
|
|
|
## End-to-end test
|
|
|
|
`make e2e` starts a real Postfix and parsimony in Docker and walks the whole path: Postfix hands list mail to
|
|
parsimony over LMTP, parsimony queues it and relays the copies back through Postfix, which delivers them to
|
|
maildirs. It checks the list headers and VERP return path, a real `550 5.1.1` from Postfix disabling a member after
|
|
three hard bounces, a real LMTP `550` for a non-member, join and leave by email, moderation, the public archive
|
|
and one-click unsubscribe. The files are in `e2e/` and behind the `e2e` build tag, so `go test ./...` stays fast.
|
|
|
|
## Postfix
|
|
|
|
Deliver the list domains to parsimony over LMTP and keep the `+` extensions intact, for example:
|
|
|
|
```
|
|
transport_maps = hash:/etc/postfix/transport
|
|
# /etc/postfix/transport
|
|
lists.example.org lmtp:inet:127.0.0.1:2424
|
|
```
|
|
|
|
## Security notes
|
|
|
|
- **LMTP has no authentication.** Whoever can connect to it can send mail as any member. Keep it on `127.0.0.1`, on a
|
|
private network, or on a unix socket, and list the clients that may connect:
|
|
|
|
```yaml
|
|
lmtp:
|
|
addr: 0.0.0.0:2424
|
|
allow: [172.16.0.0/12] # CIDR ranges or single addresses; other clients are dropped before the greeting
|
|
```
|
|
|
|
The daemon logs a warning at startup when LMTP is reachable beyond the machine and `lmtp.allow` is empty.
|
|
`allow` does not apply to unix sockets: use the permissions of the socket file.
|
|
- **Who is posting.** A post is accepted when the SMTP sender is a member *and* the `From` header names the same
|
|
address. The SMTP sender can be forged by anyone who can send mail to your MX, and the archive, the rewritten `From`
|
|
and `Reply-To` come from the header; so a post whose header names someone else (or has no usable `From`, or several
|
|
addresses) is held for moderation instead of delivered, with the reason shown to the admin (API, CLI, UI). Let Postfix
|
|
reject forged senders too (SPF, DKIM, DMARC checks on inbound mail).
|
|
- **The moderation queue is bounded.** A list holds at most 500 messages and 200 MiB, and one held message at most 10 MiB; a
|
|
full queue answers `452` (the sender retries later), a message over 10 MiB `552`. Messages nobody moderated for 30 days
|
|
are dropped. Listings (API, CLI, UI) show a summary and never read the bodies. The held view shows the From header
|
|
readers would see next to the real sender and flags a difference.
|
|
- **Hostile messages and slow clients.** The archive reads at most 10 levels of nested multipart and 100 parts per
|
|
message, then uses what it found (the post is still delivered). A `Date` header more than 24 hours ahead is replaced
|
|
by the arrival time, and one before 1990 is ignored, so a message cannot pin itself to the top of the archive. The
|
|
HTTP servers (API + admin UI, and the public web) cut a request that takes more than 30 s to read, a response that
|
|
takes more than 60 s to write, and a connection idle for 120 s; the archive import is the exception and may stream
|
|
for up to 30 minutes.
|
|
- **API token.** `parsimony serve` refuses a token shorter than 24 characters or containing whitespace
|
|
(`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is
|
|
locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy
|
|
every client shares the proxy's address, so rate-limit at the proxy as well.
|
|
- The API and the admin UI use one bearer token and no TLS: put them behind a reverse proxy with TLS and keep the `api`
|
|
listener private.
|
|
|
|
## Scale
|
|
|
|
Measured on one machine with `make scale` (SQLite file on a local disk, relay on localhost; your numbers will differ):
|
|
|
|
| | |
|
|
|---|---|
|
|
| Queue a post for a 10 001-member list (one transaction) | 0.12 s |
|
|
| Send it through a local relay, 4 deliveries at a time | about 460 messages/s (22 s for the whole list) |
|
|
| Import a 50 000-message mbox (14 MiB) | 2.9 s, about 17 000 messages/s |
|
|
| Search a 50 000-message archive (one page of 25) | 6 to 15 ms |
|
|
| Open page 1 000 of the archive, read a thread | about 4 ms, about 9 ms |
|
|
| Subscribe members one at a time (`member add`, no batching) | about 1.9 ms each at the store, plus one HTTP request |
|
|
|
|
Sending is bounded by the SMTP round trips to your relay: raise `relay.concurrency` for big lists.
|
|
|
|
## Licence
|
|
|
|
Parsimony is free software, released under the GNU General Public License, version 3 or (at your option) any later
|
|
version. See [LICENSE](LICENSE). The components in `admin/src/components/ui` come from shadcn/ui (MIT); the Go
|
|
dependencies (go-smtp, go-message, go-msgauth, modernc.org/sqlite, yaml.v3) use MIT, BSD or Apache 2.0 licences, which are
|
|
compatible with the GPL.
|