parsimony/README.md

19 KiB

parsimony

A modern, single-binary mailing list manager: an evolution of Mailman.

  • Postfix delivers to parsimony over LMTP; parsimony relays through SMTP.
  • Discussion lists with member-only posting, List-* headers, one-click unsubscribe and DMARC-safe From rewriting.
  • DKIM signing of outgoing mail with one key per list domain, so DMARC holds after the From rewrite.
  • Durable outbound queue: a post is accepted once it is stored, one delivery per recipient, and a background worker sends it with retries (1m, 5m, 15m, 1h, 3h, then every 6h, for up to 3 days). Relay outages and restarts lose nothing.
  • Self-service subscribe and unsubscribe by email with double opt-in: send to list+join@domain or list+leave@domain, then reply to the confirmation. A reply works once, a request expires after 72 hours, and a confirmation that could not be applied (closed list, database error) can be replied to again. A member disabled by bounces can rejoin the same way.
  • Moderated lists: posts wait in a queue that the API can approve or discard.
  • VERP bounces: every recipient gets a signed return address; 3 hard bounces within 30 days disable a member.
  • Archive with full-text search (SQLite FTS5) and threading. Each list has an archive_policy: private (default, API only), public (web pages) or off. Public pages hide email addresses.
  • Per-recipient one-click unsubscribe links (RFC 8058) with signed tokens.
  • SQLite storage, one YAML config file.
  • Command-line client for administration and scripts (parsimony list|member|held|archive|queue).
  • Admin REST API under /api/v1 (see docs/openapi.yaml).
  • Admin web UI (Vite, React, shadcn/ui) served by the API listener at /: lists, settings, members, held queue and archive.

Run

cp parsimony.example.yaml parsimony.yaml
PARSIMONY_API_TOKEN=<token> go run ./cmd/parsimony serve -config parsimony.yaml

The API is disabled when PARSIMONY_API_TOKEN is not set.

Three listeners, each configurable:

Listener Default Purpose
lmtp 127.0.0.1:2424 mail from Postfix; no authentication, see Security notes
api 127.0.0.1:8080 admin REST API, bearer token
web 127.0.0.1:8081 public archive and unsubscribe pages ("" disables)

Put the web listener behind a reverse proxy with TLS and set base_url to its public URL: the unsubscribe links in every message use it. It must be an https:// URL (RFC 8058 one-click unsubscribe needs it; http:// is accepted only for localhost, 127.0.0.1 and ::1) with a host and no credentials, query or fragment; the daemon refuses to start otherwise. Leave it empty to turn the one-click links off. Keep the api listener private.

curl -H "Authorization: Bearer $PARSIMONY_API_TOKEN" -d '{"name":"dev","domain":"example.org","title":"Dev"}' localhost:8080/api/v1/lists

A list name is 1 to 30 characters of a-z, 0-9, ., _ and - (upper case is lowered, no dot at either end), the domain a host name, and the title one line of at most 200 characters without control characters; anything else is a 400. The title is encoded (RFC 2047) in the List-Id header when it is not ASCII.

Command line

The same binary is an administration client. It talks to the REST API, so it works against a local or remote daemon.

export PARSIMONY_API_TOKEN=...                  # the token is never a flag
export PARSIMONY_URL=http://127.0.0.1:8080      # default; --url overrides

parsimony list add dev@example.org --title "Developers"
parsimony list set dev@example.org --post moderated --archive public
parsimony member add dev@example.org alice@example.com --name Alice
parsimony member import dev@example.org subscribers.txt   # "email[,name]" per line, "-" reads stdin
parsimony held ls dev@example.org
parsimony held approve 12
parsimony archive ls dev@example.org --q "deploy"
parsimony archive import dev@example.org dev.mbox         # an mbox file or "-" for stdin
parsimony queue ls
parsimony --json list ls                         # JSON for scripts
parsimony help

<list> is an address or a numeric id. Destructive commands (list rm, archive rm) need --yes. Exit status: 0 success, 1 failure (including lines that failed in member import), 2 usage error. parsimony serve -config FILE runs the daemon; the old parsimony -config FILE form still works. In the container: docker exec parsimony /parsimony list ls.

Migrating from Mailman

Both Mailman 2 and Mailman 3 (HyperKitty) can export a list's history as an mbox file.

parsimony list add dev@example.org --title "Developers"
parsimony member import dev@example.org subscribers.txt   # "email[,name]" per line
parsimony archive import dev@example.org dev.mbox

The import reads standard mbox (a message starts at a From addr Tue Jan 2 15:04:05 2024 line, >From escapes are restored), decodes encoded headers, charsets and multipart bodies, and stores the plain text. It is safe to repeat: a message already in the archive (same Message-ID) is skipped. Threads are rebuilt from In-Reply-To whatever the order of the messages. Messages that cannot be read are listed and counted, and the exit status is 1; the others are kept. It works whatever the list's archive policy is (the policy only governs live mail). The admin UI has the same import on the Archive tab. Search is accent-insensitive: caffe finds "Caffè".

Outgoing mail: DKIM, SPF and DMARC

Parsimony rewrites From to the list address, so a signature of the original author's domain is of no use and receivers that enforce DMARC need a signature of the list domain.

parsimony dkim keygen lists.example.org --selector mail --out /etc/parsimony/dkim-lists.pem

prints the DNS TXT record to publish and the config to add (the private key is written with mode 0600 and the command never overwrites an existing key without --force):

dkim:
  - domain: lists.example.org
    selector: mail
    key_file: /etc/parsimony/dkim-lists.pem

Once the record is in DNS, parsimony dkim check lists.example.org --key FILE --selector mail looks it up and compares it with the key. Both commands work offline of the API and need no token.

  • A message is signed with the key of the domain in its From address (exact match, else a parent domain), at send time, so the per-recipient List-Unsubscribe headers are covered. Relaxed/relaxed, SHA-256, no body length limit. Signed: From, To, Cc, Subject, Date, Message-ID, In-Reply-To, References, Reply-To, MIME-Version, Content-Type, Content-Transfer-Encoding and the List-* headers. A relay may add Received or rewrite other headers.
  • A list on a domain without a key is sent unsigned. A broken key file stops the daemon at startup. If signing fails at send time the error is logged and the message goes out unsigned: not delivering is worse than a missed signature.
  • RSA (2048 bits recommended; PKCS#1 or PKCS#8 PEM) and Ed25519 keys are supported. Some receivers do not verify Ed25519 yet.
  • Also publish SPF for the machine that relays your mail (for example v=spf1 ip4:203.0.113.10 -all on the list domain) and a DMARC record (start with _dmarc.lists.example.org TXT "v=DMARC1; p=none; rua=mailto:postmaster@example.org", then tighten the policy once the reports look clean).

Docker

The image runs as an unprivileged user (uid 65532) and works with a read-only root filesystem. Config is mounted read-only; the only writable path is the /data volume (the database).

docker build -t parsimony .
docker run -d --name parsimony --read-only --cap-drop ALL \
  -v parsimony-data:/data \
  -v $PWD/parsimony.yaml:/etc/parsimony/parsimony.yaml:ro \
  -e PARSIMONY_API_TOKEN=... -p 127.0.0.1:8080:8080 -p 8081:8081 parsimony

Use db: /data/parsimony.db in the config. Do not publish the LMTP port (2424): Postfix reaches it over a Docker network shared with the container, and lmtp.allow should list that network's range (see Security notes). A new named volume gets the right ownership by itself; for a bind mount, make the host directory writable by uid 65532.

Sending through an external relay

By default parsimony hands mail to an SMTP server on the same machine (relay.addr: 127.0.0.1:25). It can also send through an external relay such as Amazon SES, Mailgun or a company smarthost:

relay:
  addr: email-smtp.eu-west-1.amazonaws.com:587
  tls: starttls                       # none (default), starttls or tls (port 465)
  username: AKIAEXAMPLE
  password_file: /run/secrets/relay-password   # the password is never in the config
  helo: lists.example.org             # EHLO name; this machine's host name by default
  # ca_file: /etc/parsimony/private-ca.pem     # to trust a private CA
  # server_name: smtp.example.com              # when the certificate is for another name than the host in addr
  • TLS is required, never silently skipped. With starttls a relay that does not offer it makes the send fail; the certificate is always verified (TLS 1.2 or later; system roots, or ca_file).
  • Credentials only travel encrypted. SMTP AUTH PLAIN is refused over a clear connection to a remote relay, at load time and again at send time. A local relay (127.0.0.1, ::1, localhost) may be used without TLS.
  • A wrong password does not lose mail. A refused login (or a relay that wants authentication) is treated as a temporary error: the deliveries stay queued, retry with backoff, and show the reason in the queue (without the password). Fix the password and restart; the mail goes out. A missing or empty password file, or an unreadable CA file, stops the daemon at startup.
  • With starttls, the very first EHLO before the upgrade says localhost (the SMTP library offers no way to set it earlier); the configured helo is used right after, over TLS.
  • Keep SPF and DKIM right for the path your mail takes (for SES, include:amazonses.com in SPF), and check that the provider accepts the list's per-recipient return addresses (list+bounce-ID-MAC@domain) as envelope sender. Bounces come back to those addresses, so they reach parsimony only if the domain's MX delivers them to its LMTP.

Delivery and the queue

LMTP answers 250 when the post is in the queue, so a down relay never bounces mail back to Postfix. It answers 451 only when the database itself fails.

  • Temporary relay errors (4xx, timeouts, connection failures) are retried with backoff.
  • A permanent rejection of the message by the relay (5xx: policy, size, line length...) is never retried, but the delivery is not made to vanish: it is kept as failed, with the reason, for a week (queue API, parsimony queue ls, the Queue page; discard it to clear it). The same happens when a delivery is still failing after 3 days.
  • A reply to RCPT TO that says the address does not exist (5.1.1, 5.1.2, 5.1.3, 5.1.6, 5.1.10) drops the delivery and counts as a hard bounce for the member (3 disable it). Nothing else counts: not "relay access denied", not a refused sender (5.1.7, 5.1.8, or any code answering MAIL FROM), not a refusal after DATA. Those deliveries are kept as failed.
  • A bounce that comes back later to the member's return address counts only if it is a real delivery status report (RFC 3464: multipart/report with a message/delivery-status part) with Action: failed and one of the same codes. A spam or policy rejection (5.7.x), a full mailbox (5.2.x), a delay, a 4.x.x, or a Status: line in a plain or quoted message does not count.
  • Hard bounces count for 30 days: when the last one is older, the next one starts again from 1. A disabled member stays disabled until it is added again.
  • A relay that accepted the message is never asked again because QUIT went wrong.
  • The same post is queued once. If the 250 for a post never reaches Postfix, Postfix delivers the post again; a list remembers each post it queued for an hour (by Message-ID, or by a hash of the whole message when it has none), answers 250 to a copy and does not queue it again (the log says so). An approved held post is always delivered and is remembered too, so a copy that arrives later is not.
  • On stop, the daemon refuses new connections and lets the LMTP sessions and API or web requests in flight finish, so a post being received gets its 250. After 30 seconds it cuts whatever is still open.
  • What remains is at-least-once delivery across the relay: a crash between the relay accepting a message and the queue entry being removed can send one message to one recipient twice. A copy of a post that arrives more than an hour later is also delivered again.
  • GET /api/v1/queue and the Queue page show what is pending or failing, and a stuck delivery can be discarded.

Admin UI

The UI is a single-page app in admin/, built into internal/adminui/dist and embedded in the binary. It is served on the API listener together with /api, so it needs no CORS, and only when PARSIMONY_API_TOKEN is set. You sign in with that token (kept in sessionStorage, gone when the tab closes).

make admin     # npm ci && npm run build
make build     # go build, embeds the UI
make test      # go test + vitest
make ci        # what CI runs: gofmt, vet, race tests, frontend tests and build
make e2e       # end-to-end with a real Postfix in Docker (about 2 minutes, mostly the image build)

Without make admin the binary still builds and the UI answers 503 with a hint. For frontend work run the daemon, then cd admin && npm run dev (Vite proxies /api to 127.0.0.1:8080).

Continuous integration

.forgejo/workflows/ci.yaml runs on Forgejo Actions (Codeberg) for pushes to main and for pull requests: a Go job (make go-ci: gofmt, go vet, tests with the race detector) and an admin UI job (make admin-ci: npm ci, tests, type check, build). Enable Actions in the repository settings and provide a runner with the docker label (change runs-on if yours is named differently). make ci does the same locally. .forgejo/workflows/e2e.yaml runs the Postfix end-to-end test on demand, because it needs a runner that can run Docker.

End-to-end test

make e2e starts a real Postfix and parsimony in Docker and walks the whole path: Postfix hands list mail to parsimony over LMTP, parsimony queues it and relays the copies back through Postfix, which delivers them to maildirs. It checks the list headers and VERP return path, a real 550 5.1.1 from Postfix disabling a member after three hard bounces, a real LMTP 550 for a non-member, join and leave by email, moderation, the public archive and one-click unsubscribe. The files are in e2e/ and behind the e2e build tag, so go test ./... stays fast.

Postfix

Deliver the list domains to parsimony over LMTP and keep the + extensions intact, for example:

transport_maps = hash:/etc/postfix/transport
# /etc/postfix/transport
lists.example.org  lmtp:inet:127.0.0.1:2424

Security notes

  • LMTP has no authentication. Whoever can connect to it can send mail as any member. Keep it on 127.0.0.1, on a private network, or on a unix socket, and list the clients that may connect:

    lmtp:
      addr: 0.0.0.0:2424
      allow: [172.16.0.0/12]   # CIDR ranges or single addresses; other clients are dropped before the greeting
    

    The daemon logs a warning at startup when LMTP is reachable beyond the machine and lmtp.allow is empty. allow does not apply to unix sockets: use the permissions of the socket file.

  • Who is posting. A post is accepted when the SMTP sender is a member and the From header names the same address. The SMTP sender can be forged by anyone who can send mail to your MX, and the archive, the rewritten From and Reply-To come from the header; so a post whose header names someone else (or has no usable From, or several addresses) is held for moderation instead of delivered, with the reason shown to the admin (API, CLI, UI). Let Postfix reject forged senders too (SPF, DKIM, DMARC checks on inbound mail).

  • Loops and robots. Mail that carries the list's own List-Id (a loop) or is a genuine auto-reply (Auto-Submitted: auto-replied, Precedence: auto_reply) is answered 250 and dropped without a trace, so the sender's server never bounces it. Other machine-generated mail (Auto-Submitted: auto-generated, cron, CI and monitoring posts) is a normal post: the membership, From and moderation checks apply. The join, leave and confirm addresses ignore (250, nothing happens, the token is kept) a mail with a null sender, any Auto-Submitted other than no, a Precedence of auto_reply, bulk or junk, or the list's own List-Id. This stops an auto-responder from confirming a subscription someone forged in your name.

  • The moderation queue is bounded. A list holds at most 500 messages and 200 MiB, and one held message at most 10 MiB; a full queue answers 452 (the sender retries later), a message over 10 MiB 552. Messages nobody moderated for 30 days are dropped. Listings (API, CLI, UI) show a summary and never read the bodies. The held view shows the From header readers would see next to the real sender and flags a difference.

  • Hostile messages and slow clients. The archive reads at most 10 levels of nested multipart and 100 parts per message, then uses what it found (the post is still delivered). A Date header more than 24 hours ahead is replaced by the arrival time, and one before 1990 is ignored, so a message cannot pin itself to the top of the archive. The HTTP servers (API + admin UI, and the public web) cut a request that takes more than 30 s to read, a response that takes more than 60 s to write, and a connection idle for 120 s; the archive import is the exception and may stream for up to 30 minutes.

  • API token. parsimony serve refuses a token shorter than 24 characters or containing whitespace (openssl rand -hex 24 makes a good one). After 10 wrong or missing tokens within 5 minutes an address is locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy every client shares the proxy's address, so rate-limit at the proxy as well.

  • The API and the admin UI use one bearer token and no TLS: put them behind a reverse proxy with TLS and keep the api listener private.

Scale

Measured on one machine with make scale (SQLite file on a local disk, relay on localhost; your numbers will differ):

Queue a post for a 10 001-member list (one transaction) 0.12 s
Send it through a local relay, 4 deliveries at a time about 460 messages/s (22 s for the whole list)
Import a 50 000-message mbox (14 MiB) 2.9 s, about 17 000 messages/s
Search a 50 000-message archive (one page of 25) 6 to 15 ms
Open page 1 000 of the archive, read a thread about 4 ms, about 9 ms
Subscribe members one at a time (member add, no batching) about 1.9 ms each at the store, plus one HTTP request

Sending is bounded by the SMTP round trips to your relay: raise relay.concurrency for big lists.

Licence

Parsimony is free software, released under the GNU General Public License, version 3 or (at your option) any later version. See LICENSE. The components in admin/src/components/ui come from shadcn/ui (MIT); the Go dependencies (go-smtp, go-message, go-msgauth, modernc.org/sqlite, yaml.v3) use MIT, BSD or Apache 2.0 licences, which are compatible with the GPL.