Commit graph

186 commits

Author SHA1 Message Date
tgiachi
a10616e216 ci: install Node in the golang jobs so the checkout action can run
- The golang image has no Node, and actions/checkout is a Node action
- The first run on my runner failed in the go and vuln jobs for this reason
2026-11-10 15:57:29 +01:00
tgiachi
1d68f9506b ci: describe the runner the workflows expect
- ci.yaml asks for a docker-labelled runner and runs jobs in the named containers
- e2e.yaml says what it needs from a runner and that it has not been run on one
2026-11-10 15:54:29 +01:00
squidd
1f74e375fc Merge pull request 'Fix the findings of the second review' (#43) from fix/second-review into main v0.1.0 2026-11-10 15:23:45 +01:00
tgiachi
c35ad47097 test: keep both store test additions after merging the review fixes 2026-11-10 15:21:43 +01:00
tgiachi
fa96a398ee docs: per-step delivery timeouts, trusted proxies and the confirmation cap
- Describe the phases of a delivery and the unknown-outcome retry
- Document web.trusted_proxies, the form's address rules and limits.pending_per_hour
2026-11-10 15:21:24 +01:00
tgiachi
c17ff1fed1 Merge branch 'fix/sr-send-timeout' into fix/second-review 2026-11-10 15:21:24 +01:00
tgiachi
af487b8c84 fix(queue): stop putting one deadline on the whole send
Worker.Timeout defaulted to one minute over the entire SMTP
transaction, cutting relays that are slow to accept a message and
causing duplicate deliveries. The default is now no total deadline:
SMTPRelay bounds each phase itself. A positive Timeout still applies.
2026-11-10 15:20:49 +01:00
tgiachi
f14faf52d5 fix(outbound): give each SMTP phase its own deadline
A relay can take minutes to answer the final dot (content filter,
large message, slow link). One deadline over the whole transaction
cut it after the message had been accepted, so the delivery was
retried and sent again.

The connection now moves its deadline as the transaction advances:
setup (dial, greeting, TLS, AUTH, MAIL, RCPT, DATA) has
SMTPRelay.Timeout, 60 s by default; the upload has 60 s plus the size
at 32 KiB/s, at most 10 minutes; the reply to the final dot has
10 minutes (RFC 5321 4.5.3.2.6). go-smtp sets and clears its own
per-command deadlines, so a wrapper clips them to the current phase.
Context cancellation still closes the connection at once. A timeout
in any phase is temporary; after the final dot the error says the
outcome is unknown.
2026-11-10 15:20:49 +01:00
tgiachi
a63807e71b fix(web): refuse a bare IP address as the domain of a typed address
a@127.0.0.1 has dots and digit labels, so it passed the domain check
and named the daemon's own host without brackets. No top-level domain
is all digits: a domain whose last label is all digits is refused.
2026-11-10 15:16:48 +01:00
tgiachi
d22423c02c refactor(store): drop a redundant zero check in the cap warning
now.Sub of the zero time is already longer than a minute, so the first
refusal is logged without a special case.
2026-11-10 15:16:02 +01:00
tgiachi
097314e8d9 test(web): check publicDomain directly
mail.ParseAddress already refuses empty labels such as x..com, so the
form-level tests could not see the empty-label check in publicDomain.
A direct table test pins the rule on its own.
2026-11-10 15:16:02 +01:00
tgiachi
8d28e48360 feat: configurable cap on confirmation requests, with a warning
When the global cap of unconfirmed requests (300 an hour) was reached,
a subscribe request silently got no mail while the visitor saw "Check
your email"; a newsletter launch can pass 300 signups an hour.

- limits.pending_per_hour (default 300, must be a whole number >= 1)
  is validated at load and set on the store by the app.
- When the global cap refuses a request the store logs a warning naming
  the setting, at most once a minute. The visitor still gets the
  neutral page, so nothing about members is revealed. The per-address
  cap is unchanged and is not logged.
- parsimony.example.yaml documents the setting.
2026-11-10 15:14:04 +01:00
tgiachi
96a1243c2f fix(web): refuse local addresses on the subscribe form
root@localhost, a@b and a@[127.0.0.1] passed mail.ParseAddress, so the
public form could make the daemon mail itself or its own network.

The domain of a typed address must now contain a dot, not be a domain
literal, be made of non-empty labels of letters, digits and hyphens, and
not be localhost or end in .localhost, .local, .internal or .invalid.
.test and .example stay accepted (the end-to-end tests use them).
2026-11-10 15:11:43 +01:00
tgiachi
44b3f9d98a fix(web): key the subscribe limit on the real client
Behind a reverse proxy every visitor shared the proxy's address, so the
whole site got 10 signups an hour and one person could block everyone.
Over IPv6 each address of a /64 was a fresh key, so the limit did not
bind and the map grew by one entry per address.

- web.trusted_proxies (addresses or CIDR ranges, default empty) is
  validated at load and passed to the web handler. X-Forwarded-For is
  read only when the TCP peer is a trusted proxy; the client is then the
  right-most entry that is not itself a trusted proxy. An empty or
  malformed header falls back to the peer. With no trusted proxy the
  behaviour is unchanged: the header is ignored.
- IPv6 clients are keyed by their /64, IPv4 clients by address.
- The limiter remembers at most 100000 clients. When full it sweeps
  expired entries first and otherwise answers 429 to new clients until
  the oldest window ends, instead of growing.
2026-11-10 15:11:26 +01:00
tgiachi
c66fd427b2 fix(store): add the outbox post_id and priority columns separately
The two columns were added in one non-transactional Exec guarded by
post_id alone. A crash between the two statements left a database
with post_id but no priority, and every later start failed at
CREATE INDEX outbox_due_prio. Each column is now checked and added on
its own, as the members columns are.
2026-11-10 15:07:11 +01:00
tgiachi
c6cbcb3d3b docs: say what the second review found about newsletters, priority and shutdown
- Warn that forged publisher senders are not checked by the daemon
- Describe the real limits of the delivery priority
- Document the Docker stop timeout and three known gaps
2026-11-10 15:06:35 +01:00
squidd
bc1ea2e7c3 Merge pull request 'End-to-end test for newsletters' (#41) from test/e2e-newsletter into main 2026-11-10 14:49:25 +01:00
tgiachi
2030b39270 test: end-to-end newsletter through a real Postfix
- A subscriber's post is refused with a real LMTP 550 and nothing is held or delivered
- The publisher's post reaches subscribers with list headers; the post stats add up
- The public subscribe form shows the neutral page and sends one confirmation mail
2026-11-10 14:48:59 +01:00
squidd
9a51b39eef Merge pull request 'Newsletter mode' (#39) from feat/newsletter into main 2026-11-10 13:08:16 +01:00
tgiachi
275b269977 docs: Newsletters section in the README
- Publishers policy, sent-post statistics, priority, rate limit and the public subscribe form
- State what is not included: scheduled sends and open or click tracking
2026-11-10 13:07:16 +01:00
tgiachi
c42ae9a276 test: share one SetPolicies-based helper for the newsletter priority tests
- Rename the priority helper to priorityEnv and set the publishers policy through SetPolicies instead of raw SQL
- Fixes the duplicate newsletterEnv after merging the publishers and posts branches
2026-11-10 13:02:58 +01:00
tgiachi
6a819aaeaa Merge branch 'feat/nl-web-signup' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
3d25f73d9b Merge branch 'feat/nl-rate-limit' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
34f461bb6b Merge branch 'feat/nl-posts-stats' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
c853ae7e61 test(queue): cover a pacer slot given up while a later caller holds its own
- a caller that gives up must not release a slot when another caller reserved after it
- a delivery that did not get its slot is not counted in the run's stats
2026-11-10 12:55:26 +01:00
tgiachi
47dc118b98 test: tighten the posts tests found weak by mutation
- ListPosts: a limit of 201 is capped to 200
- posts ls: distinct pending, failed and dropped counts, so a swapped
  column is caught
2026-11-10 12:47:17 +01:00
tgiachi
5ca091ccbf test(web): trimming, Retry-After rounding and the exact edges of the limit window
- the typed address is trimmed before it is used
- Retry-After rounds up to whole seconds
- the hour ends after exactly an hour, and the sweep forgets such an address
- found by mutation testing: these edges survived the first tests
2026-11-10 12:45:49 +01:00
tgiachi
ee1f837a20 feat(admin): a Sent tab with the posts of a list and their progress
- route /lists/:id/sent and a Sent tab on the list page
- a table of posts (subject, date, sent of recipients with a progress
  bar, recipients, pending, failed, dropped) and an empty state; a post
  with no recipients shows no bar
- refreshes every 5 seconds while any post has pending deliveries, and
  stops when none has
- api.posts.list, the Post type and the fake API route; tests
2026-11-10 12:43:55 +01:00
tgiachi
7e43117c74 feat(cli): parsimony posts ls
- posts ls <list> [--limit N] prints ID, DATE, SUBJECT, RECIPIENTS, SENT,
  PENDING, FAILED and DROPPED, newest first, "No posts yet." when empty,
  and the API's JSON with --json
- cells go through the same control-character escaping as the other
  tables; tests for the table, the limit, JSON, errors and escaping
2026-11-10 12:42:15 +01:00
tgiachi
10f593e9bd feat(admin): publishers in the Settings and Members tabs
- Settings: the Posting select offers Publishers only (newsletter)
- Members: a Can publish checkbox in the add form (unchecked sends no flag, so
  adding an existing publisher again keeps it), a Publisher badge in the table,
  and a Make publisher / Make subscriber row action
- api client: publisher on Member, the publishers post policy, setPublisher
- the fake API keeps the flag on re-add and answers the member PATCH
- tests for each
2026-11-10 12:41:44 +01:00
tgiachi
2f2ee71f92 feat(api): list the posts of a list with their delivery counters
- GET /api/v1/lists/{id}/posts?limit= returns id, message_id, subject,
  created_at, recipients, sent, failed, dropped and pending, newest
  first; 404 for an unknown list, 400 for a bad limit, token required
- discarding a queued delivery counts it as dropped (store.DropOutbox)
- openapi: the new path and the discard note
2026-11-10 12:40:55 +01:00
tgiachi
e850547b3d feat(cli): publishers in member add, ls, set and import
- member add --publisher makes a publisher, --publisher=false demotes; without
  the flag an existing member keeps its role
- member ls has a ROLE column: publisher or subscriber
- new member set <list> <email> --publisher=true|false (usage error without the flag)
- member import reads an optional third column, the keyword publisher (any case);
  anything else there fails that line; an empty third column is ignored
- list set --post publishers; help text updated
- the old import test now uses the publisher keyword in its third column
2026-11-10 12:40:10 +01:00
tgiachi
0dc9c03bae feat(router): send newsletter deliveries with priority 1
- Deliver and Release (through outbound) queue the deliveries of a list
  whose post policy is publishers with priority 1, everything else 0;
  confirmation mail stays 0
- tests for Deliver, Release, confirmation mail and the recorded post
2026-11-10 12:39:34 +01:00
tgiachi
0bbfd43e30 feat(api): publisher flag on members and the publishers post policy
- the member DTO carries publisher
- POST /lists/{id}/members takes an optional publisher: absent leaves the flag
  of an existing member alone, true or false sets it
- PATCH /lists/{id}/members/{address} with {"publisher": bool}: 404 for a
  non-member, 400 when the field is missing
- the list PATCH accepts post_policy publishers; the validation message says so
- openapi updated
2026-11-10 12:38:35 +01:00
tgiachi
51e2bf683e feat(queue): count sent, failed and dropped deliveries of a post
- a delivery the relay accepted completes as sent, one given up as
  failed, one to a missing mailbox (or with a missing body) is dropped
  through store.DropOutbox; a retry changes nothing
- tests: each outcome of a post, giving up after MaxAge, a missing body,
  and discussion mail sent before a bigger newsletter queued earlier
2026-11-10 12:38:32 +01:00
tgiachi
922e2dc500 feat(web): public subscribe form for lists with a public archive and open joins
- GET /subscribe/{address}: email field, an off-screen honeypot named website,
  a Subscribe button; 404 (the usual page) for a private, closed or unknown list
- POST /subscribe/{address}: one plain ASCII address of at most 254 characters,
  else the form again (400) with the escaped typed value; a filled honeypot gets
  the success page and nothing happens; otherwise Router.RequestJoin and always
  the same neutral page, so a member, a rate limited address or an unknown
  one cannot be told apart; the body is limited to 4 KiB (413)
- at most 10 POSTs per hour per TCP peer address (X-Forwarded-For is ignored),
  then 429 with Retry-After; idle addresses are swept; the clock is injectable
- a Subscribe link on the archive page of a list that qualifies, nothing else
- no-store on the subscribe pages; the CSP is unchanged
- tests for each of these, including escaping, enumeration, the limit and its expiry
2026-11-10 12:38:02 +01:00
tgiachi
2abd435494 feat(store): record posts and count their deliveries
- posts table: one row per post queued to a list (message id, subject,
  time, recipients, sent, failed, dropped), deleted with the list and
  forgotten after 400 days when a new post is recorded
- EnqueuePost and EnqueueFromHeld record the post in the same
  transaction and stamp every delivery with its id; a refused duplicate
  records nothing; other mail keeps post_id 0
- CompleteOutbox counts sent, the new DropOutbox counts dropped and
  FailOutbox counts failed, each in the transaction that changes the
  delivery; a delivery that already failed is never counted twice
- outbox priority column; DueOutbox orders by priority, next_at, id
- ListPosts newest first (default 50, at most 200) with Pending derived
- additive migration of the outbox columns, tested on an old database
2026-11-10 12:37:52 +01:00
tgiachi
adeae3f451 feat(app): pass relay.rate_per_second to the queue worker
- one line of wiring, a commented example in parsimony.example.yaml
- tests: with a rate of 20/s eight deliveries to a sink take at least 350 ms, without a rate they are not paced
2026-11-10 12:37:15 +01:00
tgiachi
8ac4943e49 feat(config): relay.rate_per_second
- float, default 0 (unlimited); negative, NaN, infinite or non-numeric values are refused with an error naming the key
- tests with 14, 0.5, 0, 1e2 and -1, .nan, .inf, nan, fast, lists
2026-11-10 12:37:15 +01:00
tgiachi
67ba07b642 feat(queue): Worker.Rate paces every send attempt
- Rate (sends per second, 0 = unlimited) builds one pacer, used right before Sender.Send
- the wait comes before the send timeout starts, so waiting does not eat into it
- a cancelled wait leaves the delivery queued, with no attempt recorded
- tests: 5 deliveries at 2/s take 4 intervals of fake time, rate 0 never waits, retries share the pacer,
  cancellation while waiting, and one short real-time test (20 sends at 200/s)
2026-11-10 12:37:15 +01:00
tgiachi
b125c61fcf feat(queue): a pacer that spaces outgoing sends
- newPacer(perSecond, now, sleep): strictly spaced slots, no burst, safe for concurrent callers
- Wait reserves the next slot under a mutex and sleeps outside it
- a cancelled caller gives its slot back when it is the last one; a cancelled context returns at once
- a rate that is zero, negative, NaN or infinite means no pacer; a very low rate is capped at one interval of a year
- tests with an injected clock and sleep
2026-11-10 12:37:15 +01:00
tgiachi
b6f776530f test(ingest): match the refusal text of a non-publisher by suffix
- the client sees the recipient in front of the server message
2026-11-10 12:37:13 +01:00
tgiachi
d4f706523f feat(router): the publishers post policy refuses non-publishers
- Route on a publishers list lets through a member flagged publisher, after the
  unchanged discard and loop checks, and applies the usual From header check
- anyone else gets the new ErrNotPublisher: never held, archived or queued
- ingest answers ErrNotPublisher with 550 5.7.1 only the publishers of this list may post
- tests: publisher ok, plain member and stranger refused, disabled publisher posts,
  identity mismatch held and released, discard rules first, other policies unchanged
2026-11-10 12:37:00 +01:00
tgiachi
5bed75864f feat(store): publisher flag on members and the publishers post policy
- members.publisher column, added to old databases by an additive migration
- Member.Publisher, SetPublisher (ErrNotFound for a non-member) and IsPublisher
- a publisher disabled by bounces is still a publisher
- AddMember keeps the publisher flag of an existing member
- SetPolicies accepts the publishers post policy
2026-11-10 12:35:58 +01:00
tgiachi
daea68f69d feat(router): RequestJoin for callers that are not an LMTP session
- RequestJoin(listAddress, addr) reuses join: the same open-policy check, the
  silence for active members, the confirmation mail and the store rate limits
- it returns nil for an unknown or closed list, a member and a rate limited
  request, and an error only for internal failures
- tests: confirmation mail and confirming it, nothing revealed, a bounce
  disabled member can rejoin, a database failure is reported
2026-11-10 12:35:17 +01:00
tgiachi
e772777a2a chore: name the newsletter post policy 2026-11-10 12:33:03 +01:00
squidd
83eae529a4 Merge pull request 'Screenshots in the README' (#37) from docs/screenshots into main 2026-11-10 12:30:40 +01:00
tgiachi
0ea96c28d9 docs: screenshots of the admin UI and the public archive in the README
- Add five screenshots taken from a demo instance: lists, moderation, archive, queue and a public thread
- Cropped to their content and optimised (about 740 kB in total)
2026-11-10 12:29:54 +01:00
squidd
dac928144a Merge pull request 'A logo for Parsimony' (#35) from feat/logo into main 2026-11-10 12:16:44 +01:00
tgiachi
1dddf14cfd feat: the logo in the admin UI, the README and as PNG exports
- Show the wordmark on the login page and the mark in the header; set the page title to Parsimony
- Put the logo at the top of the README, with a dark-mode variant
- Add PNG exports (a 512 px mark for avatars, a 1200 px wordmark), both transparent
- Say that the logo is under the same licence
2026-11-10 12:14:50 +01:00