Commit graph

175 commits

Author SHA1 Message Date
tgiachi
c17ff1fed1 Merge branch 'fix/sr-send-timeout' into fix/second-review 2026-11-10 15:21:24 +01:00
tgiachi
af487b8c84 fix(queue): stop putting one deadline on the whole send
Worker.Timeout defaulted to one minute over the entire SMTP
transaction, cutting relays that are slow to accept a message and
causing duplicate deliveries. The default is now no total deadline:
SMTPRelay bounds each phase itself. A positive Timeout still applies.
2026-11-10 15:20:49 +01:00
tgiachi
f14faf52d5 fix(outbound): give each SMTP phase its own deadline
A relay can take minutes to answer the final dot (content filter,
large message, slow link). One deadline over the whole transaction
cut it after the message had been accepted, so the delivery was
retried and sent again.

The connection now moves its deadline as the transaction advances:
setup (dial, greeting, TLS, AUTH, MAIL, RCPT, DATA) has
SMTPRelay.Timeout, 60 s by default; the upload has 60 s plus the size
at 32 KiB/s, at most 10 minutes; the reply to the final dot has
10 minutes (RFC 5321 4.5.3.2.6). go-smtp sets and clears its own
per-command deadlines, so a wrapper clips them to the current phase.
Context cancellation still closes the connection at once. A timeout
in any phase is temporary; after the final dot the error says the
outcome is unknown.
2026-11-10 15:20:49 +01:00
tgiachi
c66fd427b2 fix(store): add the outbox post_id and priority columns separately
The two columns were added in one non-transactional Exec guarded by
post_id alone. A crash between the two statements left a database
with post_id but no priority, and every later start failed at
CREATE INDEX outbox_due_prio. Each column is now checked and added on
its own, as the members columns are.
2026-11-10 15:07:11 +01:00
tgiachi
c6cbcb3d3b docs: say what the second review found about newsletters, priority and shutdown
- Warn that forged publisher senders are not checked by the daemon
- Describe the real limits of the delivery priority
- Document the Docker stop timeout and three known gaps
2026-11-10 15:06:35 +01:00
squidd
bc1ea2e7c3 Merge pull request 'End-to-end test for newsletters' (#41) from test/e2e-newsletter into main 2026-11-10 14:49:25 +01:00
tgiachi
2030b39270 test: end-to-end newsletter through a real Postfix
- A subscriber's post is refused with a real LMTP 550 and nothing is held or delivered
- The publisher's post reaches subscribers with list headers; the post stats add up
- The public subscribe form shows the neutral page and sends one confirmation mail
2026-11-10 14:48:59 +01:00
squidd
9a51b39eef Merge pull request 'Newsletter mode' (#39) from feat/newsletter into main 2026-11-10 13:08:16 +01:00
tgiachi
275b269977 docs: Newsletters section in the README
- Publishers policy, sent-post statistics, priority, rate limit and the public subscribe form
- State what is not included: scheduled sends and open or click tracking
2026-11-10 13:07:16 +01:00
tgiachi
c42ae9a276 test: share one SetPolicies-based helper for the newsletter priority tests
- Rename the priority helper to priorityEnv and set the publishers policy through SetPolicies instead of raw SQL
- Fixes the duplicate newsletterEnv after merging the publishers and posts branches
2026-11-10 13:02:58 +01:00
tgiachi
6a819aaeaa Merge branch 'feat/nl-web-signup' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
3d25f73d9b Merge branch 'feat/nl-rate-limit' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
34f461bb6b Merge branch 'feat/nl-posts-stats' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
c853ae7e61 test(queue): cover a pacer slot given up while a later caller holds its own
- a caller that gives up must not release a slot when another caller reserved after it
- a delivery that did not get its slot is not counted in the run's stats
2026-11-10 12:55:26 +01:00
tgiachi
47dc118b98 test: tighten the posts tests found weak by mutation
- ListPosts: a limit of 201 is capped to 200
- posts ls: distinct pending, failed and dropped counts, so a swapped
  column is caught
2026-11-10 12:47:17 +01:00
tgiachi
5ca091ccbf test(web): trimming, Retry-After rounding and the exact edges of the limit window
- the typed address is trimmed before it is used
- Retry-After rounds up to whole seconds
- the hour ends after exactly an hour, and the sweep forgets such an address
- found by mutation testing: these edges survived the first tests
2026-11-10 12:45:49 +01:00
tgiachi
ee1f837a20 feat(admin): a Sent tab with the posts of a list and their progress
- route /lists/:id/sent and a Sent tab on the list page
- a table of posts (subject, date, sent of recipients with a progress
  bar, recipients, pending, failed, dropped) and an empty state; a post
  with no recipients shows no bar
- refreshes every 5 seconds while any post has pending deliveries, and
  stops when none has
- api.posts.list, the Post type and the fake API route; tests
2026-11-10 12:43:55 +01:00
tgiachi
7e43117c74 feat(cli): parsimony posts ls
- posts ls <list> [--limit N] prints ID, DATE, SUBJECT, RECIPIENTS, SENT,
  PENDING, FAILED and DROPPED, newest first, "No posts yet." when empty,
  and the API's JSON with --json
- cells go through the same control-character escaping as the other
  tables; tests for the table, the limit, JSON, errors and escaping
2026-11-10 12:42:15 +01:00
tgiachi
10f593e9bd feat(admin): publishers in the Settings and Members tabs
- Settings: the Posting select offers Publishers only (newsletter)
- Members: a Can publish checkbox in the add form (unchecked sends no flag, so
  adding an existing publisher again keeps it), a Publisher badge in the table,
  and a Make publisher / Make subscriber row action
- api client: publisher on Member, the publishers post policy, setPublisher
- the fake API keeps the flag on re-add and answers the member PATCH
- tests for each
2026-11-10 12:41:44 +01:00
tgiachi
2f2ee71f92 feat(api): list the posts of a list with their delivery counters
- GET /api/v1/lists/{id}/posts?limit= returns id, message_id, subject,
  created_at, recipients, sent, failed, dropped and pending, newest
  first; 404 for an unknown list, 400 for a bad limit, token required
- discarding a queued delivery counts it as dropped (store.DropOutbox)
- openapi: the new path and the discard note
2026-11-10 12:40:55 +01:00
tgiachi
e850547b3d feat(cli): publishers in member add, ls, set and import
- member add --publisher makes a publisher, --publisher=false demotes; without
  the flag an existing member keeps its role
- member ls has a ROLE column: publisher or subscriber
- new member set <list> <email> --publisher=true|false (usage error without the flag)
- member import reads an optional third column, the keyword publisher (any case);
  anything else there fails that line; an empty third column is ignored
- list set --post publishers; help text updated
- the old import test now uses the publisher keyword in its third column
2026-11-10 12:40:10 +01:00
tgiachi
0dc9c03bae feat(router): send newsletter deliveries with priority 1
- Deliver and Release (through outbound) queue the deliveries of a list
  whose post policy is publishers with priority 1, everything else 0;
  confirmation mail stays 0
- tests for Deliver, Release, confirmation mail and the recorded post
2026-11-10 12:39:34 +01:00
tgiachi
0bbfd43e30 feat(api): publisher flag on members and the publishers post policy
- the member DTO carries publisher
- POST /lists/{id}/members takes an optional publisher: absent leaves the flag
  of an existing member alone, true or false sets it
- PATCH /lists/{id}/members/{address} with {"publisher": bool}: 404 for a
  non-member, 400 when the field is missing
- the list PATCH accepts post_policy publishers; the validation message says so
- openapi updated
2026-11-10 12:38:35 +01:00
tgiachi
51e2bf683e feat(queue): count sent, failed and dropped deliveries of a post
- a delivery the relay accepted completes as sent, one given up as
  failed, one to a missing mailbox (or with a missing body) is dropped
  through store.DropOutbox; a retry changes nothing
- tests: each outcome of a post, giving up after MaxAge, a missing body,
  and discussion mail sent before a bigger newsletter queued earlier
2026-11-10 12:38:32 +01:00
tgiachi
922e2dc500 feat(web): public subscribe form for lists with a public archive and open joins
- GET /subscribe/{address}: email field, an off-screen honeypot named website,
  a Subscribe button; 404 (the usual page) for a private, closed or unknown list
- POST /subscribe/{address}: one plain ASCII address of at most 254 characters,
  else the form again (400) with the escaped typed value; a filled honeypot gets
  the success page and nothing happens; otherwise Router.RequestJoin and always
  the same neutral page, so a member, a rate limited address or an unknown
  one cannot be told apart; the body is limited to 4 KiB (413)
- at most 10 POSTs per hour per TCP peer address (X-Forwarded-For is ignored),
  then 429 with Retry-After; idle addresses are swept; the clock is injectable
- a Subscribe link on the archive page of a list that qualifies, nothing else
- no-store on the subscribe pages; the CSP is unchanged
- tests for each of these, including escaping, enumeration, the limit and its expiry
2026-11-10 12:38:02 +01:00
tgiachi
2abd435494 feat(store): record posts and count their deliveries
- posts table: one row per post queued to a list (message id, subject,
  time, recipients, sent, failed, dropped), deleted with the list and
  forgotten after 400 days when a new post is recorded
- EnqueuePost and EnqueueFromHeld record the post in the same
  transaction and stamp every delivery with its id; a refused duplicate
  records nothing; other mail keeps post_id 0
- CompleteOutbox counts sent, the new DropOutbox counts dropped and
  FailOutbox counts failed, each in the transaction that changes the
  delivery; a delivery that already failed is never counted twice
- outbox priority column; DueOutbox orders by priority, next_at, id
- ListPosts newest first (default 50, at most 200) with Pending derived
- additive migration of the outbox columns, tested on an old database
2026-11-10 12:37:52 +01:00
tgiachi
adeae3f451 feat(app): pass relay.rate_per_second to the queue worker
- one line of wiring, a commented example in parsimony.example.yaml
- tests: with a rate of 20/s eight deliveries to a sink take at least 350 ms, without a rate they are not paced
2026-11-10 12:37:15 +01:00
tgiachi
8ac4943e49 feat(config): relay.rate_per_second
- float, default 0 (unlimited); negative, NaN, infinite or non-numeric values are refused with an error naming the key
- tests with 14, 0.5, 0, 1e2 and -1, .nan, .inf, nan, fast, lists
2026-11-10 12:37:15 +01:00
tgiachi
67ba07b642 feat(queue): Worker.Rate paces every send attempt
- Rate (sends per second, 0 = unlimited) builds one pacer, used right before Sender.Send
- the wait comes before the send timeout starts, so waiting does not eat into it
- a cancelled wait leaves the delivery queued, with no attempt recorded
- tests: 5 deliveries at 2/s take 4 intervals of fake time, rate 0 never waits, retries share the pacer,
  cancellation while waiting, and one short real-time test (20 sends at 200/s)
2026-11-10 12:37:15 +01:00
tgiachi
b125c61fcf feat(queue): a pacer that spaces outgoing sends
- newPacer(perSecond, now, sleep): strictly spaced slots, no burst, safe for concurrent callers
- Wait reserves the next slot under a mutex and sleeps outside it
- a cancelled caller gives its slot back when it is the last one; a cancelled context returns at once
- a rate that is zero, negative, NaN or infinite means no pacer; a very low rate is capped at one interval of a year
- tests with an injected clock and sleep
2026-11-10 12:37:15 +01:00
tgiachi
b6f776530f test(ingest): match the refusal text of a non-publisher by suffix
- the client sees the recipient in front of the server message
2026-11-10 12:37:13 +01:00
tgiachi
d4f706523f feat(router): the publishers post policy refuses non-publishers
- Route on a publishers list lets through a member flagged publisher, after the
  unchanged discard and loop checks, and applies the usual From header check
- anyone else gets the new ErrNotPublisher: never held, archived or queued
- ingest answers ErrNotPublisher with 550 5.7.1 only the publishers of this list may post
- tests: publisher ok, plain member and stranger refused, disabled publisher posts,
  identity mismatch held and released, discard rules first, other policies unchanged
2026-11-10 12:37:00 +01:00
tgiachi
5bed75864f feat(store): publisher flag on members and the publishers post policy
- members.publisher column, added to old databases by an additive migration
- Member.Publisher, SetPublisher (ErrNotFound for a non-member) and IsPublisher
- a publisher disabled by bounces is still a publisher
- AddMember keeps the publisher flag of an existing member
- SetPolicies accepts the publishers post policy
2026-11-10 12:35:58 +01:00
tgiachi
daea68f69d feat(router): RequestJoin for callers that are not an LMTP session
- RequestJoin(listAddress, addr) reuses join: the same open-policy check, the
  silence for active members, the confirmation mail and the store rate limits
- it returns nil for an unknown or closed list, a member and a rate limited
  request, and an error only for internal failures
- tests: confirmation mail and confirming it, nothing revealed, a bounce
  disabled member can rejoin, a database failure is reported
2026-11-10 12:35:17 +01:00
tgiachi
e772777a2a chore: name the newsletter post policy 2026-11-10 12:33:03 +01:00
squidd
83eae529a4 Merge pull request 'Screenshots in the README' (#37) from docs/screenshots into main 2026-11-10 12:30:40 +01:00
tgiachi
0ea96c28d9 docs: screenshots of the admin UI and the public archive in the README
- Add five screenshots taken from a demo instance: lists, moderation, archive, queue and a public thread
- Cropped to their content and optimised (about 740 kB in total)
2026-11-10 12:29:54 +01:00
squidd
dac928144a Merge pull request 'A logo for Parsimony' (#35) from feat/logo into main 2026-11-10 12:16:44 +01:00
tgiachi
1dddf14cfd feat: the logo in the admin UI, the README and as PNG exports
- Show the wordmark on the login page and the mark in the header; set the page title to Parsimony
- Put the logo at the top of the README, with a dark-mode variant
- Add PNG exports (a 512 px mark for avatars, a 1200 px wordmark), both transparent
- Say that the logo is under the same licence
2026-11-10 12:14:50 +01:00
tgiachi
ec8ca567ff feat(web): logo and favicon on the public pages
- Show the mark on every page, linking to the index, inline so the CSP needs no image source
- Add the favicon as a data: URI and allow img-src data: only
2026-11-10 12:13:53 +01:00
tgiachi
5f5deb3e30 feat(assets): the Parsimony logo
- Add the mark (an envelope in a rounded square) and the wordmark, drawn with monoline strokes so no font is needed
- Add light and dark versions, and the generator script
- Embed them in a Go package and ship the same files with the admin UI
- Test that the SVGs are well formed, need nothing from outside, stay in sync with the admin copies and have enough contrast
2026-11-10 12:13:17 +01:00
squidd
6f745e9b25 Merge pull request 'Fix the medium and low findings of the global review' (#33) from fix/fable-mediums into main 2026-11-10 12:04:26 +01:00
tgiachi
e8c359add5 test(cli): drop an unused assignment flagged by staticcheck 2026-11-10 12:01:54 +01:00
tgiachi
b4f45035bb Merge branch 'fix/hygiene-cli-ui' into fix/fable-mediums
# Conflicts:
#	internal/router/m2_test.go
2026-11-10 12:01:40 +01:00
tgiachi
b2f447f875 Merge branch 'fix/input-validation' into fix/fable-mediums 2026-11-10 12:01:07 +01:00
tgiachi
5af20f5891 Merge branch 'fix/dup-and-shutdown' into fix/fable-mediums 2026-11-10 12:01:06 +01:00
tgiachi
266bcba354 test(app): use an HTTP client without keep-alive
- the transport could dial a connection and never use it; the HTTP
  server's graceful shutdown waits 5 s for such a connection, which made
  the Close tests flaky under -race
- the call and fetch helpers now open one connection per request
2026-11-10 12:00:04 +01:00
tgiachi
d957be3e84 fix(admin): the Held empty state names both reasons a message is held
- The hint said mail is held only on moderated lists; it is also held when the From
  header does not match the sender. It now says both, and shows for every posting policy.
- Test covers members and moderated lists and the hint staying away when messages are held.
2026-11-10 11:58:59 +01:00
tgiachi
d435f80297 docs: describe the duplicate guard and the graceful stop
- replace the sentence on at-least-once delivery with what is guarded
  now (a post queued once per list per hour, approvals always delivered)
- describe the 30 s graceful stop of LMTP and HTTP
- state what remains: at-least-once across the relay, and copies that
  arrive after an hour
2026-11-10 11:58:21 +01:00
tgiachi
dd0096093b test: pin the duplicate guard and the graceful shutdown
- router: an approved post is remembered, so the MTA's copy is refused
- store: a post with no recipients stores no message body
- app: an API request in flight during Close gets its answer
- app: finished LMTP sessions leave the connection tracker
- router: drop a TrimSpace on the Message-ID that net/mail already does
2026-11-10 11:58:07 +01:00