Commit graph parsimony/internal/store
Author SHA1 Message Date
tgiachi
c35ad47097 test: keep both store test additions after merging the review fixes 2026-11-10 15:21:43 +01:00
tgiachi
fa96a398ee docs: per-step delivery timeouts, trusted proxies and the confirmation cap
- Describe the phases of a delivery and the unknown-outcome retry
- Document web.trusted_proxies, the form's address rules and limits.pending_per_hour
2026-11-10 15:21:24 +01:00
tgiachi
d22423c02c refactor(store): drop a redundant zero check in the cap warning
now.Sub of the zero time is already longer than a minute, so the first
refusal is logged without a special case.
2026-11-10 15:16:02 +01:00
tgiachi
8d28e48360 feat: configurable cap on confirmation requests, with a warning
When the global cap of unconfirmed requests (300 an hour) was reached,
a subscribe request silently got no mail while the visitor saw "Check
your email"; a newsletter launch can pass 300 signups an hour.

- limits.pending_per_hour (default 300, must be a whole number >= 1)
  is validated at load and set on the store by the app.
- When the global cap refuses a request the store logs a warning naming
  the setting, at most once a minute. The visitor still gets the
  neutral page, so nothing about members is revealed. The per-address
  cap is unchanged and is not logged.
- parsimony.example.yaml documents the setting.
2026-11-10 15:14:04 +01:00
tgiachi
c66fd427b2 fix(store): add the outbox post_id and priority columns separately
The two columns were added in one non-transactional Exec guarded by
post_id alone. A crash between the two statements left a database
with post_id but no priority, and every later start failed at
CREATE INDEX outbox_due_prio. Each column is now checked and added on
its own, as the members columns are.
2026-11-10 15:07:11 +01:00
tgiachi
34f461bb6b Merge branch 'feat/nl-posts-stats' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
47dc118b98 test: tighten the posts tests found weak by mutation
- ListPosts: a limit of 201 is capped to 200
- posts ls: distinct pending, failed and dropped counts, so a swapped
  column is caught
2026-11-10 12:47:17 +01:00
tgiachi
2abd435494 feat(store): record posts and count their deliveries
- posts table: one row per post queued to a list (message id, subject,
  time, recipients, sent, failed, dropped), deleted with the list and
  forgotten after 400 days when a new post is recorded
- EnqueuePost and EnqueueFromHeld record the post in the same
  transaction and stamp every delivery with its id; a refused duplicate
  records nothing; other mail keeps post_id 0
- CompleteOutbox counts sent, the new DropOutbox counts dropped and
  FailOutbox counts failed, each in the transaction that changes the
  delivery; a delivery that already failed is never counted twice
- outbox priority column; DueOutbox orders by priority, next_at, id
- ListPosts newest first (default 50, at most 200) with Pending derived
- additive migration of the outbox columns, tested on an old database
2026-11-10 12:37:52 +01:00
tgiachi
5bed75864f feat(store): publisher flag on members and the publishers post policy
- members.publisher column, added to old databases by an additive migration
- Member.Publisher, SetPublisher (ErrNotFound for a non-member) and IsPublisher
- a publisher disabled by bounces is still a publisher
- AddMember keeps the publisher flag of an existing member
- SetPolicies accepts the publishers post policy
2026-11-10 12:35:58 +01:00
tgiachi
e772777a2a chore: name the newsletter post policy 2026-11-10 12:33:03 +01:00
tgiachi
b2f447f875 Merge branch 'fix/input-validation' into fix/fable-mediums 2026-11-10 12:01:07 +01:00
tgiachi
5af20f5891 Merge branch 'fix/dup-and-shutdown' into fix/fable-mediums 2026-11-10 12:01:06 +01:00
tgiachi
dd0096093b test: pin the duplicate guard and the graceful shutdown
- router: an approved post is remembered, so the MTA's copy is refused
- store: a post with no recipients stores no message body
- app: an API request in flight during Close gets its answer
- app: finished LMTP sessions leave the connection tracker
- router: drop a TrimSpace on the Message-ID that net/mail already does
2026-11-10 11:58:07 +01:00
tgiachi
f7ce18c3d4 test: close the gaps found by mutation checks
- api: empty names, '{' in a name, 253/254-character domains, title trimming and the name fallback
- config: an empty query or fragment in base_url
- router: leaving works on a list closed to new members
- store: an expired token is removed when presented
2026-11-10 11:57:49 +01:00
tgiachi
8151204a3d Merge branch 'fix/limits-timeouts' into fix/fable-mediums 2026-11-10 11:56:17 +01:00
tgiachi
555578bac5 fix: shorten the confirmation token to 20 hex characters
name(<=30) + "+confirm-" + 20 = 59 characters, inside the 64 of a local part
(a 30-character name with the old 32-digit token made 71).

- store: CreatePending makes an 80-bit token
- tests: a 30-character list name keeps the confirm and the VERP bounce address (12-digit id) within 64,
  and the shorter token still confirms
2026-11-10 11:54:28 +01:00
tgiachi
cfcfd42110 fix(router,store): consume a confirmation token only when its action happened
- store: ConfirmPending claims the row (DELETE ... RETURNING), applies join or leave and commits in one
  transaction; a refusal from check, a database error, a token of another list or a rollback leaves the
  token usable; concurrent confirmations of one token apply exactly once
- store: CreatePending purges pending rows older than the 72 h TTL
- store: IsActiveMember
- router: confirm uses it, the list policy check runs before anything is consumed
- router: +join from a member disabled by bounces sends the confirmation; confirming re-enables it
- tests: purge, refusal, forced database error, other list, expiry, 8 concurrent confirms, whole rejoin flow
- cli test: the create error text changed with the new name rule
2026-11-10 11:54:05 +01:00
tgiachi
1e8d60b72c fix(store): run the v1 members migration in one transaction
- store: migrateMembersV1 renames, creates, copies and drops inside a transaction
- test: a copy that fails on an orphan row leaves the old members table with its rows and no members_v1
2026-11-10 11:52:42 +01:00
tgiachi
bd4258ebac feat(router): queue a post to a list only once per hour
When the reply to a delivered post is lost, the MTA sends the post again
and every member got it twice. The list now remembers the posts it queued.

- store: new delivered table (list id, post id, time), created with
  CREATE TABLE IF NOT EXISTS and dropped with its list
- store: EnqueuePost records the post and queues it in one transaction,
  returns ErrDuplicate when the list queued it in the last hour, and
  purges older records
- store: EnqueueFromHeld records the post too, without refusing it: an
  approval always delivers
- router: Deliver uses EnqueuePost and logs a duplicate; the post id is
  the Message-ID, or a SHA-256 of the message when it has none
- router: Release passes the post id to EnqueueFromHeld
- tests for duplicates, other ids and lists, expiry on the store clock,
  atomicity, posts without Message-ID and approvals
2026-11-10 11:52:25 +01:00
tgiachi
ca2a1d63d7 fix(store): count only hard bounces of the last 30 days
- Add members.bounce_at (unix time of the last counted bounce) with an
  additive migration; existing counts get 0, so they are treated as old.
- RecordHardBounce restarts the count at 1 when the last bounce is older
  than BounceWindow (30 days); a disabled member stays disabled.
- Tests for the window boundary, the restart and the migration of an
  old database.
2026-11-10 11:50:49 +01:00
tgiachi
d43c47ea08 fix(store): put the pragmas in the DSN so every connection has them
- foreign_keys, busy_timeout(5000) and WAL now come from the DSN, not from
  an Exec on one connection
- test: every connection of a 2-connection pool, in memory and on disk
2026-11-10 11:49:00 +01:00
tgiachi
beff2579c2 test: pin the SMTP command named in a relay error, SMTPUTF8 and thread order
- Name the refused command in the error (MAIL FROM, RCPT TO, DATA) so the queue tells a refused sender from a refused recipient
- Test a refusal of the DATA command itself, SMTPUTF8 for non-ASCII addresses and oldest-first threads
2026-11-10 11:42:19 +01:00
tgiachi
94b5a58e93 fix(store): moderation listings never read message bodies, bound the held bytes
- Store subject, From header and size when a message is held; ListHeld returns the summary only
- Cap one held message at 10 MiB (ErrHeldTooLarge) and the bytes held per list at 200 MiB (ErrHeldFull)
- Summarise the messages already held when migrating an existing database, one at a time
- Show the From header and whether it differs from the envelope sender in the held API
- Thread listings no longer load the bodies of the thread
2026-11-10 11:39:21 +01:00
tgiachi
7005edcb9b test: pin the documented limits and close gaps found by mutation checks
- Test the lockout, the confirmation cap and Retry-After with literal numbers, not the constants
- Test that failed deliveries are listed before the most retried ones
2026-11-09 20:43:41 +01:00
tgiachi
bb0858ef87 fix: keep refused deliveries visible instead of dropping them silently
- Add failed deliveries: a permanent refusal by the relay (policy, size, line length) or the 3-day give-up marks the delivery failed, with the reason, and never retries it
- Keep failed deliveries listed for a week, then purge them and their stored message
- A non-existent mailbox still drops the delivery and counts a hard bounce
- Migrate the outbox table
2026-11-09 20:33:27 +01:00
tgiachi
e7e1afbde9 fix: approve a held message once, however many approvals race
- Release removes the held message and queues its deliveries in one transaction
- Of several simultaneous approvals one wins, the others get 404
- Extract the thread-root computation and test its linearity without the database (fast under -race)
2026-11-09 20:29:42 +01:00
tgiachi
fd113c4ac2 fix(security): bound the moderation queue
- Cap the held queue at 500 per list and drop messages nobody moderated for 30 days
- Answer 452 when it is full, so a legitimate sender retries and a spam run goes away
2026-11-09 20:26:33 +01:00
tgiachi
bad3c9bc10 fix(security): cap confirmation mail per address and in total 2026-11-09 20:25:53 +01:00
tgiachi
eb9e915805 fix(security): hold posts whose From header does not match the sender
- Judge membership on the envelope sender and require the From header to name the same address
- Hold mismatches (and missing, unparsable or multiple From addresses) for the admin instead of delivering
- Record why a message was held: new held.reason column, migrated, shown in the API, CLI and UI
- Accept an empty message in the held queue
- Document who may post
2026-11-09 20:23:07 +01:00
tgiachi
a170204ecb test: close gaps found by mutation checks on the archive import
- Cover the CRLF separator line, cycle roots entered from outside, linear rethreading of long chains
- Cover a read error mid-import, the "and N more" lines, and the mbox content type
- Cover clearing the previous result and the capped error list in the UI
2026-11-09 19:50:40 +01:00
tgiachi
ba6c37bc21 feat(mbox,store): mbox reader, batch archiving and thread rebuilding
- Add internal/mbox: streaming reader with strict From_ detection, mboxrd unescaping, size cap and envelope dates
- Add Store.ArchiveBatch: many messages in one transaction
- Add Store.Rethread: rebuild threads from In-Reply-To whatever the insertion order, tolerating cycles
2026-11-09 17:13:56 +01:00
tgiachi
6461e77a95 feat: document the queue, close mutation gaps, plain relay errors
- Document delivery semantics, retries and the queue API in the README and OpenAPI
- Add relay.concurrency to the example config
- Test that a relay problem never counts against a member, that the longest-waiting delivery goes first, and the error and limit details
- Show the plain relay error when there is a single batch
2026-11-09 16:04:45 +01:00
tgiachi
e6713962f9 feat(queue): background worker with retries, backoff and bounded concurrency
- Add Worker.RunOnce, Run and Wake over the durable outbox
- Send prefix plus shared message per recipient with a per-send timeout
- Retry temporary failures after 1m, 5m, 15m, 1h, 3h, then every 6h; give up after 3 days
- Drop permanent failures; count a hard bounce only for bad-mailbox replies
- Leave interrupted sends queued on shutdown
- Share the bounce threshold between router and worker
2026-11-09 15:47:14 +01:00
tgiachi
092cf4c350 feat(store): durable outbox with shared bodies and retry scheduling
- Add outbox_msgs and outbox tables
- Enqueue every recipient of a post in one transaction
- Add due, list, complete, retry and stats queries
- Delete the shared body with the last recipient
2026-11-09 15:45:19 +01:00
tgiachi
8253dd608e feat(router): archive accepted posts and add per-recipient unsubscribe links
- Archive the original message on delivery and on release, never when held or rejected
- Log archive failures instead of blocking delivery
- Add List-Unsubscribe per recipient: signed one-click URL and the +leave address
- Add signed unsubscribe tokens bound to their purpose
- Fix the mailto unsubscribe address, +unsubscribe was not a known command
2026-11-09 15:20:01 +01:00
tgiachi
d8275aaa1f feat(store,api): archive with full-text search, threading and archive policy
- Add archive table with FTS5 index kept in sync by triggers
- Deduplicate by Message-ID per list so MTA retries do not repeat posts
- Thread messages through In-Reply-To within the same list
- Quote every search word so user input is never FTS syntax
- Add archive_policy (public, private, off) with migration
- Add API to list, search, read and delete archived messages
2026-11-09 15:17:13 +01:00
tgiachi
d6a0f6e5a9 test: close gaps found by mutation checks
- Cover the default relay batch size
- Cover partial and total relay failure in Router.Send
- Cover stripping of spoofed List-* headers without a base URL
- Cover the end of the confirmation rate-limit window
2026-11-09 15:09:23 +01:00
tgiachi
9eb4d14a70 feat(store): schema v2 with moderation, pending requests and bounce state
- Add member id, disabled flag and bounce counter
- Add per-list post and join policies
- Add held message queue
- Add expiring, rate-limited pending subscribe and unsubscribe tokens
- Add persistent HMAC secret
- Migrate databases created by milestone 1
2026-11-09 15:03:08 +01:00
tgiachi
e1b0c7666c feat(store): list update, delete, listing and member removal
- Add ListLists, GetList, UpdateList, DeleteList, RemoveMember
- Add ErrExists for duplicate lists
2026-11-09 14:57:13 +01:00
tgiachi
015e0bbd0f feat(store): SQLite store for lists and members
- Add schema with lists and members tables
- Add CreateList, FindList, AddMember, IsMember, Members
- Normalize addresses to lowercase
2026-11-09 14:46:35 +01:00