Commit graph parsimony/internal/store/store.go
Author SHA1 Message Date
tgiachi
34f461bb6b Merge branch 'feat/nl-posts-stats' into feat/newsletter 2026-11-10 13:02:41 +01:00
tgiachi
2abd435494 feat(store): record posts and count their deliveries
- posts table: one row per post queued to a list (message id, subject,
  time, recipients, sent, failed, dropped), deleted with the list and
  forgotten after 400 days when a new post is recorded
- EnqueuePost and EnqueueFromHeld record the post in the same
  transaction and stamp every delivery with its id; a refused duplicate
  records nothing; other mail keeps post_id 0
- CompleteOutbox counts sent, the new DropOutbox counts dropped and
  FailOutbox counts failed, each in the transaction that changes the
  delivery; a delivery that already failed is never counted twice
- outbox priority column; DueOutbox orders by priority, next_at, id
- ListPosts newest first (default 50, at most 200) with Pending derived
- additive migration of the outbox columns, tested on an old database
2026-11-10 12:37:52 +01:00
tgiachi
5bed75864f feat(store): publisher flag on members and the publishers post policy
- members.publisher column, added to old databases by an additive migration
- Member.Publisher, SetPublisher (ErrNotFound for a non-member) and IsPublisher
- a publisher disabled by bounces is still a publisher
- AddMember keeps the publisher flag of an existing member
- SetPolicies accepts the publishers post policy
2026-11-10 12:35:58 +01:00
tgiachi
e772777a2a chore: name the newsletter post policy 2026-11-10 12:33:03 +01:00
tgiachi
b2f447f875 Merge branch 'fix/input-validation' into fix/fable-mediums 2026-11-10 12:01:07 +01:00
tgiachi
5af20f5891 Merge branch 'fix/dup-and-shutdown' into fix/fable-mediums 2026-11-10 12:01:06 +01:00
tgiachi
8151204a3d Merge branch 'fix/limits-timeouts' into fix/fable-mediums 2026-11-10 11:56:17 +01:00
tgiachi
555578bac5 fix: shorten the confirmation token to 20 hex characters
name(<=30) + "+confirm-" + 20 = 59 characters, inside the 64 of a local part
(a 30-character name with the old 32-digit token made 71).

- store: CreatePending makes an 80-bit token
- tests: a 30-character list name keeps the confirm and the VERP bounce address (12-digit id) within 64,
  and the shorter token still confirms
2026-11-10 11:54:28 +01:00
tgiachi
cfcfd42110 fix(router,store): consume a confirmation token only when its action happened
- store: ConfirmPending claims the row (DELETE ... RETURNING), applies join or leave and commits in one
  transaction; a refusal from check, a database error, a token of another list or a rollback leaves the
  token usable; concurrent confirmations of one token apply exactly once
- store: CreatePending purges pending rows older than the 72 h TTL
- store: IsActiveMember
- router: confirm uses it, the list policy check runs before anything is consumed
- router: +join from a member disabled by bounces sends the confirmation; confirming re-enables it
- tests: purge, refusal, forced database error, other list, expiry, 8 concurrent confirms, whole rejoin flow
- cli test: the create error text changed with the new name rule
2026-11-10 11:54:05 +01:00
tgiachi
1e8d60b72c fix(store): run the v1 members migration in one transaction
- store: migrateMembersV1 renames, creates, copies and drops inside a transaction
- test: a copy that fails on an orphan row leaves the old members table with its rows and no members_v1
2026-11-10 11:52:42 +01:00
tgiachi
bd4258ebac feat(router): queue a post to a list only once per hour
When the reply to a delivered post is lost, the MTA sends the post again
and every member got it twice. The list now remembers the posts it queued.

- store: new delivered table (list id, post id, time), created with
  CREATE TABLE IF NOT EXISTS and dropped with its list
- store: EnqueuePost records the post and queues it in one transaction,
  returns ErrDuplicate when the list queued it in the last hour, and
  purges older records
- store: EnqueueFromHeld records the post too, without refusing it: an
  approval always delivers
- router: Deliver uses EnqueuePost and logs a duplicate; the post id is
  the Message-ID, or a SHA-256 of the message when it has none
- router: Release passes the post id to EnqueueFromHeld
- tests for duplicates, other ids and lists, expiry on the store clock,
  atomicity, posts without Message-ID and approvals
2026-11-10 11:52:25 +01:00
tgiachi
ca2a1d63d7 fix(store): count only hard bounces of the last 30 days
- Add members.bounce_at (unix time of the last counted bounce) with an
  additive migration; existing counts get 0, so they are treated as old.
- RecordHardBounce restarts the count at 1 when the last bounce is older
  than BounceWindow (30 days); a disabled member stays disabled.
- Tests for the window boundary, the restart and the migration of an
  old database.
2026-11-10 11:50:49 +01:00
tgiachi
d43c47ea08 fix(store): put the pragmas in the DSN so every connection has them
- foreign_keys, busy_timeout(5000) and WAL now come from the DSN, not from
  an Exec on one connection
- test: every connection of a 2-connection pool, in memory and on disk
2026-11-10 11:49:00 +01:00
tgiachi
94b5a58e93 fix(store): moderation listings never read message bodies, bound the held bytes
- Store subject, From header and size when a message is held; ListHeld returns the summary only
- Cap one held message at 10 MiB (ErrHeldTooLarge) and the bytes held per list at 200 MiB (ErrHeldFull)
- Summarise the messages already held when migrating an existing database, one at a time
- Show the From header and whether it differs from the envelope sender in the held API
- Thread listings no longer load the bodies of the thread
2026-11-10 11:39:21 +01:00
tgiachi
bb0858ef87 fix: keep refused deliveries visible instead of dropping them silently
- Add failed deliveries: a permanent refusal by the relay (policy, size, line length) or the 3-day give-up marks the delivery failed, with the reason, and never retries it
- Keep failed deliveries listed for a week, then purge them and their stored message
- A non-existent mailbox still drops the delivery and counts a hard bounce
- Migrate the outbox table
2026-11-09 20:33:27 +01:00
tgiachi
e7e1afbde9 fix: approve a held message once, however many approvals race
- Release removes the held message and queues its deliveries in one transaction
- Of several simultaneous approvals one wins, the others get 404
- Extract the thread-root computation and test its linearity without the database (fast under -race)
2026-11-09 20:29:42 +01:00
tgiachi
fd113c4ac2 fix(security): bound the moderation queue
- Cap the held queue at 500 per list and drop messages nobody moderated for 30 days
- Answer 452 when it is full, so a legitimate sender retries and a spam run goes away
2026-11-09 20:26:33 +01:00
tgiachi
bad3c9bc10 fix(security): cap confirmation mail per address and in total 2026-11-09 20:25:53 +01:00
tgiachi
eb9e915805 fix(security): hold posts whose From header does not match the sender
- Judge membership on the envelope sender and require the From header to name the same address
- Hold mismatches (and missing, unparsable or multiple From addresses) for the admin instead of delivering
- Record why a message was held: new held.reason column, migrated, shown in the API, CLI and UI
- Accept an empty message in the held queue
- Document who may post
2026-11-09 20:23:07 +01:00
tgiachi
ba6c37bc21 feat(mbox,store): mbox reader, batch archiving and thread rebuilding
- Add internal/mbox: streaming reader with strict From_ detection, mboxrd unescaping, size cap and envelope dates
- Add Store.ArchiveBatch: many messages in one transaction
- Add Store.Rethread: rebuild threads from In-Reply-To whatever the insertion order, tolerating cycles
2026-11-09 17:13:56 +01:00
tgiachi
e6713962f9 feat(queue): background worker with retries, backoff and bounded concurrency
- Add Worker.RunOnce, Run and Wake over the durable outbox
- Send prefix plus shared message per recipient with a per-send timeout
- Retry temporary failures after 1m, 5m, 15m, 1h, 3h, then every 6h; give up after 3 days
- Drop permanent failures; count a hard bounce only for bad-mailbox replies
- Leave interrupted sends queued on shutdown
- Share the bounce threshold between router and worker
2026-11-09 15:47:14 +01:00
tgiachi
092cf4c350 feat(store): durable outbox with shared bodies and retry scheduling
- Add outbox_msgs and outbox tables
- Enqueue every recipient of a post in one transaction
- Add due, list, complete, retry and stats queries
- Delete the shared body with the last recipient
2026-11-09 15:45:19 +01:00
tgiachi
8253dd608e feat(router): archive accepted posts and add per-recipient unsubscribe links
- Archive the original message on delivery and on release, never when held or rejected
- Log archive failures instead of blocking delivery
- Add List-Unsubscribe per recipient: signed one-click URL and the +leave address
- Add signed unsubscribe tokens bound to their purpose
- Fix the mailto unsubscribe address, +unsubscribe was not a known command
2026-11-09 15:20:01 +01:00
tgiachi
d8275aaa1f feat(store,api): archive with full-text search, threading and archive policy
- Add archive table with FTS5 index kept in sync by triggers
- Deduplicate by Message-ID per list so MTA retries do not repeat posts
- Thread messages through In-Reply-To within the same list
- Quote every search word so user input is never FTS syntax
- Add archive_policy (public, private, off) with migration
- Add API to list, search, read and delete archived messages
2026-11-09 15:17:13 +01:00
tgiachi
9eb4d14a70 feat(store): schema v2 with moderation, pending requests and bounce state
- Add member id, disabled flag and bounce counter
- Add per-list post and join policies
- Add held message queue
- Add expiring, rate-limited pending subscribe and unsubscribe tokens
- Add persistent HMAC secret
- Migrate databases created by milestone 1
2026-11-09 15:03:08 +01:00
tgiachi
e1b0c7666c feat(store): list update, delete, listing and member removal
- Add ListLists, GetList, UpdateList, DeleteList, RemoveMember
- Add ErrExists for duplicate lists
2026-11-09 14:57:13 +01:00
tgiachi
015e0bbd0f feat(store): SQLite store for lists and members
- Add schema with lists and members tables
- Add CreateList, FindList, AddMember, IsMember, Members
- Normalize addresses to lowercase
2026-11-09 14:46:35 +01:00