diff --git a/README.md b/README.md index 61453bc..d635e9a 100644 --- a/README.md +++ b/README.md @@ -10,9 +10,11 @@ A modern, single-binary mailing list manager: an evolution of Mailman. and a background worker sends it with retries (1m, 5m, 15m, 1h, 3h, then every 6h, for up to 3 days). Relay outages and restarts lose nothing. - Self-service subscribe and unsubscribe by email with double opt-in: - send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation. + send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation. A reply works once, a request + expires after 72 hours, and a confirmation that could not be applied (closed list, database error) can be replied to again. + A member disabled by bounces can rejoin the same way. - Moderated lists: posts wait in a queue that the API can approve or discard. -- VERP bounces: every recipient gets a signed return address; 3 hard bounces disable a member. +- VERP bounces: every recipient gets a signed return address; 3 hard bounces within 30 days disable a member. - Archive with full-text search (SQLite FTS5) and threading. Each list has an `archive_policy`: `private` (default, API only), `public` (web pages) or `off`. Public pages hide email addresses. @@ -41,12 +43,18 @@ Three listeners, each configurable: | `web` | `127.0.0.1:8081` | public archive and unsubscribe pages (`""` disables) | Put the `web` listener behind a reverse proxy with TLS and set `base_url` to its public URL: -the unsubscribe links in every message use it. Keep the `api` listener private. +the unsubscribe links in every message use it. It must be an `https://` URL (RFC 8058 one-click unsubscribe needs +it; `http://` is accepted only for `localhost`, `127.0.0.1` and `::1`) with a host and no credentials, query or fragment; +the daemon refuses to start otherwise. Leave it empty to turn the one-click links off. Keep the `api` listener private. ``` curl -H "Authorization: Bearer $PARSIMONY_API_TOKEN" -d '{"name":"dev","domain":"example.org","title":"Dev"}' localhost:8080/api/v1/lists ``` +A list `name` is 1 to 30 characters of `a-z`, `0-9`, `.`, `_` and `-` (upper case is lowered, no dot at either end), the `domain` +a host name, and the `title` one line of at most 200 characters without control characters; anything else is a 400. +The title is encoded (RFC 2047) in the `List-Id` header when it is not ASCII. + ## Command line The same binary is an administration client. It talks to the REST API, so it works against a local or remote daemon. @@ -70,6 +78,10 @@ parsimony help `` is an address or a numeric id. Destructive commands (`list rm`, `archive rm`) need `--yes`. Exit status: 0 success, 1 failure (including lines that failed in `member import`), 2 usage error. +The client refuses to send the token over plain `http://` to any host other than `localhost`, `127.0.0.1` or `::1` +(usage error, exit 2): put the API behind HTTPS, or pass `--insecure` if you really mean it. +Subjects, senders, message bodies and error texts that come from mail are printed with control characters +(escape sequences) shown as `\x1b`-style text instead of being sent to the terminal; `--json` output is already escaped. `parsimony serve -config FILE` runs the daemon; the old `parsimony -config FILE` form still works. In the container: `docker exec parsimony /parsimony list ls`. @@ -183,9 +195,22 @@ It answers `451` only when the database itself fails. delivery and counts as a hard bounce for the member (3 disable it). Nothing else counts: not "relay access denied", not a refused sender (`5.1.7`, `5.1.8`, or any code answering `MAIL FROM`), not a refusal after `DATA`. Those deliveries are kept as failed. +- A bounce that comes back later to the member's return address counts only if it is a real delivery status report + (RFC 3464: `multipart/report` with a `message/delivery-status` part) with `Action: failed` and one of the same + codes. A spam or policy rejection (`5.7.x`), a full mailbox (`5.2.x`), a delay, a `4.x.x`, or a `Status:` line in a + plain or quoted message does not count. +- Hard bounces count for 30 days: when the last one is older, the next one starts again from 1. A disabled member + stays disabled until it is added again. - A relay that accepted the message is never asked again because `QUIT` went wrong. -- Delivery is at-least-once: a crash between the relay accepting a message and the queue entry being - removed can send one message to one recipient twice. +- The same post is queued once. If the `250` for a post never reaches Postfix, Postfix delivers the post again; a list + remembers each post it queued for an hour (by `Message-ID`, or by a hash of the whole message when it has none), + answers `250` to a copy and does not queue it again (the log says so). An approved held post is always delivered and is + remembered too, so a copy that arrives later is not. +- On stop, the daemon refuses new connections and lets the LMTP sessions and API or web requests in flight finish, so a + post being received gets its `250`. After 30 seconds it cuts whatever is still open. +- What remains is at-least-once delivery across the relay: a crash between the relay accepting a message and the queue + entry being removed can send one message to one recipient twice. A copy of a post that arrives more than an hour later + is also delivered again. - `GET /api/v1/queue` and the Queue page show what is pending or failing, and a stuck delivery can be discarded. ## Admin UI @@ -249,10 +274,23 @@ lists.example.org lmtp:inet:127.0.0.1:2424 and `Reply-To` come from the header; so a post whose header names someone else (or has no usable `From`, or several addresses) is held for moderation instead of delivered, with the reason shown to the admin (API, CLI, UI). Let Postfix reject forged senders too (SPF, DKIM, DMARC checks on inbound mail). +- **Loops and robots.** Mail that carries the list's own `List-Id` (a loop) or is a genuine auto-reply + (`Auto-Submitted: auto-replied`, `Precedence: auto_reply`) is answered `250` and dropped without a trace, so the + sender's server never bounces it. Other machine-generated mail (`Auto-Submitted: auto-generated`, cron, CI and + monitoring posts) is a normal post: the membership, `From` and moderation checks apply. The `join`, `leave` and + `confirm` addresses ignore (`250`, nothing happens, the token is kept) a mail with a null sender, any + `Auto-Submitted` other than `no`, a `Precedence` of `auto_reply`, `bulk` or `junk`, or the list's own `List-Id`. + This stops an auto-responder from confirming a subscription someone forged in your name. - **The moderation queue is bounded.** A list holds at most 500 messages and 200 MiB, and one held message at most 10 MiB; a full queue answers `452` (the sender retries later), a message over 10 MiB `552`. Messages nobody moderated for 30 days are dropped. Listings (API, CLI, UI) show a summary and never read the bodies. The held view shows the From header readers would see next to the real sender and flags a difference. +- **Hostile messages and slow clients.** The archive reads at most 10 levels of nested multipart and 100 parts per + message, then uses what it found (the post is still delivered). A `Date` header more than 24 hours ahead is replaced + by the arrival time, and one before 1990 is ignored, so a message cannot pin itself to the top of the archive. The + HTTP servers (API + admin UI, and the public web) cut a request that takes more than 30 s to read, a response that + takes more than 60 s to write, and a connection idle for 120 s; the archive import is the exception and may stream + for up to 30 minutes. - **API token.** `parsimony serve` refuses a token shorter than 24 characters or containing whitespace (`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy diff --git a/admin/src/tabs/HeldTab.test.tsx b/admin/src/tabs/HeldTab.test.tsx index 5abdc84..e1e8c3b 100644 --- a/admin/src/tabs/HeldTab.test.tsx +++ b/admin/src/tabs/HeldTab.test.tsx @@ -76,15 +76,21 @@ describe('held tab', () => { expect(await screen.findByText(/nothing is waiting/i)).toBeInTheDocument() }) - it('explains how messages get held when posting is not moderated', async () => { - await setup([], aList({ id: 1, post_policy: 'members' })) - expect(await screen.findByText(/set posting to moderated/i)).toBeInTheDocument() + it('explains both ways a message gets held, whatever the posting policy', async () => { + for (const post_policy of ['members', 'moderated'] as const) { + const { unmount } = await setup([], aList({ id: 1, post_policy })) + const hint = await screen.findByText(/messages are held when/i) + expect(hint).toHaveTextContent(/posting is set to moderated/i) + expect(hint).toHaveTextContent(/from header does not match the sender/i) + expect(hint).not.toHaveTextContent(/only/i) + unmount() + } }) - it('does not nag about the policy when the list is moderated', async () => { - await setup([]) - await screen.findByText(/nothing is waiting/i) - expect(screen.queryByText(/set posting to moderated/i)).not.toBeInTheDocument() + it('shows no hint when there are held messages', async () => { + await setup([held(1)]) + await screen.findByText('Subject 1') + expect(screen.queryByText(/messages are held when/i)).not.toBeInTheDocument() }) it('approves a message: it is sent and leaves the queue', async () => { diff --git a/admin/src/tabs/HeldTab.tsx b/admin/src/tabs/HeldTab.tsx index fa3c0e6..4b43188 100644 --- a/admin/src/tabs/HeldTab.tsx +++ b/admin/src/tabs/HeldTab.tsx @@ -51,9 +51,10 @@ export function HeldTab() { {held.data && items.length === 0 && (

Nothing is waiting for approval.

- {list.post_policy !== 'moderated' && ( -

Messages are held only when you set Posting to moderated in Settings.

- )} +

+ Messages are held when Posting is set to moderated in Settings, or when the From header does not match + the sender. +

)} {items.length > 0 && ( diff --git a/docs/openapi.yaml b/docs/openapi.yaml index d5b334b..ad0b348 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -50,8 +50,8 @@ components: properties: address: {type: string} display_name: {type: string} - disabled: {type: boolean, description: Disabled after 3 hard bounces; re-adding the member re-enables it} - bounce_count: {type: integer} + disabled: {type: boolean, description: Disabled after 3 hard bounces within 30 days; re-adding the member re-enables it} + bounce_count: {type: integer, description: Recent hard bounces (restarts at 1 when the last one is older than 30 days)} Held: type: object properties: @@ -99,9 +99,19 @@ paths: type: object required: [name, domain] properties: - name: {type: string} - domain: {type: string} - title: {type: string} + name: + type: string + minLength: 1 + maxLength: 30 + pattern: '^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$' + description: Local part of the list address, stored lowercase. Letters, digits, '.', '_' and '-' only, no dot at either end or two in a row. + domain: + type: string + description: ASCII host name (labels of 1 to 63 letters, digits and hyphens). + title: + type: string + maxLength: 200 + description: A single line without control characters. Defaults to the name. responses: '201': description: Created @@ -131,7 +141,7 @@ paths: schema: type: object properties: - title: {type: string} + title: {type: string, minLength: 1, maxLength: 200, description: A single line without control characters.} post_policy: {type: string, enum: [members, moderated]} join_policy: {type: string, enum: [open, closed]} archive_policy: {type: string, enum: [public, private, off]} diff --git a/internal/api/api.go b/internal/api/api.go index b40b9ab..eb03fd7 100644 --- a/internal/api/api.go +++ b/internal/api/api.go @@ -12,6 +12,8 @@ import ( "strconv" "strings" "time" + "unicode" + "unicode/utf8" "codeberg.org/squidd/parsimony/internal/archive" "codeberg.org/squidd/parsimony/internal/store" @@ -40,29 +42,44 @@ func New(s *store.Store, token string, rel Releaser) http.Handler { func newHandler(s *store.Store, token string, rel Releaser, now func() time.Time) http.Handler { srv := &server{store: s, token: token, releaser: rel, lim: newLimiter(now)} mux := http.NewServeMux() - mux.HandleFunc("GET /api/v1/health", func(w http.ResponseWriter, _ *http.Request) { - writeJSON(w, http.StatusOK, map[string]string{"status": "ok"}) - }) - mux.Handle("GET /api/v1/lists", srv.auth(srv.listLists)) - mux.Handle("POST /api/v1/lists", srv.auth(srv.createList)) - mux.Handle("GET /api/v1/lists/{id}", srv.auth(srv.getList)) - mux.Handle("PATCH /api/v1/lists/{id}", srv.auth(srv.updateList)) - mux.Handle("DELETE /api/v1/lists/{id}", srv.auth(srv.deleteList)) - mux.Handle("GET /api/v1/lists/{id}/members", srv.auth(srv.listMembers)) - mux.Handle("POST /api/v1/lists/{id}/members", srv.auth(srv.addMember)) - mux.Handle("DELETE /api/v1/lists/{id}/members/{address}", srv.auth(srv.removeMember)) - mux.Handle("GET /api/v1/lists/{id}/archive", srv.auth(srv.listArchive)) - mux.Handle("POST /api/v1/lists/{id}/archive/import", srv.auth(srv.importArchive)) - mux.Handle("GET /api/v1/archive/{id}", srv.auth(srv.getArchived)) - mux.Handle("DELETE /api/v1/archive/{id}", srv.auth(srv.deleteArchived)) - mux.Handle("GET /api/v1/queue", srv.auth(srv.getQueue)) - mux.Handle("DELETE /api/v1/queue/{id}", srv.auth(srv.discardQueued)) - mux.Handle("GET /api/v1/lists/{id}/held", srv.auth(srv.listHeld)) - mux.Handle("POST /api/v1/held/{id}/approve", srv.auth(srv.approveHeld)) - mux.Handle("DELETE /api/v1/held/{id}", srv.auth(srv.discardHeld)) + for _, rt := range srv.routes() { + mux.Handle(rt.pattern, rt.handler) + } return mux } +type route struct { + pattern string + handler http.Handler +} + +// routes is the only place routes are declared. Everything but the health check goes through auth. +func (s *server) routes() []route { + public := func(h http.HandlerFunc) http.Handler { return h } + return []route{ + {"GET /api/v1/health", public(func(w http.ResponseWriter, _ *http.Request) { + writeJSON(w, http.StatusOK, map[string]string{"status": "ok"}) + })}, + {"GET /api/v1/lists", s.auth(s.listLists)}, + {"POST /api/v1/lists", s.auth(s.createList)}, + {"GET /api/v1/lists/{id}", s.auth(s.getList)}, + {"PATCH /api/v1/lists/{id}", s.auth(s.updateList)}, + {"DELETE /api/v1/lists/{id}", s.auth(s.deleteList)}, + {"GET /api/v1/lists/{id}/members", s.auth(s.listMembers)}, + {"POST /api/v1/lists/{id}/members", s.auth(s.addMember)}, + {"DELETE /api/v1/lists/{id}/members/{address}", s.auth(s.removeMember)}, + {"GET /api/v1/lists/{id}/archive", s.auth(s.listArchive)}, + {"POST /api/v1/lists/{id}/archive/import", s.auth(s.importArchive)}, + {"GET /api/v1/archive/{id}", s.auth(s.getArchived)}, + {"DELETE /api/v1/archive/{id}", s.auth(s.deleteArchived)}, + {"GET /api/v1/queue", s.auth(s.getQueue)}, + {"DELETE /api/v1/queue/{id}", s.auth(s.discardQueued)}, + {"GET /api/v1/lists/{id}/held", s.auth(s.listHeld)}, + {"POST /api/v1/held/{id}/approve", s.auth(s.approveHeld)}, + {"DELETE /api/v1/held/{id}", s.auth(s.discardHeld)}, + } +} + func (s *server) auth(next http.HandlerFunc) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ip := clientIP(r) @@ -169,6 +186,53 @@ func (s *server) listLists(w http.ResponseWriter, _ *http.Request) { writeJSON(w, http.StatusOK, out) } +const ( + maxNameLen = 30 // keeps list+confirm- and list+bounce-- within 64 characters + maxTitleLen = 200 + titleHint = "title must be a non-empty single line of at most 200 characters, without control characters" +) + +// validName reports whether name (already lowercased) works as the local part of a list address. +func validName(name string) bool { + if name == "" || len(name) > maxNameLen || strings.HasPrefix(name, ".") || strings.HasSuffix(name, ".") || strings.Contains(name, "..") { + return false + } + for _, c := range name { + if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '.' && c != '_' && c != '-' { + return false + } + } + return true +} + +// validDomain reports whether d (already lowercased) is an ASCII host name: dot-separated labels of +// 1 to 63 letters, digits and hyphens, none starting or ending with a hyphen. +func validDomain(d string) bool { + if d == "" || len(d) > 253 { + return false + } + for _, label := range strings.Split(d, ".") { + if label == "" || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' { + return false + } + for _, c := range label { + if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '-' { + return false + } + } + } + return true +} + +// validTitle reports whether t is a usable list title: not empty, at most maxTitleLen characters, +// no line breaks or other control characters (it goes into mail headers). +func validTitle(t string) bool { + if t == "" || utf8.RuneCountInString(t) > maxTitleLen { + return false + } + return !strings.ContainsFunc(t, unicode.IsControl) +} + func (s *server) createList(w http.ResponseWriter, r *http.Request) { var in struct { Name string `json:"name"` @@ -178,13 +242,22 @@ func (s *server) createList(w http.ResponseWriter, r *http.Request) { if !decodeBody(w, r, &in) { return } - in.Name, in.Domain = strings.TrimSpace(in.Name), strings.TrimSpace(in.Domain) - if in.Name == "" || in.Domain == "" || strings.ContainsAny(in.Name+in.Domain, "@+ \r\n") { - writeError(w, http.StatusBadRequest, "name and domain are required and must not contain '@', '+' or spaces") + typedName := strings.TrimSpace(in.Name) + in.Name, in.Domain = strings.ToLower(typedName), strings.ToLower(strings.TrimSpace(in.Domain)) + if !validName(in.Name) { + writeError(w, http.StatusBadRequest, "name must be 1 to 30 characters of a-z, 0-9, '.', '_' or '-', without a dot at either end or two dots in a row") return } - if in.Title == "" { - in.Title = in.Name + if !validDomain(in.Domain) { + writeError(w, http.StatusBadRequest, "domain must be a host name such as lists.example.org") + return + } + if in.Title = strings.TrimSpace(in.Title); in.Title == "" { + in.Title = typedName + } + if !validTitle(in.Title) { + writeError(w, http.StatusBadRequest, titleHint) + return } l, err := s.store.CreateList(in.Name, in.Domain, in.Title) if err != nil { @@ -233,8 +306,8 @@ func (s *server) updateList(w http.ResponseWriter, r *http.Request) { } if in.Title != nil { title := strings.TrimSpace(*in.Title) - if title == "" || strings.ContainsAny(title, "\r\n") { - writeError(w, http.StatusBadRequest, "title must be a non-empty single line") + if !validTitle(title) { + writeError(w, http.StatusBadRequest, titleHint) return } l.Title = title @@ -561,6 +634,10 @@ func (s *server) discardQueued(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) } +// importTimeout is how long one archive import may read its body and write its +// summary, in place of the server's default read and write timeouts. +const importTimeout = 30 * time.Minute + // importArchive reads an mbox from the request body, streaming, and archives it. // Failed messages are reported in the summary; the request itself succeeds. func (s *server) importArchive(w http.ResponseWriter, r *http.Request) { @@ -572,6 +649,11 @@ func (s *server) importArchive(w http.ResponseWriter, r *http.Request) { storeError(w, err) return } + // A big mbox streams for longer than the server's default timeouts. Errors + // are ignored: a ResponseWriter that cannot extend deadlines has none. + rc := http.NewResponseController(w) + _ = rc.SetReadDeadline(time.Now().Add(importTimeout)) + _ = rc.SetWriteDeadline(time.Now().Add(importTimeout)) sum, err := archive.ImportMbox(s.store, id, r.Body, time.Now()) if err != nil { writeError(w, http.StatusBadRequest, fmt.Sprintf("import stopped after %d message(s): %v", sum.Imported, err)) diff --git a/internal/api/api_test.go b/internal/api/api_test.go index ac1e3ad..685971c 100644 --- a/internal/api/api_test.go +++ b/internal/api/api_test.go @@ -152,6 +152,37 @@ func TestCreateListValidation(t *testing.T) { `{"name":"a+b","domain":"example.org"}`, `{"name":"dev","domain":""}`, `{"name":"dev","domain":"example.org","bogus":1}`, + // Names that break an address: separators, brackets, whitespace, control characters, dots at the ends. + `{"name":"a/b","domain":"example.org"}`, + `{"name":"ab","domain":"example.org"}`, + `{"name":"a\tb","domain":"example.org"}`, + `{"name":"a b","domain":"example.org"}`, + `{"name":"a\"b","domain":"example.org"}`, + `{"name":"caffè","domain":"example.org"}`, + `{"name":".dev","domain":"example.org"}`, + `{"name":"dev.","domain":"example.org"}`, + `{"name":"a..b","domain":"example.org"}`, + `{"name":"a{b","domain":"example.org"}`, + `{"name":"","domain":"example.org","title":"Dev"}`, + `{"name":" ","domain":"example.org","title":"Dev"}`, + `{"name":"` + strings.Repeat("a", 31) + `","domain":"example.org"}`, + // Domains that are not host names. + `{"name":"dev","domain":"exa mple.org"}`, + `{"name":"dev","domain":"example..org"}`, + `{"name":"dev","domain":"-example.org"}`, + `{"name":"dev","domain":"example-.org"}`, + `{"name":"dev","domain":"example.org."}`, + `{"name":"dev","domain":"ex_ample.org"}`, + `{"name":"dev","domain":"exämple.org"}`, + `{"name":"dev","domain":"a/b.org"}`, + `{"name":"dev","domain":"` + strings.Repeat("a", 64) + `.org"}`, + // Titles that are not one clean line. + `{"name":"dev","domain":"example.org","title":"Dev\r\nBcc: x@y.com"}`, + `{"name":"dev","domain":"example.org","title":"Dev\nlist"}`, + `{"name":"dev","domain":"example.org","title":"Dev\tlist"}`, + `{"name":"dev","domain":"example.org","title":"Dev\u0000list"}`, + `{"name":"dev","domain":"example.org","title":"` + strings.Repeat("x", 201) + `"}`, } { rec := do(t, h, "POST", "/api/v1/lists", body, true) if rec.Code != 400 { @@ -685,3 +716,127 @@ func TestQueueShowsFailedDeliveriesApartFromPendingOnes(t *testing.T) { } expect(t, do(t, h, "DELETE", "/api/v1/queue/"+strconv.FormatInt(due[2].ID, 10), "", true), 204) } + +// TestEveryRouteNeedsTheToken walks the route table New registers, so a route added later is +// covered without touching this test: all of them but the health check answer 401 without a token +// and something else with one. +func TestEveryRouteNeedsTheToken(t *testing.T) { + s, err := store.Open(":memory:") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { s.Close() }) + h := New(s, token, nil) + + rs := (&server{}).routes() + if len(rs) < 17 { + t.Fatalf("only %d routes in the table", len(rs)) + } + for i, rt := range rs { + method, pattern, _ := strings.Cut(rt.pattern, " ") + path := strings.NewReplacer("{id}", "1", "{address}", "a@example.com").Replace(pattern) + t.Run(rt.pattern, func(t *testing.T) { + // Each request comes from its own address, so the lockout never gets in the way. + addr := fmt.Sprintf("192.0.2.%d:4000", i+1) + call := func(auth bool) int { + req := httptest.NewRequest(method, path, strings.NewReader("{}")) + req.RemoteAddr = addr + if auth { + req.Header.Set("Authorization", "Bearer "+token) + } + rec := httptest.NewRecorder() + h.ServeHTTP(rec, req) + return rec.Code + } + if pattern == "/api/v1/health" { + if got := call(false); got != 200 { + t.Fatalf("health without a token: %d", got) + } + return + } + if got := call(false); got != 401 { + t.Errorf("no token: status %d, want 401", got) + } + if got := call(true); got == 401 { + t.Errorf("good token: status %d", got) + } + }) + } +} + +func TestCreateListAcceptsTheLimits(t *testing.T) { + h := newServer(t) + for _, body := range []string{ + `{"name":"` + strings.Repeat("a", 30) + `","domain":"example.org","title":"` + strings.Repeat("è", 200) + `"}`, + `{"name":"dev-team_1.eu","domain":"lists.sub-domain.example.org"}`, + `{"name":"Dev2","domain":"localhost","title":"Développeurs ☕"}`, + } { + if rec := do(t, h, "POST", "/api/v1/lists", body, true); rec.Code != 201 { + t.Errorf("%.80q: status %d, body %s", body, rec.Code, rec.Body.String()) + } + } +} + +// A title with a line break would make every later post to the list fail to rewrite, so the list +// must not come into existence at all. +func TestCreateListWithABadTitleLeavesNothingBehind(t *testing.T) { + h := newServer(t) + expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev\r\nX: y"}`, true), 400) + rec := do(t, h, "GET", "/api/v1/lists", "", true) + if strings.TrimSpace(rec.Body.String()) != "[]" { + t.Fatalf("the refused list was created: %s", rec.Body.String()) + } + // The name is free for a good request. + expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, true), 201) +} + +func TestCreateListLowercasesNameAndDomain(t *testing.T) { + h := newServer(t) + rec := do(t, h, "POST", "/api/v1/lists", `{"name":" Dev ","domain":"Example.ORG"}`, true) + expect(t, rec, 201) + var l listJSON + decode(t, rec, &l) + if l.Address != "dev@example.org" || l.Title != "Dev" { + t.Fatalf("%+v", l) + } +} + +func TestUpdateListTitleUsesTheSameRules(t *testing.T) { + h := newServer(t) + createList(t, h) + for _, title := range []string{`"a\tb"`, `"` + strings.Repeat("x", 201) + `"`, `" "`, `"a\r\nb"`} { + expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":`+title+`}`, true), 400) + } + expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":"`+strings.Repeat("x", 200)+`"}`, true), 200) +} + +func TestValidDomainLengthLimits(t *testing.T) { + l63 := strings.Repeat("a", 63) + d253 := l63 + "." + l63 + "." + l63 + "." + strings.Repeat("a", 61) + if len(d253) != 253 || !validDomain(d253) { + t.Fatalf("a 253-character domain is valid (%d)", len(d253)) + } + if validDomain(d253 + "a") { + t.Fatal("254 characters is too long") + } + if !validDomain(l63+".org") || validDomain("") || validName("") { + t.Fatal("63-character labels are valid, empty names and domains are not") + } +} + +func TestCreateListTrimsTheTitle(t *testing.T) { + h := newServer(t) + rec := do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":" Dev team \t"}`, true) + expect(t, rec, 201) + var l listJSON + decode(t, rec, &l) + if l.Title != "Dev team" { + t.Fatalf("title = %q", l.Title) + } + rec = do(t, h, "POST", "/api/v1/lists", `{"name":"ops","domain":"example.org","title":" "}`, true) + expect(t, rec, 201) + decode(t, rec, &l) + if l.Title != "ops" { + t.Fatalf("a blank title falls back to the name: %q", l.Title) + } +} diff --git a/internal/app/app.go b/internal/app/app.go index 8909328..5115942 100644 --- a/internal/app/app.go +++ b/internal/app/app.go @@ -26,9 +26,31 @@ import ( "codeberg.org/squidd/parsimony/internal/web" ) +// shutdownTimeout bounds how long Close waits for the sessions in flight before it cuts them. +var shutdownTimeout = 30 * time.Second + // nowFunc is the clock of the store (queue schedules, token expiry). Tests move it. var nowFunc = time.Now +// HTTP timeouts of the API + admin UI server and of the public web server, so +// a slow or silent client cannot pin a goroutine. The archive import extends +// its own deadlines (see api.importArchive). Variables so tests can shorten them. +var ( + httpReadTimeout = 30 * time.Second + httpWriteTimeout = 60 * time.Second + httpIdleTimeout = 120 * time.Second +) + +func newHTTPServer(h http.Handler) *http.Server { + return &http.Server{ + Handler: h, + ReadHeaderTimeout: 10 * time.Second, + ReadTimeout: httpReadTimeout, + WriteTimeout: httpWriteTimeout, + IdleTimeout: httpIdleTimeout, + } +} + // heloName is the EHLO name: the configured one, else this machine's host name. func heloName(configured string) string { if configured != "" { @@ -90,6 +112,51 @@ func lmtpIsOpen(network, addr string) bool { return ip == nil || !ip.IsLoopback() } +// connTracker remembers the open connections of a listener, so Close can cut the ones still open +// when the grace period ends: once Shutdown has begun, smtp.Server.Close does nothing. +type connTracker struct { + net.Listener + mu sync.Mutex + conns map[*trackedConn]struct{} +} + +func (t *connTracker) Accept() (net.Conn, error) { + c, err := t.Listener.Accept() + if err != nil { + return nil, err + } + tc := &trackedConn{Conn: c, t: t} + t.mu.Lock() + t.conns[tc] = struct{}{} + t.mu.Unlock() + return tc, nil +} + +// closeAll cuts every connection still open. +func (t *connTracker) closeAll() { + t.mu.Lock() + open := make([]*trackedConn, 0, len(t.conns)) + for c := range t.conns { + open = append(open, c) + } + t.mu.Unlock() + for _, c := range open { + c.Close() + } +} + +type trackedConn struct { + net.Conn + t *connTracker +} + +func (c *trackedConn) Close() error { + c.t.mu.Lock() + delete(c.t.conns, c) + c.t.mu.Unlock() + return c.Conn.Close() +} + // App is a running daemon. type App struct { // LMTPAddr and APIAddr are the addresses actually bound. @@ -98,14 +165,16 @@ type App struct { APIAddr string WebAddr string // empty when the public web is disabled - store *store.Store - lmtp *smtp.Server - lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it - web *http.Server - pub *http.Server - wg sync.WaitGroup - stop context.CancelFunc // stops the queue worker - once sync.Once + store *store.Store + lmtp *smtp.Server + lmtpOpen *connTracker // the LMTP connections, cut by Close when the grace period ends + lmtpDone chan struct{} // closed when the LMTP Serve returns + lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it + web *http.Server + pub *http.Server + wg sync.WaitGroup + stop context.CancelFunc // stops the queue worker + once sync.Once } // releaser lets the API approve held messages through the router. @@ -196,7 +265,7 @@ func Start(cfg config.Config, apiToken string) (*App, error) { mux := http.NewServeMux() mux.Handle("/api/", api.New(st, apiToken, releaser{rt})) mux.Handle("/", adminui.Embedded()) - a.web = &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second} + a.web = newHTTPServer(mux) } var pubLn net.Listener @@ -211,9 +280,12 @@ func Start(cfg config.Config, apiToken string) (*App, error) { } a.WebAddr = pubLn.Addr().String() a.lns = append(a.lns, pubLn) - a.pub = &http.Server{Handler: web.New(st, rt), ReadHeaderTimeout: 10 * time.Second} + a.pub = newHTTPServer(web.New(st, rt)) } + a.lmtpOpen = &connTracker{Listener: serveLn, conns: map[*trackedConn]struct{}{}} + a.lmtpDone = make(chan struct{}) + var workerCtx context.Context workerCtx, a.stop = context.WithCancel(context.Background()) a.wg.Add(1) @@ -224,7 +296,8 @@ func Start(cfg config.Config, apiToken string) (*App, error) { a.wg.Add(1) go func() { defer a.wg.Done() - if err := a.lmtp.Serve(serveLn); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) { + defer close(a.lmtpDone) + if err := a.lmtp.Serve(a.lmtpOpen); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) { log.Printf("lmtp: %v", err) } }() @@ -249,18 +322,26 @@ func Start(cfg config.Config, apiToken string) (*App, error) { return a, nil } -// Close stops the servers and closes the database. It is safe to call twice. +// Close stops the daemon and closes the database. It is safe to call twice. It stops accepting +// connections, then lets the LMTP sessions and HTTP requests in flight finish, so a post being +// received is queued and gets its 250 (a cut reply makes the MTA send the post again). After +// shutdownTimeout it cuts whatever is still open. func (a *App) Close() { a.once.Do(func() { + ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout) + defer cancel() for _, ln := range a.lns { ln.Close() } - a.lmtp.Close() - if a.web != nil { - a.web.Close() + <-a.lmtpDone // Serve has returned: no connection is added while Shutdown waits + if err := a.lmtp.Shutdown(ctx); errors.Is(err, context.DeadlineExceeded) { + log.Printf("lmtp: sessions still open after %v, closing them", shutdownTimeout) + a.lmtpOpen.closeAll() } - if a.pub != nil { - a.pub.Close() + for _, srv := range []*http.Server{a.web, a.pub} { + if srv != nil && srv.Shutdown(ctx) != nil { + srv.Close() + } } a.stop() // in-flight sends are interrupted and stay queued a.wg.Wait() diff --git a/internal/app/app_test.go b/internal/app/app_test.go index 832937c..e96ea79 100644 --- a/internal/app/app_test.go +++ b/internal/app/app_test.go @@ -1,6 +1,7 @@ package app import ( + "bufio" "bytes" "crypto/ecdsa" "crypto/elliptic" @@ -12,6 +13,7 @@ import ( "encoding/base64" "encoding/pem" "errors" + "fmt" "io" "log" "math/big" @@ -120,13 +122,17 @@ func testConfig(t *testing.T, relay string) config.Config { return c } +// client opens one connection per request: a pooled connection the transport dialed and never +// used would hold an HTTP server's graceful shutdown for 5 s. +var client = &http.Client{Transport: &http.Transport{DisableKeepAlives: true}} + func call(t *testing.T, a *App, method, path, body, token string) int { t.Helper() req, _ := http.NewRequest(method, "http://"+a.APIAddr+path, strings.NewReader(body)) if token != "" { req.Header.Set("Authorization", "Bearer "+token) } - resp, err := http.DefaultClient.Do(req) + resp, err := client.Do(req) if err != nil { t.Fatal(err) } @@ -137,7 +143,7 @@ func call(t *testing.T, a *App, method, path, body, token string) int { func fetch(t *testing.T, method, url string) (int, string) { t.Helper() req, _ := http.NewRequest(method, url, nil) - resp, err := http.DefaultClient.Do(req) + resp, err := client.Do(req) if err != nil { t.Fatal(err) } @@ -353,6 +359,7 @@ func TestWebDisabledWhenListenIsEmpty(t *testing.T) { } func TestCloseDropsOpenConnections(t *testing.T) { + setShutdownTimeout(t, 200*time.Millisecond) relayAddr, _ := startSink(t) a, err := Start(testConfig(t, relayAddr), "tok") if err != nil { @@ -432,6 +439,193 @@ func deadAddr(t *testing.T) string { return addr } +func setShutdownTimeout(t *testing.T, d time.Duration) { + old := shutdownTimeout + shutdownTimeout = d + t.Cleanup(func() { shutdownTimeout = old }) +} + +// waitRefused waits until the LMTP listener no longer accepts connections. +func waitRefused(t *testing.T, addr string) { + t.Helper() + deadline := time.Now().Add(5 * time.Second) + for { + c, err := net.Dial("tcp", addr) + if err != nil { + return + } + c.Close() + if time.Now().After(deadline) { + t.Fatal("the LMTP listener still accepts connections") + } + time.Sleep(5 * time.Millisecond) + } +} + +func TestCloseLetsAnLMTPDeliveryInFlightFinish(t *testing.T) { + cfg := testConfig(t, deadAddr(t)) // the post stays queued, where it can be counted + a, err := Start(cfg, "tok") + if err != nil { + t.Fatal(err) + } + call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org"}`, "tok") + call(t, a, "POST", "/api/v1/lists/1/members", `{"address":"alice@example.com"}`, "tok") + + conn, err := net.Dial("tcp", a.LMTPAddr) + if err != nil { + t.Fatal(err) + } + c := smtp.NewClientLMTP(conn) + defer c.Close() + if err := c.Hello("localhost"); err != nil { + t.Fatal(err) + } + if err := c.Mail("alice@example.com", nil); err != nil { + t.Fatal(err) + } + if err := c.Rcpt("dev@example.org", nil); err != nil { + t.Fatal(err) + } + w, err := c.Data() + if err != nil { + t.Fatal(err) + } + w.Write([]byte("From: alice@example.com\r\nSubject: in flight\r\nMessage-ID: \r\n\r\nbody\r\n")) + + // The daemon stops while the post is being sent. + closed := make(chan struct{}) + go func() { a.Close(); close(closed) }() + waitRefused(t, a.LMTPAddr) + if _, err := w.CloseWithLMTPResponse(); err != nil { + t.Fatalf("the post in flight must get its 250: %v", err) + } + if err := c.Quit(); err != nil { + t.Fatalf("quit: %v", err) + } + select { + case <-closed: + case <-time.After(5 * time.Second): + t.Fatal("Close must return once the session is over") + } + peek, err := store.Open(cfg.DB) + if err != nil { + t.Fatal(err) + } + defer peek.Close() + if st, _ := peek.OutboxStats(); st.Pending != 1 { + t.Fatalf("the post must be queued: %+v", st) + } +} + +func TestCloseCutsAHungLMTPClientAfterTheGracePeriod(t *testing.T) { + setShutdownTimeout(t, 200*time.Millisecond) + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + conn, err := net.Dial("tcp", a.LMTPAddr) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + c := smtp.NewClientLMTP(conn) + if err := c.Hello("localhost"); err != nil { + t.Fatal(err) + } + if err := c.Mail("alice@example.com", nil); err != nil { + t.Fatal(err) + } + if err := c.Rcpt("dev@example.org", nil); err != nil { + t.Fatal(err) + } + if _, err := c.Data(); err != nil { + t.Fatal(err) + } + // The client never ends the message. + start := time.Now() + closed := make(chan struct{}) + go func() { a.Close(); close(closed) }() + select { + case <-closed: + case <-time.After(5 * time.Second): + t.Fatal("Close must not wait for a hung client beyond the grace period") + } + if d := time.Since(start); d < 150*time.Millisecond { + t.Fatalf("Close returned after %v: it must give the session its grace period", d) + } + conn.SetReadDeadline(time.Now().Add(2 * time.Second)) + buf := make([]byte, 512) + for { + if _, err := conn.Read(buf); err != nil { + if ne, ok := err.(net.Error); ok && ne.Timeout() { + t.Fatal("the hung connection is still open after Close") + } + break + } + } +} + +func TestCloseLetsAnAPIRequestInFlightFinish(t *testing.T) { + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + conn, err := net.Dial("tcp", a.APIAddr) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + body := `{"name":"dev","domain":"example.org"}` + fmt.Fprintf(conn, "POST /api/v1/lists HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s", len(body), body[:10]) + time.Sleep(100 * time.Millisecond) // the handler is now reading the body + + closed := make(chan struct{}) + go func() { a.Close(); close(closed) }() + waitRefused(t, a.APIAddr) + io.WriteString(conn, body[10:]) + conn.SetReadDeadline(time.Now().Add(5 * time.Second)) + resp, err := http.ReadResponse(bufio.NewReader(conn), nil) + if err != nil { + t.Fatalf("the request in flight must get its answer: %v", err) + } + resp.Body.Close() + if resp.StatusCode != http.StatusCreated { + t.Fatalf("status = %d", resp.StatusCode) + } + select { + case <-closed: + case <-time.After(5 * time.Second): + t.Fatal("Close must return once the request is over") + } +} + +func TestFinishedLMTPSessionsAreForgotten(t *testing.T) { + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { a.Close() }) + for i := 0; i < 3; i++ { + lmtpSend(t, a.LMTPAddr, "alice@example.com", "nope@example.org", "Subject: x\r\n\r\nx\r\n") + } + deadline := time.Now().Add(5 * time.Second) + for { + a.lmtpOpen.mu.Lock() + n := len(a.lmtpOpen.conns) + a.lmtpOpen.mu.Unlock() + if n == 0 { + break + } + if time.Now().After(deadline) { + t.Fatalf("%d closed sessions are still tracked", n) + } + time.Sleep(5 * time.Millisecond) + } +} + func TestQueuedMailSurvivesARestartAndIsRetriedWhenTheRelayReturns(t *testing.T) { var skew atomic.Int64 nowFunc = func() time.Time { return time.Now().Add(time.Duration(skew.Load())) } @@ -1004,3 +1198,115 @@ func TestTheServerNameSettingReachesTheTLSCheck(t *testing.T) { t.Fatal("nothing may be sent to a relay whose certificate does not match") } } + +// shortTimeouts makes the HTTP timeouts small enough to test. +func shortTimeouts(t *testing.T) { + t.Helper() + r, w, i := httpReadTimeout, httpWriteTimeout, httpIdleTimeout + httpReadTimeout, httpWriteTimeout, httpIdleTimeout = 300*time.Millisecond, 300*time.Millisecond, 300*time.Millisecond + t.Cleanup(func() { httpReadTimeout, httpWriteTimeout, httpIdleTimeout = r, w, i }) +} + +// stalled sends a request that promises a body and never finishes it, and +// reports whether the server hung up within a few seconds. +func stalled(t *testing.T, addr, req string) bool { + t.Helper() + c, err := net.Dial("tcp", addr) + if err != nil { + t.Fatal(err) + } + defer c.Close() + if _, err := c.Write([]byte(req)); err != nil { + t.Fatal(err) + } + c.SetReadDeadline(time.Now().Add(4 * time.Second)) + _, err = io.Copy(io.Discard, c) // returns nil on EOF, an error on our deadline + return err == nil +} + +func TestSlowRequestsAreCutOnBothServers(t *testing.T) { + shortTimeouts(t) + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { a.Close() }) + hdr := "POST %s HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: 500\r\n\r\n{\"name\":" + for name, c := range map[string]struct{ addr, path string }{ + "api": {a.APIAddr, "/api/v1/lists"}, + "web": {a.WebAddr, "/unsubscribe/tok"}, + } { + if !stalled(t, c.addr, strings.Replace(hdr, "%s", c.path, 1)) { + t.Errorf("%s: a stalled body was not cut", name) + } + } + // An idle keep-alive connection is closed too. + for name, addr := range map[string]string{"api": a.APIAddr, "web": a.WebAddr} { + if !stalled(t, addr, "GET /nothing HTTP/1.1\r\nHost: x\r\n\r\n") { + t.Errorf("%s: an idle connection was kept", name) + } + } +} + +// slowReader hands out its data in two halves, a pause apart. +type slowReader struct { + parts [][]byte + pause time.Duration +} + +func (s *slowReader) Read(p []byte) (int, error) { + if len(s.parts) == 0 { + return 0, io.EOF + } + if len(s.parts) == 1 { + time.Sleep(s.pause) + } + n := copy(p, s.parts[0]) + s.parts = s.parts[1:] + return n, nil +} + +func TestArchiveImportIsNotCutByTheDefaultTimeouts(t *testing.T) { + shortTimeouts(t) + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { a.Close() }) + call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, "tok") + + m1 := "From a@example.com Tue Jan 2 15:04:05 2024\nFrom: a@example.com\nSubject: one\nMessage-ID: <1@x>\n\nbody one\n\n" + m2 := "From a@example.com Tue Jan 2 15:04:06 2024\nFrom: a@example.com\nSubject: two\nMessage-ID: <2@x>\n\nbody two\n" + body := &slowReader{parts: [][]byte{[]byte(m1), []byte(m2)}, pause: 3 * httpReadTimeout} + req, _ := http.NewRequest("POST", "http://"+a.APIAddr+"/api/v1/lists/1/archive/import", body) + req.Header.Set("Authorization", "Bearer tok") + req.ContentLength = -1 // chunked: the body is streamed + resp, err := http.DefaultClient.Do(req) + if err != nil { + t.Fatalf("import: %v", err) + } + defer resp.Body.Close() + b, _ := io.ReadAll(resp.Body) + if resp.StatusCode != 200 || !strings.Contains(string(b), `"imported":2`) { + t.Fatalf("import = %d %s", resp.StatusCode, b) + } +} + +func TestHTTPServersCarryTheTimeouts(t *testing.T) { + relayAddr, _ := startSink(t) + a, err := Start(testConfig(t, relayAddr), "tok") + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { a.Close() }) + if httpReadTimeout != 30*time.Second || httpWriteTimeout != 60*time.Second || httpIdleTimeout != 120*time.Second { + t.Fatalf("defaults: %v %v %v", httpReadTimeout, httpWriteTimeout, httpIdleTimeout) + } + for name, s := range map[string]*http.Server{"api": a.web, "web": a.pub} { + if s.ReadTimeout != httpReadTimeout || s.WriteTimeout != httpWriteTimeout || s.IdleTimeout != httpIdleTimeout || s.ReadHeaderTimeout == 0 { + t.Errorf("%s: timeouts %v %v %v %v", name, s.ReadHeaderTimeout, s.ReadTimeout, s.WriteTimeout, s.IdleTimeout) + } + } +} diff --git a/internal/archive/archive.go b/internal/archive/archive.go index afc76d9..b8a8827 100644 --- a/internal/archive/archive.go +++ b/internal/archive/archive.go @@ -24,6 +24,19 @@ import _ "github.com/emersion/go-message/charset" // MaxBody is the largest body stored for one message, in bytes. const MaxBody = 1 << 20 +// Limits on how much of a message's structure is walked to find its body: a +// hostile message cannot make the parser recurse or loop for long. +const ( + maxDepth = 10 // levels of nested multipart + maxParts = 100 // parts visited in one message +) + +// Dates outside these bounds are not trusted: they would pin a message to the +// top (or the bottom) of the archive for good. +const maxFuture = 24 * time.Hour + +var minDate = time.Date(1990, 1, 1, 0, 0, 0, 0, time.UTC) + // Entry is the archivable content of a message. type Entry struct { MessageID string @@ -45,8 +58,8 @@ var ( emailRe = regexp.MustCompile(`([A-Za-z0-9._%+\-]+)@[A-Za-z0-9\-]+(?:\.[A-Za-z0-9\-]+)*\.[A-Za-z]{2,}`) ) -// Parse reads an RFC 5322 message. now is used when the Date header is missing -// or invalid. +// Parse reads an RFC 5322 message. now is used when the Date header is missing, +// invalid, before 1990, or more than 24 hours after now. func Parse(raw []byte, now time.Time) (Entry, error) { ent, err := message.Read(bytes.NewReader(raw)) if err != nil && !message.IsUnknownCharset(err) { @@ -58,7 +71,7 @@ func Parse(raw []byte, now time.Time) (Entry, error) { h := mail.Header{Header: ent.Header} e := Entry{Date: now} - if d, err := h.Date(); err == nil && !d.IsZero() { + if d, err := h.Date(); err == nil && !d.Before(minDate) && !d.After(now.Add(maxFuture)) { e.Date = d } if e.Subject, _ = h.Subject(); strings.TrimSpace(e.Subject) == "" { @@ -79,7 +92,9 @@ func Parse(raw []byte, now time.Time) (Entry, error) { e.InReplyTo = refs[len(refs)-1] } - plain, htmlBody := bodies(ent) + w := &walker{} + w.walk(ent, 0) + plain, htmlBody := w.plain, w.html if strings.TrimSpace(plain) == "" && htmlBody != "" { plain = stripHTML(htmlBody) } @@ -99,45 +114,55 @@ func senderName(h mail.Header) string { return clean(local) } -// bodies returns the first plain text and the first HTML body found. -func bodies(e *message.Entity) (plain, htmlBody string) { +// walker collects the first plain text and the first HTML body of a message. +type walker struct { + plain, html string + parts int +} + +// done reports whether the plain text is found: nothing else is needed. +func (w *walker) done() bool { return strings.TrimSpace(w.plain) != "" } + +func (w *walker) walk(e *message.Entity, depth int) { ct, _, _ := e.Header.ContentType() if ct == "" { ct = "text/plain" } if strings.HasPrefix(ct, "multipart/") { + if depth >= maxDepth { + return + } mr := e.MultipartReader() if mr == nil { - return "", "" + return } - for { + for !w.done() && w.parts < maxParts { part, err := mr.NextPart() if err == io.EOF { - break + return } if err != nil && !message.IsUnknownCharset(err) { - break + return } + w.parts++ if disp, _, _ := part.Header.ContentDisposition(); disp == "attachment" { continue } - p, h := bodies(part) - if plain == "" { - plain = p - } - if htmlBody == "" { - htmlBody = h - } + w.walk(part, depth+1) } - return plain, htmlBody + return } + // Read a text part only if it can still become the body. switch ct { case "text/plain": - plain = readLimited(e.Body) + if w.plain == "" { + w.plain = readLimited(e.Body) + } case "text/html": - htmlBody = readLimited(e.Body) + if w.html == "" { + w.html = readLimited(e.Body) + } } - return plain, htmlBody } func readLimited(r io.Reader) string { diff --git a/internal/archive/archive_test.go b/internal/archive/archive_test.go index fa96e29..1add6bd 100644 --- a/internal/archive/archive_test.go +++ b/internal/archive/archive_test.go @@ -1,6 +1,8 @@ package archive import ( + "fmt" + "runtime" "strings" "testing" "time" @@ -212,3 +214,198 @@ func TestCapNeverSplitsAMultibyteCharacter(t *testing.T) { t.Fatalf("body len %d", len(e.Body)) } } + +// nestedBomb is a message with a text part followed by a part that nests +// multipart/mixed levels deep. +func nestedBomb(levels int) string { + var b strings.Builder + b.WriteString("From: a@example.com\r\nSubject: bomb\r\nContent-Type: multipart/mixed; boundary=b0\r\n\r\n") + b.WriteString("--b0\r\nContent-Type: text/plain\r\n\r\nhello\r\n") + for i := 0; i < levels; i++ { + fmt.Fprintf(&b, "--b%d\r\nContent-Type: multipart/mixed; boundary=b%d\r\n\r\n", i, i+1) + } + b.WriteString("--b" + fmt.Sprint(levels) + "\r\nContent-Type: text/plain\r\n\r\ndeep\r\n") + return b.String() +} + +func parseWithin(t *testing.T, raw string, limit time.Duration) Entry { + t.Helper() + type result struct { + e Entry + err error + } + done := make(chan result, 1) + go func() { + e, err := Parse([]byte(raw), now) + done <- result{e, err} + }() + select { + case r := <-done: + if r.err != nil { + t.Fatalf("parse: %v", r.err) + } + return r.e + case <-time.After(limit): + t.Fatalf("Parse still running after %v", limit) + return Entry{} + } +} + +func TestDeeplyNestedMultipartIsBounded(t *testing.T) { + e := parseWithin(t, nestedBomb(20000), 2*time.Second) + if e.Body != "hello\n" && e.Body != "hello" { + t.Fatalf("body = %q", e.Body) + } +} + +func TestNestingDeeperThanTheLimitIsIgnored(t *testing.T) { + // text/plain at depth 3 is found, at depth 40 it is not. + shallow := "Content-Type: multipart/mixed; boundary=b0\r\n\r\n" + + "--b0\r\nContent-Type: multipart/mixed; boundary=b1\r\n\r\n" + + "--b1\r\nContent-Type: multipart/mixed; boundary=b2\r\n\r\n" + + "--b2\r\nContent-Type: text/plain\r\n\r\nfound\r\n" + if e := parse(t, "Subject: s\r\n"+shallow); !strings.Contains(e.Body, "found") { + t.Fatalf("shallow body = %q", e.Body) + } + if e := parse(t, strings.Replace(nestedBomb(40), "text/plain\r\n\r\nhello", "application/octet-stream\r\n\r\nhello", 1)); strings.Contains(e.Body, "deep") { + t.Fatalf("a part 40 levels deep was read: %q", e.Body) + } +} + +func TestManyPartsAreBounded(t *testing.T) { + var b strings.Builder + b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n") + for i := 0; i < 5000; i++ { + b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n") + } + b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nlast\r\n") + e := parseWithin(t, b.String(), 2*time.Second) + if strings.Contains(e.Body, "last") { + t.Fatalf("part 5001 was read: %q", e.Body) + } + + // A text part among the first parts is still found. + b.Reset() + b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n") + for i := 0; i < 50; i++ { + b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n") + } + b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nearly\r\n") + if e := parse(t, b.String()); !strings.Contains(e.Body, "early") { + t.Fatalf("body = %q", e.Body) + } +} + +func TestOnlyTheTextPartsThatAreNeededAreRead(t *testing.T) { + // 30 text parts of 1 MiB each: the first wins and the rest are not read. + big := strings.Repeat("a", MaxBody) + var b strings.Builder + b.WriteString("Subject: big\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n") + for i := 0; i < 30; i++ { + b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\n" + big + "\r\n") + } + raw := []byte(b.String()) + var before, after runtime.MemStats + runtime.ReadMemStats(&before) + e, err := Parse(raw, now) + runtime.ReadMemStats(&after) + if err != nil { + t.Fatal(err) + } + if len(e.Body) != MaxBody { + t.Fatalf("body = %d bytes", len(e.Body)) + } + if used := after.TotalAlloc - before.TotalAlloc; used > 24<<20 { + t.Fatalf("Parse allocated %d MiB for a 1 MiB body", used>>20) + } +} + +func TestPathologicalHTMLIsNotQuadratic(t *testing.T) { + for name, frag := range map[string]string{ + "open script": "