Merge pull request 'Fix the medium and low findings of the global review' (#33) from fix/fable-mediums into main

This commit is contained in:
squidd 2026-11-10 12:04:26 +01:00
commit 6f745e9b25
38 changed files with 3164 additions and 207 deletions

View file

@ -10,9 +10,11 @@ A modern, single-binary mailing list manager: an evolution of Mailman.
and a background worker sends it with retries (1m, 5m, 15m, 1h, 3h, then every 6h, for up to
3 days). Relay outages and restarts lose nothing.
- Self-service subscribe and unsubscribe by email with double opt-in:
send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation.
send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation. A reply works once, a request
expires after 72 hours, and a confirmation that could not be applied (closed list, database error) can be replied to again.
A member disabled by bounces can rejoin the same way.
- Moderated lists: posts wait in a queue that the API can approve or discard.
- VERP bounces: every recipient gets a signed return address; 3 hard bounces disable a member.
- VERP bounces: every recipient gets a signed return address; 3 hard bounces within 30 days disable a member.
- Archive with full-text search (SQLite FTS5) and threading. Each list has an
`archive_policy`: `private` (default, API only), `public` (web pages) or `off`.
Public pages hide email addresses.
@ -41,12 +43,18 @@ Three listeners, each configurable:
| `web` | `127.0.0.1:8081` | public archive and unsubscribe pages (`""` disables) |
Put the `web` listener behind a reverse proxy with TLS and set `base_url` to its public URL:
the unsubscribe links in every message use it. Keep the `api` listener private.
the unsubscribe links in every message use it. It must be an `https://` URL (RFC 8058 one-click unsubscribe needs
it; `http://` is accepted only for `localhost`, `127.0.0.1` and `::1`) with a host and no credentials, query or fragment;
the daemon refuses to start otherwise. Leave it empty to turn the one-click links off. Keep the `api` listener private.
```
curl -H "Authorization: Bearer $PARSIMONY_API_TOKEN" -d '{"name":"dev","domain":"example.org","title":"Dev"}' localhost:8080/api/v1/lists
```
A list `name` is 1 to 30 characters of `a-z`, `0-9`, `.`, `_` and `-` (upper case is lowered, no dot at either end), the `domain`
a host name, and the `title` one line of at most 200 characters without control characters; anything else is a 400.
The title is encoded (RFC 2047) in the `List-Id` header when it is not ASCII.
## Command line
The same binary is an administration client. It talks to the REST API, so it works against a local or remote daemon.
@ -70,6 +78,10 @@ parsimony help
`<list>` is an address or a numeric id. Destructive commands (`list rm`, `archive rm`) need `--yes`.
Exit status: 0 success, 1 failure (including lines that failed in `member import`), 2 usage error.
The client refuses to send the token over plain `http://` to any host other than `localhost`, `127.0.0.1` or `::1`
(usage error, exit 2): put the API behind HTTPS, or pass `--insecure` if you really mean it.
Subjects, senders, message bodies and error texts that come from mail are printed with control characters
(escape sequences) shown as `\x1b`-style text instead of being sent to the terminal; `--json` output is already escaped.
`parsimony serve -config FILE` runs the daemon; the old `parsimony -config FILE` form still works.
In the container: `docker exec parsimony /parsimony list ls`.
@ -183,9 +195,22 @@ It answers `451` only when the database itself fails.
delivery and counts as a hard bounce for the member (3 disable it). Nothing else counts: not "relay access denied", not a
refused sender (`5.1.7`, `5.1.8`, or any code answering `MAIL FROM`), not a refusal after `DATA`. Those deliveries are kept
as failed.
- A bounce that comes back later to the member's return address counts only if it is a real delivery status report
(RFC 3464: `multipart/report` with a `message/delivery-status` part) with `Action: failed` and one of the same
codes. A spam or policy rejection (`5.7.x`), a full mailbox (`5.2.x`), a delay, a `4.x.x`, or a `Status:` line in a
plain or quoted message does not count.
- Hard bounces count for 30 days: when the last one is older, the next one starts again from 1. A disabled member
stays disabled until it is added again.
- A relay that accepted the message is never asked again because `QUIT` went wrong.
- Delivery is at-least-once: a crash between the relay accepting a message and the queue entry being
removed can send one message to one recipient twice.
- The same post is queued once. If the `250` for a post never reaches Postfix, Postfix delivers the post again; a list
remembers each post it queued for an hour (by `Message-ID`, or by a hash of the whole message when it has none),
answers `250` to a copy and does not queue it again (the log says so). An approved held post is always delivered and is
remembered too, so a copy that arrives later is not.
- On stop, the daemon refuses new connections and lets the LMTP sessions and API or web requests in flight finish, so a
post being received gets its `250`. After 30 seconds it cuts whatever is still open.
- What remains is at-least-once delivery across the relay: a crash between the relay accepting a message and the queue
entry being removed can send one message to one recipient twice. A copy of a post that arrives more than an hour later
is also delivered again.
- `GET /api/v1/queue` and the Queue page show what is pending or failing, and a stuck delivery can be discarded.
## Admin UI
@ -249,10 +274,23 @@ lists.example.org lmtp:inet:127.0.0.1:2424
and `Reply-To` come from the header; so a post whose header names someone else (or has no usable `From`, or several
addresses) is held for moderation instead of delivered, with the reason shown to the admin (API, CLI, UI). Let Postfix
reject forged senders too (SPF, DKIM, DMARC checks on inbound mail).
- **Loops and robots.** Mail that carries the list's own `List-Id` (a loop) or is a genuine auto-reply
(`Auto-Submitted: auto-replied`, `Precedence: auto_reply`) is answered `250` and dropped without a trace, so the
sender's server never bounces it. Other machine-generated mail (`Auto-Submitted: auto-generated`, cron, CI and
monitoring posts) is a normal post: the membership, `From` and moderation checks apply. The `join`, `leave` and
`confirm` addresses ignore (`250`, nothing happens, the token is kept) a mail with a null sender, any
`Auto-Submitted` other than `no`, a `Precedence` of `auto_reply`, `bulk` or `junk`, or the list's own `List-Id`.
This stops an auto-responder from confirming a subscription someone forged in your name.
- **The moderation queue is bounded.** A list holds at most 500 messages and 200 MiB, and one held message at most 10 MiB; a
full queue answers `452` (the sender retries later), a message over 10 MiB `552`. Messages nobody moderated for 30 days
are dropped. Listings (API, CLI, UI) show a summary and never read the bodies. The held view shows the From header
readers would see next to the real sender and flags a difference.
- **Hostile messages and slow clients.** The archive reads at most 10 levels of nested multipart and 100 parts per
message, then uses what it found (the post is still delivered). A `Date` header more than 24 hours ahead is replaced
by the arrival time, and one before 1990 is ignored, so a message cannot pin itself to the top of the archive. The
HTTP servers (API + admin UI, and the public web) cut a request that takes more than 30 s to read, a response that
takes more than 60 s to write, and a connection idle for 120 s; the archive import is the exception and may stream
for up to 30 minutes.
- **API token.** `parsimony serve` refuses a token shorter than 24 characters or containing whitespace
(`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is
locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy

View file

@ -76,15 +76,21 @@ describe('held tab', () => {
expect(await screen.findByText(/nothing is waiting/i)).toBeInTheDocument()
})
it('explains how messages get held when posting is not moderated', async () => {
await setup([], aList({ id: 1, post_policy: 'members' }))
expect(await screen.findByText(/set posting to moderated/i)).toBeInTheDocument()
it('explains both ways a message gets held, whatever the posting policy', async () => {
for (const post_policy of ['members', 'moderated'] as const) {
const { unmount } = await setup([], aList({ id: 1, post_policy }))
const hint = await screen.findByText(/messages are held when/i)
expect(hint).toHaveTextContent(/posting is set to moderated/i)
expect(hint).toHaveTextContent(/from header does not match the sender/i)
expect(hint).not.toHaveTextContent(/only/i)
unmount()
}
})
it('does not nag about the policy when the list is moderated', async () => {
await setup([])
await screen.findByText(/nothing is waiting/i)
expect(screen.queryByText(/set posting to moderated/i)).not.toBeInTheDocument()
it('shows no hint when there are held messages', async () => {
await setup([held(1)])
await screen.findByText('Subject 1')
expect(screen.queryByText(/messages are held when/i)).not.toBeInTheDocument()
})
it('approves a message: it is sent and leaves the queue', async () => {

View file

@ -51,9 +51,10 @@ export function HeldTab() {
{held.data && items.length === 0 && (
<div className="space-y-1 text-muted-foreground">
<p>Nothing is waiting for approval.</p>
{list.post_policy !== 'moderated' && (
<p className="text-sm">Messages are held only when you set Posting to moderated in Settings.</p>
)}
<p className="text-sm">
Messages are held when Posting is set to moderated in Settings, or when the From header does not match
the sender.
</p>
</div>
)}
{items.length > 0 && (

View file

@ -50,8 +50,8 @@ components:
properties:
address: {type: string}
display_name: {type: string}
disabled: {type: boolean, description: Disabled after 3 hard bounces; re-adding the member re-enables it}
bounce_count: {type: integer}
disabled: {type: boolean, description: Disabled after 3 hard bounces within 30 days; re-adding the member re-enables it}
bounce_count: {type: integer, description: Recent hard bounces (restarts at 1 when the last one is older than 30 days)}
Held:
type: object
properties:
@ -99,9 +99,19 @@ paths:
type: object
required: [name, domain]
properties:
name: {type: string}
domain: {type: string}
title: {type: string}
name:
type: string
minLength: 1
maxLength: 30
pattern: '^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$'
description: Local part of the list address, stored lowercase. Letters, digits, '.', '_' and '-' only, no dot at either end or two in a row.
domain:
type: string
description: ASCII host name (labels of 1 to 63 letters, digits and hyphens).
title:
type: string
maxLength: 200
description: A single line without control characters. Defaults to the name.
responses:
'201':
description: Created
@ -131,7 +141,7 @@ paths:
schema:
type: object
properties:
title: {type: string}
title: {type: string, minLength: 1, maxLength: 200, description: A single line without control characters.}
post_policy: {type: string, enum: [members, moderated]}
join_policy: {type: string, enum: [open, closed]}
archive_policy: {type: string, enum: [public, private, off]}

View file

@ -12,6 +12,8 @@ import (
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
"codeberg.org/squidd/parsimony/internal/archive"
"codeberg.org/squidd/parsimony/internal/store"
@ -40,29 +42,44 @@ func New(s *store.Store, token string, rel Releaser) http.Handler {
func newHandler(s *store.Store, token string, rel Releaser, now func() time.Time) http.Handler {
srv := &server{store: s, token: token, releaser: rel, lim: newLimiter(now)}
mux := http.NewServeMux()
mux.HandleFunc("GET /api/v1/health", func(w http.ResponseWriter, _ *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
})
mux.Handle("GET /api/v1/lists", srv.auth(srv.listLists))
mux.Handle("POST /api/v1/lists", srv.auth(srv.createList))
mux.Handle("GET /api/v1/lists/{id}", srv.auth(srv.getList))
mux.Handle("PATCH /api/v1/lists/{id}", srv.auth(srv.updateList))
mux.Handle("DELETE /api/v1/lists/{id}", srv.auth(srv.deleteList))
mux.Handle("GET /api/v1/lists/{id}/members", srv.auth(srv.listMembers))
mux.Handle("POST /api/v1/lists/{id}/members", srv.auth(srv.addMember))
mux.Handle("DELETE /api/v1/lists/{id}/members/{address}", srv.auth(srv.removeMember))
mux.Handle("GET /api/v1/lists/{id}/archive", srv.auth(srv.listArchive))
mux.Handle("POST /api/v1/lists/{id}/archive/import", srv.auth(srv.importArchive))
mux.Handle("GET /api/v1/archive/{id}", srv.auth(srv.getArchived))
mux.Handle("DELETE /api/v1/archive/{id}", srv.auth(srv.deleteArchived))
mux.Handle("GET /api/v1/queue", srv.auth(srv.getQueue))
mux.Handle("DELETE /api/v1/queue/{id}", srv.auth(srv.discardQueued))
mux.Handle("GET /api/v1/lists/{id}/held", srv.auth(srv.listHeld))
mux.Handle("POST /api/v1/held/{id}/approve", srv.auth(srv.approveHeld))
mux.Handle("DELETE /api/v1/held/{id}", srv.auth(srv.discardHeld))
for _, rt := range srv.routes() {
mux.Handle(rt.pattern, rt.handler)
}
return mux
}
type route struct {
pattern string
handler http.Handler
}
// routes is the only place routes are declared. Everything but the health check goes through auth.
func (s *server) routes() []route {
public := func(h http.HandlerFunc) http.Handler { return h }
return []route{
{"GET /api/v1/health", public(func(w http.ResponseWriter, _ *http.Request) {
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
})},
{"GET /api/v1/lists", s.auth(s.listLists)},
{"POST /api/v1/lists", s.auth(s.createList)},
{"GET /api/v1/lists/{id}", s.auth(s.getList)},
{"PATCH /api/v1/lists/{id}", s.auth(s.updateList)},
{"DELETE /api/v1/lists/{id}", s.auth(s.deleteList)},
{"GET /api/v1/lists/{id}/members", s.auth(s.listMembers)},
{"POST /api/v1/lists/{id}/members", s.auth(s.addMember)},
{"DELETE /api/v1/lists/{id}/members/{address}", s.auth(s.removeMember)},
{"GET /api/v1/lists/{id}/archive", s.auth(s.listArchive)},
{"POST /api/v1/lists/{id}/archive/import", s.auth(s.importArchive)},
{"GET /api/v1/archive/{id}", s.auth(s.getArchived)},
{"DELETE /api/v1/archive/{id}", s.auth(s.deleteArchived)},
{"GET /api/v1/queue", s.auth(s.getQueue)},
{"DELETE /api/v1/queue/{id}", s.auth(s.discardQueued)},
{"GET /api/v1/lists/{id}/held", s.auth(s.listHeld)},
{"POST /api/v1/held/{id}/approve", s.auth(s.approveHeld)},
{"DELETE /api/v1/held/{id}", s.auth(s.discardHeld)},
}
}
func (s *server) auth(next http.HandlerFunc) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ip := clientIP(r)
@ -169,6 +186,53 @@ func (s *server) listLists(w http.ResponseWriter, _ *http.Request) {
writeJSON(w, http.StatusOK, out)
}
const (
maxNameLen = 30 // keeps list+confirm-<token> and list+bounce-<id>-<mac> within 64 characters
maxTitleLen = 200
titleHint = "title must be a non-empty single line of at most 200 characters, without control characters"
)
// validName reports whether name (already lowercased) works as the local part of a list address.
func validName(name string) bool {
if name == "" || len(name) > maxNameLen || strings.HasPrefix(name, ".") || strings.HasSuffix(name, ".") || strings.Contains(name, "..") {
return false
}
for _, c := range name {
if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '.' && c != '_' && c != '-' {
return false
}
}
return true
}
// validDomain reports whether d (already lowercased) is an ASCII host name: dot-separated labels of
// 1 to 63 letters, digits and hyphens, none starting or ending with a hyphen.
func validDomain(d string) bool {
if d == "" || len(d) > 253 {
return false
}
for _, label := range strings.Split(d, ".") {
if label == "" || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' {
return false
}
for _, c := range label {
if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '-' {
return false
}
}
}
return true
}
// validTitle reports whether t is a usable list title: not empty, at most maxTitleLen characters,
// no line breaks or other control characters (it goes into mail headers).
func validTitle(t string) bool {
if t == "" || utf8.RuneCountInString(t) > maxTitleLen {
return false
}
return !strings.ContainsFunc(t, unicode.IsControl)
}
func (s *server) createList(w http.ResponseWriter, r *http.Request) {
var in struct {
Name string `json:"name"`
@ -178,13 +242,22 @@ func (s *server) createList(w http.ResponseWriter, r *http.Request) {
if !decodeBody(w, r, &in) {
return
}
in.Name, in.Domain = strings.TrimSpace(in.Name), strings.TrimSpace(in.Domain)
if in.Name == "" || in.Domain == "" || strings.ContainsAny(in.Name+in.Domain, "@+ \r\n") {
writeError(w, http.StatusBadRequest, "name and domain are required and must not contain '@', '+' or spaces")
typedName := strings.TrimSpace(in.Name)
in.Name, in.Domain = strings.ToLower(typedName), strings.ToLower(strings.TrimSpace(in.Domain))
if !validName(in.Name) {
writeError(w, http.StatusBadRequest, "name must be 1 to 30 characters of a-z, 0-9, '.', '_' or '-', without a dot at either end or two dots in a row")
return
}
if in.Title == "" {
in.Title = in.Name
if !validDomain(in.Domain) {
writeError(w, http.StatusBadRequest, "domain must be a host name such as lists.example.org")
return
}
if in.Title = strings.TrimSpace(in.Title); in.Title == "" {
in.Title = typedName
}
if !validTitle(in.Title) {
writeError(w, http.StatusBadRequest, titleHint)
return
}
l, err := s.store.CreateList(in.Name, in.Domain, in.Title)
if err != nil {
@ -233,8 +306,8 @@ func (s *server) updateList(w http.ResponseWriter, r *http.Request) {
}
if in.Title != nil {
title := strings.TrimSpace(*in.Title)
if title == "" || strings.ContainsAny(title, "\r\n") {
writeError(w, http.StatusBadRequest, "title must be a non-empty single line")
if !validTitle(title) {
writeError(w, http.StatusBadRequest, titleHint)
return
}
l.Title = title
@ -561,6 +634,10 @@ func (s *server) discardQueued(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNoContent)
}
// importTimeout is how long one archive import may read its body and write its
// summary, in place of the server's default read and write timeouts.
const importTimeout = 30 * time.Minute
// importArchive reads an mbox from the request body, streaming, and archives it.
// Failed messages are reported in the summary; the request itself succeeds.
func (s *server) importArchive(w http.ResponseWriter, r *http.Request) {
@ -572,6 +649,11 @@ func (s *server) importArchive(w http.ResponseWriter, r *http.Request) {
storeError(w, err)
return
}
// A big mbox streams for longer than the server's default timeouts. Errors
// are ignored: a ResponseWriter that cannot extend deadlines has none.
rc := http.NewResponseController(w)
_ = rc.SetReadDeadline(time.Now().Add(importTimeout))
_ = rc.SetWriteDeadline(time.Now().Add(importTimeout))
sum, err := archive.ImportMbox(s.store, id, r.Body, time.Now())
if err != nil {
writeError(w, http.StatusBadRequest, fmt.Sprintf("import stopped after %d message(s): %v", sum.Imported, err))

View file

@ -152,6 +152,37 @@ func TestCreateListValidation(t *testing.T) {
`{"name":"a+b","domain":"example.org"}`,
`{"name":"dev","domain":""}`,
`{"name":"dev","domain":"example.org","bogus":1}`,
// Names that break an address: separators, brackets, whitespace, control characters, dots at the ends.
`{"name":"a/b","domain":"example.org"}`,
`{"name":"a<b","domain":"example.org"}`,
`{"name":"a>b","domain":"example.org"}`,
`{"name":"a\tb","domain":"example.org"}`,
`{"name":"a b","domain":"example.org"}`,
`{"name":"a\"b","domain":"example.org"}`,
`{"name":"caffè","domain":"example.org"}`,
`{"name":".dev","domain":"example.org"}`,
`{"name":"dev.","domain":"example.org"}`,
`{"name":"a..b","domain":"example.org"}`,
`{"name":"a{b","domain":"example.org"}`,
`{"name":"","domain":"example.org","title":"Dev"}`,
`{"name":" ","domain":"example.org","title":"Dev"}`,
`{"name":"` + strings.Repeat("a", 31) + `","domain":"example.org"}`,
// Domains that are not host names.
`{"name":"dev","domain":"exa mple.org"}`,
`{"name":"dev","domain":"example..org"}`,
`{"name":"dev","domain":"-example.org"}`,
`{"name":"dev","domain":"example-.org"}`,
`{"name":"dev","domain":"example.org."}`,
`{"name":"dev","domain":"ex_ample.org"}`,
`{"name":"dev","domain":"exämple.org"}`,
`{"name":"dev","domain":"a/b.org"}`,
`{"name":"dev","domain":"` + strings.Repeat("a", 64) + `.org"}`,
// Titles that are not one clean line.
`{"name":"dev","domain":"example.org","title":"Dev\r\nBcc: x@y.com"}`,
`{"name":"dev","domain":"example.org","title":"Dev\nlist"}`,
`{"name":"dev","domain":"example.org","title":"Dev\tlist"}`,
`{"name":"dev","domain":"example.org","title":"Dev\u0000list"}`,
`{"name":"dev","domain":"example.org","title":"` + strings.Repeat("x", 201) + `"}`,
} {
rec := do(t, h, "POST", "/api/v1/lists", body, true)
if rec.Code != 400 {
@ -685,3 +716,127 @@ func TestQueueShowsFailedDeliveriesApartFromPendingOnes(t *testing.T) {
}
expect(t, do(t, h, "DELETE", "/api/v1/queue/"+strconv.FormatInt(due[2].ID, 10), "", true), 204)
}
// TestEveryRouteNeedsTheToken walks the route table New registers, so a route added later is
// covered without touching this test: all of them but the health check answer 401 without a token
// and something else with one.
func TestEveryRouteNeedsTheToken(t *testing.T) {
s, err := store.Open(":memory:")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { s.Close() })
h := New(s, token, nil)
rs := (&server{}).routes()
if len(rs) < 17 {
t.Fatalf("only %d routes in the table", len(rs))
}
for i, rt := range rs {
method, pattern, _ := strings.Cut(rt.pattern, " ")
path := strings.NewReplacer("{id}", "1", "{address}", "a@example.com").Replace(pattern)
t.Run(rt.pattern, func(t *testing.T) {
// Each request comes from its own address, so the lockout never gets in the way.
addr := fmt.Sprintf("192.0.2.%d:4000", i+1)
call := func(auth bool) int {
req := httptest.NewRequest(method, path, strings.NewReader("{}"))
req.RemoteAddr = addr
if auth {
req.Header.Set("Authorization", "Bearer "+token)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code
}
if pattern == "/api/v1/health" {
if got := call(false); got != 200 {
t.Fatalf("health without a token: %d", got)
}
return
}
if got := call(false); got != 401 {
t.Errorf("no token: status %d, want 401", got)
}
if got := call(true); got == 401 {
t.Errorf("good token: status %d", got)
}
})
}
}
func TestCreateListAcceptsTheLimits(t *testing.T) {
h := newServer(t)
for _, body := range []string{
`{"name":"` + strings.Repeat("a", 30) + `","domain":"example.org","title":"` + strings.Repeat("è", 200) + `"}`,
`{"name":"dev-team_1.eu","domain":"lists.sub-domain.example.org"}`,
`{"name":"Dev2","domain":"localhost","title":"Développeurs ☕"}`,
} {
if rec := do(t, h, "POST", "/api/v1/lists", body, true); rec.Code != 201 {
t.Errorf("%.80q: status %d, body %s", body, rec.Code, rec.Body.String())
}
}
}
// A title with a line break would make every later post to the list fail to rewrite, so the list
// must not come into existence at all.
func TestCreateListWithABadTitleLeavesNothingBehind(t *testing.T) {
h := newServer(t)
expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev\r\nX: y"}`, true), 400)
rec := do(t, h, "GET", "/api/v1/lists", "", true)
if strings.TrimSpace(rec.Body.String()) != "[]" {
t.Fatalf("the refused list was created: %s", rec.Body.String())
}
// The name is free for a good request.
expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, true), 201)
}
func TestCreateListLowercasesNameAndDomain(t *testing.T) {
h := newServer(t)
rec := do(t, h, "POST", "/api/v1/lists", `{"name":" Dev ","domain":"Example.ORG"}`, true)
expect(t, rec, 201)
var l listJSON
decode(t, rec, &l)
if l.Address != "dev@example.org" || l.Title != "Dev" {
t.Fatalf("%+v", l)
}
}
func TestUpdateListTitleUsesTheSameRules(t *testing.T) {
h := newServer(t)
createList(t, h)
for _, title := range []string{`"a\tb"`, `"` + strings.Repeat("x", 201) + `"`, `" "`, `"a\r\nb"`} {
expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":`+title+`}`, true), 400)
}
expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":"`+strings.Repeat("x", 200)+`"}`, true), 200)
}
func TestValidDomainLengthLimits(t *testing.T) {
l63 := strings.Repeat("a", 63)
d253 := l63 + "." + l63 + "." + l63 + "." + strings.Repeat("a", 61)
if len(d253) != 253 || !validDomain(d253) {
t.Fatalf("a 253-character domain is valid (%d)", len(d253))
}
if validDomain(d253 + "a") {
t.Fatal("254 characters is too long")
}
if !validDomain(l63+".org") || validDomain("") || validName("") {
t.Fatal("63-character labels are valid, empty names and domains are not")
}
}
func TestCreateListTrimsTheTitle(t *testing.T) {
h := newServer(t)
rec := do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":" Dev team \t"}`, true)
expect(t, rec, 201)
var l listJSON
decode(t, rec, &l)
if l.Title != "Dev team" {
t.Fatalf("title = %q", l.Title)
}
rec = do(t, h, "POST", "/api/v1/lists", `{"name":"ops","domain":"example.org","title":" "}`, true)
expect(t, rec, 201)
decode(t, rec, &l)
if l.Title != "ops" {
t.Fatalf("a blank title falls back to the name: %q", l.Title)
}
}

View file

@ -26,9 +26,31 @@ import (
"codeberg.org/squidd/parsimony/internal/web"
)
// shutdownTimeout bounds how long Close waits for the sessions in flight before it cuts them.
var shutdownTimeout = 30 * time.Second
// nowFunc is the clock of the store (queue schedules, token expiry). Tests move it.
var nowFunc = time.Now
// HTTP timeouts of the API + admin UI server and of the public web server, so
// a slow or silent client cannot pin a goroutine. The archive import extends
// its own deadlines (see api.importArchive). Variables so tests can shorten them.
var (
httpReadTimeout = 30 * time.Second
httpWriteTimeout = 60 * time.Second
httpIdleTimeout = 120 * time.Second
)
func newHTTPServer(h http.Handler) *http.Server {
return &http.Server{
Handler: h,
ReadHeaderTimeout: 10 * time.Second,
ReadTimeout: httpReadTimeout,
WriteTimeout: httpWriteTimeout,
IdleTimeout: httpIdleTimeout,
}
}
// heloName is the EHLO name: the configured one, else this machine's host name.
func heloName(configured string) string {
if configured != "" {
@ -90,6 +112,51 @@ func lmtpIsOpen(network, addr string) bool {
return ip == nil || !ip.IsLoopback()
}
// connTracker remembers the open connections of a listener, so Close can cut the ones still open
// when the grace period ends: once Shutdown has begun, smtp.Server.Close does nothing.
type connTracker struct {
net.Listener
mu sync.Mutex
conns map[*trackedConn]struct{}
}
func (t *connTracker) Accept() (net.Conn, error) {
c, err := t.Listener.Accept()
if err != nil {
return nil, err
}
tc := &trackedConn{Conn: c, t: t}
t.mu.Lock()
t.conns[tc] = struct{}{}
t.mu.Unlock()
return tc, nil
}
// closeAll cuts every connection still open.
func (t *connTracker) closeAll() {
t.mu.Lock()
open := make([]*trackedConn, 0, len(t.conns))
for c := range t.conns {
open = append(open, c)
}
t.mu.Unlock()
for _, c := range open {
c.Close()
}
}
type trackedConn struct {
net.Conn
t *connTracker
}
func (c *trackedConn) Close() error {
c.t.mu.Lock()
delete(c.t.conns, c)
c.t.mu.Unlock()
return c.Conn.Close()
}
// App is a running daemon.
type App struct {
// LMTPAddr and APIAddr are the addresses actually bound.
@ -98,14 +165,16 @@ type App struct {
APIAddr string
WebAddr string // empty when the public web is disabled
store *store.Store
lmtp *smtp.Server
lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it
web *http.Server
pub *http.Server
wg sync.WaitGroup
stop context.CancelFunc // stops the queue worker
once sync.Once
store *store.Store
lmtp *smtp.Server
lmtpOpen *connTracker // the LMTP connections, cut by Close when the grace period ends
lmtpDone chan struct{} // closed when the LMTP Serve returns
lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it
web *http.Server
pub *http.Server
wg sync.WaitGroup
stop context.CancelFunc // stops the queue worker
once sync.Once
}
// releaser lets the API approve held messages through the router.
@ -196,7 +265,7 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
mux := http.NewServeMux()
mux.Handle("/api/", api.New(st, apiToken, releaser{rt}))
mux.Handle("/", adminui.Embedded())
a.web = &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
a.web = newHTTPServer(mux)
}
var pubLn net.Listener
@ -211,9 +280,12 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
}
a.WebAddr = pubLn.Addr().String()
a.lns = append(a.lns, pubLn)
a.pub = &http.Server{Handler: web.New(st, rt), ReadHeaderTimeout: 10 * time.Second}
a.pub = newHTTPServer(web.New(st, rt))
}
a.lmtpOpen = &connTracker{Listener: serveLn, conns: map[*trackedConn]struct{}{}}
a.lmtpDone = make(chan struct{})
var workerCtx context.Context
workerCtx, a.stop = context.WithCancel(context.Background())
a.wg.Add(1)
@ -224,7 +296,8 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
a.wg.Add(1)
go func() {
defer a.wg.Done()
if err := a.lmtp.Serve(serveLn); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) {
defer close(a.lmtpDone)
if err := a.lmtp.Serve(a.lmtpOpen); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) {
log.Printf("lmtp: %v", err)
}
}()
@ -249,18 +322,26 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
return a, nil
}
// Close stops the servers and closes the database. It is safe to call twice.
// Close stops the daemon and closes the database. It is safe to call twice. It stops accepting
// connections, then lets the LMTP sessions and HTTP requests in flight finish, so a post being
// received is queued and gets its 250 (a cut reply makes the MTA send the post again). After
// shutdownTimeout it cuts whatever is still open.
func (a *App) Close() {
a.once.Do(func() {
ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
defer cancel()
for _, ln := range a.lns {
ln.Close()
}
a.lmtp.Close()
if a.web != nil {
a.web.Close()
<-a.lmtpDone // Serve has returned: no connection is added while Shutdown waits
if err := a.lmtp.Shutdown(ctx); errors.Is(err, context.DeadlineExceeded) {
log.Printf("lmtp: sessions still open after %v, closing them", shutdownTimeout)
a.lmtpOpen.closeAll()
}
if a.pub != nil {
a.pub.Close()
for _, srv := range []*http.Server{a.web, a.pub} {
if srv != nil && srv.Shutdown(ctx) != nil {
srv.Close()
}
}
a.stop() // in-flight sends are interrupted and stay queued
a.wg.Wait()

View file

@ -1,6 +1,7 @@
package app
import (
"bufio"
"bytes"
"crypto/ecdsa"
"crypto/elliptic"
@ -12,6 +13,7 @@ import (
"encoding/base64"
"encoding/pem"
"errors"
"fmt"
"io"
"log"
"math/big"
@ -120,13 +122,17 @@ func testConfig(t *testing.T, relay string) config.Config {
return c
}
// client opens one connection per request: a pooled connection the transport dialed and never
// used would hold an HTTP server's graceful shutdown for 5 s.
var client = &http.Client{Transport: &http.Transport{DisableKeepAlives: true}}
func call(t *testing.T, a *App, method, path, body, token string) int {
t.Helper()
req, _ := http.NewRequest(method, "http://"+a.APIAddr+path, strings.NewReader(body))
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}
resp, err := http.DefaultClient.Do(req)
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
@ -137,7 +143,7 @@ func call(t *testing.T, a *App, method, path, body, token string) int {
func fetch(t *testing.T, method, url string) (int, string) {
t.Helper()
req, _ := http.NewRequest(method, url, nil)
resp, err := http.DefaultClient.Do(req)
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
@ -353,6 +359,7 @@ func TestWebDisabledWhenListenIsEmpty(t *testing.T) {
}
func TestCloseDropsOpenConnections(t *testing.T) {
setShutdownTimeout(t, 200*time.Millisecond)
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
@ -432,6 +439,193 @@ func deadAddr(t *testing.T) string {
return addr
}
func setShutdownTimeout(t *testing.T, d time.Duration) {
old := shutdownTimeout
shutdownTimeout = d
t.Cleanup(func() { shutdownTimeout = old })
}
// waitRefused waits until the LMTP listener no longer accepts connections.
func waitRefused(t *testing.T, addr string) {
t.Helper()
deadline := time.Now().Add(5 * time.Second)
for {
c, err := net.Dial("tcp", addr)
if err != nil {
return
}
c.Close()
if time.Now().After(deadline) {
t.Fatal("the LMTP listener still accepts connections")
}
time.Sleep(5 * time.Millisecond)
}
}
func TestCloseLetsAnLMTPDeliveryInFlightFinish(t *testing.T) {
cfg := testConfig(t, deadAddr(t)) // the post stays queued, where it can be counted
a, err := Start(cfg, "tok")
if err != nil {
t.Fatal(err)
}
call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org"}`, "tok")
call(t, a, "POST", "/api/v1/lists/1/members", `{"address":"alice@example.com"}`, "tok")
conn, err := net.Dial("tcp", a.LMTPAddr)
if err != nil {
t.Fatal(err)
}
c := smtp.NewClientLMTP(conn)
defer c.Close()
if err := c.Hello("localhost"); err != nil {
t.Fatal(err)
}
if err := c.Mail("alice@example.com", nil); err != nil {
t.Fatal(err)
}
if err := c.Rcpt("dev@example.org", nil); err != nil {
t.Fatal(err)
}
w, err := c.Data()
if err != nil {
t.Fatal(err)
}
w.Write([]byte("From: alice@example.com\r\nSubject: in flight\r\nMessage-ID: <f@x>\r\n\r\nbody\r\n"))
// The daemon stops while the post is being sent.
closed := make(chan struct{})
go func() { a.Close(); close(closed) }()
waitRefused(t, a.LMTPAddr)
if _, err := w.CloseWithLMTPResponse(); err != nil {
t.Fatalf("the post in flight must get its 250: %v", err)
}
if err := c.Quit(); err != nil {
t.Fatalf("quit: %v", err)
}
select {
case <-closed:
case <-time.After(5 * time.Second):
t.Fatal("Close must return once the session is over")
}
peek, err := store.Open(cfg.DB)
if err != nil {
t.Fatal(err)
}
defer peek.Close()
if st, _ := peek.OutboxStats(); st.Pending != 1 {
t.Fatalf("the post must be queued: %+v", st)
}
}
func TestCloseCutsAHungLMTPClientAfterTheGracePeriod(t *testing.T) {
setShutdownTimeout(t, 200*time.Millisecond)
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
conn, err := net.Dial("tcp", a.LMTPAddr)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
c := smtp.NewClientLMTP(conn)
if err := c.Hello("localhost"); err != nil {
t.Fatal(err)
}
if err := c.Mail("alice@example.com", nil); err != nil {
t.Fatal(err)
}
if err := c.Rcpt("dev@example.org", nil); err != nil {
t.Fatal(err)
}
if _, err := c.Data(); err != nil {
t.Fatal(err)
}
// The client never ends the message.
start := time.Now()
closed := make(chan struct{})
go func() { a.Close(); close(closed) }()
select {
case <-closed:
case <-time.After(5 * time.Second):
t.Fatal("Close must not wait for a hung client beyond the grace period")
}
if d := time.Since(start); d < 150*time.Millisecond {
t.Fatalf("Close returned after %v: it must give the session its grace period", d)
}
conn.SetReadDeadline(time.Now().Add(2 * time.Second))
buf := make([]byte, 512)
for {
if _, err := conn.Read(buf); err != nil {
if ne, ok := err.(net.Error); ok && ne.Timeout() {
t.Fatal("the hung connection is still open after Close")
}
break
}
}
}
func TestCloseLetsAnAPIRequestInFlightFinish(t *testing.T) {
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
conn, err := net.Dial("tcp", a.APIAddr)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
body := `{"name":"dev","domain":"example.org"}`
fmt.Fprintf(conn, "POST /api/v1/lists HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s", len(body), body[:10])
time.Sleep(100 * time.Millisecond) // the handler is now reading the body
closed := make(chan struct{})
go func() { a.Close(); close(closed) }()
waitRefused(t, a.APIAddr)
io.WriteString(conn, body[10:])
conn.SetReadDeadline(time.Now().Add(5 * time.Second))
resp, err := http.ReadResponse(bufio.NewReader(conn), nil)
if err != nil {
t.Fatalf("the request in flight must get its answer: %v", err)
}
resp.Body.Close()
if resp.StatusCode != http.StatusCreated {
t.Fatalf("status = %d", resp.StatusCode)
}
select {
case <-closed:
case <-time.After(5 * time.Second):
t.Fatal("Close must return once the request is over")
}
}
func TestFinishedLMTPSessionsAreForgotten(t *testing.T) {
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
for i := 0; i < 3; i++ {
lmtpSend(t, a.LMTPAddr, "alice@example.com", "nope@example.org", "Subject: x\r\n\r\nx\r\n")
}
deadline := time.Now().Add(5 * time.Second)
for {
a.lmtpOpen.mu.Lock()
n := len(a.lmtpOpen.conns)
a.lmtpOpen.mu.Unlock()
if n == 0 {
break
}
if time.Now().After(deadline) {
t.Fatalf("%d closed sessions are still tracked", n)
}
time.Sleep(5 * time.Millisecond)
}
}
func TestQueuedMailSurvivesARestartAndIsRetriedWhenTheRelayReturns(t *testing.T) {
var skew atomic.Int64
nowFunc = func() time.Time { return time.Now().Add(time.Duration(skew.Load())) }
@ -1004,3 +1198,115 @@ func TestTheServerNameSettingReachesTheTLSCheck(t *testing.T) {
t.Fatal("nothing may be sent to a relay whose certificate does not match")
}
}
// shortTimeouts makes the HTTP timeouts small enough to test.
func shortTimeouts(t *testing.T) {
t.Helper()
r, w, i := httpReadTimeout, httpWriteTimeout, httpIdleTimeout
httpReadTimeout, httpWriteTimeout, httpIdleTimeout = 300*time.Millisecond, 300*time.Millisecond, 300*time.Millisecond
t.Cleanup(func() { httpReadTimeout, httpWriteTimeout, httpIdleTimeout = r, w, i })
}
// stalled sends a request that promises a body and never finishes it, and
// reports whether the server hung up within a few seconds.
func stalled(t *testing.T, addr, req string) bool {
t.Helper()
c, err := net.Dial("tcp", addr)
if err != nil {
t.Fatal(err)
}
defer c.Close()
if _, err := c.Write([]byte(req)); err != nil {
t.Fatal(err)
}
c.SetReadDeadline(time.Now().Add(4 * time.Second))
_, err = io.Copy(io.Discard, c) // returns nil on EOF, an error on our deadline
return err == nil
}
func TestSlowRequestsAreCutOnBothServers(t *testing.T) {
shortTimeouts(t)
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
hdr := "POST %s HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: 500\r\n\r\n{\"name\":"
for name, c := range map[string]struct{ addr, path string }{
"api": {a.APIAddr, "/api/v1/lists"},
"web": {a.WebAddr, "/unsubscribe/tok"},
} {
if !stalled(t, c.addr, strings.Replace(hdr, "%s", c.path, 1)) {
t.Errorf("%s: a stalled body was not cut", name)
}
}
// An idle keep-alive connection is closed too.
for name, addr := range map[string]string{"api": a.APIAddr, "web": a.WebAddr} {
if !stalled(t, addr, "GET /nothing HTTP/1.1\r\nHost: x\r\n\r\n") {
t.Errorf("%s: an idle connection was kept", name)
}
}
}
// slowReader hands out its data in two halves, a pause apart.
type slowReader struct {
parts [][]byte
pause time.Duration
}
func (s *slowReader) Read(p []byte) (int, error) {
if len(s.parts) == 0 {
return 0, io.EOF
}
if len(s.parts) == 1 {
time.Sleep(s.pause)
}
n := copy(p, s.parts[0])
s.parts = s.parts[1:]
return n, nil
}
func TestArchiveImportIsNotCutByTheDefaultTimeouts(t *testing.T) {
shortTimeouts(t)
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, "tok")
m1 := "From a@example.com Tue Jan 2 15:04:05 2024\nFrom: a@example.com\nSubject: one\nMessage-ID: <1@x>\n\nbody one\n\n"
m2 := "From a@example.com Tue Jan 2 15:04:06 2024\nFrom: a@example.com\nSubject: two\nMessage-ID: <2@x>\n\nbody two\n"
body := &slowReader{parts: [][]byte{[]byte(m1), []byte(m2)}, pause: 3 * httpReadTimeout}
req, _ := http.NewRequest("POST", "http://"+a.APIAddr+"/api/v1/lists/1/archive/import", body)
req.Header.Set("Authorization", "Bearer tok")
req.ContentLength = -1 // chunked: the body is streamed
resp, err := http.DefaultClient.Do(req)
if err != nil {
t.Fatalf("import: %v", err)
}
defer resp.Body.Close()
b, _ := io.ReadAll(resp.Body)
if resp.StatusCode != 200 || !strings.Contains(string(b), `"imported":2`) {
t.Fatalf("import = %d %s", resp.StatusCode, b)
}
}
func TestHTTPServersCarryTheTimeouts(t *testing.T) {
relayAddr, _ := startSink(t)
a, err := Start(testConfig(t, relayAddr), "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
if httpReadTimeout != 30*time.Second || httpWriteTimeout != 60*time.Second || httpIdleTimeout != 120*time.Second {
t.Fatalf("defaults: %v %v %v", httpReadTimeout, httpWriteTimeout, httpIdleTimeout)
}
for name, s := range map[string]*http.Server{"api": a.web, "web": a.pub} {
if s.ReadTimeout != httpReadTimeout || s.WriteTimeout != httpWriteTimeout || s.IdleTimeout != httpIdleTimeout || s.ReadHeaderTimeout == 0 {
t.Errorf("%s: timeouts %v %v %v %v", name, s.ReadHeaderTimeout, s.ReadTimeout, s.WriteTimeout, s.IdleTimeout)
}
}
}

View file

@ -24,6 +24,19 @@ import _ "github.com/emersion/go-message/charset"
// MaxBody is the largest body stored for one message, in bytes.
const MaxBody = 1 << 20
// Limits on how much of a message's structure is walked to find its body: a
// hostile message cannot make the parser recurse or loop for long.
const (
maxDepth = 10 // levels of nested multipart
maxParts = 100 // parts visited in one message
)
// Dates outside these bounds are not trusted: they would pin a message to the
// top (or the bottom) of the archive for good.
const maxFuture = 24 * time.Hour
var minDate = time.Date(1990, 1, 1, 0, 0, 0, 0, time.UTC)
// Entry is the archivable content of a message.
type Entry struct {
MessageID string
@ -45,8 +58,8 @@ var (
emailRe = regexp.MustCompile(`([A-Za-z0-9._%+\-]+)@[A-Za-z0-9\-]+(?:\.[A-Za-z0-9\-]+)*\.[A-Za-z]{2,}`)
)
// Parse reads an RFC 5322 message. now is used when the Date header is missing
// or invalid.
// Parse reads an RFC 5322 message. now is used when the Date header is missing,
// invalid, before 1990, or more than 24 hours after now.
func Parse(raw []byte, now time.Time) (Entry, error) {
ent, err := message.Read(bytes.NewReader(raw))
if err != nil && !message.IsUnknownCharset(err) {
@ -58,7 +71,7 @@ func Parse(raw []byte, now time.Time) (Entry, error) {
h := mail.Header{Header: ent.Header}
e := Entry{Date: now}
if d, err := h.Date(); err == nil && !d.IsZero() {
if d, err := h.Date(); err == nil && !d.Before(minDate) && !d.After(now.Add(maxFuture)) {
e.Date = d
}
if e.Subject, _ = h.Subject(); strings.TrimSpace(e.Subject) == "" {
@ -79,7 +92,9 @@ func Parse(raw []byte, now time.Time) (Entry, error) {
e.InReplyTo = refs[len(refs)-1]
}
plain, htmlBody := bodies(ent)
w := &walker{}
w.walk(ent, 0)
plain, htmlBody := w.plain, w.html
if strings.TrimSpace(plain) == "" && htmlBody != "" {
plain = stripHTML(htmlBody)
}
@ -99,45 +114,55 @@ func senderName(h mail.Header) string {
return clean(local)
}
// bodies returns the first plain text and the first HTML body found.
func bodies(e *message.Entity) (plain, htmlBody string) {
// walker collects the first plain text and the first HTML body of a message.
type walker struct {
plain, html string
parts int
}
// done reports whether the plain text is found: nothing else is needed.
func (w *walker) done() bool { return strings.TrimSpace(w.plain) != "" }
func (w *walker) walk(e *message.Entity, depth int) {
ct, _, _ := e.Header.ContentType()
if ct == "" {
ct = "text/plain"
}
if strings.HasPrefix(ct, "multipart/") {
if depth >= maxDepth {
return
}
mr := e.MultipartReader()
if mr == nil {
return "", ""
return
}
for {
for !w.done() && w.parts < maxParts {
part, err := mr.NextPart()
if err == io.EOF {
break
return
}
if err != nil && !message.IsUnknownCharset(err) {
break
return
}
w.parts++
if disp, _, _ := part.Header.ContentDisposition(); disp == "attachment" {
continue
}
p, h := bodies(part)
if plain == "" {
plain = p
}
if htmlBody == "" {
htmlBody = h
}
w.walk(part, depth+1)
}
return plain, htmlBody
return
}
// Read a text part only if it can still become the body.
switch ct {
case "text/plain":
plain = readLimited(e.Body)
if w.plain == "" {
w.plain = readLimited(e.Body)
}
case "text/html":
htmlBody = readLimited(e.Body)
if w.html == "" {
w.html = readLimited(e.Body)
}
}
return plain, htmlBody
}
func readLimited(r io.Reader) string {

View file

@ -1,6 +1,8 @@
package archive
import (
"fmt"
"runtime"
"strings"
"testing"
"time"
@ -212,3 +214,198 @@ func TestCapNeverSplitsAMultibyteCharacter(t *testing.T) {
t.Fatalf("body len %d", len(e.Body))
}
}
// nestedBomb is a message with a text part followed by a part that nests
// multipart/mixed levels deep.
func nestedBomb(levels int) string {
var b strings.Builder
b.WriteString("From: a@example.com\r\nSubject: bomb\r\nContent-Type: multipart/mixed; boundary=b0\r\n\r\n")
b.WriteString("--b0\r\nContent-Type: text/plain\r\n\r\nhello\r\n")
for i := 0; i < levels; i++ {
fmt.Fprintf(&b, "--b%d\r\nContent-Type: multipart/mixed; boundary=b%d\r\n\r\n", i, i+1)
}
b.WriteString("--b" + fmt.Sprint(levels) + "\r\nContent-Type: text/plain\r\n\r\ndeep\r\n")
return b.String()
}
func parseWithin(t *testing.T, raw string, limit time.Duration) Entry {
t.Helper()
type result struct {
e Entry
err error
}
done := make(chan result, 1)
go func() {
e, err := Parse([]byte(raw), now)
done <- result{e, err}
}()
select {
case r := <-done:
if r.err != nil {
t.Fatalf("parse: %v", r.err)
}
return r.e
case <-time.After(limit):
t.Fatalf("Parse still running after %v", limit)
return Entry{}
}
}
func TestDeeplyNestedMultipartIsBounded(t *testing.T) {
e := parseWithin(t, nestedBomb(20000), 2*time.Second)
if e.Body != "hello\n" && e.Body != "hello" {
t.Fatalf("body = %q", e.Body)
}
}
func TestNestingDeeperThanTheLimitIsIgnored(t *testing.T) {
// text/plain at depth 3 is found, at depth 40 it is not.
shallow := "Content-Type: multipart/mixed; boundary=b0\r\n\r\n" +
"--b0\r\nContent-Type: multipart/mixed; boundary=b1\r\n\r\n" +
"--b1\r\nContent-Type: multipart/mixed; boundary=b2\r\n\r\n" +
"--b2\r\nContent-Type: text/plain\r\n\r\nfound\r\n"
if e := parse(t, "Subject: s\r\n"+shallow); !strings.Contains(e.Body, "found") {
t.Fatalf("shallow body = %q", e.Body)
}
if e := parse(t, strings.Replace(nestedBomb(40), "text/plain\r\n\r\nhello", "application/octet-stream\r\n\r\nhello", 1)); strings.Contains(e.Body, "deep") {
t.Fatalf("a part 40 levels deep was read: %q", e.Body)
}
}
func TestManyPartsAreBounded(t *testing.T) {
var b strings.Builder
b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
for i := 0; i < 5000; i++ {
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
}
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nlast\r\n")
e := parseWithin(t, b.String(), 2*time.Second)
if strings.Contains(e.Body, "last") {
t.Fatalf("part 5001 was read: %q", e.Body)
}
// A text part among the first parts is still found.
b.Reset()
b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
for i := 0; i < 50; i++ {
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
}
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nearly\r\n")
if e := parse(t, b.String()); !strings.Contains(e.Body, "early") {
t.Fatalf("body = %q", e.Body)
}
}
func TestOnlyTheTextPartsThatAreNeededAreRead(t *testing.T) {
// 30 text parts of 1 MiB each: the first wins and the rest are not read.
big := strings.Repeat("a", MaxBody)
var b strings.Builder
b.WriteString("Subject: big\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
for i := 0; i < 30; i++ {
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\n" + big + "\r\n")
}
raw := []byte(b.String())
var before, after runtime.MemStats
runtime.ReadMemStats(&before)
e, err := Parse(raw, now)
runtime.ReadMemStats(&after)
if err != nil {
t.Fatal(err)
}
if len(e.Body) != MaxBody {
t.Fatalf("body = %d bytes", len(e.Body))
}
if used := after.TotalAlloc - before.TotalAlloc; used > 24<<20 {
t.Fatalf("Parse allocated %d MiB for a 1 MiB body", used>>20)
}
}
func TestPathologicalHTMLIsNotQuadratic(t *testing.T) {
for name, frag := range map[string]string{
"open script": "<script>",
"open tag": "<a ",
"open style": "<style x>",
"break": "<br",
} {
html := strings.Repeat(frag, MaxBody/len(frag))
raw := "Subject: h\r\nContent-Type: text/html\r\n\r\n" + html
t.Run(name, func(t *testing.T) { parseWithin(t, raw, 15*time.Second) })
}
}
func TestFarFutureDateIsClamped(t *testing.T) {
// More than 24 hours ahead: the time of parsing. A little ahead is kept.
e := parse(t, "Date: Mon, 02 Jan 2090 15:04:05 +0000\r\nSubject: d\r\n\r\nx\r\n")
if !e.Date.Equal(now) {
t.Fatalf("far future date = %v", e.Date)
}
soon := now.Add(23 * time.Hour).Format(time.RFC1123Z)
if e := parse(t, "Date: "+soon+"\r\nSubject: d\r\n\r\nx\r\n"); e.Date.Equal(now) {
t.Fatalf("a date 23 hours ahead was clamped: %v", e.Date)
}
late := now.Add(25 * time.Hour).Format(time.RFC1123Z)
if e := parse(t, "Date: "+late+"\r\nSubject: d\r\n\r\nx\r\n"); !e.Date.Equal(now) {
t.Fatalf("a date 25 hours ahead = %v", e.Date)
}
}
func TestDateBefore1990IsMissing(t *testing.T) {
for _, d := range []string{"Sun, 31 Dec 1989 23:59:59 +0000", "Thu, 01 Jan 1970 00:00:00 +0000", "Mon, 01 Jan 1900 00:00:00 +0000"} {
if e := parse(t, "Date: "+d+"\r\nSubject: d\r\n\r\nx\r\n"); !e.Date.Equal(now) {
t.Fatalf("%s: date = %v", d, e.Date)
}
}
e := parse(t, "Date: Mon, 01 Jan 1990 00:00:00 +0000\r\nSubject: d\r\n\r\nx\r\n")
if !e.Date.Equal(time.Date(1990, 1, 1, 0, 0, 0, 0, time.UTC)) {
t.Fatalf("1990 date = %v", e.Date)
}
}
// textAtDepth is a message whose only text part sits inside n nested
// multipart levels.
func textAtDepth(n int) string {
var b strings.Builder
b.WriteString("Subject: s\r\nContent-Type: multipart/mixed; boundary=b0\r\n\r\n")
for i := 0; i < n-1; i++ {
fmt.Fprintf(&b, "--b%d\r\nContent-Type: multipart/mixed; boundary=b%d\r\n\r\n", i, i+1)
}
fmt.Fprintf(&b, "--b%d\r\nContent-Type: text/plain\r\n\r\nfound\r\n", n-1)
return b.String()
}
func TestNestingLimitIsTenLevels(t *testing.T) {
if e := parse(t, textAtDepth(10)); !strings.Contains(e.Body, "found") {
t.Fatalf("10 levels: body = %q", e.Body)
}
if e := parse(t, textAtDepth(11)); e.Body != "" {
t.Fatalf("11 levels: body = %q", e.Body)
}
}
func TestPartLimitIsOneHundred(t *testing.T) {
withTextAt := func(n int) string {
var b strings.Builder
b.WriteString("Subject: s\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
for i := 1; i < n; i++ {
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
}
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nfound\r\n")
return b.String()
}
if e := parse(t, withTextAt(100)); !strings.Contains(e.Body, "found") {
t.Fatalf("part 100: body = %q", e.Body)
}
if e := parse(t, withTextAt(101)); e.Body != "" {
t.Fatalf("part 101: body = %q", e.Body)
}
}
func TestFirstPartOfEachKindWinsEvenWhenPlainIsBlank(t *testing.T) {
part := func(ct, body string) string { return "--b\r\nContent-Type: " + ct + "\r\n\r\n" + body + "\r\n" }
raw := "Subject: s\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n" +
part("text/plain", " ") + part("text/plain", "second plain") +
part("text/html", "<p>first html</p>") + part("text/html", "<p>second html</p>")
if e := parse(t, raw); strings.TrimSpace(e.Body) != "first html" {
t.Fatalf("body = %q", e.Body)
}
}

View file

@ -8,6 +8,7 @@ import (
"flag"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strconv"
@ -22,7 +23,7 @@ type IO struct {
Out, Err io.Writer
}
const usageText = `Usage: parsimony [--url URL] [--json] <command> [arguments]
const usageText = `Usage: parsimony [--url URL] [--insecure] [--json] <command> [arguments]
Administration (talks to the REST API):
list ls show all lists
@ -69,6 +70,7 @@ Environment:
PARSIMONY_URL API address, default http://127.0.0.1:8080 (--url overrides)
--json prints the API's JSON instead of tables.
--insecure allows sending the token over plain http to a host other than localhost.
Exit status: 0 success, 1 failure, 2 usage error.
`
@ -79,6 +81,7 @@ func Run(args []string, io IO, getenv func(string) string) int {
fs := newFlagSet("parsimony")
fs.StringVar(&c.url, "url", "", "")
fs.BoolVar(&c.json, "json", false, "")
fs.BoolVar(&c.insecure, "insecure", false, "")
if err := fs.Parse(args); err != nil {
if errors.Is(err, flag.ErrHelp) {
fmt.Fprint(io.Out, usageText)
@ -88,7 +91,7 @@ func Run(args []string, io IO, getenv func(string) string) int {
}
rest := fs.Args()
if len(rest) == 0 {
fmt.Fprint(io.Err, "usage: parsimony [--url URL] [--json] <command> (see parsimony help)\n")
fmt.Fprint(io.Err, "usage: parsimony [--url URL] [--insecure] [--json] <command> (see parsimony help)\n")
return 2
}
if rest[0] == "help" {
@ -124,16 +127,27 @@ type cmd struct {
getenv func(string) string
url string
json bool
cl *client
// insecure allows the token to travel over plain http to a non-loopback host.
insecure bool
cl *client
}
// usageErr is an error that is the caller's mistake: exit status 2.
type usageErr string
func (e usageErr) Error() string { return string(e) }
func (c *cmd) usageError(format string, a ...any) int {
fmt.Fprintf(c.io.Err, "error: "+format+"\n", a...)
return 2
}
func (c *cmd) fail(err error) int {
fmt.Fprintf(c.io.Err, "error: %v\n", err)
fmt.Fprintf(c.io.Err, "error: %s\n", printable(err.Error()))
var u usageErr
if errors.As(err, &u) {
return 2
}
return 1
}
@ -153,9 +167,14 @@ func (c *cmd) client() (*client, error) {
if base == "" {
base = "http://127.0.0.1:8080"
}
if _, err := url.Parse(base); err != nil {
u, err := url.Parse(base)
if err != nil {
return nil, fmt.Errorf("invalid URL %q", base)
}
// url.Parse lowercases the scheme.
if u.Scheme == "http" && !isLoopback(u.Hostname()) && !c.insecure {
return nil, usageErr(fmt.Sprintf("refusing to send the token over plain http to %s: use https, or pass --insecure", u.Host))
}
c.cl = &client{
base: strings.TrimRight(base, "/"), token: token,
http: &http.Client{Timeout: 30 * time.Second},
@ -312,10 +331,44 @@ func (c *cmd) printJSON(raw json.RawMessage) int {
return 0
}
// isLoopback reports whether host is this machine: localhost or a loopback address.
func isLoopback(host string) bool {
if strings.EqualFold(strings.TrimSuffix(host, "."), "localhost") {
return true
}
ip := net.ParseIP(host)
return ip != nil && ip.IsLoopback()
}
// printable makes text that came from mail or from the API safe for a terminal: control
// characters (C0, DEL, C1) other than tab and newline are shown as \xNN / \uNNNN instead of
// being sent, so an escape sequence in a subject cannot drive the terminal. CRLF counts as a newline.
func printable(s string) string {
s = strings.ReplaceAll(s, "\r\n", "\n")
var b strings.Builder
for _, r := range s {
switch {
case r == '\t' || r == '\n':
b.WriteRune(r)
case r < 0x20 || r == 0x7f:
fmt.Fprintf(&b, `\x%02x`, r)
case r >= 0x80 && r <= 0x9f:
fmt.Fprintf(&b, `\u%04x`, r)
default:
b.WriteRune(r)
}
}
return b.String()
}
func (c *cmd) table(rows [][]string) {
w := tabwriter.NewWriter(c.io.Out, 0, 8, 2, ' ', 0)
for _, r := range rows {
fmt.Fprintln(w, strings.Join(r, "\t"))
cells := make([]string, len(r))
for i, cell := range r {
cells[i] = printable(cell)
}
fmt.Fprintln(w, strings.Join(cells, "\t"))
}
w.Flush()
}

View file

@ -153,7 +153,7 @@ func TestListLifecycle(t *testing.T) {
e.fails(t, 1, "already exists", "list", "add", "dev@example.org")
e.fails(t, 2, "name@domain", "list", "add", "nodomain")
e.fails(t, 2, "name@domain", "list", "add", "@example.org")
e.fails(t, 1, "name and domain", "list", "add", "a+b@example.org")
e.fails(t, 1, "name must be", "list", "add", "a+b@example.org")
for _, ref := range []string{"dev@example.org", "DEV@Example.org", "1"} {
out = e.ok(t, "list", "show", ref)
@ -824,3 +824,172 @@ func TestHeldListingWarnsWhenTheHeaderDiffersFromTheSender(t *testing.T) {
t.Fatalf("no usable From is flagged too: %q", lines["no from"])
}
}
const esc = "\x1b[31mRED\x1b]0;pwned\x07"
// noControl fails when s still holds a control character other than tab and newline.
func noControl(t *testing.T, what, s string) {
t.Helper()
for _, r := range s {
if (r < 0x20 && r != '\t' && r != '\n') || (r >= 0x7f && r <= 0x9f) {
t.Fatalf("%s prints control character %U:\n%q", what, r, s)
}
}
}
func TestPrintableStripsControlCharactersButKeepsTabAndNewline(t *testing.T) {
for in, want := range map[string]string{
"plain text, accents: àèì 日本": "plain text, accents: àèì 日本",
"a\tb\nc": "a\tb\nc",
"a\r\nb": "a\nb",
"\x1b[31mred\x1b[0m": `\x1b[31mred\x1b[0m`,
"us\x1f": `us\x1f`,
"bell\x07 nul\x00 del\x7f": `bell\x07 nul\x00 del\x7f`,
"c1 \u009b31m and \u0085": `c1 \u009b31m and \u0085`,
"c1 edges \u0080 \u009f, kept: ~ \u00a0 \u00a1": `c1 edges \u0080 \u009f, kept: ~ ` + "\u00a0 \u00a1",
"lone\rcarriage": `lone\x0dcarriage`,
"": "",
} {
if got := printable(in); got != want {
t.Errorf("printable(%q) = %q, want %q", in, got, want)
}
}
}
func TestControlCharactersFromMailNeverReachTheTerminal(t *testing.T) {
e := newEnv(t)
e.ok(t, "list", "add", "dev@example.org")
l, _ := e.s.FindList("dev@example.org")
e.s.ArchiveMessage(l.ID, store.Message{MessageID: "<e@x>", Subject: "subj " + esc, Sender: "from " + esc,
Date: time.Unix(1_700_000_000, 0), Body: "line one\r\nbody " + esc + "\r\nlast"})
e.s.HoldMessage(l.ID, "bob@x.com", []byte("From: Bob "+esc+" <bob@x.com>\r\nSubject: held "+esc+"\r\n\r\nb"), "reason "+esc)
e.s.Enqueue([]byte("x"), []store.NewOutbound{{From: "b@x.org", Rcpt: "lost@x.com"}})
due, _ := e.s.DueOutbox(10)
e.s.FailOutbox(due[0].ID, "refused: 554 "+esc)
for _, args := range [][]string{
{"archive", "ls", "dev@example.org"}, {"archive", "ls", "dev@example.org", "--q", "subj"},
{"archive", "show", "1"}, {"held", "ls", "dev@example.org"}, {"queue", "ls"},
} {
out := e.ok(t, args...)
noControl(t, strings.Join(args, " "), out)
if !strings.Contains(out, `\x1b[31mRED`) {
t.Errorf("%v: the escape sequence should stay visible as text:\n%s", args, out)
}
}
if out := e.ok(t, "archive", "show", "1"); !strings.Contains(out, "\nbody ") || strings.Contains(out, `\x0d`) {
t.Errorf("CRLF line ends are not control noise: %q", out)
}
}
func TestControlCharactersInListAndMemberNamesAreEscaped(t *testing.T) {
e := newEnv(t)
addr := "a\x1b[31mb\x07@x.org"
l, err := e.s.CreateList("a\x1b[31mb\x07", "x.org", "title "+esc)
if err != nil {
t.Fatal(err)
}
e.s.AddMember(l.ID, "m\x1b[2Jm@x.org", "name "+esc)
mbox := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nbody\n"
for _, c := range []struct {
stdin string
args []string
}{
{"", []string{"list", "ls"}}, {"", []string{"list", "show", addr}},
{"", []string{"list", "set", addr, "--title", "t"}},
{"", []string{"member", "ls", addr}}, {"", []string{"member", "ls", addr, "--emails"}},
{"", []string{"member", "add", addr, "new@x.com"}},
{"new2@x.com\n", []string{"member", "import", addr, "-"}},
{mbox, []string{"archive", "import", addr, "-"}},
{"", []string{"member", "rm", addr, "new@x.com"}},
{"", []string{"list", "rm", addr, "--yes"}},
} {
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": e.srv.URL}, c.stdin, c.args...)
if code != 0 {
t.Fatalf("%v: exit %d %s", c.args, code, errs)
}
noControl(t, strings.Join(c.args, " "), out+errs)
}
}
func TestControlCharactersInImportReportsAreEscaped(t *testing.T) {
e := newEnv(t)
e.ok(t, "list", "add", "dev@example.org")
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": e.srv.URL},
"bad"+esc+"@x\n", "member", "import", "dev@example.org", "-")
if code != 1 || !strings.Contains(errs, `line 1: bad\x1b[31mRED`) {
t.Fatalf("member import: %d %q", code, errs)
}
noControl(t, "member import", out+errs)
fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
b, _ := json.Marshal(map[string]any{"imported": 0, "failed": 1, "errors": []string{"message 1: " + esc}})
w.Write(b)
return
}
w.Write([]byte(`[{"id":1,"address":"dev@example.org"}]`))
}))
defer fake.Close()
var o, eb bytes.Buffer
vars := map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": fake.URL}
Run([]string{"archive", "import", "dev@example.org", "-"}, IO{In: strings.NewReader("x"), Out: &o, Err: &eb}, func(k string) string { return vars[k] })
if !strings.Contains(eb.String(), `message 1: \x1b[31mRED`) {
t.Fatalf("archive import errors: %q", eb.String())
}
noControl(t, "archive import", o.String()+eb.String())
}
func TestControlCharactersInAPIErrorsAndImportReportsAreEscaped(t *testing.T) {
api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadGateway)
msg, _ := json.Marshal(map[string]string{"error": "relay said " + esc})
w.Write(msg)
}))
defer api.Close()
var out, errb bytes.Buffer
vars := map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": api.URL}
code := Run([]string{"list", "ls"}, IO{In: strings.NewReader(""), Out: &out, Err: &errb}, func(k string) string { return vars[k] })
if code != 1 || !strings.Contains(errb.String(), `relay said \x1b[31mRED`) {
t.Fatalf("exit %d: %q", code, errb.String())
}
noControl(t, "error text", errb.String())
}
func TestTokenIsNotSentOverPlainHTTPToARemoteHost(t *testing.T) {
e := newEnv(t)
run := func(url string, extra ...string) (int, string) {
args := append(append([]string{"--url", url}, extra...), "list", "ls")
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token}, "", args...)
if out != "" {
t.Fatalf("stdout %q", out)
}
return code, errs
}
code, errs := run("http://lists.example.org:8080")
if code != 2 || !strings.Contains(errs, "--insecure") || !strings.Contains(errs, "http") {
t.Fatalf("remote http must be refused with a usage error: %d %q", code, errs)
}
if code, errs = run("HTTP://10.1.2.3"); code != 2 {
t.Fatalf("scheme case must not matter: %d %q", code, errs)
}
// Past the check the only thing left to fail is the connection (port 1 is closed, .invalid never resolves).
for _, u := range []string{"http://localhost:1", "http://LocalHost.:1", "http://127.0.0.1:1", "http://[::1]:1", "https://nowhere.invalid"} {
if code, errs = run(u); code != 1 || !strings.Contains(errs, "cannot reach") {
t.Errorf("%s must be allowed: %d %q", u, code, errs)
}
}
if code, errs = run("http://nowhere.invalid", "--insecure"); code != 1 || !strings.Contains(errs, "cannot reach") {
t.Fatalf("--insecure allows plain http: %d %q", code, errs)
}
// The environment variable is checked like the flag.
code, _, errs = e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": "http://lists.example.org"}, "", "list", "ls")
if code != 2 {
t.Fatalf("PARSIMONY_URL: %d %q", code, errs)
}
// No token to protect, no URL check: offline commands never build the client.
_, _, errs = e.runEnv(t, nil, "", "--url", "http://lists.example.org", "dkim", "record", "example.org", "--key", filepath.Join(t.TempDir(), "none"))
if strings.Contains(errs, "--insecure") {
t.Fatalf("offline commands must not care about the URL")
}
}

View file

@ -135,7 +135,7 @@ func (c *cmd) listSet(args []string) int {
if c.json {
return c.printJSON(raw)
}
fmt.Fprintf(c.io.Out, "Updated %s\n", l.Address)
fmt.Fprintf(c.io.Out, "Updated %s\n", printable(l.Address))
return 0
}
@ -168,7 +168,7 @@ func (c *cmd) done(msg string) int {
if c.json {
return c.printJSON(nil)
}
fmt.Fprintln(c.io.Out, msg)
fmt.Fprintln(c.io.Out, printable(msg))
return 0
}
@ -220,7 +220,7 @@ func (c *cmd) memberLs(args []string) int {
}
if *emails {
for _, m := range ms {
fmt.Fprintln(c.io.Out, m.Address)
fmt.Fprintln(c.io.Out, printable(m.Address))
}
return 0
}
@ -259,7 +259,7 @@ func (c *cmd) memberAdd(args []string) int {
}
var m member
json.Unmarshal(raw, &m)
fmt.Fprintf(c.io.Out, "Added %s to %s\n", m.Address, l.Address)
fmt.Fprintf(c.io.Out, "Added %s to %s\n", printable(m.Address), printable(l.Address))
return 0
}
@ -327,7 +327,7 @@ func (c *cmd) memberImport(args []string) int {
}
if _, err := c.addMember(l.ID, address, name); err != nil {
failed++
fmt.Fprintf(c.io.Err, "line %d: %s: %v\n", n, line, err)
fmt.Fprintf(c.io.Err, "line %d: %s\n", n, printable(fmt.Sprintf("%s: %v", line, err)))
continue
}
imported++
@ -339,7 +339,7 @@ func (c *cmd) memberImport(args []string) int {
if failed > 0 {
msg += fmt.Sprintf(", %d failed", failed)
}
fmt.Fprintln(c.io.Out, msg)
fmt.Fprintln(c.io.Out, printable(msg))
if failed > 0 {
return 1
}
@ -544,7 +544,7 @@ func (c *cmd) archiveShow(args []string) int {
if err := json.Unmarshal(raw, &m); err != nil {
return err
}
fmt.Fprintf(c.io.Out, "Subject: %s\nFrom: %s\nDate: %s\n\n%s\n", m.Subject, m.Sender, formatDate(m.Date), m.Body)
fmt.Fprintf(c.io.Out, "Subject: %s\nFrom: %s\nDate: %s\n\n%s\n", printable(m.Subject), printable(m.Sender), formatDate(m.Date), printable(m.Body))
return nil
}, "")
}
@ -601,10 +601,10 @@ func (c *cmd) archiveImport(args []string) int {
if sum.Failed > 0 {
msg += fmt.Sprintf(", %d failed", sum.Failed)
}
fmt.Fprintln(c.io.Out, msg)
fmt.Fprintln(c.io.Out, printable(msg))
}
for _, e := range sum.Errors {
fmt.Fprintln(c.io.Err, e)
fmt.Fprintln(c.io.Err, printable(e))
}
if sum.Failed > len(sum.Errors) {
fmt.Fprintf(c.io.Err, "... and %d more\n", sum.Failed-len(sum.Errors))

View file

@ -7,8 +7,10 @@ import (
"errors"
"fmt"
"net"
"net/url"
"os"
"strings"
"unicode"
"gopkg.in/yaml.v3"
)
@ -145,7 +147,31 @@ func (c Config) validateRelay() error {
return nil
}
// validateBaseURL checks base_url, the public URL the one-click unsubscribe links are built from.
// RFC 8058 needs https; http is accepted for a loopback host (development and tests). Empty is fine:
// there is no one-click link then.
func (c Config) validateBaseURL() error {
if c.BaseURL == "" {
return nil
}
u, err := url.Parse(c.BaseURL)
switch {
case err != nil || strings.ContainsFunc(c.BaseURL, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }):
return fmt.Errorf("base_url %q is not a valid URL", c.BaseURL)
case u.Hostname() == "":
return fmt.Errorf("base_url %q needs a scheme and a host, such as https://lists.example.org", c.BaseURL)
case u.Scheme != "https" && !(u.Scheme == "http" && isLoopbackHost(u.Hostname())):
return fmt.Errorf("base_url %q must use https (one-click unsubscribe, RFC 8058); http is only accepted for localhost", c.BaseURL)
case u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || strings.Contains(c.BaseURL, "#"):
return fmt.Errorf("base_url %q must not contain credentials, a query or a fragment", c.BaseURL)
}
return nil
}
func (c Config) validate() error {
if err := c.validateBaseURL(); err != nil {
return err
}
if err := c.validateRelay(); err != nil {
return err
}

View file

@ -246,3 +246,43 @@ func testCAPEM(t *testing.T) []byte {
}
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
}
func TestBaseURLIsChecked(t *testing.T) {
for _, ok := range []string{
"", // one-click unsubscribe is simply off
"https://lists.example.org",
"https://lists.example.org/",
"https://lists.example.org:8443/sub/path",
"http://localhost:8081",
"http://127.0.0.1:8081",
"http://[::1]:8081",
} {
if _, err := Load(write(t, "base_url: \""+ok+"\"\n")); err != nil {
t.Errorf("%q must be accepted: %v", ok, err)
}
}
for _, bad := range []string{
"http://lists.example.org", // RFC 8058 needs https
"http://10.0.0.5:8081", // only loopback may use http
"ftp://lists.example.org", // not http(s) at all
"lists.example.org", // no scheme
"//lists.example.org", // no scheme
"https://", // no host
"https:///path", // no host
"https://lists.example.org?x=1", // query
"https://lists.example.org/#top",
"https://lists.example.org/#", // an empty fragment is still one
"https://lists.example.org/?", // an empty query is still one // fragment
"https://user:pw@lists.example.org", // userinfo
"https://user@lists.example.org",
"https://lists.example.org/a b",
"https://exa mple.org",
} {
_, err := Load(write(t, "base_url: \""+bad+"\"\n"))
if err == nil {
t.Errorf("%q must be rejected", bad)
} else if !strings.Contains(err.Error(), "base_url") {
t.Errorf("%q: the error must name base_url: %v", bad, err)
}
}
}

View file

@ -86,7 +86,7 @@ func (s *session) deliver(rcpt string, data []byte) error {
// status turns a router result into an LMTP reply.
func (s *session) status(rcpt string, err error) error {
switch {
case err == nil, errors.Is(err, router.ErrHeld), errors.Is(err, router.ErrDiscarded):
case err == nil, errors.Is(err, router.ErrHeld), errors.Is(err, router.ErrDiscarded), errors.Is(err, router.ErrDuplicate):
return nil
case errors.Is(err, router.ErrUnknownList):
return &smtp.SMTPError{Code: 550, EnhancedCode: smtp.EnhancedCode{5, 1, 1}, Message: "no such list"}

View file

@ -255,7 +255,7 @@ func TestModeratedPostIsAcceptedButNotQueued(t *testing.T) {
func TestAutoReplyAndLoopsAreAcceptedButNotDelivered(t *testing.T) {
st := newStack(t, nil)
for _, h := range []string{"Auto-Submitted: auto-replied\r\n", "List-Id: Dev <dev.example.org>\r\n"} {
for _, h := range []string{"Auto-Submitted: auto-replied\r\n", "Precedence: auto_reply\r\n", "List-Id: Dev <dev.example.org>\r\n"} {
conn, err := net.Dial("tcp", st.addr)
if err != nil {
t.Fatal(err)
@ -276,6 +276,29 @@ func TestAutoReplyAndLoopsAreAcceptedButNotDelivered(t *testing.T) {
}
}
func TestAutoGeneratedPostsAreDelivered(t *testing.T) {
st := newStack(t, nil)
for _, h := range []string{"Auto-Submitted: auto-generated\r\n", "Auto-Submitted: auto-notified\r\n"} {
conn, err := net.Dial("tcp", st.addr)
if err != nil {
t.Fatal(err)
}
c := smtp.NewClientLMTP(conn)
c.Hello("localhost")
c.Mail("alice@example.com", nil)
c.Rcpt("dev@example.org", nil)
w, _ := c.Data()
w.Write([]byte("From: Alice <alice@example.com>\r\n" + h + "Subject: nightly report\r\nMessage-ID: <" + strings.TrimSpace(h) + "@x>\r\n\r\nbody\r\n"))
if _, err := w.CloseWithLMTPResponse(); err != nil {
t.Fatalf("%q: %v", h, err)
}
c.Close()
}
if st.queued(t) == 0 {
t.Fatal("a cron or CI post from a member must be queued for delivery")
}
}
func TestAFullModerationQueueAnswers452(t *testing.T) {
st := newStack(t, nil)
l, _ := st.store.FindList("dev@example.org")
@ -316,3 +339,15 @@ func TestAMessageTooBigToHoldAnswers552(t *testing.T) {
t.Fatalf("a permanent refusal: the sender is told to send it elsewhere: %v", err)
}
}
func TestTheSamePostDeliveredAgainIsAcceptedButNotQueuedTwice(t *testing.T) {
st := newStack(t, down{}) // the deliveries stay in the queue, where they can be counted
for i := 0; i < 2; i++ {
if res := deliver(t, st.addr, "alice@example.com", []string{"dev@example.org"}); res["dev@example.org"] != nil {
t.Fatalf("delivery %d: a duplicate is acknowledged with 250, got %v", i+1, res)
}
}
if n := st.queued(t); n != 2 {
t.Fatalf("queued = %d, want one delivery per member", n)
}
}

View file

@ -67,14 +67,45 @@ func NewReader(r io.Reader, maxSize int) *Reader {
return &Reader{br: bufio.NewReaderSize(r, 64*1024), max: maxSize}
}
// readLine returns the next line including its newline. At the end of the input
// it returns the last partial line (if any) with io.EOF on the following call.
func (r *Reader) readLine() ([]byte, error) {
line, err := r.br.ReadBytes('\n')
if err == io.EOF && len(line) > 0 {
return line, nil
// minKeep is the least a line may be cut to, so a "From " separator is never cut short.
const minKeep = 1024
// keep is how many bytes of a line are worth holding: a message with a longer line is too
// large anyway, so the rest is read and thrown away instead of buffered.
func (r *Reader) keep() int {
if r.max <= 0 {
return 0 // no limit
}
return max(r.max+1, minKeep)
}
// readLine returns the next line including its newline, cut to keep bytes (0 means no cut),
// and the full length of the line. At the end of the input it returns the last partial line
// (if any) with io.EOF on the following call.
func (r *Reader) readLine(keep int) ([]byte, int, error) {
var line []byte
n := 0
for {
chunk, err := r.br.ReadSlice('\n')
n += len(chunk)
if room := keep - len(line); keep == 0 || room > 0 {
if keep > 0 && len(chunk) > room {
chunk = chunk[:room]
}
line = append(line, chunk...)
}
switch err {
case nil:
return line, n, nil
case bufio.ErrBufferFull:
continue
case io.EOF:
if n > 0 {
return line, n, nil
}
}
return line, n, err
}
return line, err
}
// Next returns the next message, or io.EOF when there are no more.
@ -84,7 +115,7 @@ func (r *Reader) Next() (Message, error) {
}
// Skip anything before the first separator.
for r.next == "" {
line, err := r.readLine()
line, _, err := r.readLine(r.keep())
if err != nil {
r.eof = true
if err == io.EOF {
@ -102,8 +133,9 @@ func (r *Reader) Next() (Message, error) {
var buf bytes.Buffer
oversized := false
size := 0
for {
line, err := r.readLine()
line, n, err := r.readLine(r.keep())
if err != nil {
r.eof = true
if err != io.EOF {
@ -120,12 +152,15 @@ func (r *Reader) Next() (Message, error) {
}
if escaped.Match(line) {
line = line[1:]
n--
}
buf.Write(line)
if r.max > 0 && buf.Len() > r.max {
size += n
if r.max > 0 && size > r.max {
oversized = true
buf.Reset()
continue
}
buf.Write(line)
}
if oversized {
msg.Err = ErrTooLarge

View file

@ -166,3 +166,93 @@ func TestReadErrorsArePropagated(t *testing.T) {
type errReader struct{}
func (errReader) Read([]byte) (int, error) { return 0, errors.New("disk on fire") }
func TestAVeryLongLineIsTooLargeAndTheReaderResynchronises(t *testing.T) {
long := strings.Repeat("a", 5000) // far above the limit of 100, with no way to be a separator
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + long + "\n\n" +
"From b@x.com Wed Jan 3 15:04:05 2024\nSubject: ok\n\nfine\n"
ms := readAll(t, in, 100)
if len(ms) != 2 || !errors.Is(ms[0].Err, ErrTooLarge) || ms[0].Data != nil {
t.Fatalf("messages: %+v", ms)
}
if ms[1].Err != nil || !strings.Contains(string(ms[1].Data), "fine") {
t.Fatalf("the next message must be read: %+v", ms[1])
}
}
func TestALongLineAtTheEndOfTheInputIsTooLarge(t *testing.T) {
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + strings.Repeat("a", 5000) // no newline
ms := readAll(t, in, 100)
if len(ms) != 1 || !errors.Is(ms[0].Err, ErrTooLarge) {
t.Fatalf("messages: %+v", ms)
}
}
func TestAnOverlongLineBeforeTheFirstSeparatorIsSkipped(t *testing.T) {
in := strings.Repeat("x", 5000) + "\nFrom a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nbody\n"
ms := readAll(t, in, 100)
if len(ms) != 1 || ms[0].Err != nil || !strings.Contains(string(ms[0].Data), "body") {
t.Fatalf("messages: %+v", ms)
}
}
func TestLongLinesBelowTheLimitStillWork(t *testing.T) {
long := strings.Repeat("a", 3000)
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + long + "\n"
ms := readAll(t, in, 4000)
if len(ms) != 1 || ms[0].Err != nil || !strings.Contains(string(ms[0].Data), long+"\n") {
t.Fatalf("messages: %+v", ms)
}
if ms = readAll(t, in, 0); len(ms) != 1 || ms[0].Err != nil {
t.Fatalf("no limit: %+v", ms)
}
}
func TestReadLineKeepsAtMostTheLimitButReportsTheFullLength(t *testing.T) {
// A line far longer than the reader's buffer: only keep bytes may be held.
in := strings.Repeat("a", 300_000) + "\nnext\n"
r := NewReader(strings.NewReader(in), 0)
line, n, err := r.readLine(1000)
if err != nil || n != 300_001 || len(line) > 1000 {
t.Fatalf("n=%d kept=%d err=%v", n, len(line), err)
}
if line, n, err = r.readLine(1000); err != nil || string(line) != "next\n" || n != 5 {
t.Fatalf("the following line: %q %d %v", line, n, err)
}
if _, _, err = r.readLine(1000); err != io.EOF {
t.Fatalf("want EOF, got %v", err)
}
// keep 0 means keep everything.
r = NewReader(strings.NewReader(in), 0)
if line, n, _ = r.readLine(0); len(line) != 300_001 || n != 300_001 {
t.Fatalf("unlimited: %d %d", len(line), n)
}
}
func TestASmallLimitDoesNotCutSeparatorsShort(t *testing.T) {
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nlong enough to be too large\n" +
"From b@x.com Wed Jan 3 15:04:05 2024\nok\n"
ms := readAll(t, in, 10)
if len(ms) != 2 || !errors.Is(ms[0].Err, ErrTooLarge) || ms[1].Err != nil || string(ms[1].Data) != "ok\n" {
t.Fatalf("messages: %+v", ms)
}
}
func TestKeepIsBoundedByTheLimit(t *testing.T) {
for _, c := range []struct{ max, want int }{{0, 0}, {-1, 0}, {100, minKeep}, {5000, 5001}} {
if got := NewReader(strings.NewReader(""), c.max).keep(); got != c.want {
t.Errorf("max %d: keep %d, want %d", c.max, got, c.want)
}
}
}
func TestTheLimitIsExactAndCountsTheRestoredLine(t *testing.T) {
body := "Subject: s\n\n>From here\n" // restored to "From here\n": 22 bytes
in := "From a@x.com Tue Jan 2 15:04:05 2024\n" + body
if ms := readAll(t, in, 22); len(ms) != 1 || ms[0].Err != nil {
t.Fatalf("exactly at the limit must pass: %+v", ms)
}
if ms := readAll(t, in, 21); len(ms) != 1 || !errors.Is(ms[0].Err, ErrTooLarge) {
t.Fatalf("one over the limit must be too large: %+v", ms)
}
}

View file

@ -22,6 +22,9 @@ const (
defaultBatch = 50
defaultTimeout = time.Minute
defaultInterval = 5 * time.Second
// purgeEvery is the least time between two scans that forget old failed deliveries.
purgeEvery = 5 * time.Minute
)
var backoff = []time.Duration{time.Minute, 5 * time.Minute, 15 * time.Minute, time.Hour, 3 * time.Hour, 6 * time.Hour}
@ -61,6 +64,53 @@ type Worker struct {
wakeOnce sync.Once
wake chan struct{}
runMu sync.Mutex
lastPurge time.Time // under runMu; zero until the first purge
// Hooks for tests; nil means the Store methods.
purge func() (int, error)
readBody func(msgID int64) ([]byte, error)
}
// bodies reads each message body once per run, however many recipients share it.
type bodies struct {
read func(msgID int64) ([]byte, error)
mu sync.Mutex
m map[int64]*body
}
type body struct {
once sync.Once
data []byte
err error
}
func (b *bodies) get(msgID int64) ([]byte, error) {
b.mu.Lock()
e := b.m[msgID]
if e == nil {
e = &body{}
b.m[msgID] = e
}
b.mu.Unlock()
e.once.Do(func() { e.data, e.err = b.read(msgID) })
return e.data, e.err
}
// purgeOld forgets old failed deliveries, at most once every purgeEvery.
func (w *Worker) purgeOld() {
now := w.Store.Now()
if !w.lastPurge.IsZero() && now.Sub(w.lastPurge) < purgeEvery {
return
}
purge := w.purge
if purge == nil {
purge = w.Store.PurgeFailedOutbox
}
if _, err := purge(); err != nil {
log.Printf("queue: purge old failures: %v", err)
return // try again on the next run
}
w.lastPurge = now
}
func (w *Worker) wakeCh() chan struct{} {
@ -125,14 +175,18 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
if conc <= 0 {
conc = defaultConcurrency
}
if _, err := w.Store.PurgeFailedOutbox(); err != nil {
log.Printf("queue: purge old failures: %v", err)
}
w.purgeOld()
due, err := w.Store.DueOutbox(batch)
if err != nil {
return Stats{}, err
}
read := w.readBody
if read == nil {
read = w.Store.OutboxData
}
cache := &bodies{read: read, m: map[int64]*body{}}
var (
mu sync.Mutex
stats Stats
@ -148,7 +202,7 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
go func(it store.OutboxItem) {
defer wg.Done()
defer func() { <-sem }()
o := w.deliver(ctx, it)
o := w.deliver(ctx, cache, it)
mu.Lock()
stats.Sent += o.Sent
stats.Retried += o.Retried
@ -162,8 +216,8 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
return stats, nil
}
func (w *Worker) deliver(ctx context.Context, it store.OutboxItem) Stats {
data, err := w.Store.OutboxData(it.MsgID)
func (w *Worker) deliver(ctx context.Context, cache *bodies, it store.OutboxItem) Stats {
data, err := cache.get(it.MsgID)
if errors.Is(err, store.ErrNotFound) {
w.complete(it)
return Stats{Dropped: 1}

Some files were not shown because too many files have changed in this diff Show more