Merge pull request 'Fix the medium and low findings of the global review' (#33) from fix/fable-mediums into main
This commit is contained in:
commit
6f745e9b25
38 changed files with 3164 additions and 207 deletions
48
README.md
48
README.md
|
|
@ -10,9 +10,11 @@ A modern, single-binary mailing list manager: an evolution of Mailman.
|
|||
and a background worker sends it with retries (1m, 5m, 15m, 1h, 3h, then every 6h, for up to
|
||||
3 days). Relay outages and restarts lose nothing.
|
||||
- Self-service subscribe and unsubscribe by email with double opt-in:
|
||||
send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation.
|
||||
send to `list+join@domain` or `list+leave@domain`, then reply to the confirmation. A reply works once, a request
|
||||
expires after 72 hours, and a confirmation that could not be applied (closed list, database error) can be replied to again.
|
||||
A member disabled by bounces can rejoin the same way.
|
||||
- Moderated lists: posts wait in a queue that the API can approve or discard.
|
||||
- VERP bounces: every recipient gets a signed return address; 3 hard bounces disable a member.
|
||||
- VERP bounces: every recipient gets a signed return address; 3 hard bounces within 30 days disable a member.
|
||||
- Archive with full-text search (SQLite FTS5) and threading. Each list has an
|
||||
`archive_policy`: `private` (default, API only), `public` (web pages) or `off`.
|
||||
Public pages hide email addresses.
|
||||
|
|
@ -41,12 +43,18 @@ Three listeners, each configurable:
|
|||
| `web` | `127.0.0.1:8081` | public archive and unsubscribe pages (`""` disables) |
|
||||
|
||||
Put the `web` listener behind a reverse proxy with TLS and set `base_url` to its public URL:
|
||||
the unsubscribe links in every message use it. Keep the `api` listener private.
|
||||
the unsubscribe links in every message use it. It must be an `https://` URL (RFC 8058 one-click unsubscribe needs
|
||||
it; `http://` is accepted only for `localhost`, `127.0.0.1` and `::1`) with a host and no credentials, query or fragment;
|
||||
the daemon refuses to start otherwise. Leave it empty to turn the one-click links off. Keep the `api` listener private.
|
||||
|
||||
```
|
||||
curl -H "Authorization: Bearer $PARSIMONY_API_TOKEN" -d '{"name":"dev","domain":"example.org","title":"Dev"}' localhost:8080/api/v1/lists
|
||||
```
|
||||
|
||||
A list `name` is 1 to 30 characters of `a-z`, `0-9`, `.`, `_` and `-` (upper case is lowered, no dot at either end), the `domain`
|
||||
a host name, and the `title` one line of at most 200 characters without control characters; anything else is a 400.
|
||||
The title is encoded (RFC 2047) in the `List-Id` header when it is not ASCII.
|
||||
|
||||
## Command line
|
||||
|
||||
The same binary is an administration client. It talks to the REST API, so it works against a local or remote daemon.
|
||||
|
|
@ -70,6 +78,10 @@ parsimony help
|
|||
|
||||
`<list>` is an address or a numeric id. Destructive commands (`list rm`, `archive rm`) need `--yes`.
|
||||
Exit status: 0 success, 1 failure (including lines that failed in `member import`), 2 usage error.
|
||||
The client refuses to send the token over plain `http://` to any host other than `localhost`, `127.0.0.1` or `::1`
|
||||
(usage error, exit 2): put the API behind HTTPS, or pass `--insecure` if you really mean it.
|
||||
Subjects, senders, message bodies and error texts that come from mail are printed with control characters
|
||||
(escape sequences) shown as `\x1b`-style text instead of being sent to the terminal; `--json` output is already escaped.
|
||||
`parsimony serve -config FILE` runs the daemon; the old `parsimony -config FILE` form still works.
|
||||
In the container: `docker exec parsimony /parsimony list ls`.
|
||||
|
||||
|
|
@ -183,9 +195,22 @@ It answers `451` only when the database itself fails.
|
|||
delivery and counts as a hard bounce for the member (3 disable it). Nothing else counts: not "relay access denied", not a
|
||||
refused sender (`5.1.7`, `5.1.8`, or any code answering `MAIL FROM`), not a refusal after `DATA`. Those deliveries are kept
|
||||
as failed.
|
||||
- A bounce that comes back later to the member's return address counts only if it is a real delivery status report
|
||||
(RFC 3464: `multipart/report` with a `message/delivery-status` part) with `Action: failed` and one of the same
|
||||
codes. A spam or policy rejection (`5.7.x`), a full mailbox (`5.2.x`), a delay, a `4.x.x`, or a `Status:` line in a
|
||||
plain or quoted message does not count.
|
||||
- Hard bounces count for 30 days: when the last one is older, the next one starts again from 1. A disabled member
|
||||
stays disabled until it is added again.
|
||||
- A relay that accepted the message is never asked again because `QUIT` went wrong.
|
||||
- Delivery is at-least-once: a crash between the relay accepting a message and the queue entry being
|
||||
removed can send one message to one recipient twice.
|
||||
- The same post is queued once. If the `250` for a post never reaches Postfix, Postfix delivers the post again; a list
|
||||
remembers each post it queued for an hour (by `Message-ID`, or by a hash of the whole message when it has none),
|
||||
answers `250` to a copy and does not queue it again (the log says so). An approved held post is always delivered and is
|
||||
remembered too, so a copy that arrives later is not.
|
||||
- On stop, the daemon refuses new connections and lets the LMTP sessions and API or web requests in flight finish, so a
|
||||
post being received gets its `250`. After 30 seconds it cuts whatever is still open.
|
||||
- What remains is at-least-once delivery across the relay: a crash between the relay accepting a message and the queue
|
||||
entry being removed can send one message to one recipient twice. A copy of a post that arrives more than an hour later
|
||||
is also delivered again.
|
||||
- `GET /api/v1/queue` and the Queue page show what is pending or failing, and a stuck delivery can be discarded.
|
||||
|
||||
## Admin UI
|
||||
|
|
@ -249,10 +274,23 @@ lists.example.org lmtp:inet:127.0.0.1:2424
|
|||
and `Reply-To` come from the header; so a post whose header names someone else (or has no usable `From`, or several
|
||||
addresses) is held for moderation instead of delivered, with the reason shown to the admin (API, CLI, UI). Let Postfix
|
||||
reject forged senders too (SPF, DKIM, DMARC checks on inbound mail).
|
||||
- **Loops and robots.** Mail that carries the list's own `List-Id` (a loop) or is a genuine auto-reply
|
||||
(`Auto-Submitted: auto-replied`, `Precedence: auto_reply`) is answered `250` and dropped without a trace, so the
|
||||
sender's server never bounces it. Other machine-generated mail (`Auto-Submitted: auto-generated`, cron, CI and
|
||||
monitoring posts) is a normal post: the membership, `From` and moderation checks apply. The `join`, `leave` and
|
||||
`confirm` addresses ignore (`250`, nothing happens, the token is kept) a mail with a null sender, any
|
||||
`Auto-Submitted` other than `no`, a `Precedence` of `auto_reply`, `bulk` or `junk`, or the list's own `List-Id`.
|
||||
This stops an auto-responder from confirming a subscription someone forged in your name.
|
||||
- **The moderation queue is bounded.** A list holds at most 500 messages and 200 MiB, and one held message at most 10 MiB; a
|
||||
full queue answers `452` (the sender retries later), a message over 10 MiB `552`. Messages nobody moderated for 30 days
|
||||
are dropped. Listings (API, CLI, UI) show a summary and never read the bodies. The held view shows the From header
|
||||
readers would see next to the real sender and flags a difference.
|
||||
- **Hostile messages and slow clients.** The archive reads at most 10 levels of nested multipart and 100 parts per
|
||||
message, then uses what it found (the post is still delivered). A `Date` header more than 24 hours ahead is replaced
|
||||
by the arrival time, and one before 1990 is ignored, so a message cannot pin itself to the top of the archive. The
|
||||
HTTP servers (API + admin UI, and the public web) cut a request that takes more than 30 s to read, a response that
|
||||
takes more than 60 s to write, and a connection idle for 120 s; the archive import is the exception and may stream
|
||||
for up to 30 minutes.
|
||||
- **API token.** `parsimony serve` refuses a token shorter than 24 characters or containing whitespace
|
||||
(`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is
|
||||
locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy
|
||||
|
|
|
|||
|
|
@ -76,15 +76,21 @@ describe('held tab', () => {
|
|||
expect(await screen.findByText(/nothing is waiting/i)).toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('explains how messages get held when posting is not moderated', async () => {
|
||||
await setup([], aList({ id: 1, post_policy: 'members' }))
|
||||
expect(await screen.findByText(/set posting to moderated/i)).toBeInTheDocument()
|
||||
it('explains both ways a message gets held, whatever the posting policy', async () => {
|
||||
for (const post_policy of ['members', 'moderated'] as const) {
|
||||
const { unmount } = await setup([], aList({ id: 1, post_policy }))
|
||||
const hint = await screen.findByText(/messages are held when/i)
|
||||
expect(hint).toHaveTextContent(/posting is set to moderated/i)
|
||||
expect(hint).toHaveTextContent(/from header does not match the sender/i)
|
||||
expect(hint).not.toHaveTextContent(/only/i)
|
||||
unmount()
|
||||
}
|
||||
})
|
||||
|
||||
it('does not nag about the policy when the list is moderated', async () => {
|
||||
await setup([])
|
||||
await screen.findByText(/nothing is waiting/i)
|
||||
expect(screen.queryByText(/set posting to moderated/i)).not.toBeInTheDocument()
|
||||
it('shows no hint when there are held messages', async () => {
|
||||
await setup([held(1)])
|
||||
await screen.findByText('Subject 1')
|
||||
expect(screen.queryByText(/messages are held when/i)).not.toBeInTheDocument()
|
||||
})
|
||||
|
||||
it('approves a message: it is sent and leaves the queue', async () => {
|
||||
|
|
|
|||
|
|
@ -51,9 +51,10 @@ export function HeldTab() {
|
|||
{held.data && items.length === 0 && (
|
||||
<div className="space-y-1 text-muted-foreground">
|
||||
<p>Nothing is waiting for approval.</p>
|
||||
{list.post_policy !== 'moderated' && (
|
||||
<p className="text-sm">Messages are held only when you set Posting to moderated in Settings.</p>
|
||||
)}
|
||||
<p className="text-sm">
|
||||
Messages are held when Posting is set to moderated in Settings, or when the From header does not match
|
||||
the sender.
|
||||
</p>
|
||||
</div>
|
||||
)}
|
||||
{items.length > 0 && (
|
||||
|
|
|
|||
|
|
@ -50,8 +50,8 @@ components:
|
|||
properties:
|
||||
address: {type: string}
|
||||
display_name: {type: string}
|
||||
disabled: {type: boolean, description: Disabled after 3 hard bounces; re-adding the member re-enables it}
|
||||
bounce_count: {type: integer}
|
||||
disabled: {type: boolean, description: Disabled after 3 hard bounces within 30 days; re-adding the member re-enables it}
|
||||
bounce_count: {type: integer, description: Recent hard bounces (restarts at 1 when the last one is older than 30 days)}
|
||||
Held:
|
||||
type: object
|
||||
properties:
|
||||
|
|
@ -99,9 +99,19 @@ paths:
|
|||
type: object
|
||||
required: [name, domain]
|
||||
properties:
|
||||
name: {type: string}
|
||||
domain: {type: string}
|
||||
title: {type: string}
|
||||
name:
|
||||
type: string
|
||||
minLength: 1
|
||||
maxLength: 30
|
||||
pattern: '^[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*$'
|
||||
description: Local part of the list address, stored lowercase. Letters, digits, '.', '_' and '-' only, no dot at either end or two in a row.
|
||||
domain:
|
||||
type: string
|
||||
description: ASCII host name (labels of 1 to 63 letters, digits and hyphens).
|
||||
title:
|
||||
type: string
|
||||
maxLength: 200
|
||||
description: A single line without control characters. Defaults to the name.
|
||||
responses:
|
||||
'201':
|
||||
description: Created
|
||||
|
|
@ -131,7 +141,7 @@ paths:
|
|||
schema:
|
||||
type: object
|
||||
properties:
|
||||
title: {type: string}
|
||||
title: {type: string, minLength: 1, maxLength: 200, description: A single line without control characters.}
|
||||
post_policy: {type: string, enum: [members, moderated]}
|
||||
join_policy: {type: string, enum: [open, closed]}
|
||||
archive_policy: {type: string, enum: [public, private, off]}
|
||||
|
|
|
|||
|
|
@ -12,6 +12,8 @@ import (
|
|||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"codeberg.org/squidd/parsimony/internal/archive"
|
||||
"codeberg.org/squidd/parsimony/internal/store"
|
||||
|
|
@ -40,29 +42,44 @@ func New(s *store.Store, token string, rel Releaser) http.Handler {
|
|||
func newHandler(s *store.Store, token string, rel Releaser, now func() time.Time) http.Handler {
|
||||
srv := &server{store: s, token: token, releaser: rel, lim: newLimiter(now)}
|
||||
mux := http.NewServeMux()
|
||||
mux.HandleFunc("GET /api/v1/health", func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
})
|
||||
mux.Handle("GET /api/v1/lists", srv.auth(srv.listLists))
|
||||
mux.Handle("POST /api/v1/lists", srv.auth(srv.createList))
|
||||
mux.Handle("GET /api/v1/lists/{id}", srv.auth(srv.getList))
|
||||
mux.Handle("PATCH /api/v1/lists/{id}", srv.auth(srv.updateList))
|
||||
mux.Handle("DELETE /api/v1/lists/{id}", srv.auth(srv.deleteList))
|
||||
mux.Handle("GET /api/v1/lists/{id}/members", srv.auth(srv.listMembers))
|
||||
mux.Handle("POST /api/v1/lists/{id}/members", srv.auth(srv.addMember))
|
||||
mux.Handle("DELETE /api/v1/lists/{id}/members/{address}", srv.auth(srv.removeMember))
|
||||
mux.Handle("GET /api/v1/lists/{id}/archive", srv.auth(srv.listArchive))
|
||||
mux.Handle("POST /api/v1/lists/{id}/archive/import", srv.auth(srv.importArchive))
|
||||
mux.Handle("GET /api/v1/archive/{id}", srv.auth(srv.getArchived))
|
||||
mux.Handle("DELETE /api/v1/archive/{id}", srv.auth(srv.deleteArchived))
|
||||
mux.Handle("GET /api/v1/queue", srv.auth(srv.getQueue))
|
||||
mux.Handle("DELETE /api/v1/queue/{id}", srv.auth(srv.discardQueued))
|
||||
mux.Handle("GET /api/v1/lists/{id}/held", srv.auth(srv.listHeld))
|
||||
mux.Handle("POST /api/v1/held/{id}/approve", srv.auth(srv.approveHeld))
|
||||
mux.Handle("DELETE /api/v1/held/{id}", srv.auth(srv.discardHeld))
|
||||
for _, rt := range srv.routes() {
|
||||
mux.Handle(rt.pattern, rt.handler)
|
||||
}
|
||||
return mux
|
||||
}
|
||||
|
||||
type route struct {
|
||||
pattern string
|
||||
handler http.Handler
|
||||
}
|
||||
|
||||
// routes is the only place routes are declared. Everything but the health check goes through auth.
|
||||
func (s *server) routes() []route {
|
||||
public := func(h http.HandlerFunc) http.Handler { return h }
|
||||
return []route{
|
||||
{"GET /api/v1/health", public(func(w http.ResponseWriter, _ *http.Request) {
|
||||
writeJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
})},
|
||||
{"GET /api/v1/lists", s.auth(s.listLists)},
|
||||
{"POST /api/v1/lists", s.auth(s.createList)},
|
||||
{"GET /api/v1/lists/{id}", s.auth(s.getList)},
|
||||
{"PATCH /api/v1/lists/{id}", s.auth(s.updateList)},
|
||||
{"DELETE /api/v1/lists/{id}", s.auth(s.deleteList)},
|
||||
{"GET /api/v1/lists/{id}/members", s.auth(s.listMembers)},
|
||||
{"POST /api/v1/lists/{id}/members", s.auth(s.addMember)},
|
||||
{"DELETE /api/v1/lists/{id}/members/{address}", s.auth(s.removeMember)},
|
||||
{"GET /api/v1/lists/{id}/archive", s.auth(s.listArchive)},
|
||||
{"POST /api/v1/lists/{id}/archive/import", s.auth(s.importArchive)},
|
||||
{"GET /api/v1/archive/{id}", s.auth(s.getArchived)},
|
||||
{"DELETE /api/v1/archive/{id}", s.auth(s.deleteArchived)},
|
||||
{"GET /api/v1/queue", s.auth(s.getQueue)},
|
||||
{"DELETE /api/v1/queue/{id}", s.auth(s.discardQueued)},
|
||||
{"GET /api/v1/lists/{id}/held", s.auth(s.listHeld)},
|
||||
{"POST /api/v1/held/{id}/approve", s.auth(s.approveHeld)},
|
||||
{"DELETE /api/v1/held/{id}", s.auth(s.discardHeld)},
|
||||
}
|
||||
}
|
||||
|
||||
func (s *server) auth(next http.HandlerFunc) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
ip := clientIP(r)
|
||||
|
|
@ -169,6 +186,53 @@ func (s *server) listLists(w http.ResponseWriter, _ *http.Request) {
|
|||
writeJSON(w, http.StatusOK, out)
|
||||
}
|
||||
|
||||
const (
|
||||
maxNameLen = 30 // keeps list+confirm-<token> and list+bounce-<id>-<mac> within 64 characters
|
||||
maxTitleLen = 200
|
||||
titleHint = "title must be a non-empty single line of at most 200 characters, without control characters"
|
||||
)
|
||||
|
||||
// validName reports whether name (already lowercased) works as the local part of a list address.
|
||||
func validName(name string) bool {
|
||||
if name == "" || len(name) > maxNameLen || strings.HasPrefix(name, ".") || strings.HasSuffix(name, ".") || strings.Contains(name, "..") {
|
||||
return false
|
||||
}
|
||||
for _, c := range name {
|
||||
if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '.' && c != '_' && c != '-' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// validDomain reports whether d (already lowercased) is an ASCII host name: dot-separated labels of
|
||||
// 1 to 63 letters, digits and hyphens, none starting or ending with a hyphen.
|
||||
func validDomain(d string) bool {
|
||||
if d == "" || len(d) > 253 {
|
||||
return false
|
||||
}
|
||||
for _, label := range strings.Split(d, ".") {
|
||||
if label == "" || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' {
|
||||
return false
|
||||
}
|
||||
for _, c := range label {
|
||||
if (c < 'a' || c > 'z') && (c < '0' || c > '9') && c != '-' {
|
||||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// validTitle reports whether t is a usable list title: not empty, at most maxTitleLen characters,
|
||||
// no line breaks or other control characters (it goes into mail headers).
|
||||
func validTitle(t string) bool {
|
||||
if t == "" || utf8.RuneCountInString(t) > maxTitleLen {
|
||||
return false
|
||||
}
|
||||
return !strings.ContainsFunc(t, unicode.IsControl)
|
||||
}
|
||||
|
||||
func (s *server) createList(w http.ResponseWriter, r *http.Request) {
|
||||
var in struct {
|
||||
Name string `json:"name"`
|
||||
|
|
@ -178,13 +242,22 @@ func (s *server) createList(w http.ResponseWriter, r *http.Request) {
|
|||
if !decodeBody(w, r, &in) {
|
||||
return
|
||||
}
|
||||
in.Name, in.Domain = strings.TrimSpace(in.Name), strings.TrimSpace(in.Domain)
|
||||
if in.Name == "" || in.Domain == "" || strings.ContainsAny(in.Name+in.Domain, "@+ \r\n") {
|
||||
writeError(w, http.StatusBadRequest, "name and domain are required and must not contain '@', '+' or spaces")
|
||||
typedName := strings.TrimSpace(in.Name)
|
||||
in.Name, in.Domain = strings.ToLower(typedName), strings.ToLower(strings.TrimSpace(in.Domain))
|
||||
if !validName(in.Name) {
|
||||
writeError(w, http.StatusBadRequest, "name must be 1 to 30 characters of a-z, 0-9, '.', '_' or '-', without a dot at either end or two dots in a row")
|
||||
return
|
||||
}
|
||||
if in.Title == "" {
|
||||
in.Title = in.Name
|
||||
if !validDomain(in.Domain) {
|
||||
writeError(w, http.StatusBadRequest, "domain must be a host name such as lists.example.org")
|
||||
return
|
||||
}
|
||||
if in.Title = strings.TrimSpace(in.Title); in.Title == "" {
|
||||
in.Title = typedName
|
||||
}
|
||||
if !validTitle(in.Title) {
|
||||
writeError(w, http.StatusBadRequest, titleHint)
|
||||
return
|
||||
}
|
||||
l, err := s.store.CreateList(in.Name, in.Domain, in.Title)
|
||||
if err != nil {
|
||||
|
|
@ -233,8 +306,8 @@ func (s *server) updateList(w http.ResponseWriter, r *http.Request) {
|
|||
}
|
||||
if in.Title != nil {
|
||||
title := strings.TrimSpace(*in.Title)
|
||||
if title == "" || strings.ContainsAny(title, "\r\n") {
|
||||
writeError(w, http.StatusBadRequest, "title must be a non-empty single line")
|
||||
if !validTitle(title) {
|
||||
writeError(w, http.StatusBadRequest, titleHint)
|
||||
return
|
||||
}
|
||||
l.Title = title
|
||||
|
|
@ -561,6 +634,10 @@ func (s *server) discardQueued(w http.ResponseWriter, r *http.Request) {
|
|||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
// importTimeout is how long one archive import may read its body and write its
|
||||
// summary, in place of the server's default read and write timeouts.
|
||||
const importTimeout = 30 * time.Minute
|
||||
|
||||
// importArchive reads an mbox from the request body, streaming, and archives it.
|
||||
// Failed messages are reported in the summary; the request itself succeeds.
|
||||
func (s *server) importArchive(w http.ResponseWriter, r *http.Request) {
|
||||
|
|
@ -572,6 +649,11 @@ func (s *server) importArchive(w http.ResponseWriter, r *http.Request) {
|
|||
storeError(w, err)
|
||||
return
|
||||
}
|
||||
// A big mbox streams for longer than the server's default timeouts. Errors
|
||||
// are ignored: a ResponseWriter that cannot extend deadlines has none.
|
||||
rc := http.NewResponseController(w)
|
||||
_ = rc.SetReadDeadline(time.Now().Add(importTimeout))
|
||||
_ = rc.SetWriteDeadline(time.Now().Add(importTimeout))
|
||||
sum, err := archive.ImportMbox(s.store, id, r.Body, time.Now())
|
||||
if err != nil {
|
||||
writeError(w, http.StatusBadRequest, fmt.Sprintf("import stopped after %d message(s): %v", sum.Imported, err))
|
||||
|
|
|
|||
|
|
@ -152,6 +152,37 @@ func TestCreateListValidation(t *testing.T) {
|
|||
`{"name":"a+b","domain":"example.org"}`,
|
||||
`{"name":"dev","domain":""}`,
|
||||
`{"name":"dev","domain":"example.org","bogus":1}`,
|
||||
// Names that break an address: separators, brackets, whitespace, control characters, dots at the ends.
|
||||
`{"name":"a/b","domain":"example.org"}`,
|
||||
`{"name":"a<b","domain":"example.org"}`,
|
||||
`{"name":"a>b","domain":"example.org"}`,
|
||||
`{"name":"a\tb","domain":"example.org"}`,
|
||||
`{"name":"a b","domain":"example.org"}`,
|
||||
`{"name":"a\"b","domain":"example.org"}`,
|
||||
`{"name":"caffè","domain":"example.org"}`,
|
||||
`{"name":".dev","domain":"example.org"}`,
|
||||
`{"name":"dev.","domain":"example.org"}`,
|
||||
`{"name":"a..b","domain":"example.org"}`,
|
||||
`{"name":"a{b","domain":"example.org"}`,
|
||||
`{"name":"","domain":"example.org","title":"Dev"}`,
|
||||
`{"name":" ","domain":"example.org","title":"Dev"}`,
|
||||
`{"name":"` + strings.Repeat("a", 31) + `","domain":"example.org"}`,
|
||||
// Domains that are not host names.
|
||||
`{"name":"dev","domain":"exa mple.org"}`,
|
||||
`{"name":"dev","domain":"example..org"}`,
|
||||
`{"name":"dev","domain":"-example.org"}`,
|
||||
`{"name":"dev","domain":"example-.org"}`,
|
||||
`{"name":"dev","domain":"example.org."}`,
|
||||
`{"name":"dev","domain":"ex_ample.org"}`,
|
||||
`{"name":"dev","domain":"exämple.org"}`,
|
||||
`{"name":"dev","domain":"a/b.org"}`,
|
||||
`{"name":"dev","domain":"` + strings.Repeat("a", 64) + `.org"}`,
|
||||
// Titles that are not one clean line.
|
||||
`{"name":"dev","domain":"example.org","title":"Dev\r\nBcc: x@y.com"}`,
|
||||
`{"name":"dev","domain":"example.org","title":"Dev\nlist"}`,
|
||||
`{"name":"dev","domain":"example.org","title":"Dev\tlist"}`,
|
||||
`{"name":"dev","domain":"example.org","title":"Dev\u0000list"}`,
|
||||
`{"name":"dev","domain":"example.org","title":"` + strings.Repeat("x", 201) + `"}`,
|
||||
} {
|
||||
rec := do(t, h, "POST", "/api/v1/lists", body, true)
|
||||
if rec.Code != 400 {
|
||||
|
|
@ -685,3 +716,127 @@ func TestQueueShowsFailedDeliveriesApartFromPendingOnes(t *testing.T) {
|
|||
}
|
||||
expect(t, do(t, h, "DELETE", "/api/v1/queue/"+strconv.FormatInt(due[2].ID, 10), "", true), 204)
|
||||
}
|
||||
|
||||
// TestEveryRouteNeedsTheToken walks the route table New registers, so a route added later is
|
||||
// covered without touching this test: all of them but the health check answer 401 without a token
|
||||
// and something else with one.
|
||||
func TestEveryRouteNeedsTheToken(t *testing.T) {
|
||||
s, err := store.Open(":memory:")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { s.Close() })
|
||||
h := New(s, token, nil)
|
||||
|
||||
rs := (&server{}).routes()
|
||||
if len(rs) < 17 {
|
||||
t.Fatalf("only %d routes in the table", len(rs))
|
||||
}
|
||||
for i, rt := range rs {
|
||||
method, pattern, _ := strings.Cut(rt.pattern, " ")
|
||||
path := strings.NewReplacer("{id}", "1", "{address}", "a@example.com").Replace(pattern)
|
||||
t.Run(rt.pattern, func(t *testing.T) {
|
||||
// Each request comes from its own address, so the lockout never gets in the way.
|
||||
addr := fmt.Sprintf("192.0.2.%d:4000", i+1)
|
||||
call := func(auth bool) int {
|
||||
req := httptest.NewRequest(method, path, strings.NewReader("{}"))
|
||||
req.RemoteAddr = addr
|
||||
if auth {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
rec := httptest.NewRecorder()
|
||||
h.ServeHTTP(rec, req)
|
||||
return rec.Code
|
||||
}
|
||||
if pattern == "/api/v1/health" {
|
||||
if got := call(false); got != 200 {
|
||||
t.Fatalf("health without a token: %d", got)
|
||||
}
|
||||
return
|
||||
}
|
||||
if got := call(false); got != 401 {
|
||||
t.Errorf("no token: status %d, want 401", got)
|
||||
}
|
||||
if got := call(true); got == 401 {
|
||||
t.Errorf("good token: status %d", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateListAcceptsTheLimits(t *testing.T) {
|
||||
h := newServer(t)
|
||||
for _, body := range []string{
|
||||
`{"name":"` + strings.Repeat("a", 30) + `","domain":"example.org","title":"` + strings.Repeat("è", 200) + `"}`,
|
||||
`{"name":"dev-team_1.eu","domain":"lists.sub-domain.example.org"}`,
|
||||
`{"name":"Dev2","domain":"localhost","title":"Développeurs ☕"}`,
|
||||
} {
|
||||
if rec := do(t, h, "POST", "/api/v1/lists", body, true); rec.Code != 201 {
|
||||
t.Errorf("%.80q: status %d, body %s", body, rec.Code, rec.Body.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A title with a line break would make every later post to the list fail to rewrite, so the list
|
||||
// must not come into existence at all.
|
||||
func TestCreateListWithABadTitleLeavesNothingBehind(t *testing.T) {
|
||||
h := newServer(t)
|
||||
expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev\r\nX: y"}`, true), 400)
|
||||
rec := do(t, h, "GET", "/api/v1/lists", "", true)
|
||||
if strings.TrimSpace(rec.Body.String()) != "[]" {
|
||||
t.Fatalf("the refused list was created: %s", rec.Body.String())
|
||||
}
|
||||
// The name is free for a good request.
|
||||
expect(t, do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, true), 201)
|
||||
}
|
||||
|
||||
func TestCreateListLowercasesNameAndDomain(t *testing.T) {
|
||||
h := newServer(t)
|
||||
rec := do(t, h, "POST", "/api/v1/lists", `{"name":" Dev ","domain":"Example.ORG"}`, true)
|
||||
expect(t, rec, 201)
|
||||
var l listJSON
|
||||
decode(t, rec, &l)
|
||||
if l.Address != "dev@example.org" || l.Title != "Dev" {
|
||||
t.Fatalf("%+v", l)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpdateListTitleUsesTheSameRules(t *testing.T) {
|
||||
h := newServer(t)
|
||||
createList(t, h)
|
||||
for _, title := range []string{`"a\tb"`, `"` + strings.Repeat("x", 201) + `"`, `" "`, `"a\r\nb"`} {
|
||||
expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":`+title+`}`, true), 400)
|
||||
}
|
||||
expect(t, do(t, h, "PATCH", "/api/v1/lists/1", `{"title":"`+strings.Repeat("x", 200)+`"}`, true), 200)
|
||||
}
|
||||
|
||||
func TestValidDomainLengthLimits(t *testing.T) {
|
||||
l63 := strings.Repeat("a", 63)
|
||||
d253 := l63 + "." + l63 + "." + l63 + "." + strings.Repeat("a", 61)
|
||||
if len(d253) != 253 || !validDomain(d253) {
|
||||
t.Fatalf("a 253-character domain is valid (%d)", len(d253))
|
||||
}
|
||||
if validDomain(d253 + "a") {
|
||||
t.Fatal("254 characters is too long")
|
||||
}
|
||||
if !validDomain(l63+".org") || validDomain("") || validName("") {
|
||||
t.Fatal("63-character labels are valid, empty names and domains are not")
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateListTrimsTheTitle(t *testing.T) {
|
||||
h := newServer(t)
|
||||
rec := do(t, h, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":" Dev team \t"}`, true)
|
||||
expect(t, rec, 201)
|
||||
var l listJSON
|
||||
decode(t, rec, &l)
|
||||
if l.Title != "Dev team" {
|
||||
t.Fatalf("title = %q", l.Title)
|
||||
}
|
||||
rec = do(t, h, "POST", "/api/v1/lists", `{"name":"ops","domain":"example.org","title":" "}`, true)
|
||||
expect(t, rec, 201)
|
||||
decode(t, rec, &l)
|
||||
if l.Title != "ops" {
|
||||
t.Fatalf("a blank title falls back to the name: %q", l.Title)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,9 +26,31 @@ import (
|
|||
"codeberg.org/squidd/parsimony/internal/web"
|
||||
)
|
||||
|
||||
// shutdownTimeout bounds how long Close waits for the sessions in flight before it cuts them.
|
||||
var shutdownTimeout = 30 * time.Second
|
||||
|
||||
// nowFunc is the clock of the store (queue schedules, token expiry). Tests move it.
|
||||
var nowFunc = time.Now
|
||||
|
||||
// HTTP timeouts of the API + admin UI server and of the public web server, so
|
||||
// a slow or silent client cannot pin a goroutine. The archive import extends
|
||||
// its own deadlines (see api.importArchive). Variables so tests can shorten them.
|
||||
var (
|
||||
httpReadTimeout = 30 * time.Second
|
||||
httpWriteTimeout = 60 * time.Second
|
||||
httpIdleTimeout = 120 * time.Second
|
||||
)
|
||||
|
||||
func newHTTPServer(h http.Handler) *http.Server {
|
||||
return &http.Server{
|
||||
Handler: h,
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
ReadTimeout: httpReadTimeout,
|
||||
WriteTimeout: httpWriteTimeout,
|
||||
IdleTimeout: httpIdleTimeout,
|
||||
}
|
||||
}
|
||||
|
||||
// heloName is the EHLO name: the configured one, else this machine's host name.
|
||||
func heloName(configured string) string {
|
||||
if configured != "" {
|
||||
|
|
@ -90,6 +112,51 @@ func lmtpIsOpen(network, addr string) bool {
|
|||
return ip == nil || !ip.IsLoopback()
|
||||
}
|
||||
|
||||
// connTracker remembers the open connections of a listener, so Close can cut the ones still open
|
||||
// when the grace period ends: once Shutdown has begun, smtp.Server.Close does nothing.
|
||||
type connTracker struct {
|
||||
net.Listener
|
||||
mu sync.Mutex
|
||||
conns map[*trackedConn]struct{}
|
||||
}
|
||||
|
||||
func (t *connTracker) Accept() (net.Conn, error) {
|
||||
c, err := t.Listener.Accept()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
tc := &trackedConn{Conn: c, t: t}
|
||||
t.mu.Lock()
|
||||
t.conns[tc] = struct{}{}
|
||||
t.mu.Unlock()
|
||||
return tc, nil
|
||||
}
|
||||
|
||||
// closeAll cuts every connection still open.
|
||||
func (t *connTracker) closeAll() {
|
||||
t.mu.Lock()
|
||||
open := make([]*trackedConn, 0, len(t.conns))
|
||||
for c := range t.conns {
|
||||
open = append(open, c)
|
||||
}
|
||||
t.mu.Unlock()
|
||||
for _, c := range open {
|
||||
c.Close()
|
||||
}
|
||||
}
|
||||
|
||||
type trackedConn struct {
|
||||
net.Conn
|
||||
t *connTracker
|
||||
}
|
||||
|
||||
func (c *trackedConn) Close() error {
|
||||
c.t.mu.Lock()
|
||||
delete(c.t.conns, c)
|
||||
c.t.mu.Unlock()
|
||||
return c.Conn.Close()
|
||||
}
|
||||
|
||||
// App is a running daemon.
|
||||
type App struct {
|
||||
// LMTPAddr and APIAddr are the addresses actually bound.
|
||||
|
|
@ -98,14 +165,16 @@ type App struct {
|
|||
APIAddr string
|
||||
WebAddr string // empty when the public web is disabled
|
||||
|
||||
store *store.Store
|
||||
lmtp *smtp.Server
|
||||
lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it
|
||||
web *http.Server
|
||||
pub *http.Server
|
||||
wg sync.WaitGroup
|
||||
stop context.CancelFunc // stops the queue worker
|
||||
once sync.Once
|
||||
store *store.Store
|
||||
lmtp *smtp.Server
|
||||
lmtpOpen *connTracker // the LMTP connections, cut by Close when the grace period ends
|
||||
lmtpDone chan struct{} // closed when the LMTP Serve returns
|
||||
lns []net.Listener // closed by Close so Serve returns even if Close wins the race with it
|
||||
web *http.Server
|
||||
pub *http.Server
|
||||
wg sync.WaitGroup
|
||||
stop context.CancelFunc // stops the queue worker
|
||||
once sync.Once
|
||||
}
|
||||
|
||||
// releaser lets the API approve held messages through the router.
|
||||
|
|
@ -196,7 +265,7 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
|
|||
mux := http.NewServeMux()
|
||||
mux.Handle("/api/", api.New(st, apiToken, releaser{rt}))
|
||||
mux.Handle("/", adminui.Embedded())
|
||||
a.web = &http.Server{Handler: mux, ReadHeaderTimeout: 10 * time.Second}
|
||||
a.web = newHTTPServer(mux)
|
||||
}
|
||||
|
||||
var pubLn net.Listener
|
||||
|
|
@ -211,9 +280,12 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
|
|||
}
|
||||
a.WebAddr = pubLn.Addr().String()
|
||||
a.lns = append(a.lns, pubLn)
|
||||
a.pub = &http.Server{Handler: web.New(st, rt), ReadHeaderTimeout: 10 * time.Second}
|
||||
a.pub = newHTTPServer(web.New(st, rt))
|
||||
}
|
||||
|
||||
a.lmtpOpen = &connTracker{Listener: serveLn, conns: map[*trackedConn]struct{}{}}
|
||||
a.lmtpDone = make(chan struct{})
|
||||
|
||||
var workerCtx context.Context
|
||||
workerCtx, a.stop = context.WithCancel(context.Background())
|
||||
a.wg.Add(1)
|
||||
|
|
@ -224,7 +296,8 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
|
|||
a.wg.Add(1)
|
||||
go func() {
|
||||
defer a.wg.Done()
|
||||
if err := a.lmtp.Serve(serveLn); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) {
|
||||
defer close(a.lmtpDone)
|
||||
if err := a.lmtp.Serve(a.lmtpOpen); err != nil && !errors.Is(err, smtp.ErrServerClosed) && !errors.Is(err, net.ErrClosed) {
|
||||
log.Printf("lmtp: %v", err)
|
||||
}
|
||||
}()
|
||||
|
|
@ -249,18 +322,26 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
|
|||
return a, nil
|
||||
}
|
||||
|
||||
// Close stops the servers and closes the database. It is safe to call twice.
|
||||
// Close stops the daemon and closes the database. It is safe to call twice. It stops accepting
|
||||
// connections, then lets the LMTP sessions and HTTP requests in flight finish, so a post being
|
||||
// received is queued and gets its 250 (a cut reply makes the MTA send the post again). After
|
||||
// shutdownTimeout it cuts whatever is still open.
|
||||
func (a *App) Close() {
|
||||
a.once.Do(func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), shutdownTimeout)
|
||||
defer cancel()
|
||||
for _, ln := range a.lns {
|
||||
ln.Close()
|
||||
}
|
||||
a.lmtp.Close()
|
||||
if a.web != nil {
|
||||
a.web.Close()
|
||||
<-a.lmtpDone // Serve has returned: no connection is added while Shutdown waits
|
||||
if err := a.lmtp.Shutdown(ctx); errors.Is(err, context.DeadlineExceeded) {
|
||||
log.Printf("lmtp: sessions still open after %v, closing them", shutdownTimeout)
|
||||
a.lmtpOpen.closeAll()
|
||||
}
|
||||
if a.pub != nil {
|
||||
a.pub.Close()
|
||||
for _, srv := range []*http.Server{a.web, a.pub} {
|
||||
if srv != nil && srv.Shutdown(ctx) != nil {
|
||||
srv.Close()
|
||||
}
|
||||
}
|
||||
a.stop() // in-flight sends are interrupted and stay queued
|
||||
a.wg.Wait()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
package app
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"crypto/ecdsa"
|
||||
"crypto/elliptic"
|
||||
|
|
@ -12,6 +13,7 @@ import (
|
|||
"encoding/base64"
|
||||
"encoding/pem"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math/big"
|
||||
|
|
@ -120,13 +122,17 @@ func testConfig(t *testing.T, relay string) config.Config {
|
|||
return c
|
||||
}
|
||||
|
||||
// client opens one connection per request: a pooled connection the transport dialed and never
|
||||
// used would hold an HTTP server's graceful shutdown for 5 s.
|
||||
var client = &http.Client{Transport: &http.Transport{DisableKeepAlives: true}}
|
||||
|
||||
func call(t *testing.T, a *App, method, path, body, token string) int {
|
||||
t.Helper()
|
||||
req, _ := http.NewRequest(method, "http://"+a.APIAddr+path, strings.NewReader(body))
|
||||
if token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
}
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
|
@ -137,7 +143,7 @@ func call(t *testing.T, a *App, method, path, body, token string) int {
|
|||
func fetch(t *testing.T, method, url string) (int, string) {
|
||||
t.Helper()
|
||||
req, _ := http.NewRequest(method, url, nil)
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
|
@ -353,6 +359,7 @@ func TestWebDisabledWhenListenIsEmpty(t *testing.T) {
|
|||
}
|
||||
|
||||
func TestCloseDropsOpenConnections(t *testing.T) {
|
||||
setShutdownTimeout(t, 200*time.Millisecond)
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
|
|
@ -432,6 +439,193 @@ func deadAddr(t *testing.T) string {
|
|||
return addr
|
||||
}
|
||||
|
||||
func setShutdownTimeout(t *testing.T, d time.Duration) {
|
||||
old := shutdownTimeout
|
||||
shutdownTimeout = d
|
||||
t.Cleanup(func() { shutdownTimeout = old })
|
||||
}
|
||||
|
||||
// waitRefused waits until the LMTP listener no longer accepts connections.
|
||||
func waitRefused(t *testing.T, addr string) {
|
||||
t.Helper()
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
c, err := net.Dial("tcp", addr)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
c.Close()
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatal("the LMTP listener still accepts connections")
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloseLetsAnLMTPDeliveryInFlightFinish(t *testing.T) {
|
||||
cfg := testConfig(t, deadAddr(t)) // the post stays queued, where it can be counted
|
||||
a, err := Start(cfg, "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org"}`, "tok")
|
||||
call(t, a, "POST", "/api/v1/lists/1/members", `{"address":"alice@example.com"}`, "tok")
|
||||
|
||||
conn, err := net.Dial("tcp", a.LMTPAddr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := smtp.NewClientLMTP(conn)
|
||||
defer c.Close()
|
||||
if err := c.Hello("localhost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Mail("alice@example.com", nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Rcpt("dev@example.org", nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w, err := c.Data()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
w.Write([]byte("From: alice@example.com\r\nSubject: in flight\r\nMessage-ID: <f@x>\r\n\r\nbody\r\n"))
|
||||
|
||||
// The daemon stops while the post is being sent.
|
||||
closed := make(chan struct{})
|
||||
go func() { a.Close(); close(closed) }()
|
||||
waitRefused(t, a.LMTPAddr)
|
||||
if _, err := w.CloseWithLMTPResponse(); err != nil {
|
||||
t.Fatalf("the post in flight must get its 250: %v", err)
|
||||
}
|
||||
if err := c.Quit(); err != nil {
|
||||
t.Fatalf("quit: %v", err)
|
||||
}
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Close must return once the session is over")
|
||||
}
|
||||
peek, err := store.Open(cfg.DB)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer peek.Close()
|
||||
if st, _ := peek.OutboxStats(); st.Pending != 1 {
|
||||
t.Fatalf("the post must be queued: %+v", st)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloseCutsAHungLMTPClientAfterTheGracePeriod(t *testing.T) {
|
||||
setShutdownTimeout(t, 200*time.Millisecond)
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn, err := net.Dial("tcp", a.LMTPAddr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
c := smtp.NewClientLMTP(conn)
|
||||
if err := c.Hello("localhost"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Mail("alice@example.com", nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.Rcpt("dev@example.org", nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := c.Data(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// The client never ends the message.
|
||||
start := time.Now()
|
||||
closed := make(chan struct{})
|
||||
go func() { a.Close(); close(closed) }()
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Close must not wait for a hung client beyond the grace period")
|
||||
}
|
||||
if d := time.Since(start); d < 150*time.Millisecond {
|
||||
t.Fatalf("Close returned after %v: it must give the session its grace period", d)
|
||||
}
|
||||
conn.SetReadDeadline(time.Now().Add(2 * time.Second))
|
||||
buf := make([]byte, 512)
|
||||
for {
|
||||
if _, err := conn.Read(buf); err != nil {
|
||||
if ne, ok := err.(net.Error); ok && ne.Timeout() {
|
||||
t.Fatal("the hung connection is still open after Close")
|
||||
}
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCloseLetsAnAPIRequestInFlightFinish(t *testing.T) {
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
conn, err := net.Dial("tcp", a.APIAddr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer conn.Close()
|
||||
body := `{"name":"dev","domain":"example.org"}`
|
||||
fmt.Fprintf(conn, "POST /api/v1/lists HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: %d\r\nConnection: close\r\n\r\n%s", len(body), body[:10])
|
||||
time.Sleep(100 * time.Millisecond) // the handler is now reading the body
|
||||
|
||||
closed := make(chan struct{})
|
||||
go func() { a.Close(); close(closed) }()
|
||||
waitRefused(t, a.APIAddr)
|
||||
io.WriteString(conn, body[10:])
|
||||
conn.SetReadDeadline(time.Now().Add(5 * time.Second))
|
||||
resp, err := http.ReadResponse(bufio.NewReader(conn), nil)
|
||||
if err != nil {
|
||||
t.Fatalf("the request in flight must get its answer: %v", err)
|
||||
}
|
||||
resp.Body.Close()
|
||||
if resp.StatusCode != http.StatusCreated {
|
||||
t.Fatalf("status = %d", resp.StatusCode)
|
||||
}
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("Close must return once the request is over")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFinishedLMTPSessionsAreForgotten(t *testing.T) {
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { a.Close() })
|
||||
for i := 0; i < 3; i++ {
|
||||
lmtpSend(t, a.LMTPAddr, "alice@example.com", "nope@example.org", "Subject: x\r\n\r\nx\r\n")
|
||||
}
|
||||
deadline := time.Now().Add(5 * time.Second)
|
||||
for {
|
||||
a.lmtpOpen.mu.Lock()
|
||||
n := len(a.lmtpOpen.conns)
|
||||
a.lmtpOpen.mu.Unlock()
|
||||
if n == 0 {
|
||||
break
|
||||
}
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("%d closed sessions are still tracked", n)
|
||||
}
|
||||
time.Sleep(5 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
|
||||
func TestQueuedMailSurvivesARestartAndIsRetriedWhenTheRelayReturns(t *testing.T) {
|
||||
var skew atomic.Int64
|
||||
nowFunc = func() time.Time { return time.Now().Add(time.Duration(skew.Load())) }
|
||||
|
|
@ -1004,3 +1198,115 @@ func TestTheServerNameSettingReachesTheTLSCheck(t *testing.T) {
|
|||
t.Fatal("nothing may be sent to a relay whose certificate does not match")
|
||||
}
|
||||
}
|
||||
|
||||
// shortTimeouts makes the HTTP timeouts small enough to test.
|
||||
func shortTimeouts(t *testing.T) {
|
||||
t.Helper()
|
||||
r, w, i := httpReadTimeout, httpWriteTimeout, httpIdleTimeout
|
||||
httpReadTimeout, httpWriteTimeout, httpIdleTimeout = 300*time.Millisecond, 300*time.Millisecond, 300*time.Millisecond
|
||||
t.Cleanup(func() { httpReadTimeout, httpWriteTimeout, httpIdleTimeout = r, w, i })
|
||||
}
|
||||
|
||||
// stalled sends a request that promises a body and never finishes it, and
|
||||
// reports whether the server hung up within a few seconds.
|
||||
func stalled(t *testing.T, addr, req string) bool {
|
||||
t.Helper()
|
||||
c, err := net.Dial("tcp", addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
if _, err := c.Write([]byte(req)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c.SetReadDeadline(time.Now().Add(4 * time.Second))
|
||||
_, err = io.Copy(io.Discard, c) // returns nil on EOF, an error on our deadline
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func TestSlowRequestsAreCutOnBothServers(t *testing.T) {
|
||||
shortTimeouts(t)
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { a.Close() })
|
||||
hdr := "POST %s HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer tok\r\nContent-Type: application/json\r\nContent-Length: 500\r\n\r\n{\"name\":"
|
||||
for name, c := range map[string]struct{ addr, path string }{
|
||||
"api": {a.APIAddr, "/api/v1/lists"},
|
||||
"web": {a.WebAddr, "/unsubscribe/tok"},
|
||||
} {
|
||||
if !stalled(t, c.addr, strings.Replace(hdr, "%s", c.path, 1)) {
|
||||
t.Errorf("%s: a stalled body was not cut", name)
|
||||
}
|
||||
}
|
||||
// An idle keep-alive connection is closed too.
|
||||
for name, addr := range map[string]string{"api": a.APIAddr, "web": a.WebAddr} {
|
||||
if !stalled(t, addr, "GET /nothing HTTP/1.1\r\nHost: x\r\n\r\n") {
|
||||
t.Errorf("%s: an idle connection was kept", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// slowReader hands out its data in two halves, a pause apart.
|
||||
type slowReader struct {
|
||||
parts [][]byte
|
||||
pause time.Duration
|
||||
}
|
||||
|
||||
func (s *slowReader) Read(p []byte) (int, error) {
|
||||
if len(s.parts) == 0 {
|
||||
return 0, io.EOF
|
||||
}
|
||||
if len(s.parts) == 1 {
|
||||
time.Sleep(s.pause)
|
||||
}
|
||||
n := copy(p, s.parts[0])
|
||||
s.parts = s.parts[1:]
|
||||
return n, nil
|
||||
}
|
||||
|
||||
func TestArchiveImportIsNotCutByTheDefaultTimeouts(t *testing.T) {
|
||||
shortTimeouts(t)
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { a.Close() })
|
||||
call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, "tok")
|
||||
|
||||
m1 := "From a@example.com Tue Jan 2 15:04:05 2024\nFrom: a@example.com\nSubject: one\nMessage-ID: <1@x>\n\nbody one\n\n"
|
||||
m2 := "From a@example.com Tue Jan 2 15:04:06 2024\nFrom: a@example.com\nSubject: two\nMessage-ID: <2@x>\n\nbody two\n"
|
||||
body := &slowReader{parts: [][]byte{[]byte(m1), []byte(m2)}, pause: 3 * httpReadTimeout}
|
||||
req, _ := http.NewRequest("POST", "http://"+a.APIAddr+"/api/v1/lists/1/archive/import", body)
|
||||
req.Header.Set("Authorization", "Bearer tok")
|
||||
req.ContentLength = -1 // chunked: the body is streamed
|
||||
resp, err := http.DefaultClient.Do(req)
|
||||
if err != nil {
|
||||
t.Fatalf("import: %v", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
b, _ := io.ReadAll(resp.Body)
|
||||
if resp.StatusCode != 200 || !strings.Contains(string(b), `"imported":2`) {
|
||||
t.Fatalf("import = %d %s", resp.StatusCode, b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTTPServersCarryTheTimeouts(t *testing.T) {
|
||||
relayAddr, _ := startSink(t)
|
||||
a, err := Start(testConfig(t, relayAddr), "tok")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { a.Close() })
|
||||
if httpReadTimeout != 30*time.Second || httpWriteTimeout != 60*time.Second || httpIdleTimeout != 120*time.Second {
|
||||
t.Fatalf("defaults: %v %v %v", httpReadTimeout, httpWriteTimeout, httpIdleTimeout)
|
||||
}
|
||||
for name, s := range map[string]*http.Server{"api": a.web, "web": a.pub} {
|
||||
if s.ReadTimeout != httpReadTimeout || s.WriteTimeout != httpWriteTimeout || s.IdleTimeout != httpIdleTimeout || s.ReadHeaderTimeout == 0 {
|
||||
t.Errorf("%s: timeouts %v %v %v %v", name, s.ReadHeaderTimeout, s.ReadTimeout, s.WriteTimeout, s.IdleTimeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,6 +24,19 @@ import _ "github.com/emersion/go-message/charset"
|
|||
// MaxBody is the largest body stored for one message, in bytes.
|
||||
const MaxBody = 1 << 20
|
||||
|
||||
// Limits on how much of a message's structure is walked to find its body: a
|
||||
// hostile message cannot make the parser recurse or loop for long.
|
||||
const (
|
||||
maxDepth = 10 // levels of nested multipart
|
||||
maxParts = 100 // parts visited in one message
|
||||
)
|
||||
|
||||
// Dates outside these bounds are not trusted: they would pin a message to the
|
||||
// top (or the bottom) of the archive for good.
|
||||
const maxFuture = 24 * time.Hour
|
||||
|
||||
var minDate = time.Date(1990, 1, 1, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
// Entry is the archivable content of a message.
|
||||
type Entry struct {
|
||||
MessageID string
|
||||
|
|
@ -45,8 +58,8 @@ var (
|
|||
emailRe = regexp.MustCompile(`([A-Za-z0-9._%+\-]+)@[A-Za-z0-9\-]+(?:\.[A-Za-z0-9\-]+)*\.[A-Za-z]{2,}`)
|
||||
)
|
||||
|
||||
// Parse reads an RFC 5322 message. now is used when the Date header is missing
|
||||
// or invalid.
|
||||
// Parse reads an RFC 5322 message. now is used when the Date header is missing,
|
||||
// invalid, before 1990, or more than 24 hours after now.
|
||||
func Parse(raw []byte, now time.Time) (Entry, error) {
|
||||
ent, err := message.Read(bytes.NewReader(raw))
|
||||
if err != nil && !message.IsUnknownCharset(err) {
|
||||
|
|
@ -58,7 +71,7 @@ func Parse(raw []byte, now time.Time) (Entry, error) {
|
|||
h := mail.Header{Header: ent.Header}
|
||||
|
||||
e := Entry{Date: now}
|
||||
if d, err := h.Date(); err == nil && !d.IsZero() {
|
||||
if d, err := h.Date(); err == nil && !d.Before(minDate) && !d.After(now.Add(maxFuture)) {
|
||||
e.Date = d
|
||||
}
|
||||
if e.Subject, _ = h.Subject(); strings.TrimSpace(e.Subject) == "" {
|
||||
|
|
@ -79,7 +92,9 @@ func Parse(raw []byte, now time.Time) (Entry, error) {
|
|||
e.InReplyTo = refs[len(refs)-1]
|
||||
}
|
||||
|
||||
plain, htmlBody := bodies(ent)
|
||||
w := &walker{}
|
||||
w.walk(ent, 0)
|
||||
plain, htmlBody := w.plain, w.html
|
||||
if strings.TrimSpace(plain) == "" && htmlBody != "" {
|
||||
plain = stripHTML(htmlBody)
|
||||
}
|
||||
|
|
@ -99,45 +114,55 @@ func senderName(h mail.Header) string {
|
|||
return clean(local)
|
||||
}
|
||||
|
||||
// bodies returns the first plain text and the first HTML body found.
|
||||
func bodies(e *message.Entity) (plain, htmlBody string) {
|
||||
// walker collects the first plain text and the first HTML body of a message.
|
||||
type walker struct {
|
||||
plain, html string
|
||||
parts int
|
||||
}
|
||||
|
||||
// done reports whether the plain text is found: nothing else is needed.
|
||||
func (w *walker) done() bool { return strings.TrimSpace(w.plain) != "" }
|
||||
|
||||
func (w *walker) walk(e *message.Entity, depth int) {
|
||||
ct, _, _ := e.Header.ContentType()
|
||||
if ct == "" {
|
||||
ct = "text/plain"
|
||||
}
|
||||
if strings.HasPrefix(ct, "multipart/") {
|
||||
if depth >= maxDepth {
|
||||
return
|
||||
}
|
||||
mr := e.MultipartReader()
|
||||
if mr == nil {
|
||||
return "", ""
|
||||
return
|
||||
}
|
||||
for {
|
||||
for !w.done() && w.parts < maxParts {
|
||||
part, err := mr.NextPart()
|
||||
if err == io.EOF {
|
||||
break
|
||||
return
|
||||
}
|
||||
if err != nil && !message.IsUnknownCharset(err) {
|
||||
break
|
||||
return
|
||||
}
|
||||
w.parts++
|
||||
if disp, _, _ := part.Header.ContentDisposition(); disp == "attachment" {
|
||||
continue
|
||||
}
|
||||
p, h := bodies(part)
|
||||
if plain == "" {
|
||||
plain = p
|
||||
}
|
||||
if htmlBody == "" {
|
||||
htmlBody = h
|
||||
}
|
||||
w.walk(part, depth+1)
|
||||
}
|
||||
return plain, htmlBody
|
||||
return
|
||||
}
|
||||
// Read a text part only if it can still become the body.
|
||||
switch ct {
|
||||
case "text/plain":
|
||||
plain = readLimited(e.Body)
|
||||
if w.plain == "" {
|
||||
w.plain = readLimited(e.Body)
|
||||
}
|
||||
case "text/html":
|
||||
htmlBody = readLimited(e.Body)
|
||||
if w.html == "" {
|
||||
w.html = readLimited(e.Body)
|
||||
}
|
||||
}
|
||||
return plain, htmlBody
|
||||
}
|
||||
|
||||
func readLimited(r io.Reader) string {
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
package archive
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
|
@ -212,3 +214,198 @@ func TestCapNeverSplitsAMultibyteCharacter(t *testing.T) {
|
|||
t.Fatalf("body len %d", len(e.Body))
|
||||
}
|
||||
}
|
||||
|
||||
// nestedBomb is a message with a text part followed by a part that nests
|
||||
// multipart/mixed levels deep.
|
||||
func nestedBomb(levels int) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("From: a@example.com\r\nSubject: bomb\r\nContent-Type: multipart/mixed; boundary=b0\r\n\r\n")
|
||||
b.WriteString("--b0\r\nContent-Type: text/plain\r\n\r\nhello\r\n")
|
||||
for i := 0; i < levels; i++ {
|
||||
fmt.Fprintf(&b, "--b%d\r\nContent-Type: multipart/mixed; boundary=b%d\r\n\r\n", i, i+1)
|
||||
}
|
||||
b.WriteString("--b" + fmt.Sprint(levels) + "\r\nContent-Type: text/plain\r\n\r\ndeep\r\n")
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func parseWithin(t *testing.T, raw string, limit time.Duration) Entry {
|
||||
t.Helper()
|
||||
type result struct {
|
||||
e Entry
|
||||
err error
|
||||
}
|
||||
done := make(chan result, 1)
|
||||
go func() {
|
||||
e, err := Parse([]byte(raw), now)
|
||||
done <- result{e, err}
|
||||
}()
|
||||
select {
|
||||
case r := <-done:
|
||||
if r.err != nil {
|
||||
t.Fatalf("parse: %v", r.err)
|
||||
}
|
||||
return r.e
|
||||
case <-time.After(limit):
|
||||
t.Fatalf("Parse still running after %v", limit)
|
||||
return Entry{}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeeplyNestedMultipartIsBounded(t *testing.T) {
|
||||
e := parseWithin(t, nestedBomb(20000), 2*time.Second)
|
||||
if e.Body != "hello\n" && e.Body != "hello" {
|
||||
t.Fatalf("body = %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNestingDeeperThanTheLimitIsIgnored(t *testing.T) {
|
||||
// text/plain at depth 3 is found, at depth 40 it is not.
|
||||
shallow := "Content-Type: multipart/mixed; boundary=b0\r\n\r\n" +
|
||||
"--b0\r\nContent-Type: multipart/mixed; boundary=b1\r\n\r\n" +
|
||||
"--b1\r\nContent-Type: multipart/mixed; boundary=b2\r\n\r\n" +
|
||||
"--b2\r\nContent-Type: text/plain\r\n\r\nfound\r\n"
|
||||
if e := parse(t, "Subject: s\r\n"+shallow); !strings.Contains(e.Body, "found") {
|
||||
t.Fatalf("shallow body = %q", e.Body)
|
||||
}
|
||||
if e := parse(t, strings.Replace(nestedBomb(40), "text/plain\r\n\r\nhello", "application/octet-stream\r\n\r\nhello", 1)); strings.Contains(e.Body, "deep") {
|
||||
t.Fatalf("a part 40 levels deep was read: %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManyPartsAreBounded(t *testing.T) {
|
||||
var b strings.Builder
|
||||
b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
|
||||
for i := 0; i < 5000; i++ {
|
||||
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
|
||||
}
|
||||
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nlast\r\n")
|
||||
e := parseWithin(t, b.String(), 2*time.Second)
|
||||
if strings.Contains(e.Body, "last") {
|
||||
t.Fatalf("part 5001 was read: %q", e.Body)
|
||||
}
|
||||
|
||||
// A text part among the first parts is still found.
|
||||
b.Reset()
|
||||
b.WriteString("Subject: many\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
|
||||
for i := 0; i < 50; i++ {
|
||||
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
|
||||
}
|
||||
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nearly\r\n")
|
||||
if e := parse(t, b.String()); !strings.Contains(e.Body, "early") {
|
||||
t.Fatalf("body = %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheTextPartsThatAreNeededAreRead(t *testing.T) {
|
||||
// 30 text parts of 1 MiB each: the first wins and the rest are not read.
|
||||
big := strings.Repeat("a", MaxBody)
|
||||
var b strings.Builder
|
||||
b.WriteString("Subject: big\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
|
||||
for i := 0; i < 30; i++ {
|
||||
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\n" + big + "\r\n")
|
||||
}
|
||||
raw := []byte(b.String())
|
||||
var before, after runtime.MemStats
|
||||
runtime.ReadMemStats(&before)
|
||||
e, err := Parse(raw, now)
|
||||
runtime.ReadMemStats(&after)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(e.Body) != MaxBody {
|
||||
t.Fatalf("body = %d bytes", len(e.Body))
|
||||
}
|
||||
if used := after.TotalAlloc - before.TotalAlloc; used > 24<<20 {
|
||||
t.Fatalf("Parse allocated %d MiB for a 1 MiB body", used>>20)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPathologicalHTMLIsNotQuadratic(t *testing.T) {
|
||||
for name, frag := range map[string]string{
|
||||
"open script": "<script>",
|
||||
"open tag": "<a ",
|
||||
"open style": "<style x>",
|
||||
"break": "<br",
|
||||
} {
|
||||
html := strings.Repeat(frag, MaxBody/len(frag))
|
||||
raw := "Subject: h\r\nContent-Type: text/html\r\n\r\n" + html
|
||||
t.Run(name, func(t *testing.T) { parseWithin(t, raw, 15*time.Second) })
|
||||
}
|
||||
}
|
||||
|
||||
func TestFarFutureDateIsClamped(t *testing.T) {
|
||||
// More than 24 hours ahead: the time of parsing. A little ahead is kept.
|
||||
e := parse(t, "Date: Mon, 02 Jan 2090 15:04:05 +0000\r\nSubject: d\r\n\r\nx\r\n")
|
||||
if !e.Date.Equal(now) {
|
||||
t.Fatalf("far future date = %v", e.Date)
|
||||
}
|
||||
soon := now.Add(23 * time.Hour).Format(time.RFC1123Z)
|
||||
if e := parse(t, "Date: "+soon+"\r\nSubject: d\r\n\r\nx\r\n"); e.Date.Equal(now) {
|
||||
t.Fatalf("a date 23 hours ahead was clamped: %v", e.Date)
|
||||
}
|
||||
late := now.Add(25 * time.Hour).Format(time.RFC1123Z)
|
||||
if e := parse(t, "Date: "+late+"\r\nSubject: d\r\n\r\nx\r\n"); !e.Date.Equal(now) {
|
||||
t.Fatalf("a date 25 hours ahead = %v", e.Date)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDateBefore1990IsMissing(t *testing.T) {
|
||||
for _, d := range []string{"Sun, 31 Dec 1989 23:59:59 +0000", "Thu, 01 Jan 1970 00:00:00 +0000", "Mon, 01 Jan 1900 00:00:00 +0000"} {
|
||||
if e := parse(t, "Date: "+d+"\r\nSubject: d\r\n\r\nx\r\n"); !e.Date.Equal(now) {
|
||||
t.Fatalf("%s: date = %v", d, e.Date)
|
||||
}
|
||||
}
|
||||
e := parse(t, "Date: Mon, 01 Jan 1990 00:00:00 +0000\r\nSubject: d\r\n\r\nx\r\n")
|
||||
if !e.Date.Equal(time.Date(1990, 1, 1, 0, 0, 0, 0, time.UTC)) {
|
||||
t.Fatalf("1990 date = %v", e.Date)
|
||||
}
|
||||
}
|
||||
|
||||
// textAtDepth is a message whose only text part sits inside n nested
|
||||
// multipart levels.
|
||||
func textAtDepth(n int) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("Subject: s\r\nContent-Type: multipart/mixed; boundary=b0\r\n\r\n")
|
||||
for i := 0; i < n-1; i++ {
|
||||
fmt.Fprintf(&b, "--b%d\r\nContent-Type: multipart/mixed; boundary=b%d\r\n\r\n", i, i+1)
|
||||
}
|
||||
fmt.Fprintf(&b, "--b%d\r\nContent-Type: text/plain\r\n\r\nfound\r\n", n-1)
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func TestNestingLimitIsTenLevels(t *testing.T) {
|
||||
if e := parse(t, textAtDepth(10)); !strings.Contains(e.Body, "found") {
|
||||
t.Fatalf("10 levels: body = %q", e.Body)
|
||||
}
|
||||
if e := parse(t, textAtDepth(11)); e.Body != "" {
|
||||
t.Fatalf("11 levels: body = %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPartLimitIsOneHundred(t *testing.T) {
|
||||
withTextAt := func(n int) string {
|
||||
var b strings.Builder
|
||||
b.WriteString("Subject: s\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n")
|
||||
for i := 1; i < n; i++ {
|
||||
b.WriteString("--b\r\nContent-Type: application/octet-stream\r\n\r\nx\r\n")
|
||||
}
|
||||
b.WriteString("--b\r\nContent-Type: text/plain\r\n\r\nfound\r\n")
|
||||
return b.String()
|
||||
}
|
||||
if e := parse(t, withTextAt(100)); !strings.Contains(e.Body, "found") {
|
||||
t.Fatalf("part 100: body = %q", e.Body)
|
||||
}
|
||||
if e := parse(t, withTextAt(101)); e.Body != "" {
|
||||
t.Fatalf("part 101: body = %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstPartOfEachKindWinsEvenWhenPlainIsBlank(t *testing.T) {
|
||||
part := func(ct, body string) string { return "--b\r\nContent-Type: " + ct + "\r\n\r\n" + body + "\r\n" }
|
||||
raw := "Subject: s\r\nContent-Type: multipart/mixed; boundary=b\r\n\r\n" +
|
||||
part("text/plain", " ") + part("text/plain", "second plain") +
|
||||
part("text/html", "<p>first html</p>") + part("text/html", "<p>second html</p>")
|
||||
if e := parse(t, raw); strings.TrimSpace(e.Body) != "first html" {
|
||||
t.Fatalf("body = %q", e.Body)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import (
|
|||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
|
|
@ -22,7 +23,7 @@ type IO struct {
|
|||
Out, Err io.Writer
|
||||
}
|
||||
|
||||
const usageText = `Usage: parsimony [--url URL] [--json] <command> [arguments]
|
||||
const usageText = `Usage: parsimony [--url URL] [--insecure] [--json] <command> [arguments]
|
||||
|
||||
Administration (talks to the REST API):
|
||||
list ls show all lists
|
||||
|
|
@ -69,6 +70,7 @@ Environment:
|
|||
PARSIMONY_URL API address, default http://127.0.0.1:8080 (--url overrides)
|
||||
|
||||
--json prints the API's JSON instead of tables.
|
||||
--insecure allows sending the token over plain http to a host other than localhost.
|
||||
Exit status: 0 success, 1 failure, 2 usage error.
|
||||
`
|
||||
|
||||
|
|
@ -79,6 +81,7 @@ func Run(args []string, io IO, getenv func(string) string) int {
|
|||
fs := newFlagSet("parsimony")
|
||||
fs.StringVar(&c.url, "url", "", "")
|
||||
fs.BoolVar(&c.json, "json", false, "")
|
||||
fs.BoolVar(&c.insecure, "insecure", false, "")
|
||||
if err := fs.Parse(args); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
fmt.Fprint(io.Out, usageText)
|
||||
|
|
@ -88,7 +91,7 @@ func Run(args []string, io IO, getenv func(string) string) int {
|
|||
}
|
||||
rest := fs.Args()
|
||||
if len(rest) == 0 {
|
||||
fmt.Fprint(io.Err, "usage: parsimony [--url URL] [--json] <command> (see parsimony help)\n")
|
||||
fmt.Fprint(io.Err, "usage: parsimony [--url URL] [--insecure] [--json] <command> (see parsimony help)\n")
|
||||
return 2
|
||||
}
|
||||
if rest[0] == "help" {
|
||||
|
|
@ -124,16 +127,27 @@ type cmd struct {
|
|||
getenv func(string) string
|
||||
url string
|
||||
json bool
|
||||
cl *client
|
||||
// insecure allows the token to travel over plain http to a non-loopback host.
|
||||
insecure bool
|
||||
cl *client
|
||||
}
|
||||
|
||||
// usageErr is an error that is the caller's mistake: exit status 2.
|
||||
type usageErr string
|
||||
|
||||
func (e usageErr) Error() string { return string(e) }
|
||||
|
||||
func (c *cmd) usageError(format string, a ...any) int {
|
||||
fmt.Fprintf(c.io.Err, "error: "+format+"\n", a...)
|
||||
return 2
|
||||
}
|
||||
|
||||
func (c *cmd) fail(err error) int {
|
||||
fmt.Fprintf(c.io.Err, "error: %v\n", err)
|
||||
fmt.Fprintf(c.io.Err, "error: %s\n", printable(err.Error()))
|
||||
var u usageErr
|
||||
if errors.As(err, &u) {
|
||||
return 2
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
|
|
@ -153,9 +167,14 @@ func (c *cmd) client() (*client, error) {
|
|||
if base == "" {
|
||||
base = "http://127.0.0.1:8080"
|
||||
}
|
||||
if _, err := url.Parse(base); err != nil {
|
||||
u, err := url.Parse(base)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid URL %q", base)
|
||||
}
|
||||
// url.Parse lowercases the scheme.
|
||||
if u.Scheme == "http" && !isLoopback(u.Hostname()) && !c.insecure {
|
||||
return nil, usageErr(fmt.Sprintf("refusing to send the token over plain http to %s: use https, or pass --insecure", u.Host))
|
||||
}
|
||||
c.cl = &client{
|
||||
base: strings.TrimRight(base, "/"), token: token,
|
||||
http: &http.Client{Timeout: 30 * time.Second},
|
||||
|
|
@ -312,10 +331,44 @@ func (c *cmd) printJSON(raw json.RawMessage) int {
|
|||
return 0
|
||||
}
|
||||
|
||||
// isLoopback reports whether host is this machine: localhost or a loopback address.
|
||||
func isLoopback(host string) bool {
|
||||
if strings.EqualFold(strings.TrimSuffix(host, "."), "localhost") {
|
||||
return true
|
||||
}
|
||||
ip := net.ParseIP(host)
|
||||
return ip != nil && ip.IsLoopback()
|
||||
}
|
||||
|
||||
// printable makes text that came from mail or from the API safe for a terminal: control
|
||||
// characters (C0, DEL, C1) other than tab and newline are shown as \xNN / \uNNNN instead of
|
||||
// being sent, so an escape sequence in a subject cannot drive the terminal. CRLF counts as a newline.
|
||||
func printable(s string) string {
|
||||
s = strings.ReplaceAll(s, "\r\n", "\n")
|
||||
var b strings.Builder
|
||||
for _, r := range s {
|
||||
switch {
|
||||
case r == '\t' || r == '\n':
|
||||
b.WriteRune(r)
|
||||
case r < 0x20 || r == 0x7f:
|
||||
fmt.Fprintf(&b, `\x%02x`, r)
|
||||
case r >= 0x80 && r <= 0x9f:
|
||||
fmt.Fprintf(&b, `\u%04x`, r)
|
||||
default:
|
||||
b.WriteRune(r)
|
||||
}
|
||||
}
|
||||
return b.String()
|
||||
}
|
||||
|
||||
func (c *cmd) table(rows [][]string) {
|
||||
w := tabwriter.NewWriter(c.io.Out, 0, 8, 2, ' ', 0)
|
||||
for _, r := range rows {
|
||||
fmt.Fprintln(w, strings.Join(r, "\t"))
|
||||
cells := make([]string, len(r))
|
||||
for i, cell := range r {
|
||||
cells[i] = printable(cell)
|
||||
}
|
||||
fmt.Fprintln(w, strings.Join(cells, "\t"))
|
||||
}
|
||||
w.Flush()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -153,7 +153,7 @@ func TestListLifecycle(t *testing.T) {
|
|||
e.fails(t, 1, "already exists", "list", "add", "dev@example.org")
|
||||
e.fails(t, 2, "name@domain", "list", "add", "nodomain")
|
||||
e.fails(t, 2, "name@domain", "list", "add", "@example.org")
|
||||
e.fails(t, 1, "name and domain", "list", "add", "a+b@example.org")
|
||||
e.fails(t, 1, "name must be", "list", "add", "a+b@example.org")
|
||||
|
||||
for _, ref := range []string{"dev@example.org", "DEV@Example.org", "1"} {
|
||||
out = e.ok(t, "list", "show", ref)
|
||||
|
|
@ -824,3 +824,172 @@ func TestHeldListingWarnsWhenTheHeaderDiffersFromTheSender(t *testing.T) {
|
|||
t.Fatalf("no usable From is flagged too: %q", lines["no from"])
|
||||
}
|
||||
}
|
||||
|
||||
const esc = "\x1b[31mRED\x1b]0;pwned\x07"
|
||||
|
||||
// noControl fails when s still holds a control character other than tab and newline.
|
||||
func noControl(t *testing.T, what, s string) {
|
||||
t.Helper()
|
||||
for _, r := range s {
|
||||
if (r < 0x20 && r != '\t' && r != '\n') || (r >= 0x7f && r <= 0x9f) {
|
||||
t.Fatalf("%s prints control character %U:\n%q", what, r, s)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPrintableStripsControlCharactersButKeepsTabAndNewline(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"plain text, accents: àèì 日本": "plain text, accents: àèì 日本",
|
||||
"a\tb\nc": "a\tb\nc",
|
||||
"a\r\nb": "a\nb",
|
||||
"\x1b[31mred\x1b[0m": `\x1b[31mred\x1b[0m`,
|
||||
"us\x1f": `us\x1f`,
|
||||
"bell\x07 nul\x00 del\x7f": `bell\x07 nul\x00 del\x7f`,
|
||||
"c1 \u009b31m and \u0085": `c1 \u009b31m and \u0085`,
|
||||
"c1 edges \u0080 \u009f, kept: ~ \u00a0 \u00a1": `c1 edges \u0080 \u009f, kept: ~ ` + "\u00a0 \u00a1",
|
||||
"lone\rcarriage": `lone\x0dcarriage`,
|
||||
"": "",
|
||||
} {
|
||||
if got := printable(in); got != want {
|
||||
t.Errorf("printable(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestControlCharactersFromMailNeverReachTheTerminal(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
e.ok(t, "list", "add", "dev@example.org")
|
||||
l, _ := e.s.FindList("dev@example.org")
|
||||
e.s.ArchiveMessage(l.ID, store.Message{MessageID: "<e@x>", Subject: "subj " + esc, Sender: "from " + esc,
|
||||
Date: time.Unix(1_700_000_000, 0), Body: "line one\r\nbody " + esc + "\r\nlast"})
|
||||
e.s.HoldMessage(l.ID, "bob@x.com", []byte("From: Bob "+esc+" <bob@x.com>\r\nSubject: held "+esc+"\r\n\r\nb"), "reason "+esc)
|
||||
e.s.Enqueue([]byte("x"), []store.NewOutbound{{From: "b@x.org", Rcpt: "lost@x.com"}})
|
||||
due, _ := e.s.DueOutbox(10)
|
||||
e.s.FailOutbox(due[0].ID, "refused: 554 "+esc)
|
||||
|
||||
for _, args := range [][]string{
|
||||
{"archive", "ls", "dev@example.org"}, {"archive", "ls", "dev@example.org", "--q", "subj"},
|
||||
{"archive", "show", "1"}, {"held", "ls", "dev@example.org"}, {"queue", "ls"},
|
||||
} {
|
||||
out := e.ok(t, args...)
|
||||
noControl(t, strings.Join(args, " "), out)
|
||||
if !strings.Contains(out, `\x1b[31mRED`) {
|
||||
t.Errorf("%v: the escape sequence should stay visible as text:\n%s", args, out)
|
||||
}
|
||||
}
|
||||
if out := e.ok(t, "archive", "show", "1"); !strings.Contains(out, "\nbody ") || strings.Contains(out, `\x0d`) {
|
||||
t.Errorf("CRLF line ends are not control noise: %q", out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestControlCharactersInListAndMemberNamesAreEscaped(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
addr := "a\x1b[31mb\x07@x.org"
|
||||
l, err := e.s.CreateList("a\x1b[31mb\x07", "x.org", "title "+esc)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
e.s.AddMember(l.ID, "m\x1b[2Jm@x.org", "name "+esc)
|
||||
mbox := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nbody\n"
|
||||
for _, c := range []struct {
|
||||
stdin string
|
||||
args []string
|
||||
}{
|
||||
{"", []string{"list", "ls"}}, {"", []string{"list", "show", addr}},
|
||||
{"", []string{"list", "set", addr, "--title", "t"}},
|
||||
{"", []string{"member", "ls", addr}}, {"", []string{"member", "ls", addr, "--emails"}},
|
||||
{"", []string{"member", "add", addr, "new@x.com"}},
|
||||
{"new2@x.com\n", []string{"member", "import", addr, "-"}},
|
||||
{mbox, []string{"archive", "import", addr, "-"}},
|
||||
{"", []string{"member", "rm", addr, "new@x.com"}},
|
||||
{"", []string{"list", "rm", addr, "--yes"}},
|
||||
} {
|
||||
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": e.srv.URL}, c.stdin, c.args...)
|
||||
if code != 0 {
|
||||
t.Fatalf("%v: exit %d %s", c.args, code, errs)
|
||||
}
|
||||
noControl(t, strings.Join(c.args, " "), out+errs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestControlCharactersInImportReportsAreEscaped(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
e.ok(t, "list", "add", "dev@example.org")
|
||||
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": e.srv.URL},
|
||||
"bad"+esc+"@x\n", "member", "import", "dev@example.org", "-")
|
||||
if code != 1 || !strings.Contains(errs, `line 1: bad\x1b[31mRED`) {
|
||||
t.Fatalf("member import: %d %q", code, errs)
|
||||
}
|
||||
noControl(t, "member import", out+errs)
|
||||
|
||||
fake := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Method == "POST" {
|
||||
b, _ := json.Marshal(map[string]any{"imported": 0, "failed": 1, "errors": []string{"message 1: " + esc}})
|
||||
w.Write(b)
|
||||
return
|
||||
}
|
||||
w.Write([]byte(`[{"id":1,"address":"dev@example.org"}]`))
|
||||
}))
|
||||
defer fake.Close()
|
||||
var o, eb bytes.Buffer
|
||||
vars := map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": fake.URL}
|
||||
Run([]string{"archive", "import", "dev@example.org", "-"}, IO{In: strings.NewReader("x"), Out: &o, Err: &eb}, func(k string) string { return vars[k] })
|
||||
if !strings.Contains(eb.String(), `message 1: \x1b[31mRED`) {
|
||||
t.Fatalf("archive import errors: %q", eb.String())
|
||||
}
|
||||
noControl(t, "archive import", o.String()+eb.String())
|
||||
}
|
||||
|
||||
func TestControlCharactersInAPIErrorsAndImportReportsAreEscaped(t *testing.T) {
|
||||
api := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(http.StatusBadGateway)
|
||||
msg, _ := json.Marshal(map[string]string{"error": "relay said " + esc})
|
||||
w.Write(msg)
|
||||
}))
|
||||
defer api.Close()
|
||||
var out, errb bytes.Buffer
|
||||
vars := map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": api.URL}
|
||||
code := Run([]string{"list", "ls"}, IO{In: strings.NewReader(""), Out: &out, Err: &errb}, func(k string) string { return vars[k] })
|
||||
if code != 1 || !strings.Contains(errb.String(), `relay said \x1b[31mRED`) {
|
||||
t.Fatalf("exit %d: %q", code, errb.String())
|
||||
}
|
||||
noControl(t, "error text", errb.String())
|
||||
}
|
||||
|
||||
func TestTokenIsNotSentOverPlainHTTPToARemoteHost(t *testing.T) {
|
||||
e := newEnv(t)
|
||||
run := func(url string, extra ...string) (int, string) {
|
||||
args := append(append([]string{"--url", url}, extra...), "list", "ls")
|
||||
code, out, errs := e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token}, "", args...)
|
||||
if out != "" {
|
||||
t.Fatalf("stdout %q", out)
|
||||
}
|
||||
return code, errs
|
||||
}
|
||||
code, errs := run("http://lists.example.org:8080")
|
||||
if code != 2 || !strings.Contains(errs, "--insecure") || !strings.Contains(errs, "http") {
|
||||
t.Fatalf("remote http must be refused with a usage error: %d %q", code, errs)
|
||||
}
|
||||
if code, errs = run("HTTP://10.1.2.3"); code != 2 {
|
||||
t.Fatalf("scheme case must not matter: %d %q", code, errs)
|
||||
}
|
||||
// Past the check the only thing left to fail is the connection (port 1 is closed, .invalid never resolves).
|
||||
for _, u := range []string{"http://localhost:1", "http://LocalHost.:1", "http://127.0.0.1:1", "http://[::1]:1", "https://nowhere.invalid"} {
|
||||
if code, errs = run(u); code != 1 || !strings.Contains(errs, "cannot reach") {
|
||||
t.Errorf("%s must be allowed: %d %q", u, code, errs)
|
||||
}
|
||||
}
|
||||
if code, errs = run("http://nowhere.invalid", "--insecure"); code != 1 || !strings.Contains(errs, "cannot reach") {
|
||||
t.Fatalf("--insecure allows plain http: %d %q", code, errs)
|
||||
}
|
||||
// The environment variable is checked like the flag.
|
||||
code, _, errs = e.runEnv(t, map[string]string{"PARSIMONY_API_TOKEN": token, "PARSIMONY_URL": "http://lists.example.org"}, "", "list", "ls")
|
||||
if code != 2 {
|
||||
t.Fatalf("PARSIMONY_URL: %d %q", code, errs)
|
||||
}
|
||||
// No token to protect, no URL check: offline commands never build the client.
|
||||
_, _, errs = e.runEnv(t, nil, "", "--url", "http://lists.example.org", "dkim", "record", "example.org", "--key", filepath.Join(t.TempDir(), "none"))
|
||||
if strings.Contains(errs, "--insecure") {
|
||||
t.Fatalf("offline commands must not care about the URL")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -135,7 +135,7 @@ func (c *cmd) listSet(args []string) int {
|
|||
if c.json {
|
||||
return c.printJSON(raw)
|
||||
}
|
||||
fmt.Fprintf(c.io.Out, "Updated %s\n", l.Address)
|
||||
fmt.Fprintf(c.io.Out, "Updated %s\n", printable(l.Address))
|
||||
return 0
|
||||
}
|
||||
|
||||
|
|
@ -168,7 +168,7 @@ func (c *cmd) done(msg string) int {
|
|||
if c.json {
|
||||
return c.printJSON(nil)
|
||||
}
|
||||
fmt.Fprintln(c.io.Out, msg)
|
||||
fmt.Fprintln(c.io.Out, printable(msg))
|
||||
return 0
|
||||
}
|
||||
|
||||
|
|
@ -220,7 +220,7 @@ func (c *cmd) memberLs(args []string) int {
|
|||
}
|
||||
if *emails {
|
||||
for _, m := range ms {
|
||||
fmt.Fprintln(c.io.Out, m.Address)
|
||||
fmt.Fprintln(c.io.Out, printable(m.Address))
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
|
@ -259,7 +259,7 @@ func (c *cmd) memberAdd(args []string) int {
|
|||
}
|
||||
var m member
|
||||
json.Unmarshal(raw, &m)
|
||||
fmt.Fprintf(c.io.Out, "Added %s to %s\n", m.Address, l.Address)
|
||||
fmt.Fprintf(c.io.Out, "Added %s to %s\n", printable(m.Address), printable(l.Address))
|
||||
return 0
|
||||
}
|
||||
|
||||
|
|
@ -327,7 +327,7 @@ func (c *cmd) memberImport(args []string) int {
|
|||
}
|
||||
if _, err := c.addMember(l.ID, address, name); err != nil {
|
||||
failed++
|
||||
fmt.Fprintf(c.io.Err, "line %d: %s: %v\n", n, line, err)
|
||||
fmt.Fprintf(c.io.Err, "line %d: %s\n", n, printable(fmt.Sprintf("%s: %v", line, err)))
|
||||
continue
|
||||
}
|
||||
imported++
|
||||
|
|
@ -339,7 +339,7 @@ func (c *cmd) memberImport(args []string) int {
|
|||
if failed > 0 {
|
||||
msg += fmt.Sprintf(", %d failed", failed)
|
||||
}
|
||||
fmt.Fprintln(c.io.Out, msg)
|
||||
fmt.Fprintln(c.io.Out, printable(msg))
|
||||
if failed > 0 {
|
||||
return 1
|
||||
}
|
||||
|
|
@ -544,7 +544,7 @@ func (c *cmd) archiveShow(args []string) int {
|
|||
if err := json.Unmarshal(raw, &m); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Fprintf(c.io.Out, "Subject: %s\nFrom: %s\nDate: %s\n\n%s\n", m.Subject, m.Sender, formatDate(m.Date), m.Body)
|
||||
fmt.Fprintf(c.io.Out, "Subject: %s\nFrom: %s\nDate: %s\n\n%s\n", printable(m.Subject), printable(m.Sender), formatDate(m.Date), printable(m.Body))
|
||||
return nil
|
||||
}, "")
|
||||
}
|
||||
|
|
@ -601,10 +601,10 @@ func (c *cmd) archiveImport(args []string) int {
|
|||
if sum.Failed > 0 {
|
||||
msg += fmt.Sprintf(", %d failed", sum.Failed)
|
||||
}
|
||||
fmt.Fprintln(c.io.Out, msg)
|
||||
fmt.Fprintln(c.io.Out, printable(msg))
|
||||
}
|
||||
for _, e := range sum.Errors {
|
||||
fmt.Fprintln(c.io.Err, e)
|
||||
fmt.Fprintln(c.io.Err, printable(e))
|
||||
}
|
||||
if sum.Failed > len(sum.Errors) {
|
||||
fmt.Fprintf(c.io.Err, "... and %d more\n", sum.Failed-len(sum.Errors))
|
||||
|
|
|
|||
|
|
@ -7,8 +7,10 @@ import (
|
|||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"unicode"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
|
@ -145,7 +147,31 @@ func (c Config) validateRelay() error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// validateBaseURL checks base_url, the public URL the one-click unsubscribe links are built from.
|
||||
// RFC 8058 needs https; http is accepted for a loopback host (development and tests). Empty is fine:
|
||||
// there is no one-click link then.
|
||||
func (c Config) validateBaseURL() error {
|
||||
if c.BaseURL == "" {
|
||||
return nil
|
||||
}
|
||||
u, err := url.Parse(c.BaseURL)
|
||||
switch {
|
||||
case err != nil || strings.ContainsFunc(c.BaseURL, func(r rune) bool { return unicode.IsSpace(r) || unicode.IsControl(r) }):
|
||||
return fmt.Errorf("base_url %q is not a valid URL", c.BaseURL)
|
||||
case u.Hostname() == "":
|
||||
return fmt.Errorf("base_url %q needs a scheme and a host, such as https://lists.example.org", c.BaseURL)
|
||||
case u.Scheme != "https" && !(u.Scheme == "http" && isLoopbackHost(u.Hostname())):
|
||||
return fmt.Errorf("base_url %q must use https (one-click unsubscribe, RFC 8058); http is only accepted for localhost", c.BaseURL)
|
||||
case u.User != nil || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || strings.Contains(c.BaseURL, "#"):
|
||||
return fmt.Errorf("base_url %q must not contain credentials, a query or a fragment", c.BaseURL)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c Config) validate() error {
|
||||
if err := c.validateBaseURL(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.validateRelay(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -246,3 +246,43 @@ func testCAPEM(t *testing.T) []byte {
|
|||
}
|
||||
return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der})
|
||||
}
|
||||
|
||||
func TestBaseURLIsChecked(t *testing.T) {
|
||||
for _, ok := range []string{
|
||||
"", // one-click unsubscribe is simply off
|
||||
"https://lists.example.org",
|
||||
"https://lists.example.org/",
|
||||
"https://lists.example.org:8443/sub/path",
|
||||
"http://localhost:8081",
|
||||
"http://127.0.0.1:8081",
|
||||
"http://[::1]:8081",
|
||||
} {
|
||||
if _, err := Load(write(t, "base_url: \""+ok+"\"\n")); err != nil {
|
||||
t.Errorf("%q must be accepted: %v", ok, err)
|
||||
}
|
||||
}
|
||||
for _, bad := range []string{
|
||||
"http://lists.example.org", // RFC 8058 needs https
|
||||
"http://10.0.0.5:8081", // only loopback may use http
|
||||
"ftp://lists.example.org", // not http(s) at all
|
||||
"lists.example.org", // no scheme
|
||||
"//lists.example.org", // no scheme
|
||||
"https://", // no host
|
||||
"https:///path", // no host
|
||||
"https://lists.example.org?x=1", // query
|
||||
"https://lists.example.org/#top",
|
||||
"https://lists.example.org/#", // an empty fragment is still one
|
||||
"https://lists.example.org/?", // an empty query is still one // fragment
|
||||
"https://user:pw@lists.example.org", // userinfo
|
||||
"https://user@lists.example.org",
|
||||
"https://lists.example.org/a b",
|
||||
"https://exa mple.org",
|
||||
} {
|
||||
_, err := Load(write(t, "base_url: \""+bad+"\"\n"))
|
||||
if err == nil {
|
||||
t.Errorf("%q must be rejected", bad)
|
||||
} else if !strings.Contains(err.Error(), "base_url") {
|
||||
t.Errorf("%q: the error must name base_url: %v", bad, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -86,7 +86,7 @@ func (s *session) deliver(rcpt string, data []byte) error {
|
|||
// status turns a router result into an LMTP reply.
|
||||
func (s *session) status(rcpt string, err error) error {
|
||||
switch {
|
||||
case err == nil, errors.Is(err, router.ErrHeld), errors.Is(err, router.ErrDiscarded):
|
||||
case err == nil, errors.Is(err, router.ErrHeld), errors.Is(err, router.ErrDiscarded), errors.Is(err, router.ErrDuplicate):
|
||||
return nil
|
||||
case errors.Is(err, router.ErrUnknownList):
|
||||
return &smtp.SMTPError{Code: 550, EnhancedCode: smtp.EnhancedCode{5, 1, 1}, Message: "no such list"}
|
||||
|
|
|
|||
|
|
@ -255,7 +255,7 @@ func TestModeratedPostIsAcceptedButNotQueued(t *testing.T) {
|
|||
|
||||
func TestAutoReplyAndLoopsAreAcceptedButNotDelivered(t *testing.T) {
|
||||
st := newStack(t, nil)
|
||||
for _, h := range []string{"Auto-Submitted: auto-replied\r\n", "List-Id: Dev <dev.example.org>\r\n"} {
|
||||
for _, h := range []string{"Auto-Submitted: auto-replied\r\n", "Precedence: auto_reply\r\n", "List-Id: Dev <dev.example.org>\r\n"} {
|
||||
conn, err := net.Dial("tcp", st.addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
|
|
@ -276,6 +276,29 @@ func TestAutoReplyAndLoopsAreAcceptedButNotDelivered(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestAutoGeneratedPostsAreDelivered(t *testing.T) {
|
||||
st := newStack(t, nil)
|
||||
for _, h := range []string{"Auto-Submitted: auto-generated\r\n", "Auto-Submitted: auto-notified\r\n"} {
|
||||
conn, err := net.Dial("tcp", st.addr)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c := smtp.NewClientLMTP(conn)
|
||||
c.Hello("localhost")
|
||||
c.Mail("alice@example.com", nil)
|
||||
c.Rcpt("dev@example.org", nil)
|
||||
w, _ := c.Data()
|
||||
w.Write([]byte("From: Alice <alice@example.com>\r\n" + h + "Subject: nightly report\r\nMessage-ID: <" + strings.TrimSpace(h) + "@x>\r\n\r\nbody\r\n"))
|
||||
if _, err := w.CloseWithLMTPResponse(); err != nil {
|
||||
t.Fatalf("%q: %v", h, err)
|
||||
}
|
||||
c.Close()
|
||||
}
|
||||
if st.queued(t) == 0 {
|
||||
t.Fatal("a cron or CI post from a member must be queued for delivery")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFullModerationQueueAnswers452(t *testing.T) {
|
||||
st := newStack(t, nil)
|
||||
l, _ := st.store.FindList("dev@example.org")
|
||||
|
|
@ -316,3 +339,15 @@ func TestAMessageTooBigToHoldAnswers552(t *testing.T) {
|
|||
t.Fatalf("a permanent refusal: the sender is told to send it elsewhere: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheSamePostDeliveredAgainIsAcceptedButNotQueuedTwice(t *testing.T) {
|
||||
st := newStack(t, down{}) // the deliveries stay in the queue, where they can be counted
|
||||
for i := 0; i < 2; i++ {
|
||||
if res := deliver(t, st.addr, "alice@example.com", []string{"dev@example.org"}); res["dev@example.org"] != nil {
|
||||
t.Fatalf("delivery %d: a duplicate is acknowledged with 250, got %v", i+1, res)
|
||||
}
|
||||
}
|
||||
if n := st.queued(t); n != 2 {
|
||||
t.Fatalf("queued = %d, want one delivery per member", n)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -67,14 +67,45 @@ func NewReader(r io.Reader, maxSize int) *Reader {
|
|||
return &Reader{br: bufio.NewReaderSize(r, 64*1024), max: maxSize}
|
||||
}
|
||||
|
||||
// readLine returns the next line including its newline. At the end of the input
|
||||
// it returns the last partial line (if any) with io.EOF on the following call.
|
||||
func (r *Reader) readLine() ([]byte, error) {
|
||||
line, err := r.br.ReadBytes('\n')
|
||||
if err == io.EOF && len(line) > 0 {
|
||||
return line, nil
|
||||
// minKeep is the least a line may be cut to, so a "From " separator is never cut short.
|
||||
const minKeep = 1024
|
||||
|
||||
// keep is how many bytes of a line are worth holding: a message with a longer line is too
|
||||
// large anyway, so the rest is read and thrown away instead of buffered.
|
||||
func (r *Reader) keep() int {
|
||||
if r.max <= 0 {
|
||||
return 0 // no limit
|
||||
}
|
||||
return max(r.max+1, minKeep)
|
||||
}
|
||||
|
||||
// readLine returns the next line including its newline, cut to keep bytes (0 means no cut),
|
||||
// and the full length of the line. At the end of the input it returns the last partial line
|
||||
// (if any) with io.EOF on the following call.
|
||||
func (r *Reader) readLine(keep int) ([]byte, int, error) {
|
||||
var line []byte
|
||||
n := 0
|
||||
for {
|
||||
chunk, err := r.br.ReadSlice('\n')
|
||||
n += len(chunk)
|
||||
if room := keep - len(line); keep == 0 || room > 0 {
|
||||
if keep > 0 && len(chunk) > room {
|
||||
chunk = chunk[:room]
|
||||
}
|
||||
line = append(line, chunk...)
|
||||
}
|
||||
switch err {
|
||||
case nil:
|
||||
return line, n, nil
|
||||
case bufio.ErrBufferFull:
|
||||
continue
|
||||
case io.EOF:
|
||||
if n > 0 {
|
||||
return line, n, nil
|
||||
}
|
||||
}
|
||||
return line, n, err
|
||||
}
|
||||
return line, err
|
||||
}
|
||||
|
||||
// Next returns the next message, or io.EOF when there are no more.
|
||||
|
|
@ -84,7 +115,7 @@ func (r *Reader) Next() (Message, error) {
|
|||
}
|
||||
// Skip anything before the first separator.
|
||||
for r.next == "" {
|
||||
line, err := r.readLine()
|
||||
line, _, err := r.readLine(r.keep())
|
||||
if err != nil {
|
||||
r.eof = true
|
||||
if err == io.EOF {
|
||||
|
|
@ -102,8 +133,9 @@ func (r *Reader) Next() (Message, error) {
|
|||
|
||||
var buf bytes.Buffer
|
||||
oversized := false
|
||||
size := 0
|
||||
for {
|
||||
line, err := r.readLine()
|
||||
line, n, err := r.readLine(r.keep())
|
||||
if err != nil {
|
||||
r.eof = true
|
||||
if err != io.EOF {
|
||||
|
|
@ -120,12 +152,15 @@ func (r *Reader) Next() (Message, error) {
|
|||
}
|
||||
if escaped.Match(line) {
|
||||
line = line[1:]
|
||||
n--
|
||||
}
|
||||
buf.Write(line)
|
||||
if r.max > 0 && buf.Len() > r.max {
|
||||
size += n
|
||||
if r.max > 0 && size > r.max {
|
||||
oversized = true
|
||||
buf.Reset()
|
||||
continue
|
||||
}
|
||||
buf.Write(line)
|
||||
}
|
||||
if oversized {
|
||||
msg.Err = ErrTooLarge
|
||||
|
|
|
|||
|
|
@ -166,3 +166,93 @@ func TestReadErrorsArePropagated(t *testing.T) {
|
|||
type errReader struct{}
|
||||
|
||||
func (errReader) Read([]byte) (int, error) { return 0, errors.New("disk on fire") }
|
||||
|
||||
func TestAVeryLongLineIsTooLargeAndTheReaderResynchronises(t *testing.T) {
|
||||
long := strings.Repeat("a", 5000) // far above the limit of 100, with no way to be a separator
|
||||
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + long + "\n\n" +
|
||||
"From b@x.com Wed Jan 3 15:04:05 2024\nSubject: ok\n\nfine\n"
|
||||
ms := readAll(t, in, 100)
|
||||
if len(ms) != 2 || !errors.Is(ms[0].Err, ErrTooLarge) || ms[0].Data != nil {
|
||||
t.Fatalf("messages: %+v", ms)
|
||||
}
|
||||
if ms[1].Err != nil || !strings.Contains(string(ms[1].Data), "fine") {
|
||||
t.Fatalf("the next message must be read: %+v", ms[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestALongLineAtTheEndOfTheInputIsTooLarge(t *testing.T) {
|
||||
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + strings.Repeat("a", 5000) // no newline
|
||||
ms := readAll(t, in, 100)
|
||||
if len(ms) != 1 || !errors.Is(ms[0].Err, ErrTooLarge) {
|
||||
t.Fatalf("messages: %+v", ms)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnOverlongLineBeforeTheFirstSeparatorIsSkipped(t *testing.T) {
|
||||
in := strings.Repeat("x", 5000) + "\nFrom a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nbody\n"
|
||||
ms := readAll(t, in, 100)
|
||||
if len(ms) != 1 || ms[0].Err != nil || !strings.Contains(string(ms[0].Data), "body") {
|
||||
t.Fatalf("messages: %+v", ms)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLongLinesBelowTheLimitStillWork(t *testing.T) {
|
||||
long := strings.Repeat("a", 3000)
|
||||
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\n" + long + "\n"
|
||||
ms := readAll(t, in, 4000)
|
||||
if len(ms) != 1 || ms[0].Err != nil || !strings.Contains(string(ms[0].Data), long+"\n") {
|
||||
t.Fatalf("messages: %+v", ms)
|
||||
}
|
||||
if ms = readAll(t, in, 0); len(ms) != 1 || ms[0].Err != nil {
|
||||
t.Fatalf("no limit: %+v", ms)
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadLineKeepsAtMostTheLimitButReportsTheFullLength(t *testing.T) {
|
||||
// A line far longer than the reader's buffer: only keep bytes may be held.
|
||||
in := strings.Repeat("a", 300_000) + "\nnext\n"
|
||||
r := NewReader(strings.NewReader(in), 0)
|
||||
line, n, err := r.readLine(1000)
|
||||
if err != nil || n != 300_001 || len(line) > 1000 {
|
||||
t.Fatalf("n=%d kept=%d err=%v", n, len(line), err)
|
||||
}
|
||||
if line, n, err = r.readLine(1000); err != nil || string(line) != "next\n" || n != 5 {
|
||||
t.Fatalf("the following line: %q %d %v", line, n, err)
|
||||
}
|
||||
if _, _, err = r.readLine(1000); err != io.EOF {
|
||||
t.Fatalf("want EOF, got %v", err)
|
||||
}
|
||||
// keep 0 means keep everything.
|
||||
r = NewReader(strings.NewReader(in), 0)
|
||||
if line, n, _ = r.readLine(0); len(line) != 300_001 || n != 300_001 {
|
||||
t.Fatalf("unlimited: %d %d", len(line), n)
|
||||
}
|
||||
}
|
||||
|
||||
func TestASmallLimitDoesNotCutSeparatorsShort(t *testing.T) {
|
||||
in := "From a@x.com Tue Jan 2 15:04:05 2024\nSubject: s\n\nlong enough to be too large\n" +
|
||||
"From b@x.com Wed Jan 3 15:04:05 2024\nok\n"
|
||||
ms := readAll(t, in, 10)
|
||||
if len(ms) != 2 || !errors.Is(ms[0].Err, ErrTooLarge) || ms[1].Err != nil || string(ms[1].Data) != "ok\n" {
|
||||
t.Fatalf("messages: %+v", ms)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKeepIsBoundedByTheLimit(t *testing.T) {
|
||||
for _, c := range []struct{ max, want int }{{0, 0}, {-1, 0}, {100, minKeep}, {5000, 5001}} {
|
||||
if got := NewReader(strings.NewReader(""), c.max).keep(); got != c.want {
|
||||
t.Errorf("max %d: keep %d, want %d", c.max, got, c.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheLimitIsExactAndCountsTheRestoredLine(t *testing.T) {
|
||||
body := "Subject: s\n\n>From here\n" // restored to "From here\n": 22 bytes
|
||||
in := "From a@x.com Tue Jan 2 15:04:05 2024\n" + body
|
||||
if ms := readAll(t, in, 22); len(ms) != 1 || ms[0].Err != nil {
|
||||
t.Fatalf("exactly at the limit must pass: %+v", ms)
|
||||
}
|
||||
if ms := readAll(t, in, 21); len(ms) != 1 || !errors.Is(ms[0].Err, ErrTooLarge) {
|
||||
t.Fatalf("one over the limit must be too large: %+v", ms)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -22,6 +22,9 @@ const (
|
|||
defaultBatch = 50
|
||||
defaultTimeout = time.Minute
|
||||
defaultInterval = 5 * time.Second
|
||||
|
||||
// purgeEvery is the least time between two scans that forget old failed deliveries.
|
||||
purgeEvery = 5 * time.Minute
|
||||
)
|
||||
|
||||
var backoff = []time.Duration{time.Minute, 5 * time.Minute, 15 * time.Minute, time.Hour, 3 * time.Hour, 6 * time.Hour}
|
||||
|
|
@ -61,6 +64,53 @@ type Worker struct {
|
|||
wakeOnce sync.Once
|
||||
wake chan struct{}
|
||||
runMu sync.Mutex
|
||||
|
||||
lastPurge time.Time // under runMu; zero until the first purge
|
||||
// Hooks for tests; nil means the Store methods.
|
||||
purge func() (int, error)
|
||||
readBody func(msgID int64) ([]byte, error)
|
||||
}
|
||||
|
||||
// bodies reads each message body once per run, however many recipients share it.
|
||||
type bodies struct {
|
||||
read func(msgID int64) ([]byte, error)
|
||||
mu sync.Mutex
|
||||
m map[int64]*body
|
||||
}
|
||||
|
||||
type body struct {
|
||||
once sync.Once
|
||||
data []byte
|
||||
err error
|
||||
}
|
||||
|
||||
func (b *bodies) get(msgID int64) ([]byte, error) {
|
||||
b.mu.Lock()
|
||||
e := b.m[msgID]
|
||||
if e == nil {
|
||||
e = &body{}
|
||||
b.m[msgID] = e
|
||||
}
|
||||
b.mu.Unlock()
|
||||
e.once.Do(func() { e.data, e.err = b.read(msgID) })
|
||||
return e.data, e.err
|
||||
}
|
||||
|
||||
// purgeOld forgets old failed deliveries, at most once every purgeEvery.
|
||||
func (w *Worker) purgeOld() {
|
||||
now := w.Store.Now()
|
||||
if !w.lastPurge.IsZero() && now.Sub(w.lastPurge) < purgeEvery {
|
||||
return
|
||||
}
|
||||
purge := w.purge
|
||||
if purge == nil {
|
||||
purge = w.Store.PurgeFailedOutbox
|
||||
}
|
||||
if _, err := purge(); err != nil {
|
||||
log.Printf("queue: purge old failures: %v", err)
|
||||
return // try again on the next run
|
||||
}
|
||||
w.lastPurge = now
|
||||
}
|
||||
|
||||
func (w *Worker) wakeCh() chan struct{} {
|
||||
|
|
@ -125,14 +175,18 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
|
|||
if conc <= 0 {
|
||||
conc = defaultConcurrency
|
||||
}
|
||||
if _, err := w.Store.PurgeFailedOutbox(); err != nil {
|
||||
log.Printf("queue: purge old failures: %v", err)
|
||||
}
|
||||
w.purgeOld()
|
||||
due, err := w.Store.DueOutbox(batch)
|
||||
if err != nil {
|
||||
return Stats{}, err
|
||||
}
|
||||
|
||||
read := w.readBody
|
||||
if read == nil {
|
||||
read = w.Store.OutboxData
|
||||
}
|
||||
cache := &bodies{read: read, m: map[int64]*body{}}
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
stats Stats
|
||||
|
|
@ -148,7 +202,7 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
|
|||
go func(it store.OutboxItem) {
|
||||
defer wg.Done()
|
||||
defer func() { <-sem }()
|
||||
o := w.deliver(ctx, it)
|
||||
o := w.deliver(ctx, cache, it)
|
||||
mu.Lock()
|
||||
stats.Sent += o.Sent
|
||||
stats.Retried += o.Retried
|
||||
|
|
@ -162,8 +216,8 @@ func (w *Worker) RunOnce(ctx context.Context) (Stats, error) {
|
|||
return stats, nil
|
||||
}
|
||||
|
||||
func (w *Worker) deliver(ctx context.Context, it store.OutboxItem) Stats {
|
||||
data, err := w.Store.OutboxData(it.MsgID)
|
||||
func (w *Worker) deliver(ctx context.Context, cache *bodies, it store.OutboxItem) Stats {
|
||||
data, err := cache.get(it.MsgID)
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
w.complete(it)
|
||||
return Stats{Dropped: 1}
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue