Merge pull request 'Fix the findings of the second review' (#43) from fix/second-review into main

This commit is contained in:
squidd 2026-11-10 15:23:45 +01:00
commit 1f74e375fc
15 changed files with 947 additions and 61 deletions

View file

@ -71,13 +71,17 @@ In the admin UI, pick "Publishers only (newsletter)" in the list's Settings tab
A publisher's post passes the same identity check as any post: the `From` header must name the SMTP sender, or the post is held. A post from anyone else, member or not, is refused with `550 5.7.1`; it is never held, archived or queued. A publisher disabled by bounces can still post. A publisher's post passes the same identity check as any post: the `From` header must name the SMTP sender, or the post is held. A post from anyone else, member or not, is refused with `550 5.7.1`; it is never held, archived or queued. A publisher disabled by bounces can still post.
**Protect the publishers' addresses.** Parsimony does not verify SPF, DKIM or DMARC on inbound mail, and every issue carries the publisher's address in `Reply-To`, so subscribers learn it. Anyone who can send mail to your MX with that address as the sender and in `From` can mail the whole subscriber base. Make Postfix reject forged senders (see Security notes), and keep the list of publishers short.
**Sent posts.** Each post queued to a list is recorded with its subject, time and number of recipients, and the queue counts how its deliveries end: **sent** (accepted by the relay), **failed** (refused, or still failing after 3 days), **dropped** (the mailbox does not exist, or the delivery was discarded) and **pending**. A count changes in the same transaction as the delivery. See them with `parsimony posts ls news@example.org`, `GET /api/v1/lists/{id}/posts`, or the Sent tab of a list, which refreshes every 5 seconds while something is pending. Posts are kept for 400 days. Parsimony does not track opens or clicks. **Sent posts.** Each post queued to a list is recorded with its subject, time and number of recipients, and the queue counts how its deliveries end: **sent** (accepted by the relay), **failed** (refused, or still failing after 3 days), **dropped** (the mailbox does not exist, or the delivery was discarded) and **pending**. A count changes in the same transaction as the delivery. See them with `parsimony posts ls news@example.org`, `GET /api/v1/lists/{id}/posts`, or the Sent tab of a list, which refreshes every 5 seconds while something is pending. Posts are kept for 400 days. Parsimony does not track opens or clicks.
**Priority.** Newsletter deliveries go out after everything else that is due, so a big newsletter never delays discussion mail or confirmations. **Priority.** Newsletter deliveries go out after everything else that is due, so discussion mail and confirmations that are due go first. A batch of newsletter deliveries already taken by the worker is paced to the end, so with a low `relay.rate_per_second` a confirmation can wait for up to that batch (50 deliveries divided by the rate). Sustained discussion traffic at the paced rate can delay a newsletter without limit; a delivery that waited more than 3 days is given up at its first temporary failure.
**Rate limiting.** Relays such as Amazon SES throttle sending. Set `relay.rate_per_second` to the most sends the daemon may start in one second (default `0`, unlimited). One pacer sits in front of every send attempt, retries included, and spaces them evenly. Set it a little under your relay's limit (12 for a limit of 14). Each recipient is one message, so a post to N members takes about N divided by the rate in seconds. **Rate limiting.** Relays such as Amazon SES throttle sending. Set `relay.rate_per_second` to the most sends the daemon may start in one second (default `0`, unlimited). One pacer sits in front of every send attempt, retries included, and spaces them evenly. Set it a little under your relay's limit (12 for a limit of 14). Each recipient is one message, so a post to N members takes about N divided by the rate in seconds.
**Subscribe form.** A list whose archive is public and whose join policy is open gets a signup page at `/subscribe/{address}` on the public web listener, linked from its archive page. It sends the usual double opt-in message. The page that follows is always the same, whether or not the address is a member or the list qualifies for mail, so the form cannot be used to find out who is subscribed. A hidden field catches bots; each IP (the TCP peer, `X-Forwarded-For` is ignored) may post 10 times per hour, then gets `429`; behind a reverse proxy every visitor shares its address, so limit at the proxy too. The body is capped at 4 KiB. **Subscribe form.** A list whose archive is public and whose join policy is open gets a signup page at `/subscribe/{address}` on the public web listener, linked from its archive page. It sends the usual double opt-in message. The page that follows is always the same, whether or not the address is a member or the list qualifies for mail, so the form cannot be used to find out who is subscribed. A hidden field catches bots; each client (an IPv4 address or an IPv6 /64) may post 10 times per hour, then gets `429`. Behind a reverse proxy, list the proxy in `web.trusted_proxies` (addresses or CIDR ranges, default empty): `X-Forwarded-For` is then read only from those proxies, and the client is its right-most address that is not a trusted proxy. With the list empty the header is ignored and the TCP peer is the client, so every visitor behind a proxy shares one budget. Make sure the proxy appends to `X-Forwarded-For` and cannot be bypassed. At most 100000 clients are tracked; past that, new clients get `429` until a window ends. The form refuses addresses at `localhost`, at bare IPs or `[literals]`, and at names ending in `.local`, `.localhost`, `.internal` or `.invalid`. The body is capped at 4 KiB.
**Confirmation cap.** Each subscribe or unsubscribe request sends a confirmation mail. At most `limits.pending_per_hour` of them (default 300) are accepted per hour across all lists. Past that, requests get no mail, the visitor still sees the usual page, and the daemon logs a warning at most once a minute. Raise it before a newsletter launch.
Not included: scheduled sends and open or click tracking. Not included: scheduled sends and open or click tracking.
@ -240,6 +244,8 @@ relay:
## Delivery and the queue ## Delivery and the queue
Each delivery has a time limit per step rather than one overall. Connecting, the greeting, TLS, authentication and the envelope get 60 seconds together. Sending the message gets 60 seconds plus one second per 32 KiB, at most 10 minutes. The relay then has 10 minutes to accept it, as RFC 5321 recommends, so a relay that runs content filters is not cut off and does not receive the message twice. If the relay never answers after the whole message was sent, the delivery is retried and the error in the queue says the outcome is unknown: the relay may already have it.
LMTP answers `250` when the post is in the queue, so a down relay never bounces mail back to Postfix. LMTP answers `250` when the post is in the queue, so a down relay never bounces mail back to Postfix.
It answers `451` only when the database itself fails. It answers `451` only when the database itself fails.
@ -351,6 +357,8 @@ lists.example.org lmtp:inet:127.0.0.1:2424
(`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is (`openssl rand -hex 24` makes a good one). After 10 wrong or missing tokens within 5 minutes an address is
locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy locked out for 5 minutes (HTTP 429, valid token included). The address is the TCP peer: behind a reverse proxy
every client shares the proxy's address, so rate-limit at the proxy as well. every client shares the proxy's address, so rate-limit at the proxy as well.
- **Stopping.** The daemon finishes in-flight mail on SIGTERM and waits up to 30 s, idle LMTP connections included; Docker's default `docker stop` grace period is 10 s, so run the container with `--stop-timeout 35`.
- **Known gaps.** A held message that the sender retries after a lost `250` is held twice, and approving both delivers both; the admin sees a held message's summary but not its body before approving; a member's responder that tags its replies `Auto-Submitted: auto-generated` and answers the list address is delivered as a normal post until a human notices.
- The API and the admin UI use one bearer token and no TLS: put them behind a reverse proxy with TLS and keep the `api` - The API and the admin UI use one bearer token and no TLS: put them behind a reverse proxy with TLS and keep the `api`
listener private. listener private.

View file

@ -218,6 +218,9 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
return nil, fmt.Errorf("store: %w", err) return nil, fmt.Errorf("store: %w", err)
} }
st.Now = nowFunc st.Now = nowFunc
if cfg.Limits.PendingPerHour > 0 { // zero: a Config not made by config.Load, keep the default
st.PendingPerHour = int(cfg.Limits.PendingPerHour)
}
a := &App{store: st} a := &App{store: st}
rt := router.New(st) rt := router.New(st)
@ -281,7 +284,7 @@ func Start(cfg config.Config, apiToken string) (*App, error) {
} }
a.WebAddr = pubLn.Addr().String() a.WebAddr = pubLn.Addr().String()
a.lns = append(a.lns, pubLn) a.lns = append(a.lns, pubLn)
a.pub = newHTTPServer(web.New(st, rt)) a.pub = newHTTPServer(web.New(st, rt, cfg.WebTrustedProxies()))
} }
a.lmtpOpen = &connTracker{Listener: serveLn, conns: map[*trackedConn]struct{}{}} a.lmtpOpen = &connTracker{Listener: serveLn, conns: map[*trackedConn]struct{}{}}

View file

@ -344,6 +344,60 @@ func TestArchiveAndUnsubscribeEndToEnd(t *testing.T) {
} }
} }
func TestWebBelievesForwardedForOnlyFromTrustedProxies(t *testing.T) {
relayAddr, _ := startSink(t)
cfg := testConfig(t, relayAddr)
cfg.Web.TrustedProxies = []string{"127.0.0.1"} // the test client is the proxy
a, err := Start(cfg, "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
// The form's rate limit counts every POST, before the list is looked up: an unknown list will do.
post := func(xff string) int {
req, _ := http.NewRequest("POST", "http://"+a.WebAddr+"/subscribe/ghost@example.org", strings.NewReader("email=a%40x.com"))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Set("X-Forwarded-For", xff)
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
resp.Body.Close()
return resp.StatusCode
}
for i := 0; i < 10; i++ {
post("203.0.113.1")
}
if c := post("203.0.113.1"); c != 429 {
t.Fatalf("one client over its limit: %d, want 429", c)
}
if c := post("203.0.113.2"); c != 404 {
t.Fatalf("another client behind the same proxy: %d, want 404 (not limited)", c)
}
}
func TestPendingPerHourReachesTheStore(t *testing.T) {
relayAddr, relay := startSink(t)
cfg := testConfig(t, relayAddr)
cfg.Limits.PendingPerHour = 1
a, err := Start(cfg, "tok")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { a.Close() })
call(t, a, "POST", "/api/v1/lists", `{"name":"dev","domain":"example.org","title":"Dev"}`, "tok")
for _, from := range []string{"carol@x.com", "dave@x.com"} {
if err := lmtpSend(t, a.LMTPAddr, from, "dev+join@example.org", ""); err != nil {
t.Fatal(err)
}
}
relay.waitCount(t, 1)
time.Sleep(200 * time.Millisecond)
if n := len(relay.all()); n != 1 {
t.Fatalf("limits.pending_per_hour = 1: %d confirmation mails, want 1", n)
}
}
func TestWebDisabledWhenListenIsEmpty(t *testing.T) { func TestWebDisabledWhenListenIsEmpty(t *testing.T) {
relayAddr, _ := startSink(t) relayAddr, _ := startSink(t)
cfg := testConfig(t, relayAddr) cfg := testConfig(t, relayAddr)

View file

@ -32,7 +32,16 @@ type Config struct {
} `yaml:"api"` } `yaml:"api"`
Web struct { Web struct {
Listen string `yaml:"listen"` // public archive and unsubscribe pages; empty disables them Listen string `yaml:"listen"` // public archive and unsubscribe pages; empty disables them
// TrustedProxies are the reverse proxies (CIDR ranges or single addresses) whose
// X-Forwarded-For names the client of the subscribe form's rate limit. Empty, the default:
// the TCP peer is the client and the header is ignored.
TrustedProxies []string `yaml:"trusted_proxies"`
} `yaml:"web"` } `yaml:"web"`
Limits struct {
// PendingPerHour is the most subscribe and unsubscribe requests (each one sends a
// confirmation mail) accepted in an hour, for all lists together.
PendingPerHour Count `yaml:"pending_per_hour"`
} `yaml:"limits"`
// DKIM lists the signing keys, one per domain. The private keys stay in files. // DKIM lists the signing keys, one per domain. The private keys stay in files.
DKIM []DKIMKey `yaml:"dkim"` DKIM []DKIMKey `yaml:"dkim"`
Relay struct { Relay struct {
@ -68,6 +77,20 @@ func (r *Rate) UnmarshalYAML(n *yaml.Node) error {
return nil return nil
} }
// Count is limits.pending_per_hour. It exists so that a value that is not a whole number is reported
// with the name of the key.
type Count int
// UnmarshalYAML reads a YAML integer.
func (c *Count) UnmarshalYAML(n *yaml.Node) error {
var i int
if n.Tag != "!!int" || n.Decode(&i) != nil {
return fmt.Errorf("line %d: limits.pending_per_hour must be a whole number, not %q", n.Line, n.Value)
}
*c = Count(i)
return nil
}
// DKIMKey is the signing key of one domain. // DKIMKey is the signing key of one domain.
type DKIMKey struct { type DKIMKey struct {
Domain string `yaml:"domain"` Domain string `yaml:"domain"`
@ -76,9 +99,15 @@ type DKIMKey struct {
} }
// LMTPAllowed returns the parsed lmtp.allow list (empty: every client is accepted). // LMTPAllowed returns the parsed lmtp.allow list (empty: every client is accepted).
func (c Config) LMTPAllowed() []*net.IPNet { func (c Config) LMTPAllowed() []*net.IPNet { return parseNets(c.LMTP.Allow) }
nets := make([]*net.IPNet, 0, len(c.LMTP.Allow))
for _, a := range c.LMTP.Allow { // WebTrustedProxies returns the parsed web.trusted_proxies list (empty: X-Forwarded-For is ignored).
func (c Config) WebTrustedProxies() []*net.IPNet { return parseNets(c.Web.TrustedProxies) }
// parseNets parses a list that validate has already checked.
func parseNets(list []string) []*net.IPNet {
nets := make([]*net.IPNet, 0, len(list))
for _, a := range list {
if n, err := parseNet(a); err == nil { if n, err := parseNet(a); err == nil {
nets = append(nets, n) nets = append(nets, n)
} }
@ -201,6 +230,14 @@ func (c Config) validate() error {
return fmt.Errorf("lmtp.allow[%d]: %w", i, err) return fmt.Errorf("lmtp.allow[%d]: %w", i, err)
} }
} }
if c.Limits.PendingPerHour < 1 {
return fmt.Errorf("limits.pending_per_hour must be 1 or more, not %d", c.Limits.PendingPerHour)
}
for i, a := range c.Web.TrustedProxies {
if _, err := parseNet(a); err != nil {
return fmt.Errorf("web.trusted_proxies[%d]: %w", i, err)
}
}
seen := map[string]bool{} seen := map[string]bool{}
for i, k := range c.DKIM { for i, k := range c.DKIM {
switch { switch {
@ -230,6 +267,7 @@ func Load(path string) (Config, error) {
c.Relay.Concurrency = 4 c.Relay.Concurrency = 4
c.API.Listen = "127.0.0.1:8080" c.API.Listen = "127.0.0.1:8080"
c.Web.Listen = "127.0.0.1:8081" c.Web.Listen = "127.0.0.1:8081"
c.Limits.PendingPerHour = 300
b, err := os.ReadFile(path) b, err := os.ReadFile(path)
if err != nil { if err != nil {

View file

@ -308,3 +308,80 @@ func TestBaseURLIsChecked(t *testing.T) {
} }
} }
} }
func TestLoadWebTrustedProxies(t *testing.T) {
c, err := Load(write(t, "web:\n trusted_proxies: [10.0.0.0/8, 192.168.1.1, \"::1\", \"fd00::/8\"]\n"))
if err != nil {
t.Fatal(err)
}
nets := c.WebTrustedProxies()
if len(nets) != 4 {
t.Fatalf("nets: %v", nets)
}
for ip, want := range map[string]bool{
"10.9.8.7": true, "192.168.1.1": true, "192.168.1.2": false, "::1": true, "fd00::5": true, "2001:db8::1": false,
} {
got := false
for _, n := range nets {
if n.Contains(net.ParseIP(ip)) {
got = true
}
}
if got != want {
t.Errorf("%s trusted = %v, want %v", ip, got, want)
}
}
for _, empty := range []string{"{}\n", "web:\n trusted_proxies: []\n"} {
if c, err := Load(write(t, empty)); err != nil || len(c.WebTrustedProxies()) != 0 {
t.Fatalf("%q: no trusted proxy by default: %v %v", empty, c.WebTrustedProxies(), err)
}
}
}
func TestLoadRejectsABadTrustedProxy(t *testing.T) {
for _, bad := range []string{"not-an-ip", "10.0.0.0/33", "fd00::/129", "10.0.0/8", "", "10.0.0.1/", "proxy.example.org"} {
_, err := Load(write(t, "web:\n trusted_proxies: [\"127.0.0.1\", \""+bad+"\"]\n"))
if err == nil || !strings.Contains(err.Error(), "web.trusted_proxies[1]") {
t.Errorf("%q: want an error naming the entry, got %v", bad, err)
}
}
}
func TestExampleConfigLoadsAndDocumentsTheSettings(t *testing.T) {
b, err := os.ReadFile("../../parsimony.example.yaml")
if err != nil {
t.Fatal(err)
}
if _, err := Load(write(t, string(b))); err != nil {
t.Fatal(err)
}
// Uncommented, the documented lines are valid.
uncommented := string(b)
for _, line := range []string{" # trusted_proxies:", "# limits:", "# pending_per_hour:"} {
if !strings.Contains(uncommented, line) {
t.Fatalf("%q is not documented", line)
}
uncommented = strings.ReplaceAll(uncommented, line, strings.Replace(line, "# ", "", 1))
}
c, err := Load(write(t, uncommented))
if err != nil || len(c.WebTrustedProxies()) == 0 || c.Limits.PendingPerHour != 300 {
t.Fatalf("documented settings: %v %d %v", c.WebTrustedProxies(), c.Limits.PendingPerHour, err)
}
}
func TestLimitsPendingPerHour(t *testing.T) {
c, err := Load(write(t, "{}\n"))
if err != nil || c.Limits.PendingPerHour != 300 {
t.Fatalf("default: %d %v", c.Limits.PendingPerHour, err)
}
c, err = Load(write(t, "limits:\n pending_per_hour: 2000\n"))
if err != nil || c.Limits.PendingPerHour != 2000 {
t.Fatalf("value: %d %v", c.Limits.PendingPerHour, err)
}
for _, bad := range []string{"0", "-1", "lots", "1.5"} {
_, err := Load(write(t, "limits:\n pending_per_hour: "+bad+"\n"))
if err == nil || !strings.Contains(err.Error(), "pending_per_hour") {
t.Errorf("%s: want an error naming limits.pending_per_hour, got %v", bad, err)
}
}
}

View file

@ -45,6 +45,59 @@ type SMTPRelay struct {
Username string // SMTP AUTH PLAIN; only over TLS or to a loopback address Username string // SMTP AUTH PLAIN; only over TLS or to a loopback address
Password string Password string
Helo string // EHLO name; "localhost" by default Helo string // EHLO name; "localhost" by default
// Timeout bounds the setup of each transaction: connection, greeting, TLS, AUTH, MAIL FROM,
// RCPT TO and the DATA command. DefaultTimeout if zero. The upload of the message and the
// wait for its acceptance have their own budgets (see uploadBudget and finalReplyTimeout).
Timeout time.Duration
}
// DefaultTimeout is the default SMTPRelay.Timeout.
const DefaultTimeout = time.Minute
// Budgets of the later phases of a transaction. Variables so that tests can shrink them.
var (
dataFloor = time.Minute // least time to upload a message
dataRate = 32 << 10 // slowest upload we wait for, in bytes per second
dataCap = 10 * time.Minute // most time to upload a message
// finalReplyTimeout is how long the relay may take to answer the final dot: RFC 5321 4.5.3.2.6
// asks for 10 minutes, because content filters run at that point.
finalReplyTimeout = 10 * time.Minute
)
// uploadBudget is the time allowed to upload a message of size bytes: dataFloor plus the time
// it takes at dataRate, at most dataCap.
func uploadBudget(size int) time.Duration {
b := dataFloor + time.Duration(float64(size)/float64(dataRate)*float64(time.Second))
return min(b, dataCap)
}
func (r *SMTPRelay) timeout() time.Duration {
if r.Timeout > 0 {
return r.Timeout
}
return DefaultTimeout
}
// phasedConn keeps the connection within the budget of the current phase. go-smtp sets its own
// deadline around each command and clears it afterwards; here no deadline it sets, cleared or not,
// may go past the limit of the phase.
type phasedConn struct {
net.Conn
limit time.Time
}
func (p *phasedConn) SetDeadline(t time.Time) error {
if t.IsZero() || p.limit.Before(t) {
t = p.limit
}
return p.Conn.SetDeadline(t)
}
// phase gives the connection d to finish the next phase. The context's own deadline and its
// cancellation close the connection instead (see dial), so that the error is the context's.
func (p *phasedConn) phase(d time.Duration) {
p.limit = time.Now().Add(d)
p.Conn.SetDeadline(p.limit)
} }
// Send splits recipients in batches and sends one transaction per batch. // Send splits recipients in batches and sends one transaction per batch.
@ -65,20 +118,21 @@ func (r *SMTPRelay) Send(ctx context.Context, from string, to []string, data []b
return nil return nil
} }
// dial opens the connection and speaks EHLO, TLS and AUTH as configured. // dial opens the connection and speaks EHLO, TLS and AUTH as configured. The connection's deadline
// The returned release function must be called when the client is done: until then the connection // is the end of the setup budget, which starts now. The returned release function must be called when the client is done: until then the connection
// is closed if ctx is cancelled, even in the middle of EHLO, STARTTLS or AUTH with a silent relay. // is closed if ctx is cancelled, even in the middle of EHLO, STARTTLS or AUTH with a silent relay.
func (r *SMTPRelay) dial(ctx context.Context) (c *smtp.Client, release func(), err error) { func (r *SMTPRelay) dial(ctx context.Context) (c *smtp.Client, conn *phasedConn, release func(), err error) {
setupEnd := time.Now().Add(r.timeout())
host, _, err := net.SplitHostPort(r.Addr) host, _, err := net.SplitHostPort(r.Addr)
if err != nil { if err != nil {
return nil, nil, fmt.Errorf("relay address %q: %w", r.Addr, err) return nil, nil, nil, fmt.Errorf("relay address %q: %w", r.Addr, err)
} }
mode := r.TLS mode := r.TLS
if mode == "" { if mode == "" {
mode = TLSNone mode = TLSNone
} }
if r.Username != "" && mode == TLSNone && !isLoopback(host) { if r.Username != "" && mode == TLSNone && !isLoopback(host) {
return nil, nil, errors.New("refusing to send credentials in clear to a remote relay: set relay.tls to starttls or tls") return nil, nil, nil, errors.New("refusing to send credentials in clear to a remote relay: set relay.tls to starttls or tls")
} }
helo := r.Helo helo := r.Helo
if helo == "" { if helo == "" {
@ -89,21 +143,20 @@ func (r *SMTPRelay) dial(ctx context.Context) (c *smtp.Client, release func(), e
tlsConf.ServerName = r.ServerName tlsConf.ServerName = r.ServerName
} }
d := &net.Dialer{Timeout: 10 * time.Second} d := &net.Dialer{Timeout: 10 * time.Second, Deadline: setupEnd}
var conn net.Conn var raw net.Conn
if mode == TLSImplicit { if mode == TLSImplicit {
conn, err = (&tls.Dialer{NetDialer: d, Config: tlsConf}).DialContext(ctx, "tcp", r.Addr) raw, err = (&tls.Dialer{NetDialer: d, Config: tlsConf}).DialContext(ctx, "tcp", r.Addr)
} else { } else {
conn, err = d.DialContext(ctx, "tcp", r.Addr) raw, err = d.DialContext(ctx, "tcp", r.Addr)
} }
if err != nil { if err != nil {
return nil, nil, err return nil, nil, nil, err
}
// A relay that stops answering must not hold us past the context: apply its deadline to the
// connection and close the connection if the context is cancelled.
if dl, ok := ctx.Deadline(); ok {
conn.SetDeadline(dl)
} }
// A relay that stops answering must not hold us past the budget or the context: apply the
// earlier deadline to the connection, and close the connection if the context is cancelled.
conn = &phasedConn{Conn: raw}
conn.phase(time.Until(setupEnd))
stop := make(chan struct{}) stop := make(chan struct{})
go func() { go func() {
select { select {
@ -113,10 +166,10 @@ func (r *SMTPRelay) dial(ctx context.Context) (c *smtp.Client, release func(), e
} }
}() }()
release = func() { close(stop) } release = func() { close(stop) }
fail := func(err error) (*smtp.Client, func(), error) { fail := func(err error) (*smtp.Client, *phasedConn, func(), error) {
release() release()
conn.Close() conn.Close()
return nil, nil, err return nil, nil, nil, err
} }
if mode == TLSStartTLS { if mode == TLSStartTLS {
@ -139,7 +192,7 @@ func (r *SMTPRelay) dial(ctx context.Context) (c *smtp.Client, release func(), e
return fail(authError(err)) return fail(authError(err))
} }
} }
return c, release, nil return c, conn, release, nil
} }
func isLoopback(host string) bool { func isLoopback(host string) bool {
@ -162,7 +215,7 @@ func authError(err error) error {
} }
func (r *SMTPRelay) sendBatch(ctx context.Context, from string, to []string, data []byte) (err error) { func (r *SMTPRelay) sendBatch(ctx context.Context, from string, to []string, data []byte) (err error) {
c, release, err := r.dial(ctx) c, conn, release, err := r.dial(ctx)
if err != nil { if err != nil {
if ctx.Err() != nil { if ctx.Err() != nil {
err = ctx.Err() // report the cause, not "use of closed connection" err = ctx.Err() // report the cause, not "use of closed connection"
@ -194,13 +247,23 @@ func (r *SMTPRelay) sendBatch(ctx context.Context, from string, to []string, dat
if err != nil { if err != nil {
return authError(&PhaseError{Phase: "DATA", Err: err}) return authError(&PhaseError{Phase: "DATA", Err: err})
} }
conn.phase(uploadBudget(len(data)))
if _, err := io.Copy(w, bytes.NewReader(data)); err != nil { if _, err := io.Copy(w, bytes.NewReader(data)); err != nil {
return &PhaseError{Phase: "DATA", Err: err} return &PhaseError{Phase: "DATA", Err: err}
} }
// Close sends the rest of the message and the final dot, then waits for the verdict.
conn.phase(finalReplyTimeout)
if err := w.Close(); err != nil { if err := w.Close(); err != nil {
var ne net.Error
if errors.As(err, &ne) && ne.Timeout() {
// Temporary like any timeout, but the relay may have taken the message: a retry can
// deliver it twice. Say so, so the admin can tell.
err = fmt.Errorf("no reply to the end of the message, outcome unknown: %w", err)
}
return &PhaseError{Phase: "DATA", Err: err} return &PhaseError{Phase: "DATA", Err: err}
} }
// The relay has accepted the message. If QUIT now fails, retrying would deliver it twice. // The relay has accepted the message. If QUIT now fails, retrying would deliver it twice.
conn.phase(r.timeout())
c.Quit() c.Quit()
return nil return nil
} }

View file

@ -0,0 +1,231 @@
package outbound
import (
"bufio"
"context"
"errors"
"net"
"strings"
"sync"
"testing"
"time"
)
// scaleTimeouts shrinks the phase budgets so the tests run in milliseconds.
func scaleTimeouts(t *testing.T, floor, finalReply time.Duration) {
t.Helper()
oldFloor, oldRate, oldCap, oldReply := dataFloor, dataRate, dataCap, finalReplyTimeout
dataFloor, finalReplyTimeout = floor, finalReply
t.Cleanup(func() { dataFloor, dataRate, dataCap, finalReplyTimeout = oldFloor, oldRate, oldCap, oldReply })
}
// slowRelay is a bare SMTP server that waits dotDelay before answering the final dot
// (forever when dotDelay is negative).
type slowRelay struct {
mu sync.Mutex
accepted int
}
func startSlowRelay(t *testing.T, dotDelay time.Duration) (*slowRelay, string) {
t.Helper()
s := &slowRelay{}
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
done := make(chan struct{})
t.Cleanup(func() { close(done); ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
defer c.Close()
r := bufio.NewReader(c)
c.Write([]byte("220 slow ESMTP\r\n"))
inData := false
for {
l, err := r.ReadString('\n')
if err != nil {
return
}
l = strings.TrimRight(l, "\r\n")
switch {
case inData:
if l != "." {
continue
}
inData = false
if dotDelay < 0 {
<-done
return
}
time.Sleep(dotDelay)
s.mu.Lock()
s.accepted++
s.mu.Unlock()
c.Write([]byte("250 2.0.0 queued\r\n"))
case strings.HasPrefix(l, "EHLO"), strings.HasPrefix(l, "HELO"):
c.Write([]byte("250 slow\r\n"))
case strings.HasPrefix(l, "MAIL"), strings.HasPrefix(l, "RCPT"):
c.Write([]byte("250 2.1.0 ok\r\n"))
case l == "DATA":
inData = true
c.Write([]byte("354 go ahead\r\n"))
case l == "QUIT":
c.Write([]byte("221 bye\r\n"))
return
}
}
}()
}
}()
return s, ln.Addr().String()
}
func TestASlowReplyToTheFinalDotIsWaitedForAndDeliversOnce(t *testing.T) {
// The relay takes longer than the whole setup budget to accept the message (a content filter):
// that is not a failure, or the retry would deliver it a second time.
scaleTimeouts(t, 100*time.Millisecond, 3*time.Second)
s, addr := startSlowRelay(t, 400*time.Millisecond)
r := &SMTPRelay{Addr: addr, Timeout: 100 * time.Millisecond}
if err := r.Send(context.Background(), "b@x.org", []string{"a@x.com"}, []byte("Subject: x\r\n\r\nx\r\n")); err != nil {
t.Fatalf("the relay accepted the message: %v", err)
}
s.mu.Lock()
defer s.mu.Unlock()
if s.accepted != 1 {
t.Fatalf("accepted %d times", s.accepted)
}
}
func TestNoReplyToTheFinalDotTimesOutAsATemporaryFailure(t *testing.T) {
scaleTimeouts(t, 100*time.Millisecond, 300*time.Millisecond)
_, addr := startSlowRelay(t, -1)
r := &SMTPRelay{Addr: addr, Timeout: 5 * time.Second}
start := time.Now()
err := r.Send(context.Background(), "b@x.org", []string{"a@x.com"}, []byte("Subject: x\r\n\r\nx\r\n"))
var ne net.Error
if !errors.As(err, &ne) || !ne.Timeout() {
t.Fatalf("want a timeout, got %v", err)
}
if IsPermanent(err) {
t.Fatalf("an unknown outcome is retried, not failed: %v", err)
}
if !strings.Contains(err.Error(), "outcome unknown") {
t.Fatalf("the error says the relay may have the message: %v", err)
}
if took := time.Since(start); took < 250*time.Millisecond || took > 3*time.Second {
t.Fatalf("the final reply has its own budget, took %v", took)
}
}
func TestARelayThatStallsAtTheGreetingTimesOutInTheShortBudget(t *testing.T) {
r := &SMTPRelay{Addr: silent(t), Timeout: 150 * time.Millisecond}
start := time.Now()
err := r.Send(context.Background(), "b@x.org", []string{"a@x.com"}, []byte("x"))
var ne net.Error
if !errors.As(err, &ne) || !ne.Timeout() {
t.Fatalf("want a timeout, got %v", err)
}
if IsPermanent(err) {
t.Fatalf("a timeout is temporary: %v", err)
}
if took := time.Since(start); took > 2*time.Second {
t.Fatalf("took %v", took)
}
}
func TestTheUploadBudgetGrowsWithTheMessageSize(t *testing.T) {
if got := uploadBudget(0); got != time.Minute {
t.Fatalf("floor: %v", got)
}
small, big := uploadBudget(10<<10), uploadBudget(10<<20)
if big <= small {
t.Fatalf("a big message gets more time: small %v, big %v", small, big)
}
if want := time.Minute + 320*time.Second; big != want { // 10 MiB at 32 KiB/s
t.Fatalf("10 MiB: got %v, want %v", big, want)
}
if got := uploadBudget(1 << 40); got != 10*time.Minute {
t.Fatalf("cap: %v", got)
}
}
func TestCancellingTheContextAbortsTheWaitForTheFinalReply(t *testing.T) {
_, addr := startSlowRelay(t, -1) // the full 10 minutes apply
ctx, cancel := context.WithCancel(context.Background())
errc := make(chan error, 1)
go func() {
errc <- (&SMTPRelay{Addr: addr}).Send(ctx, "b@x.org", []string{"a@x.com"}, []byte("Subject: x\r\n\r\nx\r\n"))
}()
time.Sleep(200 * time.Millisecond)
cancel()
select {
case err := <-errc:
if !errors.Is(err, context.Canceled) {
t.Fatalf("want context.Canceled, got %v", err)
}
case <-time.After(2 * time.Second):
t.Fatal("Send ignored the cancelled context while waiting for the final reply")
}
}
// stopsReadingAtData answers up to DATA, then never reads again.
func stopsReadingAtData(t *testing.T) string {
t.Helper()
ln, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
done := make(chan struct{})
t.Cleanup(func() { close(done); ln.Close() })
go func() {
for {
c, err := ln.Accept()
if err != nil {
return
}
go func() {
defer c.Close()
r := bufio.NewReader(c)
c.Write([]byte("220 stuck ESMTP\r\n"))
for {
l, err := r.ReadString('\n')
if err != nil {
return
}
switch l = strings.TrimRight(l, "\r\n"); {
case l == "DATA":
c.Write([]byte("354 go ahead\r\n"))
<-done
return
case strings.HasPrefix(l, "EHLO"), strings.HasPrefix(l, "HELO"):
c.Write([]byte("250 stuck\r\n"))
default:
c.Write([]byte("250 ok\r\n"))
}
}
}()
}
}()
return ln.Addr().String()
}
func TestAStalledUploadTimesOutInItsOwnBudget(t *testing.T) {
scaleTimeouts(t, 200*time.Millisecond, time.Minute)
dataRate = 1 << 40 // the budget is the floor
r := &SMTPRelay{Addr: stopsReadingAtData(t), Timeout: time.Minute}
big := []byte(strings.Repeat(strings.Repeat("x", 998)+"\r\n", 32<<10)) // 32 MiB, more than the socket buffers
start := time.Now()
err := r.Send(context.Background(), "b@x.org", []string{"a@x.com"}, big)
var ne net.Error
if !errors.As(err, &ne) || !ne.Timeout() || IsPermanent(err) {
t.Fatalf("want a temporary timeout, got %v", err)
}
if took := time.Since(start); took > 5*time.Second {
t.Fatalf("the upload has its own budget, took %v", took)
}
}

View file

@ -20,7 +20,6 @@ const MaxAge = 72 * time.Hour
const ( const (
defaultConcurrency = 4 defaultConcurrency = 4
defaultBatch = 50 defaultBatch = 50
defaultTimeout = time.Minute
defaultInterval = 5 * time.Second defaultInterval = 5 * time.Second
// purgeEvery is the least time between two scans that forget old failed deliveries. // purgeEvery is the least time between two scans that forget old failed deliveries.
@ -55,7 +54,7 @@ type Worker struct {
Sender outbound.Sender Sender outbound.Sender
Concurrency int // simultaneous sends Concurrency int // simultaneous sends
Batch int // deliveries taken per run Batch int // deliveries taken per run
Timeout time.Duration // per send Timeout time.Duration // per send; zero leaves the deadlines to Sender (SMTPRelay bounds each SMTP phase)
Interval time.Duration // idle polling, to pick up retries that have become due Interval time.Duration // idle polling, to pick up retries that have become due
// Sign, when set, signs the final message (DKIM). On error the message is sent unsigned: // Sign, when set, signs the final message (DKIM). On error the message is sent unsigned:
// not delivering is worse than a missed signature. // not delivering is worse than a missed signature.
@ -245,15 +244,16 @@ func (w *Worker) deliver(ctx context.Context, cache *bodies, it store.OutboxItem
} }
} }
timeout := w.Timeout
if timeout <= 0 {
timeout = defaultTimeout
}
// Wait for the slot before starting the timeout, so waiting does not eat into the send. // Wait for the slot before starting the timeout, so waiting does not eat into the send.
if err := w.pace().Wait(ctx); err != nil { if err := w.pace().Wait(ctx); err != nil {
return Stats{} // shutting down: the delivery stays queued, unchanged return Stats{} // shutting down: the delivery stays queued, unchanged
} }
sctx, cancel := context.WithTimeout(ctx, timeout) // No total deadline by default: one that covers the whole transaction cut relays that take
// minutes to accept a message, which was then retried and delivered twice.
sctx, cancel := ctx, context.CancelFunc(func() {})
if w.Timeout > 0 {
sctx, cancel = context.WithTimeout(ctx, w.Timeout)
}
err = w.Sender.Send(sctx, it.From, []string{it.Rcpt}, msg) err = w.Sender.Send(sctx, it.From, []string{it.Rcpt}, msg)
cancel() cancel()

View file

@ -405,6 +405,24 @@ func TestSlowRelayTimesOutAndIsRetried(t *testing.T) {
} }
} }
func TestTheQueueLeavesTheSendDeadlinesToTheSender(t *testing.T) {
// One total deadline cut a relay that took over a minute to answer the final dot: the message
// was accepted, then retried and delivered again. The sender bounds each SMTP phase itself.
e := setup(t)
e.enqueue(t, 1)
var hasDeadline bool
e.snd.fn = func(ctx context.Context, _ string, _ []string) error {
_, hasDeadline = ctx.Deadline()
return nil
}
if st, _ := e.w.RunOnce(context.Background()); st.Sent != 1 {
t.Fatalf("sent: %+v", st)
}
if hasDeadline {
t.Fatal("the queue must not put a deadline on the whole send")
}
}
func waitFor(t *testing.T, what string, cond func() bool) { func waitFor(t *testing.T, what string, cond func() bool) {
t.Helper() t.Helper()
deadline := time.Now().Add(3 * time.Second) deadline := time.Now().Add(3 * time.Second)

View file

@ -717,3 +717,90 @@ func TestConfirmPendingIsSingleUseUnderConcurrency(t *testing.T) {
} }
} }
} }
func TestMigrationAddsTheOutboxPriorityAloneAfterAHalfDoneUpgrade(t *testing.T) {
// A crash between adding post_id and priority leaves outbox with the first and not the second.
path := t.TempDir() + "/old.db"
s, _ := Open(path)
if _, err := s.db.Exec(`DROP INDEX outbox_due_prio`); err != nil {
t.Fatal(err)
}
if _, err := s.db.Exec(`ALTER TABLE outbox DROP COLUMN priority`); err != nil {
t.Fatal(err)
}
s.Close()
s2, err := Open(path)
if err != nil {
t.Fatalf("migrate: %v", err)
}
defer s2.Close()
if ok, err := hasColumn(s2.db, "outbox", "priority"); err != nil || !ok {
t.Fatalf("priority is added back: %v %v", ok, err)
}
}
func TestConfirmationMailGlobalCapIsConfigurable(t *testing.T) {
s := newTestStore(t)
if s.PendingPerHour != 300 {
t.Fatalf("default PendingPerHour = %d, want 300", s.PendingPerHour)
}
s.PendingPerHour = 2
s.Logf = func(string, ...any) {}
now := time.Unix(1_000_000, 0)
s.Now = func() time.Time { return now }
l, _ := s.CreateList("dev", "example.org", "Dev")
for i := 0; i < 2; i++ {
if _, created, err := s.CreatePending(l.ID, fmt.Sprintf("u%d@x.com", i), "join"); err != nil || !created {
t.Fatalf("request %d: %v %v", i+1, created, err)
}
}
if _, created, err := s.CreatePending(l.ID, "u2@x.com", "join"); err != nil || created {
t.Fatalf("the third request in the hour: created %v err %v, want refused", created, err)
}
}
func TestGlobalCapRefusalsAreLoggedOncePerMinute(t *testing.T) {
s := newTestStore(t)
s.PendingPerHour = 1
var logged []string
s.Logf = func(format string, args ...any) { logged = append(logged, fmt.Sprintf(format, args...)) }
t0 := time.Unix(1_000_000, 0)
now := t0
s.Now = func() time.Time { return now }
l, _ := s.CreateList("dev", "example.org", "Dev")
s.CreatePending(l.ID, "first@x.com", "join")
s.CreatePending(l.ID, "first@x.com", "join") // a repeat is not the global cap
if len(logged) != 0 {
t.Fatalf("nothing refused by the global cap yet: %q", logged)
}
for i := 0; i < 5; i++ {
s.CreatePending(l.ID, fmt.Sprintf("u%d@x.com", i), "join")
}
if len(logged) != 1 || !strings.Contains(logged[0], "limits.pending_per_hour") || !strings.Contains(logged[0], "1") {
t.Fatalf("one warning naming the setting: %q", logged)
}
now = t0.Add(time.Minute - time.Second)
s.CreatePending(l.ID, "late@x.com", "join")
if len(logged) != 1 {
t.Fatalf("at most one line a minute: %q", logged)
}
now = t0.Add(time.Minute)
s.CreatePending(l.ID, "later@x.com", "join")
if len(logged) != 2 {
t.Fatalf("a minute later the next refusal is logged: %q", logged)
}
}
func TestPerAddressCapRefusalsAreNotLogged(t *testing.T) {
s := newTestStore(t)
var logged []string
s.Logf = func(format string, args ...any) { logged = append(logged, fmt.Sprintf(format, args...)) }
for i := 0; i <= maxPendingPerAddress; i++ {
l, _ := s.CreateList(fmt.Sprintf("l%d", i), "example.org", "L")
s.CreatePending(l.ID, "victim@x.com", "join")
}
if len(logged) != 0 {
t.Fatalf("the per-address cap is not the global one: %q", logged)
}
}

View file

@ -8,9 +8,11 @@ import (
"encoding/hex" "encoding/hex"
"errors" "errors"
"fmt" "fmt"
"log"
"mime" "mime"
"net/mail" "net/mail"
"strings" "strings"
"sync"
"time" "time"
"unicode" "unicode"
@ -70,7 +72,7 @@ const (
// Caps on confirmation mail, per hour. Anyone can name any address in a join request, so the // Caps on confirmation mail, per hour. Anyone can name any address in a join request, so the
// daemon must not mail one victim many times (maxPendingPerAddress, across all lists) or send // daemon must not mail one victim many times (maxPendingPerAddress, across all lists) or send
// an unbounded amount in total (maxPendingPerHour). // an unbounded amount in total (Store.PendingPerHour, maxPendingPerHour by default).
// A moderated list could be flooded with mail that nobody reads: the queue per list is bounded, // A moderated list could be flooded with mail that nobody reads: the queue per list is bounded,
// and a message nobody moderated for a month is dropped. // and a message nobody moderated for a month is dropped.
maxHeldPerList = 500 maxHeldPerList = 500
@ -78,6 +80,9 @@ const (
maxPendingPerAddress = 3 maxPendingPerAddress = 3
maxPendingPerHour = 300 maxPendingPerHour = 300
// pendingCapLogEvery spaces the warnings about requests refused by the global cap.
pendingCapLogEvery = time.Minute
) )
// List is a mailing list. // List is a mailing list.
@ -133,6 +138,14 @@ type Store struct {
db *sql.DB db *sql.DB
// Now returns the current time; tests replace it. // Now returns the current time; tests replace it.
Now func() time.Time Now func() time.Time
// PendingPerHour is the most join and leave requests recorded in an hour, for all lists and
// addresses together (limits.pending_per_hour).
PendingPerHour int
// Logf receives the warning written when PendingPerHour refuses a request; tests replace it.
Logf func(format string, args ...any)
capLogMu sync.Mutex
capLoggedAt time.Time
} }
const schema = ` const schema = `
@ -261,7 +274,7 @@ func Open(path string) (*Store, error) {
db.Close() db.Close()
return nil, fmt.Errorf("migrate: %w", err) return nil, fmt.Errorf("migrate: %w", err)
} }
return &Store{db: db, Now: time.Now}, nil return &Store{db: db, Now: time.Now, PendingPerHour: maxPendingPerHour, Logf: log.Printf}, nil
} }
func hasColumn(db *sql.DB, table, column string) (bool, error) { func hasColumn(db *sql.DB, table, column string) (bool, error) {
@ -325,8 +338,12 @@ func migrate(db *sql.DB) error {
} }
} }
if has, _ := hasColumn(db, "outbox", "post_id"); !has { if has, _ := hasColumn(db, "outbox", "post_id"); !has {
if _, err := db.Exec(`ALTER TABLE outbox ADD COLUMN post_id INTEGER NOT NULL DEFAULT 0; if _, err := db.Exec(`ALTER TABLE outbox ADD COLUMN post_id INTEGER NOT NULL DEFAULT 0`); err != nil {
ALTER TABLE outbox ADD COLUMN priority INTEGER NOT NULL DEFAULT 0;`); err != nil { return err
}
}
if has, _ := hasColumn(db, "outbox", "priority"); !has {
if _, err := db.Exec(`ALTER TABLE outbox ADD COLUMN priority INTEGER NOT NULL DEFAULT 0`); err != nil {
return err return err
} }
} }
@ -773,9 +790,13 @@ func (s *Store) CreatePending(listID int64, addr, action string) (token string,
if err = s.db.QueryRow(`SELECT COUNT(*) FROM pending WHERE address=? AND created_at>?`, addr, since).Scan(&n); err != nil || n >= maxPendingPerAddress { if err = s.db.QueryRow(`SELECT COUNT(*) FROM pending WHERE address=? AND created_at>?`, addr, since).Scan(&n); err != nil || n >= maxPendingPerAddress {
return "", false, err return "", false, err
} }
if err = s.db.QueryRow(`SELECT COUNT(*) FROM pending WHERE created_at>?`, since).Scan(&n); err != nil || n >= maxPendingPerHour { if err = s.db.QueryRow(`SELECT COUNT(*) FROM pending WHERE created_at>?`, since).Scan(&n); err != nil {
return "", false, err return "", false, err
} }
if n >= s.PendingPerHour {
s.warnPendingCap(now, n)
return "", false, nil
}
// 80 bits: unguessable within the 72 h life of a token, and short enough that // 80 bits: unguessable within the 72 h life of a token, and short enough that
// name(<=30)+"+confirm-"+token fits the 64 characters of a local part. // name(<=30)+"+confirm-"+token fits the 64 characters of a local part.
token, err = randomHex(10) token, err = randomHex(10)
@ -790,6 +811,18 @@ func (s *Store) CreatePending(listID int64, addr, action string) (token string,
return token, true, nil return token, true, nil
} }
// warnPendingCap tells the operator, at most once every pendingCapLogEvery, that requests are being
// dropped: the visitor still sees the neutral page and gets no mail.
func (s *Store) warnPendingCap(now time.Time, n int) {
s.capLogMu.Lock()
defer s.capLogMu.Unlock()
if now.Sub(s.capLoggedAt) < pendingCapLogEvery {
return
}
s.capLoggedAt = now
s.Logf("warning: %d join and leave requests in the last hour reached limits.pending_per_hour (%d): new requests get no confirmation mail until older ones are an hour old", n, s.PendingPerHour)
}
// ConfirmPending applies the request behind a token and consumes the token, in one transaction: a // ConfirmPending applies the request behind a token and consumes the token, in one transaction: a
// token is only used up when its action happened, and two confirmations of the same token cannot // token is only used up when its action happened, and two confirmations of the same token cannot
// both apply (the first one to claim the row wins, the other gets ErrNotFound). // both apply (the first one to claim the row wins, the other gets ErrNotFound).

View file

@ -3,17 +3,21 @@ package web
import ( import (
"net" "net"
"net/http" "net/http"
"strings"
"sync" "sync"
"time" "time"
) )
// A person subscribes once; a script posting the form for other people's addresses is what this // A person subscribes once; a script posting the form for other people's addresses is what this
// slows down: at most subscribeMax POSTs per subscribeWindow from one address. The window starts // slows down: at most subscribeMax POSTs per subscribeWindow from one client. A client is an IPv4
// at the first request of an address and everything about it is forgotten when it ends. // address or an IPv6 /64 (one machine usually holds a whole /64). The window starts at the first
// request of a client and everything about it is forgotten when it ends. At most maxClients are
// remembered: past that, new clients are refused until a window ends, rather than the map growing.
const ( const (
subscribeMax = 10 subscribeMax = 10
subscribeWindow = time.Hour subscribeWindow = time.Hour
sweepEvery = time.Minute sweepEvery = time.Minute
maxClients = 100000
) )
type visits struct { type visits struct {
@ -26,20 +30,27 @@ type subscribeLimiter struct {
now func() time.Time now func() time.Time
byIP map[string]*visits byIP map[string]*visits
lastSweep time.Time lastSweep time.Time
max int
} }
func newSubscribeLimiter(now func() time.Time) *subscribeLimiter { func newSubscribeLimiter(now func() time.Time) *subscribeLimiter {
return &subscribeLimiter{now: now, byIP: map[string]*visits{}, lastSweep: now()} return &subscribeLimiter{now: now, byIP: map[string]*visits{}, lastSweep: now(), max: maxClients}
} }
// allow counts a request from ip. When the address is over its limit it returns false and how long // allow counts a request from ip. When the address is over its limit, or it is new and the map is
// until it may try again; a refused request is not counted. // full, it returns false and how long until it may try again; a refused request is not counted.
func (l *subscribeLimiter) allow(ip string) (retryAfter time.Duration, ok bool) { func (l *subscribeLimiter) allow(ip string) (retryAfter time.Duration, ok bool) {
l.mu.Lock() l.mu.Lock()
defer l.mu.Unlock() defer l.mu.Unlock()
now := l.now() now := l.now()
l.sweep(now) l.sweep(now)
v := l.byIP[ip] v := l.byIP[ip]
if v == nil && len(l.byIP) >= l.max {
l.lastSweep = now.Add(-sweepEvery) // force it
if wait := l.sweep(now); len(l.byIP) >= l.max {
return wait, false
}
}
if v == nil || now.Sub(v.first) >= subscribeWindow { if v == nil || now.Sub(v.first) >= subscribeWindow {
v = &visits{first: now} v = &visits{first: now}
l.byIP[ip] = v l.byIP[ip] = v
@ -51,17 +62,22 @@ func (l *subscribeLimiter) allow(ip string) (retryAfter time.Duration, ok bool)
return 0, true return 0, true
} }
// sweep forgets addresses whose window is over, so the map cannot grow forever. // sweep forgets addresses whose window is over, at most once every sweepEvery. It returns how long
func (l *subscribeLimiter) sweep(now time.Time) { // until the next window of a kept address ends (zero when it did not run).
func (l *subscribeLimiter) sweep(now time.Time) (next time.Duration) {
if now.Sub(l.lastSweep) < sweepEvery { if now.Sub(l.lastSweep) < sweepEvery {
return return 0
} }
l.lastSweep = now l.lastSweep = now
next = subscribeWindow
for ip, v := range l.byIP { for ip, v := range l.byIP {
if now.Sub(v.first) >= subscribeWindow { if left := v.first.Add(subscribeWindow).Sub(now); left <= 0 {
delete(l.byIP, ip) delete(l.byIP, ip)
} else {
next = min(next, left)
} }
} }
return next
} }
func (l *subscribeLimiter) size() int { func (l *subscribeLimiter) size() int {
@ -70,12 +86,51 @@ func (l *subscribeLimiter) size() int {
return len(l.byIP) return len(l.byIP)
} }
// clientIP is the address of the TCP peer. X-Forwarded-For is deliberately ignored: anyone can // clientKey names the client of r for the limiter: an IPv4 address, or the /64 of an IPv6 one.
// send it, so trusting it would let a script pick a fresh identity for every request. //
func clientIP(r *http.Request) string { // The client is the TCP peer. X-Forwarded-For is read only when the peer is one of the trusted
// proxies: anyone can send the header, so trusting it from anybody would let a script pick a fresh
// identity for every request. Even then only the right-most entry that is not itself a trusted
// proxy is taken, since the entries on its left were written by the client. An empty or malformed
// header, or one listing only trusted proxies, leaves the peer.
func clientKey(r *http.Request, trusted []*net.IPNet) string {
host, _, err := net.SplitHostPort(r.RemoteAddr) host, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil { if err != nil {
return r.RemoteAddr host = r.RemoteAddr
} }
return host ip := net.ParseIP(host)
if ip == nil {
return host
}
if inAny(ip, trusted) {
ip = forwardedClient(r, trusted, ip)
}
if ip4 := ip.To4(); ip4 != nil {
return ip4.String()
}
return ip.Mask(net.CIDRMask(64, 128)).String() + "/64"
}
// forwardedClient walks X-Forwarded-For (all its lines, in order) from the right.
func forwardedClient(r *http.Request, trusted []*net.IPNet, peer net.IP) net.IP {
entries := strings.Split(strings.Join(r.Header.Values("X-Forwarded-For"), ","), ",")
for i := len(entries) - 1; i >= 0; i-- {
ip := net.ParseIP(strings.TrimSpace(entries[i]))
if ip == nil {
return peer
}
if !inAny(ip, trusted) {
return ip
}
}
return peer
}
func inAny(ip net.IP, nets []*net.IPNet) bool {
for _, n := range nets {
if n.Contains(ip) {
return true
}
}
return false
} }

View file

@ -7,6 +7,7 @@ import (
"errors" "errors"
"html/template" "html/template"
"log" "log"
"net"
"net/http" "net/http"
"net/mail" "net/mail"
"net/url" "net/url"
@ -31,14 +32,19 @@ type server struct {
router *router.Router router *router.Router
tmpl *template.Template tmpl *template.Template
limit *subscribeLimiter limit *subscribeLimiter
// trusted are the reverse proxies whose X-Forwarded-For is believed.
trusted []*net.IPNet
} }
// New returns the public web handler. // New returns the public web handler. trustedProxies are the reverse proxies allowed to name the
func New(s *store.Store, rt *router.Router) http.Handler { return newHandler(s, rt, time.Now) } // client in X-Forwarded-For (none: the TCP peer is the client).
func New(s *store.Store, rt *router.Router, trustedProxies []*net.IPNet) http.Handler {
return newHandler(s, rt, time.Now, trustedProxies)
}
// newHandler is New with an injectable clock, for the tests. // newHandler is New with an injectable clock, for the tests.
func newHandler(s *store.Store, rt *router.Router, now func() time.Time) http.Handler { func newHandler(s *store.Store, rt *router.Router, now func() time.Time, trusted []*net.IPNet) http.Handler {
srv := &server{store: s, router: rt, limit: newSubscribeLimiter(now), tmpl: template.Must(template.New("").Funcs(template.FuncMap{ srv := &server{store: s, router: rt, limit: newSubscribeLimiter(now), trusted: trusted, tmpl: template.Must(template.New("").Funcs(template.FuncMap{
"hide": archive.Obfuscate, "hide": archive.Obfuscate,
// The logo is a trusted static asset: inline, so the strict CSP needs no image source for it. // The logo is a trusted static asset: inline, so the strict CSP needs no image source for it.
"mark": func() template.HTML { return template.HTML(assets.Mark) }, "mark": func() template.HTML { return template.HTML(assets.Mark) },
@ -262,7 +268,7 @@ func (s *server) subscribeForm(w http.ResponseWriter, r *http.Request) {
const maxSubscribeBody = 4 << 10 const maxSubscribeBody = 4 << 10
func (s *server) subscribe(w http.ResponseWriter, r *http.Request) { func (s *server) subscribe(w http.ResponseWriter, r *http.Request) {
if wait, ok := s.limit.allow(clientIP(r)); !ok { if wait, ok := s.limit.allow(clientKey(r, s.trusted)); !ok {
w.Header().Set("Retry-After", strconv.Itoa(int((wait+time.Second-1)/time.Second))) w.Header().Set("Retry-After", strconv.Itoa(int((wait+time.Second-1)/time.Second)))
s.render(w, http.StatusTooManyRequests, "toomany", nil) s.render(w, http.StatusTooManyRequests, "toomany", nil)
return return
@ -313,7 +319,7 @@ func printable(s string) string {
} }
// plainAddress accepts one bare ASCII address of at most 254 characters: no display name, no angle // plainAddress accepts one bare ASCII address of at most 254 characters: no display name, no angle
// brackets, no list, no quoting, no control characters. // brackets, no list, no quoting, no control characters, and a public domain (see publicDomain).
func plainAddress(s string) (string, bool) { func plainAddress(s string) (string, bool) {
s = strings.TrimSpace(s) s = strings.TrimSpace(s)
if s == "" || len(s) > 254 { if s == "" || len(s) > 254 {
@ -325,12 +331,40 @@ func plainAddress(s string) (string, bool) {
} }
} }
a, err := mail.ParseAddress(s) a, err := mail.ParseAddress(s)
if err != nil || a.Name != "" || a.Address != s { if err != nil || a.Name != "" || a.Address != s || !publicDomain(s[strings.LastIndexByte(s, '@')+1:]) {
return "", false return "", false
} }
return s, true return s, true
} }
// publicDomain accepts a domain name that can be on the Internet: dotted labels of letters, digits
// and hyphens, not an IP address, and not one of the names reserved for local use. A domain literal
// ([127.0.0.1]), a bare host (localhost) or a .local name would make the daemon mail itself or its
// network.
func publicDomain(d string) bool {
d = strings.ToLower(d)
if !strings.Contains(d, ".") {
return false
}
for _, label := range strings.Split(d, ".") {
if label == "" || strings.ContainsFunc(label, func(r rune) bool {
return !(r >= 'a' && r <= 'z' || r >= '0' && r <= '9' || r == '-')
}) {
return false
}
}
// No top-level domain is all digits: such a name is an IP address (a@127.0.0.1).
if strings.Trim(d[strings.LastIndexByte(d, '.')+1:], "0123456789") == "" {
return false
}
for _, local := range []string{".localhost", ".local", ".internal", ".invalid"} {
if strings.HasSuffix(d, local) {
return false
}
}
return true
}
func (s *server) unsubscribePage(w http.ResponseWriter, r *http.Request) { func (s *server) unsubscribePage(w http.ResponseWriter, r *http.Request) {
l, err := s.router.CheckUnsubscribe(r.PathValue("token")) l, err := s.router.CheckUnsubscribe(r.PathValue("token"))
switch { switch {

View file

@ -5,6 +5,7 @@ import (
"encoding/base64" "encoding/base64"
"fmt" "fmt"
"html" "html"
"net"
"net/http" "net/http"
"net/http/httptest" "net/http/httptest"
"net/url" "net/url"
@ -53,10 +54,24 @@ func newFixture(t *testing.T) *fixture {
rt := router.New(s) rt := router.New(s)
f := &fixture{s: s, rt: rt, pub: pub, alice: ms[0].ID, bobTok: rt.UnsubscribeToken(ms[1].ID), f := &fixture{s: s, rt: rt, pub: pub, alice: ms[0].ID, bobTok: rt.UnsubscribeToken(ms[1].ID),
out: &outbound.Fake{}, clock: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)} out: &outbound.Fake{}, clock: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC)}
f.h = newHandler(s, rt, func() time.Time { return f.clock }) f.h = newHandler(s, rt, func() time.Time { return f.clock }, nil)
return f return f
} }
// trust rebuilds the handler so that it believes X-Forwarded-For from the given proxies.
func (f *fixture) trust(t *testing.T, cidrs ...string) {
t.Helper()
var nets []*net.IPNet
for _, c := range cidrs {
_, n, err := net.ParseCIDR(c)
if err != nil {
t.Fatal(err)
}
nets = append(nets, n)
}
f.h = newHandler(f.s, f.rt, func() time.Time { return f.clock }, nets)
}
// flush sends what is queued, as the background worker would, and returns everything sent so far. // flush sends what is queued, as the background worker would, and returns everything sent so far.
func (f *fixture) flush(t *testing.T) []outbound.Sent { func (f *fixture) flush(t *testing.T) []outbound.Sent {
t.Helper() t.Helper()
@ -138,7 +153,7 @@ func TestIndexEscapesTitles(t *testing.T) {
func TestIndexWithoutPublicLists(t *testing.T) { func TestIndexWithoutPublicLists(t *testing.T) {
s, _ := store.Open(":memory:") s, _ := store.Open(":memory:")
defer s.Close() defer s.Close()
rec := get(t, New(s, router.New(s)), "/") rec := get(t, New(s, router.New(s), nil), "/")
wantStatus(t, rec, 200) wantStatus(t, rec, 200)
wantBody(t, rec, "No public archives") wantBody(t, rec, "No public archives")
} }
@ -367,7 +382,7 @@ func subscribeRaw(h http.Handler, list, ip, body string, hdr map[string]string)
for k, v := range hdr { for k, v := range hdr {
req.Header.Set(k, v) req.Header.Set(k, v)
} }
req.RemoteAddr = ip + ":4321" req.RemoteAddr = net.JoinHostPort(ip, "4321")
rec := httptest.NewRecorder() rec := httptest.NewRecorder()
h.ServeHTTP(rec, req) h.ServeHTTP(rec, req)
return rec return rec
@ -517,6 +532,20 @@ func TestSubscribeRejectsBadAddressesAndEscapesWhatWasTyped(t *testing.T) {
{"line break", "a@x.com\r\nBcc: v@x.com"}, {"line break", "a@x.com\r\nBcc: v@x.com"},
{"non-ASCII", "cärol@x.com"}, {"non-ASCII", "cärol@x.com"},
{"too long", long}, {"too long", long},
{"no dot in the domain", "a@b"},
{"localhost", "root@localhost"},
{"localhost in capitals", "root@LOCALHOST"},
{"under localhost", "a@app.localhost"},
{"mDNS", "a@printer.local"},
{"internal", "a@svc.internal"},
{"invalid", "a@x.invalid"},
{"domain literal", "a@[127.0.0.1]"},
{"IPv6 domain literal", "a@[IPv6:::1]"},
{"trailing dot", "a@x.com."},
{"underscore in the domain", "a@x_y.com"},
{"symbol in the domain", "a@x!y.com"},
{"plus in the domain", "a@x+y.com"},
{"bare IP address", "a@127.0.0.1"},
} { } {
rec := subscribe(t, f, "dev@example.org", fmt.Sprintf("192.0.2.%d", i+1), email(c.in)) // one IP each: the limit is per IP rec := subscribe(t, f, "dev@example.org", fmt.Sprintf("192.0.2.%d", i+1), email(c.in)) // one IP each: the limit is per IP
if rec.Code != 400 { if rec.Code != 400 {
@ -540,6 +569,15 @@ func TestSubscribeRejectsBadAddressesAndEscapesWhatWasTyped(t *testing.T) {
wantBody(t, subscribe(t, f, "dev@example.org", "192.0.2.101", email("a"+ok)), "valid email address") wantBody(t, subscribe(t, f, "dev@example.org", "192.0.2.101", email("a"+ok)), "valid email address")
} }
func TestSubscribeAcceptsOrdinaryDomains(t *testing.T) {
f := newFixture(t)
for i, addr := range []string{"a@members.test", "a@example.com", "a@mail.example", "a@my-host.Example.ORG", "a@x1.co", "a@localhost.example.com"} {
rec := subscribe(t, f, "dev@example.org", fmt.Sprintf("192.0.2.%d", i+1), email(addr))
wantStatus(t, rec, 200)
wantBody(t, rec, neutralText)
}
}
func TestSubscribeBodyIsLimitedTo4KiB(t *testing.T) { func TestSubscribeBodyIsLimitedTo4KiB(t *testing.T) {
f := newFixture(t) f := newFixture(t)
body := func(n int) string { body := func(n int) string {
@ -714,3 +752,141 @@ func TestSubscribeLimiterSweepsAnAddressWhoseHourIsOver(t *testing.T) {
t.Fatalf("size = %d, want 1 (the first address is forgotten)", l.size()) t.Fatalf("size = %d, want 1 (the first address is forgotten)", l.size())
} }
} }
func TestSubscribeBehindATrustedProxyLimitsEachClient(t *testing.T) {
f := newFixture(t)
f.trust(t, "10.0.0.0/8")
post := func(xff string) int {
return subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), map[string]string{"X-Forwarded-For": xff}).Code
}
for i := 0; i < 10; i++ {
if c := post("203.0.113.1"); c != 200 {
t.Fatalf("request %d: %d", i+1, c)
}
}
if c := post("203.0.113.1"); c != 429 {
t.Fatalf("the 11th request of one client: %d, want 429", c)
}
// Every visitor comes through the same proxy, yet another one has a budget of their own.
if c := post("203.0.113.2"); c != 200 {
t.Fatalf("another client behind the proxy: %d, want 200", c)
}
}
func TestSubscribeForwardedForFromAnUntrustedPeerIsIgnored(t *testing.T) {
f := newFixture(t)
f.trust(t, "10.0.0.0/8")
for i := 0; i < 10; i++ {
hdr := map[string]string{"X-Forwarded-For": fmt.Sprintf("203.0.113.%d", i)}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "198.51.100.7", email("a@x.com").Encode(), hdr), 200)
}
hdr := map[string]string{"X-Forwarded-For": "203.0.113.200"}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "198.51.100.7", email("a@x.com").Encode(), hdr), 429)
}
func TestSubscribeTakesTheRightMostUntrustedForwardedAddress(t *testing.T) {
f := newFixture(t)
f.trust(t, "10.0.0.0/8", "192.168.0.0/16")
// The client may write anything on the left; the proxies append on the right.
for i := 0; i < 10; i++ {
hdr := map[string]string{"X-Forwarded-For": fmt.Sprintf("198.51.100.%d, 203.0.113.5, 192.168.1.1", i)}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), hdr), 200)
}
hdr := map[string]string{"X-Forwarded-For": "1.1.1.1,203.0.113.5"}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), hdr), 429)
// The header may arrive as several lines: they are one list, in order.
req := httptest.NewRequest("POST", "/subscribe/dev@example.org", strings.NewReader(email("a@x.com").Encode()))
req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
req.Header.Add("X-Forwarded-For", "198.51.100.77")
req.Header.Add("X-Forwarded-For", "203.0.113.5")
req.RemoteAddr = "10.0.0.2:4321"
rec := httptest.NewRecorder()
f.h.ServeHTTP(rec, req)
wantStatus(t, rec, 429)
// The proxy's own address does not count against the client.
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), map[string]string{"X-Forwarded-For": "203.0.113.6"}), 200)
}
func TestSubscribeBadForwardedForFallsBackToThePeer(t *testing.T) {
f := newFixture(t)
f.trust(t, "10.0.0.0/8")
for i, xff := range []string{"", "garbage", "203.0.113.1, nonsense", "203.0.113.1:5555", "10.0.0.7", " , ", "", "x", "", ""} {
hdr := map[string]string{}
if xff != "" {
hdr["X-Forwarded-For"] = xff
}
if c := subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), hdr).Code; c != 200 {
t.Fatalf("request %d (%q): %d", i+1, xff, c)
}
}
// All ten were counted against the proxy itself.
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), nil), 429)
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), map[string]string{"X-Forwarded-For": "203.0.113.1"}), 200)
}
func TestSubscribeIPv6ClientsShareTheirSlash64(t *testing.T) {
f := newFixture(t)
for i := 0; i < 10; i++ {
wantStatus(t, subscribe(t, f, "dev@example.org", fmt.Sprintf("2001:db8:1:2::%x", i+1), email("a@x.com")), 200)
}
wantStatus(t, subscribe(t, f, "dev@example.org", "2001:db8:1:2:ffff:ffff:ffff:ffff", email("a@x.com")), 429)
wantStatus(t, subscribe(t, f, "dev@example.org", "2001:db8:1:3::1", email("a@x.com")), 200)
// IPv4 addresses stay separate, one each.
for i := 0; i < 10; i++ {
wantStatus(t, subscribe(t, f, "dev@example.org", "192.0.2.1", email("a@x.com")), 200)
}
wantStatus(t, subscribe(t, f, "dev@example.org", "192.0.2.2", email("a@x.com")), 200)
// The same goes for an IPv6 client named by a trusted proxy.
f.trust(t, "10.0.0.0/8")
for i := 0; i < 10; i++ {
hdr := map[string]string{"X-Forwarded-For": fmt.Sprintf("2001:db8:9:9::%x", i+1)}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), hdr), 200)
}
hdr := map[string]string{"X-Forwarded-For": "2001:db8:9:9:1::1"}
wantStatus(t, subscribeRaw(f.h, "dev@example.org", "10.0.0.2", email("a@x.com").Encode(), hdr), 429)
}
func TestSubscribeLimiterIsCapped(t *testing.T) {
t0 := time.Date(2026, 1, 1, 0, 0, 0, 0, time.UTC)
now := t0
l := newSubscribeLimiter(func() time.Time { return now })
if l.max != 100000 {
t.Fatalf("max = %d, want 100000", l.max)
}
l.max = 2
l.allow("192.0.2.1")
now = t0.Add(59*time.Minute + 50*time.Second)
l.allow("192.0.2.2") // the periodic sweep runs here and finds nothing to forget
if wait, ok := l.allow("192.0.2.3"); ok || wait != 10*time.Second {
t.Fatalf("a new address when the map is full: wait %v ok %v, want refused for 10s (until the oldest window ends)", wait, ok)
}
if _, ok := l.allow("192.0.2.2"); !ok {
t.Fatal("an address already known keeps its budget")
}
if l.size() != 2 {
t.Fatalf("size = %d, want 2", l.size())
}
// 20 s later the periodic sweep is not due yet, but the first address has expired: a full map
// sweeps before it refuses.
now = t0.Add(time.Hour + 10*time.Second)
if _, ok := l.allow("192.0.2.3"); !ok {
t.Fatal("room was made by forgetting an expired address")
}
if l.size() != 2 {
t.Fatalf("size = %d, want 2", l.size())
}
}
// mail.ParseAddress already refuses most of these; publicDomain must not depend on it.
func TestPublicDomain(t *testing.T) {
for d, want := range map[string]bool{
"example.com": true, "members.test": true, "a-b.example": true, "X1.Example.ORG": true,
"x..com": false, ".x.com": false, "x.com.": false, "x": false, "localhost": false, "[127.0.0.1]": false,
"a.localhost": false, "a.local": false, "a.internal": false, "a.invalid": false, "x_y.com": false,
"127.0.0.1": false, "10.1.2.3": false, "x.123": false, "123.example": true, "x.c0m": true,
} {
if got := publicDomain(d); got != want {
t.Errorf("publicDomain(%q) = %v, want %v", d, got, want)
}
}
}

View file

@ -15,8 +15,17 @@ relay:
# rate_per_second: 14 # at most this many sends started per second, for relays that throttle (Amazon SES); 0 = unlimited # rate_per_second: 14 # at most this many sends started per second, for relays that throttle (Amazon SES); 0 = unlimited
web: web:
listen: 0.0.0.0:8081 # public archive and unsubscribe pages; "" disables them listen: 0.0.0.0:8081 # public archive and unsubscribe pages; "" disables them
# Reverse proxies (addresses or CIDR ranges) allowed to name the visitor in X-Forwarded-For, for
# the subscribe form's limit of 10 posts per hour per IPv4 address or IPv6 /64. Empty (the
# default): the TCP peer is the visitor and the header is ignored.
# trusted_proxies: [172.16.0.0/12]
api: api:
listen: 0.0.0.0:8080 listen: 0.0.0.0:8080
# Each subscribe or unsubscribe request sends a confirmation mail. At most this many are accepted
# per hour, for all lists together (default 300); past it, requests get no mail and a warning is
# logged once a minute. Raise it before a newsletter launch.
# limits:
# pending_per_hour: 300
# The API token is read from the PARSIMONY_API_TOKEN environment variable. # The API token is read from the PARSIMONY_API_TOKEN environment variable.
# Sign outgoing mail with DKIM, one key per domain. The private key stays in a file # Sign outgoing mail with DKIM, one key per domain. The private key stays in a file
# (create it with: parsimony dkim keygen lists.example.org --selector mail). # (create it with: parsimony dkim keygen lists.example.org --selector mail).