feat(providers): provider settings in a table and a careful outbound HTTP client
- feat(storage): provider_settings table (provider, enabled, api_key, contact);
migration 0007 for both drivers. A provider with no row, or turned off, is
never contacted
- feat(providers): a service that validates and applies settings: a Last.fm key
is 32 hexadecimal characters, enabling a provider that needs a key without
one is refused, clearing the key turns the provider off, a contact cannot
contain parentheses or control characters. Each provider states what it sends
and under which terms, and what it never sends
- feat(api): admin-only GET /providers and PUT /providers/{name}. The API key is
write-only: responses carry only has_api_key, errors never repeat it and the
request log never contains it (tested)
- feat(providers/outbound): the only way Sonda talks to the outside: plain GET
with an identifiable User-Agent, no cookies, no Referer, no Authorization, no
redirects, bounded answers, a timeout per attempt, a minimum interval between
requests, retries with exponential backoff that honour Retry-After without
waiting past the caller's deadline or for an absurd time, and a circuit
breaker that makes a failing provider fail fast
- feat(providers): UserAgent builds "Sonda/version ( contact )" as MusicBrainz
asks, defaulting to the project address so no personal address is sent
- breaking(config): providers.lastfm and SONDA_LASTFM_API_KEY are gone; an old
config, or the variable, fails with a hint instead of being ignored
- docs: OpenAPI operations and schemas, README section on the providers and on
what leaves the server
- test: fake servers and a fake clock only; no network in the suite
This commit is contained in:
parent
d8ec5f6ab2
commit
9f4da2e77e
20 changed files with 2032 additions and 31 deletions
29
README.md
29
README.md
|
|
@ -14,11 +14,11 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Catalog.** Browse and search artists, albums and tracks, with paging, by library, artist or album. Only tracks you can actually play are listed.
|
||||
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
|
||||
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database; nothing is sent anywhere. Sonda makes no outbound network requests at all.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere. Today Sonda makes no outbound network requests at all.
|
||||
|
||||
## What is not there yet
|
||||
|
||||
Similar-artist discovery, recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and Last.fm/MusicBrainz/ListenBrainz integrations. The `[providers.lastfm]` section of the config is reserved for a later milestone and is not used yet.
|
||||
Similar-artist discovery, recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Last.fm and MusicBrainz can already be set up (see [External providers](#external-providers)), but nothing asks them anything yet.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -58,9 +58,8 @@ Precedence is: built-in defaults, then the file, then environment variables. An
|
|||
| `auth.access_ttl_minutes` | | `15` | Lifetime of an access token. |
|
||||
| `auth.refresh_ttl_hours` | | `720` | Lifetime of a refresh token (30 days). |
|
||||
| `listening.min_listen_seconds` | | `20` | A play counts as a listen after this long (1 to 3600), or when it is `complete`. |
|
||||
| `providers.lastfm.api_key` | `SONDA_LASTFM_API_KEY` | | Reserved, not used yet. |
|
||||
|
||||
Libraries are not in the config: you add them through the API (see the demo).
|
||||
Libraries and the external providers are not in the config: you set them up through the API (see the demo and [External providers](#external-providers)).
|
||||
|
||||
### Using PostgreSQL
|
||||
|
||||
|
|
@ -174,6 +173,25 @@ AUTH="Authorization: Bearer $TOKEN"
|
|||
curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another long password"}' | jq
|
||||
```
|
||||
|
||||
## External providers
|
||||
|
||||
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today a provider can be set up but nothing uses it yet.
|
||||
|
||||
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
|
||||
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
|
||||
|
||||
Set them up with `PUT /api/v1/providers/{name}` (admin), for example:
|
||||
|
||||
```bash
|
||||
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"api_key":"<your 32 character key>"}'
|
||||
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"enabled":true}'
|
||||
curl -s $BASE/providers -H "$AUTH" | jq
|
||||
```
|
||||
|
||||
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
|
||||
|
||||
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down.
|
||||
|
||||
## The API at a glance
|
||||
|
||||
Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"error":{"code":"...","message":"..."}}`.
|
||||
|
|
@ -193,6 +211,7 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
|
|||
| signed in | `GET /library/scan` | Scan status and the last result. |
|
||||
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}` | See and set up Last.fm and MusicBrainz. The API key is write-only. |
|
||||
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
|
||||
|
||||
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.
|
||||
|
|
@ -203,6 +222,7 @@ A library path must be an absolute, existing folder, never the filesystem root,
|
|||
- Passwords are hashed with Argon2id. Refresh tokens are stored only as hashes.
|
||||
- Music folders are read-only to Sonda. Streaming opens files through `os.Root`, so a path can never leave its library, even if a symlink is swapped in later.
|
||||
- File paths are never returned by the catalog, and audio file paths are not logged.
|
||||
- Provider API keys are write-only through the API and never logged. They sit in the database, so protect your backups.
|
||||
|
||||
## Back up
|
||||
|
||||
|
|
@ -238,6 +258,7 @@ internal/api/openapi/ the OpenAPI spec, embedded in the binary
|
|||
internal/auth/ users, roles, passwords, access and refresh tokens
|
||||
internal/catalog/ browsing and searching artists, albums, tracks
|
||||
internal/library/ libraries, the scanner, the event hub, the tag reader
|
||||
internal/providers/ external providers: settings, and the careful outbound HTTP client
|
||||
internal/listening/ listen events
|
||||
internal/storage/ SQLite and PostgreSQL through bun, the repositories
|
||||
internal/streaming/ opening a track's file safely
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import (
|
|||
"codeberg.org/squidd/sonda/internal/config"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
"codeberg.org/squidd/sonda/migrations"
|
||||
|
|
@ -105,6 +106,7 @@ func run() error {
|
|||
Scan: scanControl{ctx: ctx, m: scans},
|
||||
Libraries: libraries,
|
||||
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
|
||||
Providers: providers.NewService(storage.NewProviderRepo(db)),
|
||||
Streams: streaming.NewService(libraryRepo),
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ import (
|
|||
"codeberg.org/squidd/sonda/internal/auth"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
"codeberg.org/squidd/sonda/migrations"
|
||||
|
|
@ -80,6 +81,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
Listening: listening.NewService(storage.NewListeningRepo(db), 20*time.Second),
|
||||
Scan: &fakeScan{},
|
||||
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
|
||||
Providers: providers.NewService(storage.NewProviderRepo(db)),
|
||||
Streams: streaming.NewService(noFiles{}),
|
||||
}
|
||||
mutate(&deps, db)
|
||||
|
|
|
|||
|
|
@ -277,6 +277,7 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("POST", "/users", token, map[string]any{"username": "x", "password": "bobs long pass"}), 400, "create invalid", "invalid_input")
|
||||
bobToken := c.expect(c.call("POST", "/auth/login", "", map[string]string{"username": "bob", "password": "bobs long pass"}), 200, "bob logs in")["access_token"].(string)
|
||||
c.expect(c.call("GET", "/users", bobToken, nil), 403, "users as non-admin", "admin_required")
|
||||
c.expect(c.call("GET", "/providers", bobToken, nil), 403, "providers as non-admin", "admin_required")
|
||||
c.expect(c.call("GET", "/users", token, nil), 200, "list users")
|
||||
bobID := idOf(bob)
|
||||
c.expect(c.call("PATCH", "/users/"+bobID, token, map[string]bool{"is_admin": true}), 200, "promote")
|
||||
|
|
@ -371,6 +372,23 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
t.Fatalf("track.album = %#v, want an explicit null", v)
|
||||
}
|
||||
|
||||
// ---- providers ----
|
||||
listed := c.expect(c.call("GET", "/providers", token, nil), 200, "list providers")["providers"].([]any)
|
||||
if len(listed) != 2 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false {
|
||||
t.Fatalf("providers = %v, want two, both off", listed)
|
||||
}
|
||||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/spotify", token, map[string]any{"enabled": true}), 404, "unknown provider", "provider_not_found")
|
||||
saved := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "0123456789abcdef0123456789abcdef"}), 200, "save a key")
|
||||
if saved["has_api_key"] != true || saved["updated_at"] == nil {
|
||||
t.Fatalf("saved = %v", saved)
|
||||
}
|
||||
if on := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 200, "enable"); on["enabled"] != true {
|
||||
t.Fatalf("enabled = %v", on)
|
||||
}
|
||||
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
|
||||
|
||||
// ---- playback ----
|
||||
stream := "/tracks/" + trackID + "/stream"
|
||||
c.expect(c.call("GET", stream, "", nil), 401, "stream without token", "unauthorized")
|
||||
|
|
|
|||
81
internal/api/http/handlers_providers.go
Normal file
81
internal/api/http/handlers_providers.go
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
)
|
||||
|
||||
// providersHandlers are the admin-only routes that set up the external
|
||||
// providers. The API key can be written here but is never returned.
|
||||
type providersHandlers struct{ svc providerService }
|
||||
|
||||
type providerJSON struct {
|
||||
Name string `json:"name"`
|
||||
Enabled bool `json:"enabled"`
|
||||
NeedsAPIKey bool `json:"needs_api_key"`
|
||||
HasAPIKey bool `json:"has_api_key"`
|
||||
Contact string `json:"contact"`
|
||||
Sends string `json:"sends"`
|
||||
Notes string `json:"notes"`
|
||||
UpdatedAt *string `json:"updated_at"`
|
||||
}
|
||||
|
||||
func toProviderJSON(in providers.Info) providerJSON {
|
||||
out := providerJSON{
|
||||
Name: in.Name, Enabled: in.Enabled, NeedsAPIKey: in.NeedsAPIKey, HasAPIKey: in.HasAPIKey,
|
||||
Contact: in.Contact, Sends: in.Sends, Notes: in.Notes,
|
||||
}
|
||||
if !in.UpdatedAt.IsZero() {
|
||||
out.UpdatedAt = rfc3339(in.UpdatedAt)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func writeProviderError(c *gin.Context, err error) {
|
||||
switch {
|
||||
case errors.Is(err, providers.ErrUnknownProvider):
|
||||
writeError(c, http.StatusNotFound, "provider_not_found", "no such provider")
|
||||
case errors.Is(err, providers.ErrInvalidSettings):
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
|
||||
default:
|
||||
_ = c.Error(err)
|
||||
writeError(c, http.StatusInternalServerError, "internal", "internal error")
|
||||
}
|
||||
}
|
||||
|
||||
func (h *providersHandlers) list(c *gin.Context) {
|
||||
infos, err := h.svc.List(c.Request.Context())
|
||||
if err != nil {
|
||||
writeProviderError(c, err)
|
||||
return
|
||||
}
|
||||
out := make([]providerJSON, len(infos))
|
||||
for i, in := range infos {
|
||||
out[i] = toProviderJSON(in)
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"providers": out})
|
||||
}
|
||||
|
||||
type updateProviderRequest struct {
|
||||
Enabled *bool `json:"enabled"` // pointers: an absent field is left alone
|
||||
APIKey *string `json:"api_key"`
|
||||
Contact *string `json:"contact"`
|
||||
}
|
||||
|
||||
func (h *providersHandlers) update(c *gin.Context) {
|
||||
var req updateProviderRequest
|
||||
if !bindJSON(c, &req) {
|
||||
return
|
||||
}
|
||||
info, err := h.svc.Update(c.Request.Context(), c.Param("name"),
|
||||
providers.Patch{Enabled: req.Enabled, APIKey: req.APIKey, Contact: req.Contact})
|
||||
if err != nil {
|
||||
writeProviderError(c, err)
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, toProviderJSON(info))
|
||||
}
|
||||
171
internal/api/http/providers_test.go
Normal file
171
internal/api/http/providers_test.go
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest/observer"
|
||||
)
|
||||
|
||||
const lastfmKey = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
func TestProviderRoutesAreAdminOnly(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
_, bob := addUser(t, h, admin, "bob", false)
|
||||
|
||||
for _, c := range []struct{ method, path string }{
|
||||
{"GET", "/api/v1/providers"},
|
||||
{"PUT", "/api/v1/providers/lastfm"},
|
||||
} {
|
||||
if r := do(t, h, c.method, c.path, "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("%s %s without a token: %d", c.method, c.path, r.Code)
|
||||
}
|
||||
r := do(t, h, c.method, c.path, bob, map[string]any{"enabled": true})
|
||||
if r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
|
||||
t.Errorf("%s %s as a normal user: %d %v", c.method, c.path, r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProvidersStartOffAndSayWhatTheySend(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
|
||||
r := do(t, h, "GET", "/api/v1/providers", admin, nil)
|
||||
if r.Code != http.StatusOK {
|
||||
t.Fatalf("list: %d %v", r.Code, r.Body)
|
||||
}
|
||||
list := r.Body["providers"].([]any)
|
||||
if len(list) != 2 {
|
||||
t.Fatalf("providers = %v", list)
|
||||
}
|
||||
for _, it := range list {
|
||||
p := it.(map[string]any)
|
||||
sameKeys(t, "provider", p, "name", "enabled", "needs_api_key", "has_api_key", "contact", "sends", "notes", "updated_at")
|
||||
if p["enabled"] != false || p["has_api_key"] != false {
|
||||
t.Errorf("%v: a provider must start off and without a key", p["name"])
|
||||
}
|
||||
if v, present := p["updated_at"]; !present || v != nil {
|
||||
t.Errorf("%v: updated_at = %#v, want an explicit null before any change", p["name"], v)
|
||||
}
|
||||
if !strings.Contains(p["sends"].(string), "Never") {
|
||||
t.Errorf("%v: sends = %q", p["name"], p["sends"])
|
||||
}
|
||||
}
|
||||
if list[0].(map[string]any)["name"] != "lastfm" || list[0].(map[string]any)["needs_api_key"] != true ||
|
||||
list[1].(map[string]any)["name"] != "musicbrainz" || list[1].(map[string]any)["needs_api_key"] != false {
|
||||
t.Errorf("order or needs_api_key wrong: %v", list)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingAProviderUp(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
|
||||
// Enabling before there is a key is refused.
|
||||
r := do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"enabled": true})
|
||||
if r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
|
||||
t.Fatalf("enable without a key: %d %v", r.Code, r.Body)
|
||||
}
|
||||
|
||||
// The key is written, not read back.
|
||||
r = do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": lastfmKey})
|
||||
if r.Code != http.StatusOK || r.Body["has_api_key"] != true || r.Body["enabled"] != false {
|
||||
t.Fatalf("save the key: %d %v", r.Code, r.Body)
|
||||
}
|
||||
if r.Body["updated_at"] == nil {
|
||||
t.Error("updated_at should be set after a change")
|
||||
}
|
||||
raw, _ := json.Marshal(r.Body)
|
||||
if strings.Contains(string(raw), lastfmKey) {
|
||||
t.Fatal("the response contains the API key")
|
||||
}
|
||||
|
||||
r = do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"enabled": true})
|
||||
if r.Code != http.StatusOK || r.Body["enabled"] != true {
|
||||
t.Fatalf("enable: %d %v", r.Code, r.Body)
|
||||
}
|
||||
|
||||
listed := do(t, h, "GET", "/api/v1/providers", admin, nil)
|
||||
rawList, _ := json.Marshal(listed.Body)
|
||||
if strings.Contains(string(rawList), lastfmKey) {
|
||||
t.Fatal("the list contains the API key")
|
||||
}
|
||||
first := listed.Body["providers"].([]any)[0].(map[string]any)
|
||||
if first["enabled"] != true || first["has_api_key"] != true {
|
||||
t.Errorf("listed = %v", first)
|
||||
}
|
||||
|
||||
// Clearing the key turns the provider off.
|
||||
r = do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": ""})
|
||||
if r.Code != http.StatusOK || r.Body["has_api_key"] != false || r.Body["enabled"] != false {
|
||||
t.Fatalf("clear the key: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMusicBrainzTakesAContactAndNoKey(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
|
||||
r := do(t, h, "PUT", "/api/v1/providers/musicbrainz", admin, map[string]any{"enabled": true, "contact": "https://example.org/me"})
|
||||
if r.Code != http.StatusOK || r.Body["enabled"] != true || r.Body["contact"] != "https://example.org/me" {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
r = do(t, h, "PUT", "/api/v1/providers/musicbrainz", admin, map[string]any{"api_key": lastfmKey})
|
||||
if r.Code != http.StatusBadRequest {
|
||||
t.Errorf("a key for a provider without keys: %d", r.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderUpdateRejectsBadInputWithoutEchoingTheKey(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
badKey := "not-a-valid-key-but-secret-looking-12345"
|
||||
|
||||
for name, tt := range map[string]struct {
|
||||
path string
|
||||
body any
|
||||
code int
|
||||
err string
|
||||
}{
|
||||
"empty body": {"/api/v1/providers/lastfm", map[string]any{}, 400, "invalid_input"},
|
||||
"malformed json": {"/api/v1/providers/lastfm", "{oops", 400, "invalid_input"},
|
||||
"bad key format": {"/api/v1/providers/lastfm", map[string]any{"api_key": badKey}, 400, "invalid_input"},
|
||||
"unknown": {"/api/v1/providers/spotify", map[string]any{"enabled": true}, 404, "provider_not_found"},
|
||||
"contact with ()": {"/api/v1/providers/musicbrainz", map[string]any{"contact": "a (b)"}, 400, "invalid_input"},
|
||||
} {
|
||||
r := do(t, h, "PUT", tt.path, admin, tt.body)
|
||||
if r.Code != tt.code || errCode(r) != tt.err {
|
||||
t.Errorf("%s: %d %v, want %d %s", name, r.Code, r.Body, tt.code, tt.err)
|
||||
}
|
||||
raw, _ := json.Marshal(r.Body)
|
||||
if strings.Contains(string(raw), badKey) {
|
||||
t.Errorf("%s: the error repeats the key", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The key is a secret: it must not appear in the request log either.
|
||||
func TestTheKeyNeverReachesTheLog(t *testing.T) {
|
||||
core, logs := observer.New(zap.DebugLevel)
|
||||
h, _ := newTestEnvDB(t, func(d *Deps, _ *bun.DB) { d.Logger = zap.New(core) })
|
||||
admin := registerAndLogin(t, h)
|
||||
|
||||
do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": lastfmKey})
|
||||
do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": "bad" + lastfmKey})
|
||||
|
||||
if logs.Len() == 0 {
|
||||
t.Fatal("the test saw no log entries, so it proves nothing")
|
||||
}
|
||||
for _, e := range logs.All() {
|
||||
dump, _ := json.Marshal(map[string]any{"message": e.Message, "fields": e.ContextMap()})
|
||||
if strings.Contains(string(dump), lastfmKey) {
|
||||
t.Fatalf("a log entry contains the key: %s", dump)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -14,6 +14,7 @@ import (
|
|||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
)
|
||||
|
||||
|
|
@ -53,6 +54,11 @@ type scanService interface {
|
|||
Status() library.Status
|
||||
}
|
||||
|
||||
type providerService interface {
|
||||
List(ctx context.Context) ([]providers.Info, error)
|
||||
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
|
||||
}
|
||||
|
||||
type catalogService interface {
|
||||
Artists(ctx context.Context, f catalog.Filter) (catalog.Page[catalog.Artist], error)
|
||||
Albums(ctx context.Context, f catalog.Filter) (catalog.Page[catalog.Album], error)
|
||||
|
|
@ -78,6 +84,7 @@ type Deps struct {
|
|||
Scan scanService
|
||||
Libraries libraryService
|
||||
Catalog catalogService
|
||||
Providers providerService
|
||||
Streams streamService
|
||||
}
|
||||
|
||||
|
|
@ -133,6 +140,9 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
admin.PATCH("/libraries/:id", libs.update)
|
||||
admin.DELETE("/libraries/:id", libs.remove)
|
||||
admin.POST("/libraries/:id/scan", lib.startLibraryScan)
|
||||
prov := &providersHandlers{svc: d.Providers}
|
||||
admin.GET("/providers", prov.list)
|
||||
admin.PUT("/providers/:name", prov.update)
|
||||
u := &usersHandlers{svc: d.Auth}
|
||||
admin.GET("/users", u.list)
|
||||
admin.POST("/users", u.create)
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ tags:
|
|||
description: Browsing and searching what can be played.
|
||||
- name: Playback
|
||||
description: Streaming and listening history.
|
||||
- name: Providers
|
||||
description: External services Sonda can ask about artists, set up by an admin.
|
||||
|
||||
paths:
|
||||
/health:
|
||||
|
|
@ -751,6 +753,70 @@ paths:
|
|||
'412': {description: '`If-Unmodified-Since` names a date before the last change of the file.'}
|
||||
'416': {description: The range cannot be satisfied.}
|
||||
|
||||
/providers:
|
||||
get:
|
||||
tags: [Providers]
|
||||
operationId: listProviders
|
||||
summary: List the external providers
|
||||
description: |
|
||||
Admin only. Every provider Sonda knows (Last.fm, MusicBrainz), with whether it is on and
|
||||
what it would send. Providers are off by default: nothing leaves the server until an admin
|
||||
enables one. The API key is never returned; `has_api_key` says whether one is stored.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
description: The providers, in a fixed order.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [providers]
|
||||
properties:
|
||||
providers:
|
||||
type: array
|
||||
items: {$ref: '#/components/schemas/Provider'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
|
||||
/providers/{name}:
|
||||
parameters:
|
||||
- name: name
|
||||
in: path
|
||||
required: true
|
||||
schema: {type: string, enum: [lastfm, musicbrainz]}
|
||||
put:
|
||||
tags: [Providers]
|
||||
operationId: updateProvider
|
||||
summary: Set up a provider
|
||||
description: |
|
||||
Admin only. Send any of `enabled`, `api_key` and `contact`; fields you leave out are kept.
|
||||
|
||||
The key is write-only: it is never returned and never logged. Sending an empty `api_key`
|
||||
clears it and turns the provider off. A provider that needs a key (Last.fm) cannot be
|
||||
enabled without one. `contact` goes into the User-Agent sent to the provider (MusicBrainz
|
||||
asks for a URL or an e-mail address); empty means the project address.
|
||||
security: [{bearerAuth: []}]
|
||||
requestBody:
|
||||
required: true
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/ProviderUpdate'}
|
||||
responses:
|
||||
'200':
|
||||
description: The provider as it is now.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Provider'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'404':
|
||||
description: No such provider (`provider_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/listens:
|
||||
get:
|
||||
tags: [Playback]
|
||||
|
|
@ -1168,6 +1234,41 @@ components:
|
|||
description: Milliseconds actually played. Up to twice the track length plus 10 seconds when that length is known.
|
||||
idempotency_key: {type: string, description: "One per play, 1 to 128 bytes (UTF-8). A key that was already used is ignored."}
|
||||
|
||||
Provider:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [name, enabled, needs_api_key, has_api_key, contact, sends, notes, updated_at]
|
||||
properties:
|
||||
name: {type: string, enum: [lastfm, musicbrainz]}
|
||||
enabled: {type: boolean, description: Whether Sonda may contact this provider.}
|
||||
needs_api_key: {type: boolean}
|
||||
has_api_key: {type: boolean, description: Whether a key is stored. The key itself is never returned.}
|
||||
contact: {type: string, description: Contact in the User-Agent; empty means the project address.}
|
||||
sends: {type: string, description: "Plain statement of what is sent to the provider, and what never is."}
|
||||
notes: {type: string, description: Terms of use and attribution that apply to the provider's data.}
|
||||
updated_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: When the settings last changed; null if they never did.
|
||||
|
||||
ProviderUpdate:
|
||||
type: object
|
||||
anyOf:
|
||||
- required: [enabled]
|
||||
- required: [api_key]
|
||||
- required: [contact]
|
||||
properties:
|
||||
enabled: {type: boolean}
|
||||
api_key:
|
||||
type: string
|
||||
format: password
|
||||
description: A Last.fm key is 32 hexadecimal characters. Empty clears it.
|
||||
contact:
|
||||
type: string
|
||||
maxLength: 200
|
||||
description: A URL or e-mail address, without parentheses. Empty restores the default.
|
||||
|
||||
ListenResult:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
|
|||
|
|
@ -23,7 +23,6 @@ type Config struct {
|
|||
Log Log `toml:"log"`
|
||||
Auth Auth `toml:"auth"`
|
||||
Listening Listening `toml:"listening"`
|
||||
Providers Providers `toml:"providers"`
|
||||
}
|
||||
|
||||
// Server holds HTTP listener settings.
|
||||
|
|
@ -81,16 +80,6 @@ func (l Listening) MinListen() time.Duration {
|
|||
return time.Duration(l.MinListenSeconds) * time.Second
|
||||
}
|
||||
|
||||
// Providers holds external provider settings.
|
||||
type Providers struct {
|
||||
LastFM LastFM `toml:"lastfm"`
|
||||
}
|
||||
|
||||
// LastFM holds the Last.fm API settings.
|
||||
type LastFM struct {
|
||||
APIKey string `toml:"api_key"`
|
||||
}
|
||||
|
||||
// Default returns the configuration used when nothing else is set.
|
||||
func Default() Config {
|
||||
return Config{
|
||||
|
|
@ -106,6 +95,12 @@ func Default() Config {
|
|||
// Load builds a Config from defaults, the TOML file at path, and environment
|
||||
// variables. An empty path skips the file. Unknown keys in the file are errors.
|
||||
func Load(path string) (Config, error) {
|
||||
// Provider settings live in the database now. A variable that used to carry
|
||||
// the Last.fm key must not be ignored in silence.
|
||||
if os.Getenv("SONDA_LASTFM_API_KEY") != "" {
|
||||
return Config{}, errors.New("SONDA_LASTFM_API_KEY is no longer read: set the key with " +
|
||||
"PUT /api/v1/providers/lastfm and unset the variable")
|
||||
}
|
||||
cfg := Default()
|
||||
|
||||
if path != "" {
|
||||
|
|
@ -143,9 +138,6 @@ func applyEnv(cfg *Config) {
|
|||
if v := os.Getenv("SONDA_JWT_SECRET"); v != "" {
|
||||
cfg.Auth.JWTSecret = v
|
||||
}
|
||||
if v := os.Getenv("SONDA_LASTFM_API_KEY"); v != "" {
|
||||
cfg.Providers.LastFM.APIKey = v
|
||||
}
|
||||
}
|
||||
|
||||
// Validate checks that the configuration is usable.
|
||||
|
|
@ -196,6 +188,10 @@ func explainDecodeError(path string, err error) error {
|
|||
if errors.As(err, &missing) {
|
||||
for _, e := range missing.Errors {
|
||||
key := e.Key()
|
||||
if len(key) > 0 && key[0] == "providers" {
|
||||
return fmt.Errorf("parse config %s: the providers section was removed; "+
|
||||
"remove it and set the settings with PUT /api/v1/providers/{name}", path)
|
||||
}
|
||||
if len(key) == 2 && key[0] == "library" && key[1] == "roots" {
|
||||
return fmt.Errorf("parse config %s: library.roots was removed; "+
|
||||
"remove it and add each directory as a library with POST /api/v1/libraries", path)
|
||||
|
|
|
|||
|
|
@ -57,8 +57,6 @@ listen = "0.0.0.0:9000"
|
|||
[library]
|
||||
scan_workers = 9
|
||||
|
||||
[providers.lastfm]
|
||||
api_key = "k"
|
||||
`)
|
||||
cfg, err := Load(p)
|
||||
if err != nil {
|
||||
|
|
@ -70,9 +68,6 @@ api_key = "k"
|
|||
if cfg.Library.ScanWorkers != 9 {
|
||||
t.Errorf("scan workers = %d", cfg.Library.ScanWorkers)
|
||||
}
|
||||
if cfg.Providers.LastFM.APIKey != "k" {
|
||||
t.Errorf("api key = %q", cfg.Providers.LastFM.APIKey)
|
||||
}
|
||||
// untouched keys keep their defaults
|
||||
if cfg.Database.Path != "sonda.db" {
|
||||
t.Errorf("db path = %q", cfg.Database.Path)
|
||||
|
|
@ -87,14 +82,13 @@ func TestLoadEnvOverridesFile(t *testing.T) {
|
|||
t.Setenv("SONDA_DB_PATH", "x.db")
|
||||
t.Setenv("SONDA_DB_DRIVER", "postgres")
|
||||
t.Setenv("SONDA_DB_DSN", "postgres://u:p@h/db")
|
||||
t.Setenv("SONDA_LASTFM_API_KEY", "envkey")
|
||||
t.Setenv("SONDA_JWT_SECRET", "envsecret")
|
||||
cfg, err := Load(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Log.Env != "dev" || cfg.Server.Listen != "127.0.0.1:1" ||
|
||||
cfg.Database.Path != "x.db" || cfg.Providers.LastFM.APIKey != "envkey" ||
|
||||
cfg.Database.Path != "x.db" ||
|
||||
cfg.Auth.JWTSecret != "envsecret" ||
|
||||
cfg.Database.Driver != "postgres" || cfg.Database.DSN != "postgres://u:p@h/db" {
|
||||
t.Errorf("env not applied: %+v", cfg)
|
||||
|
|
@ -186,3 +180,33 @@ func TestListeningSettingIsReadFromTheFile(t *testing.T) {
|
|||
t.Errorf("min listen = %v", cfg.Listening.MinListen())
|
||||
}
|
||||
}
|
||||
|
||||
// Provider settings moved into the database. An old config must fail with a
|
||||
// hint, never be ignored in silence.
|
||||
func TestLoadExplainsRemovedProviderSettings(t *testing.T) {
|
||||
p := writeFile(t, t.TempDir(), "[providers.lastfm]\napi_key = \"0123456789abcdef0123456789abcdef\"\n")
|
||||
_, err := Load(p)
|
||||
if err == nil {
|
||||
t.Fatal("expected an error for the providers section")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "providers") || !strings.Contains(err.Error(), "/api/v1/providers") {
|
||||
t.Errorf("error = %v, want a pointer to the providers API", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "0123456789abcdef") {
|
||||
t.Errorf("the error repeats the key: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadRefusesTheOldLastFMEnvironmentVariable(t *testing.T) {
|
||||
t.Setenv("SONDA_LASTFM_API_KEY", "0123456789abcdef0123456789abcdef")
|
||||
_, err := Load("")
|
||||
if err == nil {
|
||||
t.Fatal("expected an error: the variable is no longer read, so it must not be ignored silently")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "SONDA_LASTFM_API_KEY") || !strings.Contains(err.Error(), "/api/v1/providers") {
|
||||
t.Errorf("error = %v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "0123456789abcdef") {
|
||||
t.Errorf("the error repeats the key: %v", err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
376
internal/providers/outbound/outbound.go
Normal file
376
internal/providers/outbound/outbound.go
Normal file
|
|
@ -0,0 +1,376 @@
|
|||
// Package outbound is the only way Sonda talks to the outside world.
|
||||
//
|
||||
// It is built for a server that must not misbehave toward the services it asks
|
||||
// and must not leak anything about its users:
|
||||
//
|
||||
// - every request is a plain GET with an identifiable User-Agent and nothing
|
||||
// else that identifies anyone: no cookies are kept, no Referer, no
|
||||
// Authorization header, and redirects are never followed;
|
||||
// - answers are size-bounded and every attempt has a timeout;
|
||||
// - requests to one provider are spaced by a minimum interval;
|
||||
// - temporary failures (429, 502, 503, 504, dropped connections) are retried
|
||||
// with exponential backoff, honouring Retry-After, never waiting past the
|
||||
// caller's deadline and never waiting an unreasonable time;
|
||||
// - after repeated failures a circuit breaker answers at once instead of
|
||||
// asking a provider that is down, so a dead provider cannot slow Sonda.
|
||||
package outbound
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"math/rand/v2"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Errors returned by the client.
|
||||
var (
|
||||
ErrCircuitOpen = errors.New("provider keeps failing: not asking again yet")
|
||||
ErrResponseTooLarge = errors.New("response is larger than the allowed size")
|
||||
ErrRedirect = errors.New("provider answered with a redirect, which is never followed")
|
||||
ErrBadURL = errors.New("not an http(s) URL")
|
||||
)
|
||||
|
||||
// RateLimitedError means the provider asked us to wait longer than we are
|
||||
// willing to, so the request was not retried.
|
||||
type RateLimitedError struct{ RetryAfter time.Duration }
|
||||
|
||||
func (e *RateLimitedError) Error() string {
|
||||
return fmt.Sprintf("provider asked us to wait %v", e.RetryAfter)
|
||||
}
|
||||
|
||||
// NoRetries as Options.Retries disables retrying (zero means "the default").
|
||||
const NoRetries = -1
|
||||
|
||||
// Clock lets tests control time. The zero Options use the real one.
|
||||
type Clock interface {
|
||||
Now() time.Time
|
||||
Sleep(ctx context.Context, d time.Duration) error
|
||||
}
|
||||
|
||||
type realClock struct{}
|
||||
|
||||
func (realClock) Now() time.Time { return time.Now() }
|
||||
|
||||
func (realClock) Sleep(ctx context.Context, d time.Duration) error {
|
||||
t := time.NewTimer(d)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-t.C:
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
// Options configure a Client. Zero values mean sensible defaults.
|
||||
type Options struct {
|
||||
UserAgent string
|
||||
Timeout time.Duration // per attempt; default 10s
|
||||
|
||||
MinInterval time.Duration // minimum time between requests; 0 = no limit
|
||||
MaxBody int64 // largest accepted answer; default 4 MiB
|
||||
|
||||
Retries int // retries after the first attempt; 0 = default (2), NoRetries = none
|
||||
BackoffBase time.Duration // first backoff; doubles each retry; default 1s
|
||||
MaxRetryAfter time.Duration // longest Retry-After we will wait for; default 30s
|
||||
|
||||
BreakerThreshold int // consecutive failures that open the breaker; default 5
|
||||
BreakerOpenFor time.Duration // first open window; doubles on each new failure; default 30s
|
||||
BreakerMaxOpen time.Duration // longest open window; default 10 min
|
||||
|
||||
Transport http.RoundTripper // default: http.DefaultTransport
|
||||
Clock Clock // default: the real clock
|
||||
Jitter func(max time.Duration) time.Duration // random extra backoff; default: uniform in [0, max]
|
||||
}
|
||||
|
||||
// Response is a completed exchange. Any HTTP status can be returned: it is up
|
||||
// to the caller to read the status (and the provider's own error format).
|
||||
type Response struct {
|
||||
Status int
|
||||
Header http.Header
|
||||
Body []byte
|
||||
}
|
||||
|
||||
// Client talks to one provider. It is safe for concurrent use.
|
||||
type Client struct {
|
||||
userAgent string
|
||||
timeout time.Duration
|
||||
minInterval time.Duration
|
||||
maxBody int64
|
||||
retries int
|
||||
backoffBase time.Duration
|
||||
maxRetryAfter time.Duration
|
||||
breakerThreshold int
|
||||
breakerOpenFor time.Duration
|
||||
breakerMaxOpen time.Duration
|
||||
|
||||
hc *http.Client
|
||||
clock Clock
|
||||
jitter func(time.Duration) time.Duration
|
||||
|
||||
mu sync.Mutex
|
||||
next time.Time // earliest start of the next request
|
||||
failures int // consecutive failures
|
||||
openFor time.Duration
|
||||
openUntil time.Time
|
||||
}
|
||||
|
||||
// New builds a Client.
|
||||
func New(o Options) *Client {
|
||||
def := func(v, d time.Duration) time.Duration {
|
||||
if v <= 0 {
|
||||
return d
|
||||
}
|
||||
return v
|
||||
}
|
||||
c := &Client{
|
||||
userAgent: o.UserAgent,
|
||||
timeout: def(o.Timeout, 10*time.Second),
|
||||
minInterval: o.MinInterval,
|
||||
maxBody: o.MaxBody,
|
||||
retries: o.Retries,
|
||||
backoffBase: def(o.BackoffBase, time.Second),
|
||||
maxRetryAfter: def(o.MaxRetryAfter, 30*time.Second),
|
||||
breakerThreshold: o.BreakerThreshold,
|
||||
breakerOpenFor: def(o.BreakerOpenFor, 30*time.Second),
|
||||
breakerMaxOpen: def(o.BreakerMaxOpen, 10*time.Minute),
|
||||
clock: o.Clock,
|
||||
jitter: o.Jitter,
|
||||
}
|
||||
if c.maxBody <= 0 {
|
||||
c.maxBody = 4 << 20
|
||||
}
|
||||
switch {
|
||||
case c.retries == 0:
|
||||
c.retries = 2
|
||||
case c.retries < 0:
|
||||
c.retries = 0
|
||||
}
|
||||
if c.breakerThreshold <= 0 {
|
||||
c.breakerThreshold = 5
|
||||
}
|
||||
if c.clock == nil {
|
||||
c.clock = realClock{}
|
||||
}
|
||||
if c.jitter == nil {
|
||||
c.jitter = func(max time.Duration) time.Duration {
|
||||
if max <= 0 {
|
||||
return 0
|
||||
}
|
||||
return time.Duration(rand.Int64N(int64(max) + 1))
|
||||
}
|
||||
}
|
||||
|
||||
transport := o.Transport
|
||||
if transport == nil {
|
||||
transport = http.DefaultTransport
|
||||
}
|
||||
c.hc = &http.Client{
|
||||
Transport: transport, // no cookie jar: nothing a provider sets is ever sent back
|
||||
CheckRedirect: func(*http.Request, []*http.Request) error {
|
||||
return http.ErrUseLastResponse // never follow: a redirect could lead anywhere
|
||||
},
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
// Get fetches rawURL. It returns a Response for any HTTP status; an error means
|
||||
// no usable answer (network failure, timeout, breaker open, answer too large,
|
||||
// redirect, or a Retry-After we refuse to wait for).
|
||||
func (c *Client) Get(ctx context.Context, rawURL string) (*Response, error) {
|
||||
u, err := url.Parse(rawURL)
|
||||
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" {
|
||||
return nil, ErrBadURL
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if err := c.allow(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var lastResp *Response
|
||||
var lastErr error
|
||||
for attempt := 0; ; attempt++ {
|
||||
if err := c.throttle(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resp, err := c.once(ctx, rawURL)
|
||||
switch {
|
||||
case errors.Is(err, ErrRedirect), errors.Is(err, ErrResponseTooLarge):
|
||||
return nil, err
|
||||
case err != nil:
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
lastResp, lastErr = nil, err
|
||||
case retryable(resp.Status):
|
||||
lastResp, lastErr = resp, nil
|
||||
default:
|
||||
// An answer that is not worth retrying. A server error still counts
|
||||
// against the provider; anything else (including 403 and 404) shows
|
||||
// the provider is alive.
|
||||
if resp.Status >= 500 {
|
||||
c.failed()
|
||||
} else {
|
||||
c.succeeded()
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
if attempt >= c.retries {
|
||||
break
|
||||
}
|
||||
delay := c.backoff(attempt)
|
||||
if lastResp != nil {
|
||||
if wait, ok := retryAfter(lastResp.Header, c.clock.Now()); ok {
|
||||
if wait > c.maxRetryAfter {
|
||||
c.failed()
|
||||
return nil, &RateLimitedError{RetryAfter: wait}
|
||||
}
|
||||
if wait > delay {
|
||||
delay = wait
|
||||
}
|
||||
}
|
||||
}
|
||||
// Do not sleep through the caller's own deadline: give back what we have.
|
||||
if dl, ok := ctx.Deadline(); ok && time.Until(dl) < delay {
|
||||
break
|
||||
}
|
||||
if err := c.clock.Sleep(ctx, delay); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
// Only temporary failures get here: retried out, so the provider is struggling.
|
||||
c.failed()
|
||||
if lastResp != nil {
|
||||
return lastResp, nil
|
||||
}
|
||||
return nil, lastErr
|
||||
}
|
||||
|
||||
// once makes a single attempt.
|
||||
func (c *Client) once(ctx context.Context, rawURL string) (*Response, error) {
|
||||
actx, cancel := context.WithTimeout(ctx, c.timeout)
|
||||
defer cancel()
|
||||
|
||||
req, err := http.NewRequestWithContext(actx, http.MethodGet, rawURL, nil)
|
||||
if err != nil {
|
||||
return nil, ErrBadURL
|
||||
}
|
||||
req.Header.Set("User-Agent", c.userAgent)
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := c.hc.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode >= 300 && resp.StatusCode < 400 {
|
||||
return nil, ErrRedirect
|
||||
}
|
||||
body, err := io.ReadAll(io.LimitReader(resp.Body, c.maxBody+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if int64(len(body)) > c.maxBody {
|
||||
return nil, ErrResponseTooLarge
|
||||
}
|
||||
return &Response{Status: resp.StatusCode, Header: resp.Header.Clone(), Body: body}, nil
|
||||
}
|
||||
|
||||
func retryable(status int) bool {
|
||||
switch status {
|
||||
case http.StatusTooManyRequests, http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// backoff is base * 2^attempt plus a little random spread, capped.
|
||||
func (c *Client) backoff(attempt int) time.Duration {
|
||||
d := c.backoffBase << attempt
|
||||
if d > c.maxRetryAfter || d <= 0 {
|
||||
d = c.maxRetryAfter
|
||||
}
|
||||
return d + c.jitter(c.backoffBase/4)
|
||||
}
|
||||
|
||||
// retryAfter reads the Retry-After header: seconds, or an HTTP date.
|
||||
func retryAfter(h http.Header, now time.Time) (time.Duration, bool) {
|
||||
v := h.Get("Retry-After")
|
||||
if v == "" {
|
||||
return 0, false
|
||||
}
|
||||
if secs, err := strconv.Atoi(v); err == nil && secs >= 0 {
|
||||
return time.Duration(secs) * time.Second, true
|
||||
}
|
||||
if t, err := http.ParseTime(v); err == nil {
|
||||
if d := t.Sub(now); d > 0 {
|
||||
return d, true
|
||||
}
|
||||
return 0, true
|
||||
}
|
||||
return 0, false
|
||||
}
|
||||
|
||||
// throttle reserves the next free slot and waits for it.
|
||||
func (c *Client) throttle(ctx context.Context) error {
|
||||
if c.minInterval <= 0 {
|
||||
return nil
|
||||
}
|
||||
c.mu.Lock()
|
||||
now := c.clock.Now()
|
||||
at := c.next
|
||||
if at.Before(now) {
|
||||
at = now
|
||||
}
|
||||
c.next = at.Add(c.minInterval)
|
||||
c.mu.Unlock()
|
||||
|
||||
if wait := at.Sub(now); wait > 0 {
|
||||
return c.clock.Sleep(ctx, wait)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// allow refuses the request while the breaker is open.
|
||||
func (c *Client) allow() error {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
if c.clock.Now().Before(c.openUntil) {
|
||||
return ErrCircuitOpen
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (c *Client) succeeded() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.failures, c.openFor, c.openUntil = 0, 0, time.Time{}
|
||||
}
|
||||
|
||||
// failed records a failure. Once the threshold is reached the breaker opens;
|
||||
// each further failure (a probe after the window) reopens it for twice as long.
|
||||
func (c *Client) failed() {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.failures++
|
||||
if c.failures < c.breakerThreshold {
|
||||
return
|
||||
}
|
||||
if c.openFor == 0 {
|
||||
c.openFor = c.breakerOpenFor
|
||||
} else if c.openFor *= 2; c.openFor > c.breakerMaxOpen {
|
||||
c.openFor = c.breakerMaxOpen
|
||||
}
|
||||
c.openUntil = c.clock.Now().Add(c.openFor)
|
||||
}
|
||||
498
internal/providers/outbound/outbound_test.go
Normal file
498
internal/providers/outbound/outbound_test.go
Normal file
|
|
@ -0,0 +1,498 @@
|
|||
package outbound
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// fakeClock never really sleeps: it moves its own time forward and remembers by
|
||||
// how much, so retry and rate-limit timing can be asserted exactly.
|
||||
type fakeClock struct {
|
||||
mu sync.Mutex
|
||||
now time.Time
|
||||
slept []time.Duration
|
||||
}
|
||||
|
||||
func newClock() *fakeClock { return &fakeClock{now: time.Unix(1_700_000_000, 0)} }
|
||||
|
||||
func (f *fakeClock) Now() time.Time {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.now
|
||||
}
|
||||
|
||||
func (f *fakeClock) Sleep(ctx context.Context, d time.Duration) error {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.slept = append(f.slept, d)
|
||||
f.now = f.now.Add(d)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeClock) advance(d time.Duration) {
|
||||
f.mu.Lock()
|
||||
f.now = f.now.Add(d)
|
||||
f.mu.Unlock()
|
||||
}
|
||||
|
||||
func (f *fakeClock) sleeps() []time.Duration {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return append([]time.Duration(nil), f.slept...)
|
||||
}
|
||||
|
||||
func noJitter(time.Duration) time.Duration { return 0 }
|
||||
|
||||
func newClient(clock *fakeClock, mutate func(*Options)) *Client {
|
||||
o := Options{UserAgent: "Sonda/test ( https://example.org/sonda )", Clock: clock, Jitter: noJitter}
|
||||
if mutate != nil {
|
||||
mutate(&o)
|
||||
}
|
||||
return New(o)
|
||||
}
|
||||
|
||||
// server counts the requests it receives and answers with the next handler of a list.
|
||||
func server(t *testing.T, handlers ...http.HandlerFunc) (*httptest.Server, *atomic.Int64) {
|
||||
t.Helper()
|
||||
var n atomic.Int64
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
i := int(n.Add(1)) - 1
|
||||
if i >= len(handlers) {
|
||||
i = len(handlers) - 1
|
||||
}
|
||||
handlers[i](w, r)
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, &n
|
||||
}
|
||||
|
||||
func status(code int) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(code) }
|
||||
}
|
||||
|
||||
func ok(body string) http.HandlerFunc {
|
||||
return func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte(body)) }
|
||||
}
|
||||
|
||||
// ---- what leaves the server ----
|
||||
|
||||
func TestRequestCarriesOnlyWhatItShould(t *testing.T) {
|
||||
var got http.Header
|
||||
var method string
|
||||
srv, _ := server(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
got, method = r.Header.Clone(), r.Method
|
||||
_, _ = w.Write([]byte("{}"))
|
||||
})
|
||||
c := newClient(newClock(), nil)
|
||||
|
||||
if _, err := c.Get(context.Background(), srv.URL+"/x?artist=Creed"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if method != http.MethodGet {
|
||||
t.Errorf("method = %s", method)
|
||||
}
|
||||
if ua := got.Get("User-Agent"); ua != "Sonda/test ( https://example.org/sonda )" {
|
||||
t.Errorf("User-Agent = %q", ua)
|
||||
}
|
||||
if got.Get("Accept") != "application/json" {
|
||||
t.Errorf("Accept = %q", got.Get("Accept"))
|
||||
}
|
||||
for _, h := range []string{"Cookie", "Referer", "Authorization", "Origin", "X-Forwarded-For", "Proxy-Authorization"} {
|
||||
if v := got.Get(h); v != "" {
|
||||
t.Errorf("header %s = %q must never be sent to a provider", h, v)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCookiesAreNeverRemembered(t *testing.T) {
|
||||
var second http.Header
|
||||
srv, n := server(t,
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
http.SetCookie(w, &http.Cookie{Name: "track", Value: "me"})
|
||||
_, _ = w.Write([]byte("{}"))
|
||||
},
|
||||
func(w http.ResponseWriter, r *http.Request) {
|
||||
second = r.Header.Clone()
|
||||
_, _ = w.Write([]byte("{}"))
|
||||
})
|
||||
c := newClient(newClock(), nil)
|
||||
for i := 0; i < 2; i++ {
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if n.Load() != 2 || second.Get("Cookie") != "" {
|
||||
t.Errorf("a cookie set by the provider came back: %q", second.Get("Cookie"))
|
||||
}
|
||||
}
|
||||
|
||||
func TestRedirectsAreRefused(t *testing.T) {
|
||||
elsewhere, hits := server(t, ok("you should never see this"))
|
||||
srv, _ := server(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Redirect(w, r, elsewhere.URL+"/steal", http.StatusFound)
|
||||
})
|
||||
c := newClient(newClock(), nil)
|
||||
|
||||
_, err := c.Get(context.Background(), srv.URL+"/x")
|
||||
if !errors.Is(err, ErrRedirect) {
|
||||
t.Fatalf("err = %v, want ErrRedirect", err)
|
||||
}
|
||||
if hits.Load() != 0 {
|
||||
t.Fatal("the client followed a redirect to another host")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyHTTPURLsAreAccepted(t *testing.T) {
|
||||
c := newClient(newClock(), nil)
|
||||
for _, u := range []string{"", "ftp://example.org/x", "file:///etc/passwd", "http:///nohost", "://bad", "javascript:alert(1)"} {
|
||||
if _, err := c.Get(context.Background(), u); !errors.Is(err, ErrBadURL) {
|
||||
t.Errorf("%q: err = %v, want ErrBadURL", u, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- bounds ----
|
||||
|
||||
func TestResponseBodyIsBounded(t *testing.T) {
|
||||
srv, _ := server(t, ok("0123456789"))
|
||||
c := newClient(newClock(), func(o *Options) { o.MaxBody = 9 })
|
||||
if _, err := c.Get(context.Background(), srv.URL); !errors.Is(err, ErrResponseTooLarge) {
|
||||
t.Fatalf("10 bytes over a limit of 9: err = %v", err)
|
||||
}
|
||||
c = newClient(newClock(), func(o *Options) { o.MaxBody = 10 })
|
||||
resp, err := c.Get(context.Background(), srv.URL)
|
||||
if err != nil || string(resp.Body) != "0123456789" {
|
||||
t.Fatalf("exactly the limit: %v %v", resp, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEachAttemptHasATimeout(t *testing.T) {
|
||||
srv, n := server(t, func(w http.ResponseWriter, r *http.Request) {
|
||||
select {
|
||||
case <-time.After(2 * time.Second):
|
||||
case <-r.Context().Done():
|
||||
}
|
||||
})
|
||||
c := newClient(newClock(), func(o *Options) { o.Timeout = 50 * time.Millisecond; o.Retries = 1 })
|
||||
start := time.Now()
|
||||
if _, err := c.Get(context.Background(), srv.URL); err == nil {
|
||||
t.Fatal("a stuck server must end in an error")
|
||||
}
|
||||
if elapsed := time.Since(start); elapsed > time.Second {
|
||||
t.Errorf("took %v: the per-attempt timeout did not apply", elapsed)
|
||||
}
|
||||
if n.Load() != 2 {
|
||||
t.Errorf("attempts = %d, want 2 (one try and one retry)", n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestACancelledContextSendsNothing(t *testing.T) {
|
||||
srv, n := server(t, ok("{}"))
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if _, err := newClient(newClock(), nil).Get(ctx, srv.URL); !errors.Is(err, context.Canceled) {
|
||||
t.Fatalf("err = %v, want context.Canceled", err)
|
||||
}
|
||||
if n.Load() != 0 {
|
||||
t.Error("a request was sent for a cancelled context")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- rate limit ----
|
||||
|
||||
func TestRequestsAreSpacedByTheMinimumInterval(t *testing.T) {
|
||||
srv, _ := server(t, ok("{}"))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.MinInterval = time.Second })
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
// The first goes at once; each next one waits for the interval.
|
||||
if got := clock.sleeps(); len(got) != 2 || got[0] != time.Second || got[1] != time.Second {
|
||||
t.Errorf("sleeps = %v, want [1s 1s]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlotsAreNotWastedWhenTimePasses(t *testing.T) {
|
||||
srv, _ := server(t, ok("{}"))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.MinInterval = time.Second })
|
||||
_, _ = c.Get(context.Background(), srv.URL)
|
||||
clock.advance(5 * time.Second) // plenty of time passed
|
||||
_, _ = c.Get(context.Background(), srv.URL)
|
||||
if got := clock.sleeps(); len(got) != 0 {
|
||||
t.Errorf("sleeps = %v, want none: the interval had already passed", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Many goroutines at once must still be spaced, with a real clock.
|
||||
func TestConcurrentRequestsStaySpaced(t *testing.T) {
|
||||
var mu sync.Mutex
|
||||
var times []time.Time
|
||||
srv, _ := server(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
mu.Lock()
|
||||
times = append(times, time.Now())
|
||||
mu.Unlock()
|
||||
_, _ = w.Write([]byte("{}"))
|
||||
})
|
||||
const gap = 20 * time.Millisecond
|
||||
c := New(Options{UserAgent: "t", MinInterval: gap, Jitter: noJitter})
|
||||
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 10; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(times) != 10 {
|
||||
t.Fatalf("%d requests arrived, want 10", len(times))
|
||||
}
|
||||
for i := 1; i < len(times); i++ {
|
||||
for j := 0; j < i; j++ { // order of arrival may differ: compare every pair
|
||||
d := times[i].Sub(times[j])
|
||||
if d < 0 {
|
||||
d = -d
|
||||
}
|
||||
if d < gap/2 {
|
||||
t.Fatalf("two requests arrived %v apart, want at least about %v", d, gap)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- retries ----
|
||||
|
||||
func TestRetriesTemporaryFailuresWithBackoff(t *testing.T) {
|
||||
srv, n := server(t, status(503), status(502), ok("fine"))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.BackoffBase = time.Second })
|
||||
|
||||
resp, err := c.Get(context.Background(), srv.URL)
|
||||
if err != nil || resp.Status != 200 || string(resp.Body) != "fine" {
|
||||
t.Fatalf("resp=%+v err=%v", resp, err)
|
||||
}
|
||||
if n.Load() != 3 {
|
||||
t.Errorf("attempts = %d, want 3", n.Load())
|
||||
}
|
||||
if got := clock.sleeps(); len(got) != 2 || got[0] != time.Second || got[1] != 2*time.Second {
|
||||
t.Errorf("backoff = %v, want [1s 2s]", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGivesUpAfterTheRetriesAndReturnsTheLastAnswer(t *testing.T) {
|
||||
srv, n := server(t, status(503))
|
||||
c := newClient(newClock(), func(o *Options) { o.Retries = 2 })
|
||||
resp, err := c.Get(context.Background(), srv.URL)
|
||||
if err != nil || resp.Status != 503 {
|
||||
t.Fatalf("resp=%+v err=%v, want the final 503 as an answer", resp, err)
|
||||
}
|
||||
if n.Load() != 3 {
|
||||
t.Errorf("attempts = %d, want 3 (one try, two retries)", n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestTooManyRequestsIsRetried(t *testing.T) {
|
||||
srv, n := server(t, status(429), ok("{}"))
|
||||
resp, err := newClient(newClock(), nil).Get(context.Background(), srv.URL)
|
||||
if err != nil || resp.Status != 200 || n.Load() != 2 {
|
||||
t.Fatalf("resp=%+v err=%v attempts=%d", resp, err, n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientErrorsAreNotRetried(t *testing.T) {
|
||||
for _, code := range []int{400, 403, 404} {
|
||||
srv, n := server(t, status(code))
|
||||
resp, err := newClient(newClock(), nil).Get(context.Background(), srv.URL)
|
||||
if err != nil || resp.Status != code || n.Load() != 1 {
|
||||
t.Errorf("%d: resp=%+v err=%v attempts=%d, want one attempt and the answer", code, resp, err, n.Load())
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryAfterIsHonoured(t *testing.T) {
|
||||
srv, _ := server(t,
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Retry-After", "5")
|
||||
w.WriteHeader(503)
|
||||
}, ok("{}"))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.BackoffBase = time.Second })
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := clock.sleeps(); len(got) != 1 || got[0] != 5*time.Second {
|
||||
t.Errorf("sleeps = %v, want the 5 s the server asked for (more than the 1 s backoff)", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryAfterAsADate(t *testing.T) {
|
||||
clock := newClock()
|
||||
srv, _ := server(t,
|
||||
func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Retry-After", clock.Now().Add(7*time.Second).UTC().Format(http.TimeFormat))
|
||||
w.WriteHeader(503)
|
||||
}, ok("{}"))
|
||||
c := newClient(clock, nil)
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := clock.sleeps(); len(got) != 1 || got[0] < 6*time.Second || got[0] > 7*time.Second {
|
||||
t.Errorf("sleeps = %v, want about 7 s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnAbsurdRetryAfterIsNotWaitedFor(t *testing.T) {
|
||||
srv, n := server(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Retry-After", "3600")
|
||||
w.WriteHeader(429)
|
||||
})
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.MaxRetryAfter = 30 * time.Second })
|
||||
|
||||
_, err := c.Get(context.Background(), srv.URL)
|
||||
var rl *RateLimitedError
|
||||
if !errors.As(err, &rl) || rl.RetryAfter != time.Hour {
|
||||
t.Fatalf("err = %v, want RateLimitedError with RetryAfter 1h", err)
|
||||
}
|
||||
if n.Load() != 1 || len(clock.sleeps()) != 0 {
|
||||
t.Errorf("attempts=%d sleeps=%v: it must give up at once, not wait an hour", n.Load(), clock.sleeps())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDoesNotWaitPastTheCallersDeadline(t *testing.T) {
|
||||
srv, n := server(t, func(w http.ResponseWriter, _ *http.Request) {
|
||||
w.Header().Set("Retry-After", "20")
|
||||
w.WriteHeader(503)
|
||||
})
|
||||
clock := newClock()
|
||||
c := newClient(clock, nil)
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second) // real time: the fake clock cannot see it
|
||||
defer cancel()
|
||||
|
||||
start := time.Now()
|
||||
resp, err := c.Get(ctx, srv.URL)
|
||||
if time.Since(start) > time.Second {
|
||||
t.Errorf("took %v: it waited although the deadline was 2 s away and the server asked for 20 s", time.Since(start))
|
||||
}
|
||||
if n.Load() != 1 || len(clock.sleeps()) != 0 {
|
||||
t.Errorf("attempts=%d sleeps=%v, want a single attempt and no wait", n.Load(), clock.sleeps())
|
||||
}
|
||||
if err == nil && resp.Status != 503 {
|
||||
t.Errorf("resp=%+v", resp)
|
||||
}
|
||||
}
|
||||
|
||||
func TestATransportErrorIsRetried(t *testing.T) {
|
||||
var n atomic.Int64
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if n.Add(1) == 1 {
|
||||
conn, _, _ := w.(http.Hijacker).Hijack()
|
||||
_ = conn.Close() // drop the connection without an answer
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte("{}"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
resp, err := newClient(newClock(), nil).Get(context.Background(), srv.URL)
|
||||
if err != nil || resp.Status != 200 || n.Load() != 2 {
|
||||
t.Fatalf("resp=%+v err=%v attempts=%d", resp, err, n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
// ---- circuit breaker ----
|
||||
|
||||
func TestABrokenProviderFailsFast(t *testing.T) {
|
||||
srv, n := server(t, status(500))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) {
|
||||
o.Retries, o.BreakerThreshold, o.BreakerOpenFor = NoRetries, 3, 30*time.Second
|
||||
})
|
||||
ctx := context.Background()
|
||||
|
||||
for i := 0; i < 3; i++ {
|
||||
if resp, err := c.Get(ctx, srv.URL); err != nil || resp.Status != 500 {
|
||||
t.Fatalf("failure %d: %+v %v", i, resp, err)
|
||||
}
|
||||
}
|
||||
// Open: no network at all.
|
||||
before := n.Load()
|
||||
if _, err := c.Get(ctx, srv.URL); !errors.Is(err, ErrCircuitOpen) {
|
||||
t.Fatalf("err = %v, want ErrCircuitOpen", err)
|
||||
}
|
||||
if n.Load() != before {
|
||||
t.Error("a request reached the server while the breaker was open")
|
||||
}
|
||||
|
||||
// After the window one request is let through; failing again reopens it for longer.
|
||||
clock.advance(31 * time.Second)
|
||||
if resp, err := c.Get(ctx, srv.URL); err != nil || resp.Status != 500 {
|
||||
t.Fatalf("probe: %+v %v", resp, err)
|
||||
}
|
||||
clock.advance(31 * time.Second) // the window is now 60 s, so this is not enough
|
||||
if _, err := c.Get(ctx, srv.URL); !errors.Is(err, ErrCircuitOpen) {
|
||||
t.Fatalf("after a failed probe the window must double: err = %v", err)
|
||||
}
|
||||
clock.advance(30 * time.Second)
|
||||
if _, err := c.Get(ctx, srv.URL); errors.Is(err, ErrCircuitOpen) {
|
||||
t.Fatal("still open after the doubled window")
|
||||
}
|
||||
}
|
||||
|
||||
func TestASuccessClosesTheBreaker(t *testing.T) {
|
||||
srv, _ := server(t, status(500), status(500), ok("{}"), status(500), status(500), ok("{}"))
|
||||
clock := newClock()
|
||||
c := newClient(clock, func(o *Options) { o.Retries, o.BreakerThreshold = NoRetries, 3 })
|
||||
ctx := context.Background()
|
||||
|
||||
// Two failures, a success, two failures: never three in a row, so it stays closed.
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := c.Get(ctx, srv.URL); errors.Is(err, ErrCircuitOpen) {
|
||||
t.Fatalf("request %d: the breaker opened although failures were never consecutive", i)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientErrorsDoNotOpenTheBreaker(t *testing.T) {
|
||||
srv, n := server(t, status(404))
|
||||
c := newClient(newClock(), func(o *Options) { o.Retries, o.BreakerThreshold = NoRetries, 2 })
|
||||
for i := 0; i < 6; i++ {
|
||||
if _, err := c.Get(context.Background(), srv.URL); err != nil {
|
||||
t.Fatalf("request %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
if n.Load() != 6 {
|
||||
t.Errorf("attempts = %d: 404 is an answer, not a failure of the provider", n.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestDefaultsAreSane(t *testing.T) {
|
||||
c := New(Options{UserAgent: "x"})
|
||||
if c.timeout <= 0 || c.maxBody <= 0 || c.retries != 2 || c.breakerThreshold <= 0 {
|
||||
t.Errorf("defaults: timeout=%v maxBody=%d retries=%d threshold=%d", c.timeout, c.maxBody, c.retries, c.breakerThreshold)
|
||||
}
|
||||
if !strings.Contains(strings.ToLower(c.userAgent), "x") {
|
||||
t.Errorf("user agent = %q", c.userAgent)
|
||||
}
|
||||
}
|
||||
266
internal/providers/settings.go
Normal file
266
internal/providers/settings.go
Normal file
|
|
@ -0,0 +1,266 @@
|
|||
// Package providers manages the external services Sonda can ask about artists
|
||||
// (Last.fm, MusicBrainz) and the rules for talking to them.
|
||||
//
|
||||
// Nothing here makes a request on its own. Every provider is off until an admin
|
||||
// turns it on, and the settings (including any API key) live in the database.
|
||||
package providers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
// The providers Sonda knows.
|
||||
const (
|
||||
LastFM = "lastfm"
|
||||
MusicBrainz = "musicbrainz"
|
||||
)
|
||||
|
||||
// Errors returned by the service.
|
||||
var (
|
||||
ErrUnknownProvider = errors.New("unknown provider")
|
||||
ErrInvalidSettings = errors.New("invalid provider settings")
|
||||
)
|
||||
|
||||
const maxContactLen = 200
|
||||
|
||||
var lastfmKeyRe = regexp.MustCompile(`^[0-9a-f]{32}$`)
|
||||
|
||||
// Settings are the stored settings of one provider, API key included. They are
|
||||
// only for code that talks to the provider; the API never returns the key.
|
||||
type Settings struct {
|
||||
Provider string
|
||||
Enabled bool
|
||||
APIKey string
|
||||
Contact string // goes into the User-Agent; empty means the project address
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
// Info is what the API shows about a provider: the settings without the key,
|
||||
// plus a plain statement of what leaves the server.
|
||||
type Info struct {
|
||||
Name string
|
||||
Enabled bool
|
||||
NeedsAPIKey bool
|
||||
HasAPIKey bool
|
||||
Contact string
|
||||
Sends string // what is sent to the provider, and what never is
|
||||
Notes string // terms of use and attribution
|
||||
UpdatedAt time.Time // zero when the provider was never configured
|
||||
}
|
||||
|
||||
// Patch changes some settings. A nil field is left alone. An empty APIKey or
|
||||
// Contact clears it.
|
||||
type Patch struct {
|
||||
Enabled *bool
|
||||
APIKey *string
|
||||
Contact *string
|
||||
}
|
||||
|
||||
// Store persists provider settings.
|
||||
type Store interface {
|
||||
// Get returns found=false when the provider was never configured.
|
||||
Get(ctx context.Context, name string) (s Settings, found bool, err error)
|
||||
List(ctx context.Context) ([]Settings, error)
|
||||
Save(ctx context.Context, s Settings) error
|
||||
}
|
||||
|
||||
type meta struct {
|
||||
name string
|
||||
needsKey bool
|
||||
sends, note string
|
||||
}
|
||||
|
||||
// known lists the providers in the order the API shows them.
|
||||
var known = []meta{
|
||||
{
|
||||
name: LastFM,
|
||||
needsKey: true,
|
||||
sends: "The artist name (or MusicBrainz id) you look up, your API key, your IP address and the " +
|
||||
"User-Agent. Never your users, your listens, track titles or file paths.",
|
||||
note: "Last.fm data is for non-commercial use only. Credit Last.fm when you show it. " +
|
||||
"Images are not licensed and are never stored.",
|
||||
},
|
||||
{
|
||||
name: MusicBrainz,
|
||||
sends: "The artist name or MusicBrainz id you look up, your IP address and the User-Agent, " +
|
||||
"which carries the contact below. Never your users, your listens, track titles or file paths.",
|
||||
note: "Core data (artist names and ids) is public domain. Tags and genres are CC BY-NC-SA 3.0: " +
|
||||
"non-commercial, with attribution to MusicBrainz.",
|
||||
},
|
||||
}
|
||||
|
||||
func lookup(name string) (meta, bool) {
|
||||
for _, m := range known {
|
||||
if m.name == name {
|
||||
return m, true
|
||||
}
|
||||
}
|
||||
return meta{}, false
|
||||
}
|
||||
|
||||
// Service reads and changes provider settings.
|
||||
type Service struct {
|
||||
store Store
|
||||
now func() time.Time
|
||||
mu sync.Mutex // an update reads then writes: keep two of them apart
|
||||
}
|
||||
|
||||
// NewService builds a Service.
|
||||
func NewService(store Store) *Service { return &Service{store: store, now: time.Now} }
|
||||
|
||||
func info(m meta, s Settings) Info {
|
||||
return Info{
|
||||
Name: m.name, NeedsAPIKey: m.needsKey, Sends: m.sends, Notes: m.note,
|
||||
Enabled: s.Enabled, HasAPIKey: s.APIKey != "", Contact: s.Contact, UpdatedAt: s.UpdatedAt,
|
||||
}
|
||||
}
|
||||
|
||||
// List returns every known provider, configured or not, in a fixed order.
|
||||
func (s *Service) List(ctx context.Context) ([]Info, error) {
|
||||
rows, err := s.store.List(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("list provider settings: %w", err)
|
||||
}
|
||||
byName := map[string]Settings{}
|
||||
for _, r := range rows {
|
||||
byName[r.Provider] = r
|
||||
}
|
||||
out := make([]Info, 0, len(known))
|
||||
for _, m := range known {
|
||||
out = append(out, info(m, byName[m.name]))
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Update applies a patch and returns the provider as it is afterwards.
|
||||
func (s *Service) Update(ctx context.Context, name string, p Patch) (Info, error) {
|
||||
m, ok := lookup(name)
|
||||
if !ok {
|
||||
return Info{}, ErrUnknownProvider
|
||||
}
|
||||
if p.Enabled == nil && p.APIKey == nil && p.Contact == nil {
|
||||
return Info{}, fmt.Errorf("%w: nothing to change", ErrInvalidSettings)
|
||||
}
|
||||
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
|
||||
cur, _, err := s.store.Get(ctx, name)
|
||||
if err != nil {
|
||||
return Info{}, fmt.Errorf("read provider settings: %w", err)
|
||||
}
|
||||
cur.Provider = name
|
||||
|
||||
if p.APIKey != nil {
|
||||
key, err := validateKey(m, *p.APIKey)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
cur.APIKey = key
|
||||
if key == "" {
|
||||
cur.Enabled = false // a provider that needs a key is off without one
|
||||
}
|
||||
}
|
||||
if p.Contact != nil {
|
||||
contact, err := validateContact(*p.Contact)
|
||||
if err != nil {
|
||||
return Info{}, err
|
||||
}
|
||||
cur.Contact = contact
|
||||
}
|
||||
if p.Enabled != nil {
|
||||
if *p.Enabled && m.needsKey && cur.APIKey == "" {
|
||||
return Info{}, fmt.Errorf("%w: %s needs an API key before it can be enabled", ErrInvalidSettings, name)
|
||||
}
|
||||
cur.Enabled = *p.Enabled
|
||||
}
|
||||
|
||||
cur.UpdatedAt = s.now()
|
||||
if err := s.store.Save(ctx, cur); err != nil {
|
||||
return Info{}, fmt.Errorf("save provider settings: %w", err)
|
||||
}
|
||||
return info(m, cur), nil
|
||||
}
|
||||
|
||||
// Active returns the settings of a provider when it can be used right now: it
|
||||
// is on and, if it needs a key, has one. Provider clients call this before
|
||||
// every outbound request; false means "do not contact anyone".
|
||||
func (s *Service) Active(ctx context.Context, name string) (Settings, bool, error) {
|
||||
m, ok := lookup(name)
|
||||
if !ok {
|
||||
return Settings{}, false, ErrUnknownProvider
|
||||
}
|
||||
cur, found, err := s.store.Get(ctx, name)
|
||||
if err != nil {
|
||||
return Settings{}, false, fmt.Errorf("read provider settings: %w", err)
|
||||
}
|
||||
if !found || !cur.Enabled || (m.needsKey && cur.APIKey == "") {
|
||||
return Settings{}, false, nil
|
||||
}
|
||||
return cur, true, nil
|
||||
}
|
||||
|
||||
func validateKey(m meta, raw string) (string, error) {
|
||||
key := strings.ToLower(strings.TrimSpace(raw))
|
||||
if key == "" {
|
||||
return "", nil
|
||||
}
|
||||
if !m.needsKey {
|
||||
return "", fmt.Errorf("%w: %s does not use an API key", ErrInvalidSettings, m.name)
|
||||
}
|
||||
if m.name == LastFM && !lastfmKeyRe.MatchString(key) {
|
||||
return "", fmt.Errorf("%w: a Last.fm API key is 32 hexadecimal characters", ErrInvalidSettings)
|
||||
}
|
||||
return key, nil
|
||||
}
|
||||
|
||||
// validateContact accepts what MusicBrainz asks for in a User-Agent: a URL or an
|
||||
// e-mail address. Parentheses are refused because they delimit it there.
|
||||
func validateContact(raw string) (string, error) {
|
||||
c := strings.TrimSpace(raw)
|
||||
if utf8.RuneCountInString(c) > maxContactLen {
|
||||
return "", fmt.Errorf("%w: contact is longer than %d characters", ErrInvalidSettings, maxContactLen)
|
||||
}
|
||||
for _, r := range c {
|
||||
if unicode.IsControl(r) || r == '(' || r == ')' {
|
||||
return "", fmt.Errorf("%w: contact cannot contain control characters or parentheses", ErrInvalidSettings)
|
||||
}
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// ProjectURL is the contact used in the User-Agent when no other is set.
|
||||
const ProjectURL = "https://codeberg.org/squidd/sonda"
|
||||
|
||||
// UserAgent builds the header sent to providers, in the shape MusicBrainz asks
|
||||
// for: "Application/version ( contact )". The version and the contact are
|
||||
// cleaned, so the result is always a single safe line.
|
||||
func UserAgent(version, contact string) string {
|
||||
clean := func(s string) string {
|
||||
var b strings.Builder
|
||||
for _, r := range s {
|
||||
if unicode.IsControl(r) || r == '(' || r == ')' {
|
||||
continue
|
||||
}
|
||||
b.WriteRune(r)
|
||||
}
|
||||
return strings.TrimSpace(b.String())
|
||||
}
|
||||
v := strings.ReplaceAll(clean(version), " ", "")
|
||||
if v == "" {
|
||||
v = "dev"
|
||||
}
|
||||
c := clean(contact)
|
||||
if c == "" {
|
||||
c = ProjectURL
|
||||
}
|
||||
return fmt.Sprintf("Sonda/%s ( %s )", v, c)
|
||||
}
|
||||
261
internal/providers/settings_test.go
Normal file
261
internal/providers/settings_test.go
Normal file
|
|
@ -0,0 +1,261 @@
|
|||
package providers
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// memStore is an in-memory Store.
|
||||
type memStore struct {
|
||||
rows map[string]Settings
|
||||
err error
|
||||
}
|
||||
|
||||
func newMemStore() *memStore { return &memStore{rows: map[string]Settings{}} }
|
||||
|
||||
func (m *memStore) Get(_ context.Context, name string) (Settings, bool, error) {
|
||||
s, ok := m.rows[name]
|
||||
return s, ok, m.err
|
||||
}
|
||||
|
||||
func (m *memStore) List(context.Context) ([]Settings, error) {
|
||||
var out []Settings
|
||||
for _, s := range m.rows {
|
||||
out = append(out, s)
|
||||
}
|
||||
return out, m.err
|
||||
}
|
||||
|
||||
func (m *memStore) Save(_ context.Context, s Settings) error {
|
||||
if m.err != nil {
|
||||
return m.err
|
||||
}
|
||||
m.rows[s.Provider] = s
|
||||
return nil
|
||||
}
|
||||
|
||||
const goodKey = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
func boolp(b bool) *bool { return &b }
|
||||
func strp(s string) *string { return &s }
|
||||
func newSvc() (*Service, *memStore) {
|
||||
st := newMemStore()
|
||||
return NewService(st), st
|
||||
}
|
||||
|
||||
func TestEveryProviderIsOffByDefault(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
infos, err := s.List(context.Background())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(infos) != 2 || infos[0].Name != LastFM || infos[1].Name != MusicBrainz {
|
||||
t.Fatalf("providers = %+v, want lastfm then musicbrainz", infos)
|
||||
}
|
||||
for _, in := range infos {
|
||||
if in.Enabled || in.HasAPIKey {
|
||||
t.Errorf("%s is on or has a key before anything was configured: %+v", in.Name, in)
|
||||
}
|
||||
if strings.TrimSpace(in.Sends) == "" || strings.TrimSpace(in.Notes) == "" {
|
||||
t.Errorf("%s does not say what it sends or under which terms", in.Name)
|
||||
}
|
||||
}
|
||||
if !infos[0].NeedsAPIKey || infos[1].NeedsAPIKey {
|
||||
t.Errorf("needs_api_key: lastfm=%v musicbrainz=%v, want true/false", infos[0].NeedsAPIKey, infos[1].NeedsAPIKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhatEachProviderSendsNeverMentionsPrivateData(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
infos, _ := s.List(context.Background())
|
||||
for _, in := range infos {
|
||||
if !strings.Contains(in.Sends, "Never") {
|
||||
t.Errorf("%s: the description of what is sent must say what is never sent: %q", in.Name, in.Sends)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestInfoNeverCarriesTheKey(t *testing.T) {
|
||||
typ := reflect.TypeOf(Info{})
|
||||
for i := 0; i < typ.NumField(); i++ {
|
||||
if name := strings.ToLower(typ.Field(i).Name); name == "apikey" || strings.Contains(name, "secret") || name == "key" {
|
||||
t.Fatalf("Info has a field %q that could carry the key", typ.Field(i).Name)
|
||||
}
|
||||
}
|
||||
s, _ := newSvc()
|
||||
info, err := s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !info.HasAPIKey {
|
||||
t.Error("HasAPIKey should be true after saving a key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingAndEnablingLastFM(t *testing.T) {
|
||||
s, st := newSvc()
|
||||
ctx := context.Background()
|
||||
|
||||
// Enabling without a key is refused.
|
||||
if _, err := s.Update(ctx, LastFM, Patch{Enabled: boolp(true)}); !errors.Is(err, ErrInvalidSettings) {
|
||||
t.Fatalf("enable without a key: err = %v", err)
|
||||
}
|
||||
if _, ok := st.rows[LastFM]; ok {
|
||||
t.Error("a refused update left a row behind")
|
||||
}
|
||||
|
||||
info, err := s.Update(ctx, LastFM, Patch{APIKey: strp(" " + strings.ToUpper(goodKey) + " ")})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !info.HasAPIKey || info.Enabled {
|
||||
t.Errorf("after saving a key: %+v, want a key and still off", info)
|
||||
}
|
||||
if st.rows[LastFM].APIKey != goodKey {
|
||||
t.Errorf("stored key = %q, want it trimmed and lower-cased", st.rows[LastFM].APIKey)
|
||||
}
|
||||
|
||||
if info, err = s.Update(ctx, LastFM, Patch{Enabled: boolp(true)}); err != nil || !info.Enabled {
|
||||
t.Fatalf("enable: %+v err=%v", info, err)
|
||||
}
|
||||
|
||||
// Clearing the key turns the provider off with it.
|
||||
info, err = s.Update(ctx, LastFM, Patch{APIKey: strp("")})
|
||||
if err != nil || info.HasAPIKey || info.Enabled {
|
||||
t.Fatalf("clear the key: %+v err=%v, want no key and off", info, err)
|
||||
}
|
||||
if st.rows[LastFM].APIKey != "" || st.rows[LastFM].Enabled {
|
||||
t.Errorf("stored row after clearing: %+v", st.rows[LastFM])
|
||||
}
|
||||
}
|
||||
|
||||
func TestUpdateRejectsBadInput(t *testing.T) {
|
||||
ctx := context.Background()
|
||||
tests := []struct {
|
||||
name string
|
||||
provider string
|
||||
patch Patch
|
||||
}{
|
||||
{"empty patch", LastFM, Patch{}},
|
||||
{"key too short", LastFM, Patch{APIKey: strp("abc123")}},
|
||||
{"key not hex", LastFM, Patch{APIKey: strp(strings.Repeat("z", 32))}},
|
||||
{"key too long", LastFM, Patch{APIKey: strp(goodKey + "0")}},
|
||||
{"a key for a provider without keys", MusicBrainz, Patch{APIKey: strp(goodKey)}},
|
||||
{"contact with parentheses", MusicBrainz, Patch{Contact: strp("me (at) example.org")}},
|
||||
{"contact with a line break", MusicBrainz, Patch{Contact: strp("a\nb")}},
|
||||
{"contact too long", MusicBrainz, Patch{Contact: strp(strings.Repeat("x", 201))}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s, st := newSvc()
|
||||
if _, err := s.Update(ctx, tt.provider, tt.patch); !errors.Is(err, ErrInvalidSettings) {
|
||||
t.Fatalf("err = %v, want ErrInvalidSettings", err)
|
||||
}
|
||||
if len(st.rows) != 0 {
|
||||
t.Error("a rejected update was stored")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownProvider(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
if _, err := s.Update(context.Background(), "spotify", Patch{Enabled: boolp(true)}); !errors.Is(err, ErrUnknownProvider) {
|
||||
t.Fatalf("err = %v, want ErrUnknownProvider", err)
|
||||
}
|
||||
if _, _, err := s.Active(context.Background(), "spotify"); !errors.Is(err, ErrUnknownProvider) {
|
||||
t.Fatalf("Active: err = %v, want ErrUnknownProvider", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMusicBrainzNeedsNoKeyAndTakesAContact(t *testing.T) {
|
||||
s, st := newSvc()
|
||||
ctx := context.Background()
|
||||
info, err := s.Update(ctx, MusicBrainz, Patch{Enabled: boolp(true), Contact: strp(" https://example.org/me ")})
|
||||
if err != nil || !info.Enabled || info.Contact != "https://example.org/me" {
|
||||
t.Fatalf("%+v err=%v", info, err)
|
||||
}
|
||||
if st.rows[MusicBrainz].Contact != "https://example.org/me" {
|
||||
t.Errorf("stored contact = %q", st.rows[MusicBrainz].Contact)
|
||||
}
|
||||
// An empty contact goes back to the default (the project's address).
|
||||
info, err = s.Update(ctx, MusicBrainz, Patch{Contact: strp("")})
|
||||
if err != nil || info.Contact != "" || !info.Enabled {
|
||||
t.Fatalf("clear the contact: %+v err=%v", info, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestActiveOnlyReturnsAUsableProvider(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
ctx := context.Background()
|
||||
|
||||
if _, ok, err := s.Active(ctx, LastFM); err != nil || ok {
|
||||
t.Fatalf("nothing configured: ok=%v err=%v", ok, err)
|
||||
}
|
||||
_, _ = s.Update(ctx, LastFM, Patch{APIKey: strp(goodKey)})
|
||||
if _, ok, _ := s.Active(ctx, LastFM); ok {
|
||||
t.Error("a provider with a key but turned off was reported active")
|
||||
}
|
||||
_, _ = s.Update(ctx, LastFM, Patch{Enabled: boolp(true)})
|
||||
cfg, ok, err := s.Active(ctx, LastFM)
|
||||
if err != nil || !ok || cfg.APIKey != goodKey {
|
||||
t.Fatalf("active: %+v ok=%v err=%v", cfg, ok, err)
|
||||
}
|
||||
_, _ = s.Update(ctx, LastFM, Patch{Enabled: boolp(false)})
|
||||
if _, ok, _ := s.Active(ctx, LastFM); ok {
|
||||
t.Error("a provider turned off was reported active")
|
||||
}
|
||||
|
||||
// MusicBrainz is usable as soon as it is on.
|
||||
_, _ = s.Update(ctx, MusicBrainz, Patch{Enabled: boolp(true)})
|
||||
if _, ok, _ := s.Active(ctx, MusicBrainz); !ok {
|
||||
t.Error("musicbrainz on, no key needed: should be active")
|
||||
}
|
||||
}
|
||||
|
||||
func TestListMergesStoredRowsInAFixedOrder(t *testing.T) {
|
||||
s, st := newSvc()
|
||||
st.rows[MusicBrainz] = Settings{Provider: MusicBrainz, Enabled: true, Contact: "c", UpdatedAt: time.Unix(1, 0)}
|
||||
st.rows["stale-unknown"] = Settings{Provider: "stale-unknown", Enabled: true} // a row of a provider that no longer exists
|
||||
infos, err := s.List(context.Background())
|
||||
if err != nil || len(infos) != 2 {
|
||||
t.Fatalf("infos = %+v err=%v, want exactly the known providers", infos, err)
|
||||
}
|
||||
if infos[0].Name != LastFM || infos[1].Name != MusicBrainz || !infos[1].Enabled || infos[1].Contact != "c" {
|
||||
t.Errorf("infos = %+v", infos)
|
||||
}
|
||||
}
|
||||
|
||||
func TestStoreErrorsPassThrough(t *testing.T) {
|
||||
boom := errors.New("db down")
|
||||
s := NewService(&memStore{rows: map[string]Settings{}, err: boom})
|
||||
if _, err := s.List(context.Background()); !errors.Is(err, boom) {
|
||||
t.Errorf("List: %v", err)
|
||||
}
|
||||
if _, err := s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)}); !errors.Is(err, boom) {
|
||||
t.Errorf("Update: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestUserAgentIsIdentifiableAndSafe(t *testing.T) {
|
||||
if got := UserAgent("1.2.3", ""); got != "Sonda/1.2.3 ( https://codeberg.org/squidd/sonda )" {
|
||||
t.Errorf("default contact: %q", got)
|
||||
}
|
||||
if got := UserAgent("1.2.3", " https://example.org/me "); got != "Sonda/1.2.3 ( https://example.org/me )" {
|
||||
t.Errorf("with a contact: %q", got)
|
||||
}
|
||||
if got := UserAgent("", ""); !strings.HasPrefix(got, "Sonda/dev ( ") {
|
||||
t.Errorf("no version: %q", got)
|
||||
}
|
||||
// Whatever is passed in, the header stays one line and keeps its shape.
|
||||
for _, evil := range []string{"a\r\nX-Evil: 1", "a ) b ( c", "\x00\x07"} {
|
||||
ua := UserAgent("1.0\r\nX: y", evil)
|
||||
if strings.ContainsAny(ua, "\r\n\x00") || strings.Count(ua, "(") != 1 || strings.Count(ua, ")") != 1 {
|
||||
t.Errorf("UserAgent(%q) = %q is not a safe single-line header", evil, ua)
|
||||
}
|
||||
}
|
||||
}
|
||||
81
internal/storage/providers.go
Normal file
81
internal/storage/providers.go
Normal file
|
|
@ -0,0 +1,81 @@
|
|||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
)
|
||||
|
||||
type providerRow struct {
|
||||
bun.BaseModel `bun:"table:provider_settings"`
|
||||
|
||||
Provider string `bun:"provider,pk"`
|
||||
Enabled bool `bun:"enabled,notnull"`
|
||||
APIKey string `bun:"api_key,notnull"`
|
||||
Contact string `bun:"contact,notnull"`
|
||||
UpdatedAt int64 `bun:"updated_at,notnull"`
|
||||
}
|
||||
|
||||
func (r providerRow) toSettings() providers.Settings {
|
||||
return providers.Settings{
|
||||
Provider: r.Provider, Enabled: r.Enabled, APIKey: r.APIKey, Contact: r.Contact,
|
||||
UpdatedAt: time.Unix(r.UpdatedAt, 0),
|
||||
}
|
||||
}
|
||||
|
||||
// ProviderRepo stores provider settings. It implements providers.Store.
|
||||
type ProviderRepo struct{ db *bun.DB }
|
||||
|
||||
// NewProviderRepo builds a ProviderRepo.
|
||||
func NewProviderRepo(db *bun.DB) *ProviderRepo { return &ProviderRepo{db: db} }
|
||||
|
||||
// Get returns the settings of one provider, with found=false if there are none.
|
||||
func (r *ProviderRepo) Get(ctx context.Context, name string) (providers.Settings, bool, error) {
|
||||
var row providerRow
|
||||
err := r.db.NewSelect().Model(&row).Where("provider = ?", name).Limit(1).Scan(ctx)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return providers.Settings{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return providers.Settings{}, false, fmt.Errorf("select provider settings: %w", err)
|
||||
}
|
||||
return row.toSettings(), true, nil
|
||||
}
|
||||
|
||||
// List returns every stored row.
|
||||
func (r *ProviderRepo) List(ctx context.Context) ([]providers.Settings, error) {
|
||||
var rows []providerRow
|
||||
if err := r.db.NewSelect().Model(&rows).OrderExpr("provider").Scan(ctx); err != nil {
|
||||
return nil, fmt.Errorf("list provider settings: %w", err)
|
||||
}
|
||||
out := make([]providers.Settings, len(rows))
|
||||
for i, row := range rows {
|
||||
out[i] = row.toSettings()
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Save inserts or updates the row of a provider.
|
||||
func (r *ProviderRepo) Save(ctx context.Context, s providers.Settings) error {
|
||||
row := &providerRow{
|
||||
Provider: s.Provider, Enabled: s.Enabled, APIKey: s.APIKey,
|
||||
Contact: s.Contact, UpdatedAt: s.UpdatedAt.Unix(),
|
||||
}
|
||||
_, err := r.db.NewInsert().Model(row).
|
||||
On("CONFLICT (provider) DO UPDATE").
|
||||
Set("enabled = EXCLUDED.enabled").
|
||||
Set("api_key = EXCLUDED.api_key").
|
||||
Set("contact = EXCLUDED.contact").
|
||||
Set("updated_at = EXCLUDED.updated_at").
|
||||
Exec(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("save provider settings: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
72
internal/storage/providers_test.go
Normal file
72
internal/storage/providers_test.go
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
)
|
||||
|
||||
func TestProviderRepoSaveGetList(t *testing.T) {
|
||||
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
|
||||
ctx := context.Background()
|
||||
repo := NewProviderRepo(db)
|
||||
|
||||
if _, found, err := repo.Get(ctx, providers.LastFM); err != nil || found {
|
||||
t.Fatalf("never configured: found=%v err=%v", found, err)
|
||||
}
|
||||
if rows, err := repo.List(ctx); err != nil || len(rows) != 0 {
|
||||
t.Fatalf("empty list: %v err=%v", rows, err)
|
||||
}
|
||||
|
||||
at := time.Unix(1_700_000_000, 0)
|
||||
want := providers.Settings{Provider: providers.LastFM, Enabled: true, APIKey: "0123456789abcdef0123456789abcdef", Contact: "x", UpdatedAt: at}
|
||||
if err := repo.Save(ctx, want); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, found, err := repo.Get(ctx, providers.LastFM)
|
||||
if err != nil || !found {
|
||||
t.Fatalf("Get: found=%v err=%v", found, err)
|
||||
}
|
||||
if got.Provider != want.Provider || !got.Enabled || got.APIKey != want.APIKey || got.Contact != "x" || !got.UpdatedAt.Equal(at) {
|
||||
t.Errorf("got %+v, want %+v", got, want)
|
||||
}
|
||||
|
||||
// Saving again updates the same row.
|
||||
want.Enabled, want.APIKey, want.Contact = false, "", ""
|
||||
want.UpdatedAt = at.Add(time.Hour)
|
||||
if err := repo.Save(ctx, want); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, _, _ = repo.Get(ctx, providers.LastFM)
|
||||
if got.Enabled || got.APIKey != "" || got.Contact != "" || !got.UpdatedAt.Equal(at.Add(time.Hour)) {
|
||||
t.Errorf("after the second save: %+v", got)
|
||||
}
|
||||
if n := count(t, db, "SELECT COUNT(*) FROM provider_settings"); n != 1 {
|
||||
t.Errorf("rows = %d, want 1: a save must update, not duplicate", n)
|
||||
}
|
||||
|
||||
_ = repo.Save(ctx, providers.Settings{Provider: providers.MusicBrainz, Enabled: true, UpdatedAt: at})
|
||||
rows, err := repo.List(ctx)
|
||||
if err != nil || len(rows) != 2 {
|
||||
t.Fatalf("list: %v err=%v", rows, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A provider that was never configured must not appear as enabled: the table
|
||||
// default is "off".
|
||||
func TestProviderTableDefaultsToOff(t *testing.T) {
|
||||
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
|
||||
if _, err := db.Exec("INSERT INTO provider_settings (provider, updated_at) VALUES ('lastfm', 1)"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, found, err := NewProviderRepo(db).Get(context.Background(), providers.LastFM)
|
||||
if err != nil || !found || got.Enabled || got.APIKey != "" || got.Contact != "" {
|
||||
t.Fatalf("defaults: %+v found=%v err=%v", got, found, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
@ -147,7 +147,7 @@ func TestMigrateCreatesSchemaAndIsIdempotent(t *testing.T) {
|
|||
if err := Migrate(context.Background(), db, fsys); err != nil {
|
||||
t.Fatalf("second Migrate: %v", err)
|
||||
}
|
||||
for _, table := range []string{"artists", "albums", "tracks", "track_artists", "audio_files", "users", "listening_events", "refresh_tokens", "libraries"} {
|
||||
for _, table := range []string{"artists", "albums", "tracks", "track_artists", "audio_files", "users", "listening_events", "refresh_tokens", "libraries", "provider_settings"} {
|
||||
if !tableExists(t, db, driver, table) {
|
||||
t.Errorf("table %s missing", table)
|
||||
}
|
||||
|
|
@ -156,8 +156,8 @@ func TestMigrateCreatesSchemaAndIsIdempotent(t *testing.T) {
|
|||
if err := db.QueryRow("SELECT COUNT(*) FROM schema_migrations").Scan(&applied); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if applied != 6 {
|
||||
t.Errorf("applied migrations = %d, want 6", applied)
|
||||
if applied != 7 {
|
||||
t.Errorf("applied migrations = %d, want 7", applied)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
|
|
|||
11
migrations/postgres/0007_provider_settings.sql
Normal file
11
migrations/postgres/0007_provider_settings.sql
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
-- Settings of the external providers (Last.fm, MusicBrainz). A provider with no
|
||||
-- row, or with enabled = false, is never contacted. The API key is stored as given
|
||||
-- and is never returned by the API.
|
||||
|
||||
CREATE TABLE provider_settings (
|
||||
provider TEXT PRIMARY KEY,
|
||||
enabled BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
api_key TEXT NOT NULL DEFAULT '',
|
||||
contact TEXT NOT NULL DEFAULT '', -- goes into the User-Agent; '' = the project address
|
||||
updated_at BIGINT NOT NULL -- unix seconds
|
||||
);
|
||||
11
migrations/sqlite/0007_provider_settings.sql
Normal file
11
migrations/sqlite/0007_provider_settings.sql
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
-- Settings of the external providers (Last.fm, MusicBrainz). A provider with no
|
||||
-- row, or with enabled = 0, is never contacted. The API key is stored as given
|
||||
-- and is never returned by the API.
|
||||
|
||||
CREATE TABLE provider_settings (
|
||||
provider TEXT PRIMARY KEY,
|
||||
enabled INTEGER NOT NULL DEFAULT 0,
|
||||
api_key TEXT NOT NULL DEFAULT '',
|
||||
contact TEXT NOT NULL DEFAULT '', -- goes into the User-Agent; '' = the project address
|
||||
updated_at INTEGER NOT NULL -- unix seconds
|
||||
);
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
# Copy to sonda.toml and adjust. sonda.toml is git-ignored.
|
||||
# Precedence: defaults < this file < environment variables
|
||||
# (SONDA_ENV, SONDA_LISTEN, SONDA_DB_DRIVER, SONDA_DB_PATH, SONDA_DB_DSN,
|
||||
# SONDA_JWT_SECRET, SONDA_LASTFM_API_KEY).
|
||||
# SONDA_JWT_SECRET).
|
||||
|
||||
[server]
|
||||
listen = "127.0.0.1:8080" # localhost by default
|
||||
|
|
@ -26,5 +26,4 @@ refresh_ttl_hours = 720 # lifetime of a refresh token (30 days)
|
|||
[listening]
|
||||
min_listen_seconds = 20 # a play counts as a listen after this long, or when it completes
|
||||
|
||||
[providers.lastfm]
|
||||
api_key = ""
|
||||
# Last.fm and MusicBrainz are set up through the API, not here: PUT /api/v1/providers/{name}.
|
||||
|
|
|
|||
Loading…
Reference in a new issue