feat(discovery): artist tags from Last.fm and MusicBrainz, and ids from my files

- feat: GET /api/v1/artists/{id}/tags lists, for each provider that is on, the
  tags it gives to an artist with its own weights, kept as they came: Last.fm's
  are relative to the artist, MusicBrainz's are counts of votes, and they are
  never mixed. MusicBrainz also marks the tags it lists as genres. The same
  rules as similar artists apply: asked only on demand, kept 7 days (1 for an
  unknown artist), old data served when a provider fails, and one call for
  simultaneous requests. A provider that is off, failing or lacking an id does
  not hide the other
- feat: a thin MusicBrainz client that reads an artist's tags and genres by its
  id. It never searches by name, because a search returns look-alikes and
  picking one would be a guess. The id is checked before it reaches the path,
  and errors never repeat the provider's text
- feat: the User-Agent can change while the server runs, so the contact an
  admin sets for MusicBrainz applies to the next request; if the settings
  cannot be read it falls back to the project address
- feat: Last.fm top tags, by id when known and by name otherwise, never
  autocorrected
- feat: the scan reads MusicBrainz artist ids from the file tags, offline, and
  keeps them on the artists. An id is only used when it can be paired with its
  artist for certain: the same number of ids as artists, all valid. The same id
  is the same artist whatever the spelling, an artist without an id takes the
  one its file gives, and another id under the same name is another artist. A
  file that confirms the id a provider gave to a discovered artist keeps it
- feat: POST /providers/musicbrainz/test checks that the service answers
- test: what leaves the server for MusicBrainz is checked from the other end:
  the id of the artist and the admin's contact, nothing from the library;
  removing the safeguards on ids and pairing makes tests fail
- docs: README, OpenAPI and the provider text updated
This commit is contained in:
tgiachi 2026-11-11 17:01:43 +01:00
commit 07ea997cd0
29 changed files with 2099 additions and 62 deletions

View file

@ -15,11 +15,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights. See [External providers](#external-providers).
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
## What is not there yet
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. MusicBrainz can be set up (see [External providers](#external-providers)) but nothing asks it anything yet.
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Genres from your own files, and the removal of provider data on request.
## Requirements
@ -176,7 +177,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
## External providers
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today only Last.fm is used, to find similar artists.
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, and MusicBrainz for tags.
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
@ -188,14 +189,19 @@ curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"api_key":"<your 32 charac
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"enabled":true}'
curl -s $BASE/providers -H "$AUTH" | jq
curl -s -X POST $BASE/providers/lastfm/test -H "$AUTH" | jq # does the key work?
curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"contact":"https://example.org/me"}'
```
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
**Tags.** `GET /api/v1/artists/{id}/tags` asks each provider that is on, at the same time, and keeps the answers with the same rules (7 days, 1 for an unknown artist, old data if a provider fails). Each provider has its own entry with its own weights: Last.fm's are relative to the artist (0 to 100), MusicBrainz's are counts of votes, and they are never mixed. MusicBrainz also marks the tags it lists as genres. Its tags and genres are CC BY-NC-SA, so show the `attribution` that comes with them.
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. The scan reads a file's tags when the file is new or changed, so to pick up ids for music that is already indexed, the files have to be touched or re-added.
The name of the artist is sent exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
## The API at a glance
@ -211,6 +217,7 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
| anyone | `POST /auth/login`, `/auth/refresh`, `/auth/logout` | Open, renew and end a session. |
| signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). |
| signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. |
| signed in | `GET /artists/{id}/tags` | What Last.fm and MusicBrainz say about the artist's kind of music, from each provider that is on. |
| signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 50). |
| signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. |
| signed in | `POST /listens` | Record a `play`, `complete` or `skip`; the answer says whether it counts as a listen. |
@ -266,7 +273,7 @@ internal/auth/ users, roles, passwords, access and refresh tokens
internal/catalog/ browsing and searching artists, albums, tracks
internal/library/ libraries, the scanner, the event hub, the tag reader
internal/discovery/ similar artists: what the providers said, kept and refreshed
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm client
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm and MusicBrainz clients
internal/listening/ listen events
internal/storage/ SQLite and PostgreSQL through bun, the repositories
internal/streaming/ opening a track's file safely

View file

@ -23,6 +23,7 @@ import (
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
"codeberg.org/squidd/sonda/internal/providers/outbound"
"codeberg.org/squidd/sonda/internal/storage"
"codeberg.org/squidd/sonda/internal/streaming"
@ -37,6 +38,9 @@ const version = "0.1.0"
// lastfmMinInterval spaces the requests to Last.fm well below its limit.
const lastfmMinInterval = 250 * time.Millisecond
// musicbrainzMinInterval keeps a little above MusicBrainz's rule of one request a second.
const musicbrainzMinInterval = 1100 * time.Millisecond
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "sondad:", err)
@ -114,7 +118,21 @@ func run() error {
MinInterval: lastfmMinInterval,
}), lastfm.BaseURL)
providerSettings.SetChecker(providers.LastFM, lastfmClient.Probe)
discoveries := discovery.NewService(storage.NewDiscoveryRepo(db), lastfmClient, providerSettings, logger)
// MusicBrainz wants a contact in the User-Agent, which an admin can change: it
// is read again for every request.
musicbrainzClient := musicbrainz.New(outbound.New(outbound.Options{
UserAgentFunc: providerSettings.UserAgentFor(providers.MusicBrainz, version),
MinInterval: musicbrainzMinInterval,
}), musicbrainz.BaseURL)
providerSettings.SetChecker(providers.MusicBrainz, func(ctx context.Context, _ string) error {
return musicbrainzClient.Probe(ctx)
})
discoveryRepo := storage.NewDiscoveryRepo(db)
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
providers.LastFM: discovery.LastfmTags{Client: lastfmClient},
providers.MusicBrainz: discovery.MusicBrainzTags{Client: musicbrainzClient},
}, providerSettings, logger)
router := httpapi.NewRouter(httpapi.Deps{
Logger: logger,
@ -126,6 +144,7 @@ func run() error {
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
Providers: providerSettings,
Discovery: discoveries,
Tags: tags,
Streams: streaming.NewService(libraryRepo),
})

View file

@ -85,6 +85,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
Providers: prov,
Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()),
Tags: discovery.NewTagService(storage.NewDiscoveryRepo(db), map[string]discovery.TagSource{}, prov, zap.NewNop()),
Streams: streaming.NewService(noFiles{}),
}
mutate(&deps, db)

View file

@ -98,12 +98,17 @@ type contract struct {
func newContract(t *testing.T) *contract {
t.Helper()
reader := &contractReader{}
tagsLF := &fakeTags{tags: []discovery.Tag{{Name: "rock", Weight: 100}}}
tagsMB := &fakeTags{tags: []discovery.Tag{{Name: "grunge", Weight: 12, Genre: true}}}
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
d.Providers = prov
prov.SetChecker(providers.MusicBrainz, func(context.Context, string) error { return nil })
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), lf, prov, zap.NewNop())
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
map[string]discovery.TagSource{providers.LastFM: tagsLF, providers.MusicBrainz: tagsMB}, prov, zap.NewNop())
repo := storage.NewLibraryRepo(db)
mgr := library.NewManager(library.NewScanner(repo, reader, library.NewHub(), 2), repo)
libs := library.NewLibraries(repo)
@ -391,7 +396,9 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
c.expect(c.call("PUT", "/providers/spotify", token, map[string]any{"enabled": true}), 404, "unknown provider", "provider_not_found")
c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test without a key")
c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 400, "test a provider that has no test", "invalid_input")
if mb := c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 200, "test musicbrainz"); mb["ok"] != true {
t.Fatalf("musicbrainz test = %v", mb)
}
c.expect(c.call("POST", "/providers/spotify/test", token, nil), 404, "test an unknown provider", "provider_not_found")
c.expect(c.call("POST", "/providers/lastfm/test", "", nil), 401, "test without a token", "unauthorized")
off := c.expect(c.call("GET", "/artists/"+artistID+"/similar", token, nil), 200, "similar artists with the provider off")
@ -415,6 +422,12 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.call("GET", "/artists/999/similar", token, nil), 404, "similar to an unknown artist", "artist_not_found")
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
tags := c.expect(c.call("GET", "/artists/"+artistID+"/tags", token, nil), 200, "tags")["sources"].([]any)
if len(tags) != 2 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" {
t.Fatalf("tags = %v, want Last.fm ok and MusicBrainz without an id", tags)
}
c.expect(c.call("GET", "/artists/999/tags", token, nil), 404, "tags of an unknown artist", "artist_not_found")
c.expect(c.call("GET", "/artists/abc/tags", token, nil), 400, "tags of a bad id", "invalid_input")
// ---- playback ----
stream := "/tracks/" + trackID + "/stream"

View file

@ -75,3 +75,54 @@ func (h *discoveryHandlers) similar(c *gin.Context) {
c.JSON(http.StatusOK, toSimilarJSON(res))
}
}
// tagsHandlers serve the tags that providers gave to an artist.
type tagsHandlers struct{ svc tagService }
type tagJSON struct {
Name string `json:"name"`
Weight float64 `json:"weight"`
Genre bool `json:"genre"`
}
type tagSourceJSON struct {
Provider string `json:"provider"`
ProviderStatus string `json:"provider_status"`
Stale bool `json:"stale"`
RetrievedAt *string `json:"retrieved_at"`
Attribution string `json:"attribution"`
Tags []tagJSON `json:"tags"`
}
func (h *tagsHandlers) tags(c *gin.Context) {
id, ok := pathID(c)
if !ok {
return
}
res, err := h.svc.Tags(c.Request.Context(), id)
switch {
case errors.Is(err, discovery.ErrNotFound):
writeError(c, http.StatusNotFound, "artist_not_found", "not found")
case errors.Is(err, context.Canceled):
c.AbortWithStatus(statusClientClosed)
case err != nil:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
default:
sources := make([]tagSourceJSON, len(res.Sources))
for i, s := range res.Sources {
out := tagSourceJSON{
Provider: s.Provider, ProviderStatus: s.Status, Stale: s.Stale,
Attribution: s.Attribution, Tags: make([]tagJSON, len(s.Tags)),
}
if !s.RetrievedAt.IsZero() {
out.RetrievedAt = rfc3339(s.RetrievedAt)
}
for j, t := range s.Tags {
out.Tags[j] = tagJSON{Name: t.Name, Weight: t.Weight, Genre: t.Genre}
}
sources[i] = out
}
c.JSON(http.StatusOK, gin.H{"sources": sources})
}
}

View file

@ -61,6 +61,10 @@ type providerService interface {
Test(ctx context.Context, name string) (providers.TestResult, error)
}
type tagService interface {
Tags(ctx context.Context, artistID int64) (discovery.TagResult, error)
}
type discoveryService interface {
Similar(ctx context.Context, artistID int64, limit int) (discovery.Result, error)
}
@ -92,6 +96,7 @@ type Deps struct {
Catalog catalogService
Providers providerService
Discovery discoveryService
Tags tagService
Streams streamService
}
@ -128,6 +133,8 @@ func NewRouter(d Deps) *gin.Engine {
protected.GET("/artists/:id", cat.getArtist)
disc := &discoveryHandlers{svc: d.Discovery}
protected.GET("/artists/:id/similar", disc.similar)
tg := &tagsHandlers{svc: d.Tags}
protected.GET("/artists/:id/tags", tg.tags)
protected.GET("/albums", cat.listAlbums)
protected.GET("/albums/:id", cat.getAlbum)
protected.GET("/tracks", cat.listTracks)

View file

@ -67,6 +67,12 @@ type similarEnv struct {
// database that holds two local artists, Alpha and Beta, with Last.fm switched
// off and without a key.
func newSimilarEnv(t *testing.T) *similarEnv {
t.Helper()
return newSimilarEnvWith(t, func(*Deps, *bun.DB, *providers.Service) {})
}
// newSimilarEnvWith is newSimilarEnv that lets a test add more to Deps.
func newSimilarEnvWith(t *testing.T, more func(*Deps, *bun.DB, *providers.Service)) *similarEnv {
t.Helper()
src := &fakeLastfm{result: []lastfm.Similar{
{Name: "Beta", Match: 1},
@ -78,9 +84,16 @@ func newSimilarEnv(t *testing.T) *similarEnv {
d.Providers = prov
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
more(d, db, prov)
})
e.admin = registerAndLogin(t, e.h)
e.seedLibrary(t)
return e
}
// seedLibrary adds a library with the local artists Alpha and Beta.
func (e *similarEnv) seedLibrary(t *testing.T) {
t.Helper()
lib := storage.NewLibraryRepo(e.db)
l, err := lib.CreateLibrary(context.Background(), "music", "/lib")
if err != nil {
@ -99,7 +112,6 @@ func newSimilarEnv(t *testing.T) *similarEnv {
}
}
e.alpha, e.beta = e.artist(t, "Alpha"), e.artist(t, "Beta")
return e
}
func (e *similarEnv) artist(t *testing.T, name string) int64 {

View file

@ -0,0 +1,234 @@
package httpapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"sync"
"testing"
"github.com/uptrace/bun"
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
"codeberg.org/squidd/sonda/internal/providers/outbound"
"codeberg.org/squidd/sonda/internal/storage"
)
const alphaMBID = "aaaaaaaa-0000-4000-8000-0000000000a1"
// fakeTags answers for a provider and remembers what it was asked.
type fakeTags struct {
mu sync.Mutex
calls int
artist discovery.Artist
tags []discovery.Tag
err error
}
func (f *fakeTags) ArtistTags(_ context.Context, _ string, a discovery.Artist) ([]discovery.Tag, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls++
f.artist = a
return f.tags, f.err
}
type tagsEnv struct {
*similarEnv
lf, mb *fakeTags
}
func newTagsEnv(t *testing.T) *tagsEnv {
t.Helper()
e := &tagsEnv{
lf: &fakeTags{tags: []discovery.Tag{{Name: "rock", Weight: 100}, {Name: "grunge", Weight: 40}}},
mb: &fakeTags{tags: []discovery.Tag{{Name: "alternative rock", Weight: 43, Genre: true}}},
}
e.similarEnv = newSimilarEnvWith(t, func(d *Deps, db *bun.DB, prov *providers.Service) {
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
map[string]discovery.TagSource{providers.LastFM: e.lf, providers.MusicBrainz: e.mb}, prov, zap.NewNop())
})
return e
}
func (e *tagsEnv) path(id int64) string {
return "/api/v1/artists/" + strconv.FormatInt(id, 10) + "/tags"
}
func (e *tagsEnv) enableMusicBrainz(t *testing.T) {
t.Helper()
if r := do(t, e.h, "PUT", "/api/v1/providers/musicbrainz", e.admin, map[string]any{"enabled": true}); r.Code != http.StatusOK {
t.Fatalf("enable musicbrainz: %d %v", r.Code, r.Body)
}
}
func TestTagsNeedASignedInUserAndAnArtist(t *testing.T) {
e := newTagsEnv(t)
if r := do(t, e.h, "GET", e.path(e.alpha), "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("no token: %d", r.Code)
}
if r := do(t, e.h, "GET", e.path(99999), e.admin, nil); r.Code != http.StatusNotFound || errCode(r) != "artist_not_found" {
t.Errorf("unknown artist: %d %v", r.Code, r.Body)
}
if r := do(t, e.h, "GET", "/api/v1/artists/abc/tags", e.admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
t.Errorf("bad id: %d %v", r.Code, r.Body)
}
}
func TestTagsWithEveryProviderOffAskNobody(t *testing.T) {
e := newTagsEnv(t)
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if r.Code != http.StatusOK {
t.Fatalf("%d %v", r.Code, r.Body)
}
sameKeys(t, "tags", r.Body, "sources")
sources := r.Body["sources"].([]any)
if len(sources) != 2 {
t.Fatalf("sources = %v", sources)
}
for i, name := range []string{"lastfm", "musicbrainz"} {
s := sources[i].(map[string]any)
sameKeys(t, "source", s, "provider", "provider_status", "stale", "retrieved_at", "attribution", "tags")
if s["provider"] != name || s["provider_status"] != "disabled" || s["stale"] != false || len(s["tags"].([]any)) != 0 {
t.Errorf("%s = %v", name, s)
}
if v, present := s["retrieved_at"]; !present || v != nil {
t.Errorf("%s retrieved_at = %#v, want an explicit null", name, v)
}
}
if e.lf.calls+e.mb.calls != 0 {
t.Error("a provider that is off was contacted")
}
}
func TestTagsFromEachProvider(t *testing.T) {
e := newTagsEnv(t)
e.enableLastfm(t)
e.enableMusicBrainz(t)
// Alpha has no id yet: Last.fm is asked by name, MusicBrainz cannot be asked.
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
lf, mb := r.Body["sources"].([]any)[0].(map[string]any), r.Body["sources"].([]any)[1].(map[string]any)
if lf["provider_status"] != "ok" || lf["retrieved_at"] == nil || len(lf["tags"].([]any)) != 2 {
t.Fatalf("lastfm = %v", lf)
}
first := lf["tags"].([]any)[0].(map[string]any)
sameKeys(t, "tag", first, "name", "weight", "genre")
if first["name"] != "rock" || first["weight"] != float64(100) || first["genre"] != false {
t.Errorf("first tag = %v", first)
}
if mb["provider_status"] != "no_id" || len(mb["tags"].([]any)) != 0 || e.mb.calls != 0 {
t.Errorf("musicbrainz = %v calls = %d: it must not be asked without an id", mb, e.mb.calls)
}
if !strings.Contains(lf["attribution"].(string), "Last.fm") || !strings.Contains(mb["attribution"].(string), "MusicBrainz") {
t.Errorf("attributions = %v / %v", lf["attribution"], mb["attribution"])
}
// Once the id is known (from the file tags), MusicBrainz is asked with it.
if _, err := e.db.Exec("UPDATE artists SET mbid = ? WHERE id = ?", alphaMBID, e.alpha); err != nil {
t.Fatal(err)
}
r = do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
mb = r.Body["sources"].([]any)[1].(map[string]any)
tag := mb["tags"].([]any)[0].(map[string]any)
if mb["provider_status"] != "ok" || tag["name"] != "alternative rock" || tag["genre"] != true || tag["weight"] != float64(43) {
t.Fatalf("musicbrainz = %v", mb)
}
if e.mb.artist.MBID != alphaMBID {
t.Errorf("musicbrainz was asked about %+v", e.mb.artist)
}
raw, _ := json.Marshal(r.Body)
if strings.Contains(string(raw), lastfmKey) {
t.Error("the answer contains the API key")
}
// A second look is served from the database.
callsBefore := e.lf.calls + e.mb.calls
do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if e.lf.calls+e.mb.calls != callsBefore {
t.Error("fresh tags were asked for again")
}
}
func TestTagsWhenAProviderFailsAreStillAnAnswer(t *testing.T) {
e := newTagsEnv(t)
e.enableLastfm(t)
e.lf.err = lastfm.ErrUnavailable
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if r.Code != http.StatusOK || r.Body["sources"].([]any)[0].(map[string]any)["provider_status"] != "error" {
t.Fatalf("%d %v", r.Code, r.Body)
}
}
// What leaves the server for MusicBrainz, seen from the other end: the id of the
// artist and a User-Agent that carries the contact the admin chose, and nothing
// from the library.
func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
var (
mu sync.Mutex
seen []string
hdr http.Header
)
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
seen = append(seen, r.URL.RequestURI())
hdr = r.Header.Clone()
mu.Unlock()
_, _ = w.Write([]byte(`{"id":"` + alphaMBID + `","name":"Alpha","tags":[{"name":"rock","count":3}]}`))
}))
defer remote.Close()
e := &similarEnv{}
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
d.Providers = prov
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
mbClient := musicbrainz.New(outbound.New(outbound.Options{UserAgentFunc: prov.UserAgentFor(providers.MusicBrainz, "1.2.3")}), remote.URL+"/ws/2/")
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
map[string]discovery.TagSource{providers.MusicBrainz: discovery.MusicBrainzTags{Client: mbClient}}, prov, zap.NewNop())
})
e.admin = registerAndLogin(t, e.h)
e.seedLibrary(t)
if _, err := e.db.Exec("UPDATE artists SET mbid = ? WHERE id = ?", alphaMBID, e.alpha); err != nil {
t.Fatal(err)
}
if r := do(t, e.h, "PUT", "/api/v1/providers/musicbrainz", e.admin, map[string]any{"enabled": true, "contact": "https://example.org/me"}); r.Code != http.StatusOK {
t.Fatal(r.Body)
}
r := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(e.alpha, 10)+"/tags", e.admin, nil)
mb := r.Body["sources"].([]any)[1].(map[string]any)
if r.Code != http.StatusOK || mb["provider_status"] != "ok" {
t.Fatalf("%d %v", r.Code, r.Body)
}
mu.Lock()
defer mu.Unlock()
if len(seen) != 1 || seen[0] != "/ws/2/artist/"+alphaMBID+"?inc=tags+genres&fmt=json" {
t.Fatalf("requests = %v", seen)
}
if hdr.Get("User-Agent") != "Sonda/1.2.3 ( https://example.org/me )" {
t.Errorf("User-Agent = %q", hdr.Get("User-Agent"))
}
for name := range hdr {
switch http.CanonicalHeaderKey(name) {
case "User-Agent", "Accept", "Accept-Encoding":
default:
t.Errorf("unexpected header %s", name)
}
}
everything := strings.ToLower(seen[0] + hdr.Get("User-Agent"))
for _, secret := range []string{"alice", "alpha", "/lib", "flac"} {
if strings.Contains(everything, secret) {
t.Errorf("the request mentions %q: %s", secret, everything)
}
}
}

View file

@ -597,6 +597,41 @@ paths:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/artists/{id}/tags:
parameters:
- {$ref: '#/components/parameters/Id'}
get:
tags: [Discovery]
operationId: getArtistTags
summary: Tags of an artist, from each provider
description: |
What each provider says kind of music this artist plays: Last.fm's top tags and
MusicBrainz's tags and genres. There is always one entry for each, Last.fm first. The
weights are the providers' own numbers, kept as they came and never mixed: Last.fm's is
relative to the artist (0 to 100, the top tag is usually 100), MusicBrainz's is a count of
votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres.
A provider is only asked when an admin switched it on and its stored answer is missing or
old (7 days; a "no such artist" answer is kept 1 day), with the same rules as
`GET /artists/{id}/similar`. What is sent is the artist's name or MusicBrainz id to
Last.fm, and only the MusicBrainz id to MusicBrainz: Sonda never looks an id up by name,
so for an artist whose id is not known (it comes from the tags in the music files) the
MusicBrainz entry says `no_id`. Show each `attribution` where you show the tags.
security: [{bearerAuth: []}]
responses:
'200':
description: One entry per provider.
content:
application/json:
schema: {$ref: '#/components/schemas/ArtistTags'}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: No such artist (`artist_not_found`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/albums:
get:
tags: [Catalog]
@ -883,7 +918,8 @@ paths:
sends nothing but the key. A provider that cannot be reached or refuses the key is not an
error here: the answer is `200` with `ok` false and the reason in `status`.
Only Last.fm can be tested so far; asking for MusicBrainz is a `400`.
MusicBrainz has no key: its test only asks whether the service answers. A provider that
has no test is a `400`.
security: [{bearerAuth: []}]
responses:
'200':
@ -1365,6 +1401,45 @@ components:
`no_api_key`: none is stored, so nobody was asked. `key_refused`: the provider does not accept
it. `rate_limited`: try again later. `unreachable`: the provider did not answer. `error`: anything else.
ArtistTags:
type: object
additionalProperties: false
required: [sources]
properties:
sources:
type: array
items:
type: object
additionalProperties: false
required: [provider, provider_status, stale, retrieved_at, attribution, tags]
properties:
provider: {type: string, enum: [lastfm, musicbrainz]}
provider_status:
type: string
enum: [ok, disabled, error, no_id]
description: |
`ok`: the data is as fresh as it needs to be. `disabled`: the provider is off or has no
key, so it was not asked. `error`: it was asked and failed. `no_id`: it needs the
MusicBrainz id of the artist, which is not known, so it was not asked.
stale: {type: boolean, description: "True when `tags` are older than they should be because the provider could not be asked."}
retrieved_at:
type: string
format: date-time
nullable: true
description: When the provider was last read for this artist; null if never.
attribution: {type: string, description: Credit to show where the tags are shown.}
tags:
type: array
description: The heaviest tags first, at most 50.
items:
type: object
additionalProperties: false
required: [name, weight, genre]
properties:
name: {type: string, description: Lower case.}
weight: {type: number, description: "The provider's own number; see the operation."}
genre: {type: boolean}
SimilarArtists:
type: object
additionalProperties: false

View file

@ -213,11 +213,16 @@ func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Resul
}
func (s *Service) fresh(snap Snapshot) bool {
return fresh(s.now(), snap.RetrievedAt, snap.NotFound)
}
// fresh says whether what was read at retrievedAt is still believed.
func fresh(now, retrievedAt time.Time, notFound bool) bool {
ttl := PositiveTTL
if snap.NotFound {
if notFound {
ttl = NegativeTTL
}
return s.now().Sub(snap.RetrievedAt) < ttl
return now.Sub(retrievedAt) < ttl
}
// refresh reads the provider and stores what it said.

250
internal/discovery/tags.go Normal file
View file

@ -0,0 +1,250 @@
package discovery
import (
"context"
"errors"
"fmt"
"sort"
"sync"
"time"
"go.uber.org/zap"
"golang.org/x/sync/singleflight"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
)
// MaxTags is how many tags are kept for an artist from one provider, the most
// weighty ones. Beyond that they are noise.
const MaxTags = 50
// StatusNoID says a provider that needs the MusicBrainz id of the artist cannot
// be asked, because the id is not known. Sonda never looks one up by name.
const StatusNoID = "no_id"
// ErrSourceNotFound is what a TagSource returns when the provider does not know
// the artist.
var ErrSourceNotFound = errors.New("the provider does not know this artist")
// Tag is one tag of an artist. Weight is the provider's own number, kept as it
// came: Last.fm's is relative to the artist (0 to 100), MusicBrainz's is an
// absolute count of votes, so weights of different providers are never compared.
// Genre is set when the provider also lists the tag among its curated genres.
type Tag struct {
Name string
Weight float64
Genre bool
}
// TagSnapshot is what is stored for one artist and one provider.
type TagSnapshot struct {
Fetched bool
NotFound bool
RetrievedAt time.Time
Tags []Tag
}
// TagSourceResult is what one provider says about an artist. RetrievedAt is zero
// when nothing is known.
type TagSourceResult struct {
Provider string
Status string // StatusOK, StatusDisabled, StatusError or StatusNoID
Stale bool
RetrievedAt time.Time
Attribution string
Tags []Tag
}
// TagResult has one entry per provider, always in the same order.
type TagResult struct{ Sources []TagSourceResult }
// TagStore keeps the tags. Artist returns ErrNotFound for an id the catalog does
// not know.
type TagStore interface {
Artist(ctx context.Context, id int64) (Artist, error)
Tags(ctx context.Context, id int64, provider string) (TagSnapshot, error)
SaveTags(ctx context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error
}
// TagSource asks one provider for the tags of an artist. It returns
// ErrSourceNotFound when the provider does not know the artist.
type TagSource interface {
ArtistTags(ctx context.Context, key string, a Artist) ([]Tag, error)
}
type tagProvider struct {
name string
needsID bool
attribution string
}
// tagProviders is the fixed order the providers are listed in.
var tagProviders = []tagProvider{
{providers.LastFM, false, "Tags from Last.fm (https://www.last.fm), for non-commercial use."},
{providers.MusicBrainz, true, "Tags and genres from MusicBrainz (https://musicbrainz.org), CC BY-NC-SA 3.0: non-commercial, with attribution."},
}
// TagService answers "what kind of music does this artist play", from every
// provider that is on, with the same rules as Service: only on demand, kept for
// 7 days (1 for an unknown artist), old data served when a provider fails, and
// one call for any number of simultaneous requests.
type TagService struct {
store TagStore
sources map[string]TagSource
settings Settings
log *zap.Logger
now func() time.Time
flight singleflight.Group
}
// NewTagService builds a TagService. A provider without an entry in sources is
// reported as disabled.
func NewTagService(store TagStore, sources map[string]TagSource, settings Settings, log *zap.Logger) *TagService {
if log == nil {
log = zap.NewNop()
}
return &TagService{store: store, sources: sources, settings: settings, log: log, now: time.Now}
}
// Tags returns what each provider says about an artist. A provider that is off,
// cannot be asked or fails is not an error: its entry says so. The providers are
// asked at the same time.
func (s *TagService) Tags(ctx context.Context, artistID int64) (TagResult, error) {
if artistID <= 0 {
return TagResult{}, ErrNotFound
}
artist, err := s.store.Artist(ctx, artistID)
if err != nil {
return TagResult{}, err
}
out := make([]TagSourceResult, len(tagProviders))
errs := make([]error, len(tagProviders))
var wg sync.WaitGroup
for i, p := range tagProviders {
wg.Add(1)
go func() {
defer wg.Done()
out[i], errs[i] = s.one(ctx, artist, p)
}()
}
wg.Wait()
for _, err := range errs {
if err != nil {
return TagResult{}, err
}
}
return TagResult{Sources: out}, nil
}
func (s *TagService) one(ctx context.Context, a Artist, p tagProvider) (TagSourceResult, error) {
snap, err := s.store.Tags(ctx, a.ID, p.name)
if err != nil {
return TagSourceResult{}, err
}
res := func(status string, stale bool, snap TagSnapshot) TagSourceResult {
r := TagSourceResult{Provider: p.name, Status: status, Stale: stale && snap.Fetched, Attribution: p.attribution, Tags: snap.Tags}
if r.Tags == nil {
r.Tags = []Tag{}
}
if snap.Fetched {
r.RetrievedAt = snap.RetrievedAt
}
return r
}
if snap.Fetched && fresh(s.now(), snap.RetrievedAt, snap.NotFound) {
return res(StatusOK, false, snap), nil
}
src := s.sources[p.name]
cfg, active, err := s.settings.Active(ctx, p.name)
if err != nil {
return TagSourceResult{}, err
}
if !active || src == nil {
return res(StatusDisabled, snap.Fetched, snap), nil
}
if p.needsID && a.MBID == "" {
return res(StatusNoID, snap.Fetched, snap), nil
}
done := s.flight.DoChan(fmt.Sprintf("tags/%s/%d", p.name, a.ID), func() (_ any, err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("reading tags panicked: %v", r)
}
}()
return nil, s.refresh(src, cfg.APIKey, a, p.name)
})
select {
case <-ctx.Done():
return TagSourceResult{}, ctx.Err()
case r := <-done:
if r.Err != nil {
s.log.Warn("tags could not be read", zap.String("provider", p.name), zap.Error(r.Err))
return res(StatusError, snap.Fetched, snap), nil
}
}
snap, err = s.store.Tags(ctx, a.ID, p.name)
if err != nil {
return TagSourceResult{}, err
}
return res(StatusOK, false, snap), nil
}
// refresh reads one provider and stores what it said.
func (s *TagService) refresh(src TagSource, key string, a Artist, provider string) error {
ctx, cancel := context.WithTimeout(context.Background(), fetchTimeout)
defer cancel()
tags, err := src.ArtistTags(ctx, key, a)
switch {
case errors.Is(err, ErrSourceNotFound):
tags = nil
case err != nil:
return err
}
sort.SliceStable(tags, func(i, j int) bool { return tags[i].Weight > tags[j].Weight })
if len(tags) > MaxTags {
tags = tags[:MaxTags]
}
return s.store.SaveTags(ctx, a.ID, provider, s.now(), len(tags) > 0, tags)
}
// LastfmTags adapts the Last.fm client to a TagSource.
type LastfmTags struct{ Client *lastfm.Client }
// ArtistTags implements TagSource.
func (l LastfmTags) ArtistTags(ctx context.Context, key string, a Artist) ([]Tag, error) {
got, err := l.Client.TopTags(ctx, key, lastfm.Artist{Name: a.Name, MBID: a.MBID})
if errors.Is(err, lastfm.ErrNotFound) {
return nil, ErrSourceNotFound
}
if err != nil {
return nil, err
}
out := make([]Tag, len(got))
for i, t := range got {
out[i] = Tag{Name: t.Name, Weight: float64(t.Count)}
}
return out, nil
}
// MusicBrainzTags adapts the MusicBrainz client to a TagSource. It only ever
// asks by the id of the artist.
type MusicBrainzTags struct{ Client *musicbrainz.Client }
// ArtistTags implements TagSource.
func (m MusicBrainzTags) ArtistTags(ctx context.Context, _ string, a Artist) ([]Tag, error) {
got, err := m.Client.ArtistTags(ctx, a.MBID)
if errors.Is(err, musicbrainz.ErrNotFound) {
return nil, ErrSourceNotFound
}
if err != nil {
return nil, err
}
out := make([]Tag, len(got))
for i, t := range got {
out[i] = Tag{Name: t.Name, Weight: float64(t.Count), Genre: t.Genre}
}
return out, nil
}

View file

@ -0,0 +1,295 @@
package discovery
import (
"context"
"errors"
"sync"
"testing"
"time"
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/providers"
)
type savedTags struct {
artistID int64
provider string
at time.Time
found bool
tags []Tag
}
// tagStore keeps one snapshot per artist and provider.
type tagStore struct {
mu sync.Mutex
artists map[int64]Artist
snaps map[string]TagSnapshot
saves []savedTags
}
func newTagStore() *tagStore {
return &tagStore{
artists: map[int64]Artist{
1: {ID: 1, Name: "Source"},
2: {ID: 2, Name: "With Id", MBID: "aaaaaaaa-0000-4000-8000-000000000001"},
},
snaps: map[string]TagSnapshot{},
}
}
func tkey(id int64, provider string) string { return provider + "/" + string(rune('0'+id)) }
func (f *tagStore) Artist(_ context.Context, id int64) (Artist, error) {
f.mu.Lock()
defer f.mu.Unlock()
a, ok := f.artists[id]
if !ok {
return Artist{}, ErrNotFound
}
return a, nil
}
func (f *tagStore) Tags(_ context.Context, id int64, provider string) (TagSnapshot, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.snaps[tkey(id, provider)], nil
}
func (f *tagStore) SaveTags(_ context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error {
f.mu.Lock()
defer f.mu.Unlock()
f.saves = append(f.saves, savedTags{id, provider, at, found, tags})
f.snaps[tkey(id, provider)] = TagSnapshot{Fetched: true, NotFound: !found, RetrievedAt: at, Tags: tags}
return nil
}
type fakeTagSource struct {
mu sync.Mutex
calls int
key string
artist Artist
tags []Tag
err error
release chan struct{}
boom any
}
func (s *fakeTagSource) ArtistTags(_ context.Context, key string, a Artist) ([]Tag, error) {
s.mu.Lock()
s.calls++
s.key, s.artist = key, a
release := s.release
s.mu.Unlock()
if release != nil {
<-release
}
if s.boom != nil {
panic(s.boom)
}
return s.tags, s.err
}
func (s *fakeTagSource) count() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.calls
}
// perProvider answers Active for each provider on its own.
type perProvider map[string]bool
func (p perProvider) Active(_ context.Context, name string) (providers.Settings, bool, error) {
return providers.Settings{Provider: name, Enabled: p[name], APIKey: key}, p[name], nil
}
type tagRig struct {
svc *TagService
store *tagStore
lf *fakeTagSource
mb *fakeTagSource
on perProvider
now time.Time
}
func newTagRig(lastfm, musicbrainz bool) *tagRig {
r := &tagRig{
store: newTagStore(), lf: &fakeTagSource{}, mb: &fakeTagSource{},
on: perProvider{providers.LastFM: lastfm, providers.MusicBrainz: musicbrainz},
now: t0,
}
r.lf.tags = []Tag{{Name: "rock", Weight: 100}, {Name: "grunge", Weight: 40}}
r.mb.tags = []Tag{{Name: "alternative rock", Weight: 43, Genre: true}, {Name: "british", Weight: 2}}
r.svc = NewTagService(r.store, map[string]TagSource{providers.LastFM: r.lf, providers.MusicBrainz: r.mb}, r.on, zap.NewNop())
r.svc.now = func() time.Time { return r.now }
return r
}
func (r *tagRig) tags(t *testing.T, id int64) TagResult {
t.Helper()
res, err := r.svc.Tags(context.Background(), id)
if err != nil {
t.Fatalf("Tags: %v", err)
}
return res
}
func TestTagsOfAnUnknownArtistAreNotFoundAndNobodyIsAsked(t *testing.T) {
r := newTagRig(true, true)
for _, id := range []int64{0, -1, 99} {
if _, err := r.svc.Tags(context.Background(), id); !errors.Is(err, ErrNotFound) {
t.Errorf("id %d: err = %v", id, err)
}
}
if r.lf.count()+r.mb.count() != 0 {
t.Error("a provider was asked about an artist that does not exist")
}
}
func TestTagsAlwaysListBothProvidersInTheSameOrder(t *testing.T) {
r := newTagRig(false, false)
res := r.tags(t, 1)
if len(res.Sources) != 2 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz {
t.Fatalf("sources = %+v", res.Sources)
}
for _, s := range res.Sources {
if s.Status != StatusDisabled || len(s.Tags) != 0 || !s.RetrievedAt.IsZero() || s.Stale || s.Attribution == "" {
t.Errorf("%s = %+v", s.Provider, s)
}
if s.Tags == nil {
t.Errorf("%s: tags is nil, want an empty list", s.Provider)
}
}
if r.lf.count()+r.mb.count() != 0 {
t.Error("a provider that is off was contacted")
}
}
// MusicBrainz is only asked for an artist whose id is known; Last.fm can be
// asked by name, and by id when there is one.
func TestMusicBrainzIsNeverAskedWithoutAnID(t *testing.T) {
r := newTagRig(true, true)
res := r.tags(t, 1)
if res.Sources[0].Status != StatusOK || len(res.Sources[0].Tags) != 2 || r.lf.artist.Name != "Source" || r.lf.artist.MBID != "" || r.lf.key != key {
t.Errorf("lastfm = %+v asked about %+v with %q", res.Sources[0], r.lf.artist, r.lf.key)
}
if res.Sources[1].Status != StatusNoID || len(res.Sources[1].Tags) != 0 || r.mb.count() != 0 {
t.Errorf("musicbrainz = %+v calls = %d", res.Sources[1], r.mb.count())
}
res = r.tags(t, 2)
if r.mb.count() != 1 || r.mb.artist.MBID != "aaaaaaaa-0000-4000-8000-000000000001" {
t.Fatalf("musicbrainz was asked about %+v (%d calls)", r.mb.artist, r.mb.count())
}
if r.lf.artist.MBID != "aaaaaaaa-0000-4000-8000-000000000001" {
t.Errorf("lastfm was not given the id: %+v", r.lf.artist)
}
mb := res.Sources[1]
if mb.Status != StatusOK || len(mb.Tags) != 2 || !mb.Tags[0].Genre || mb.Tags[0].Name != "alternative rock" || mb.Tags[0].Weight != 43 || !mb.RetrievedAt.Equal(t0) {
t.Errorf("musicbrainz = %+v", mb)
}
}
func TestTagsOfOneProviderDoNotDependOnTheOther(t *testing.T) {
r := newTagRig(true, true)
r.lf.err = errors.New("last.fm is down")
res := r.tags(t, 2)
if res.Sources[0].Status != StatusError || len(res.Sources[0].Tags) != 0 {
t.Errorf("lastfm = %+v", res.Sources[0])
}
if res.Sources[1].Status != StatusOK || len(res.Sources[1].Tags) != 2 {
t.Errorf("musicbrainz = %+v: one provider failing must not hide the other", res.Sources[1])
}
}
func TestFreshTagsAreNotAskedAgainAndOldOnesAre(t *testing.T) {
r := newTagRig(true, false)
r.tags(t, 1)
r.now = t0.Add(PositiveTTL - time.Second)
res := r.tags(t, 1)
if r.lf.count() != 1 || res.Sources[0].Stale || res.Sources[0].Status != StatusOK {
t.Errorf("calls = %d result = %+v", r.lf.count(), res.Sources[0])
}
r.now = t0.Add(PositiveTTL)
r.tags(t, 1)
if r.lf.count() != 2 {
t.Errorf("calls = %d, want 2", r.lf.count())
}
}
func TestUnknownAtTheProviderIsRememberedForADayAndOldDataSurvivesFailure(t *testing.T) {
r := newTagRig(true, false)
r.lf.err = ErrSourceNotFound
res := r.tags(t, 1)
if res.Sources[0].Status != StatusOK || len(res.Sources[0].Tags) != 0 || len(r.store.saves) != 1 || r.store.saves[0].found {
t.Fatalf("result = %+v saves = %+v", res.Sources[0], r.store.saves)
}
r.now = t0.Add(NegativeTTL - time.Second)
r.tags(t, 1)
if r.lf.count() != 1 {
t.Errorf("calls = %d: a negative answer is kept for %v", r.lf.count(), NegativeTTL)
}
// Old data is served when the provider fails.
r2 := newTagRig(true, false)
r2.tags(t, 1)
r2.now = t0.Add(PositiveTTL + time.Hour)
r2.lf.err = errors.New("down")
res = r2.tags(t, 1)
s := res.Sources[0]
if !s.Stale || s.Status != StatusError || len(s.Tags) != 2 || !s.RetrievedAt.Equal(t0) || len(r2.store.saves) != 1 {
t.Errorf("result = %+v saves = %d", s, len(r2.store.saves))
}
}
func TestAnEmptyTagListIsRememberedLikeAnUnknownArtist(t *testing.T) {
r := newTagRig(true, false)
r.lf.tags = nil
r.tags(t, 1)
if len(r.store.saves) != 1 || r.store.saves[0].found {
t.Errorf("saves = %+v", r.store.saves)
}
}
func TestOnlyTheBestTagsAreKeptAndNamesAreClean(t *testing.T) {
r := newTagRig(true, false)
var many []Tag
for i := 0; i < MaxTags+20; i++ {
many = append(many, Tag{Name: "tag" + string(rune('a'+i%26)) + string(rune('a'+i/26)), Weight: float64(1000 - i)})
}
r.lf.tags = many
res := r.tags(t, 1)
if got := len(res.Sources[0].Tags); got != MaxTags || res.Sources[0].Tags[0].Weight != 1000 {
t.Errorf("kept %d tags, first weight %v; want %d, best first", got, res.Sources[0].Tags[0].Weight, MaxTags)
}
}
func TestAPanicWhileReadingTagsIsAFailureNotACrash(t *testing.T) {
r := newTagRig(true, false)
r.lf.boom = "bug"
res, err := r.svc.Tags(context.Background(), 1)
if err != nil || res.Sources[0].Status != StatusError {
t.Fatalf("result = %+v err = %v", res.Sources[0], err)
}
}
func TestConcurrentTagLooksShareOneCall(t *testing.T) {
r := newTagRig(true, false)
r.lf.release = make(chan struct{})
var wg sync.WaitGroup
for i := 0; i < 8; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if res, err := r.svc.Tags(context.Background(), 1); err != nil || len(res.Sources[0].Tags) != 2 {
t.Errorf("result = %+v err = %v", res, err)
}
}()
}
time.Sleep(300 * time.Millisecond)
close(r.lf.release)
wg.Wait()
if r.lf.count() != 1 {
t.Errorf("calls = %d, want 1", r.lf.count())
}
}

View file

@ -3,6 +3,7 @@ package library
import (
"fmt"
"path/filepath"
"regexp"
"strconv"
"strings"
@ -31,14 +32,17 @@ func (TaglibReader) Read(path string) (Metadata, error) {
return Metadata{}, fmt.Errorf("%w: no audio stream found", ErrUnreadable)
}
artists := nonEmpty(tags["ARTIST"])
m := Metadata{
Tags: Tags{
Title: first(tags, "TITLE"),
Album: first(tags, "ALBUM"),
AlbumArtist: first(tags, "ALBUMARTIST"),
Artists: nonEmpty(tags["ARTIST"]),
TrackNo: leadingInt(first(tags, "TRACKNUMBER")),
DiscNo: leadingInt(first(tags, "DISCNUMBER")),
Title: first(tags, "TITLE"),
Album: first(tags, "ALBUM"),
AlbumArtist: first(tags, "ALBUMARTIST"),
Artists: artists,
ArtistMBIDs: pairArtistIDs(artists, tags["MUSICBRAINZ_ARTISTID"]),
AlbumArtistMBID: singleID(tags["MUSICBRAINZ_ALBUMARTISTID"]),
TrackNo: leadingInt(first(tags, "TRACKNUMBER")),
DiscNo: leadingInt(first(tags, "DISCNUMBER")),
},
Properties: Properties{
Format: strings.ToLower(props.Format),
@ -54,6 +58,7 @@ func (TaglibReader) Read(path string) (Metadata, error) {
}
if len(m.Artists) == 0 {
m.Artists = []string{UnknownArtist}
m.ArtistMBIDs = nil // an id for an artist the file never named is not ours to use
}
if m.AlbumArtist == "" {
m.AlbumArtist = m.Artists[0]
@ -61,6 +66,47 @@ func (TaglibReader) Read(path string) (Metadata, error) {
return m, nil
}
var mbidRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
// splitIDs reads the MusicBrainz ids out of the values of one tag. Taggers write
// several artists either as several values or as one value joined with "/" or ";".
// ok is false when any piece is not a MusicBrainz id.
func splitIDs(values []string) (ids []string, ok bool) {
for _, v := range values {
for _, piece := range strings.FieldsFunc(v, func(r rune) bool { return r == '/' || r == ';' || r == ',' }) {
piece = strings.TrimSpace(piece)
if piece == "" {
continue
}
if !mbidRe.MatchString(piece) {
return nil, false
}
ids = append(ids, strings.ToLower(piece))
}
}
return ids, true
}
// pairArtistIDs returns one id per artist, in order, or nothing. Fewer or more
// ids than artists, or an id that is not one, means the pairing would be a guess,
// and a wrong id on an artist is worse than none.
func pairArtistIDs(artists, values []string) []string {
ids, ok := splitIDs(values)
if !ok || len(ids) == 0 || len(ids) != len(artists) {
return nil
}
return ids
}
// singleID returns the id when the tag holds exactly one valid id.
func singleID(values []string) string {
ids, ok := splitIDs(values)
if !ok || len(ids) != 1 {
return ""
}
return ids[0]
}
func first(tags map[string][]string, key string) string {
for _, v := range tags[key] {
if v = strings.TrimSpace(v); v != "" {

View file

@ -121,3 +121,72 @@ func TestScanWithRealFiles(t *testing.T) {
t.Errorf("a.mp3 row = %+v", r)
}
}
// ---- MusicBrainz ids in the file tags ----
func TestArtistIDsAreOnlyKeptWhenTheyCanBePairedWithTheirArtists(t *testing.T) {
const a = "11111111-2222-4333-8444-555555555555"
const b = "66666666-7777-4888-8999-aaaaaaaaaaaa"
tests := []struct {
name string
artists []string
ids []string
want []string
}{
{"one and one", []string{"X"}, []string{a}, []string{a}},
{"two and two, in order", []string{"X", "Y"}, []string{a, b}, []string{a, b}},
{"upper case is lowered", []string{"X"}, []string{"11111111-2222-4333-8444-55555555555A"}, []string{"11111111-2222-4333-8444-55555555555a"}},
{"ids joined in one value", []string{"X", "Y"}, []string{a + "/" + b}, []string{a, b}},
{"ids joined with a semicolon", []string{"X", "Y"}, []string{a + "; " + b}, []string{a, b}},
{"fewer ids than artists is a guess", []string{"X", "Y"}, []string{a}, nil},
{"more ids than artists is a guess", []string{"X"}, []string{a, b}, nil},
{"no ids", []string{"X"}, nil, nil},
{"one bad id spoils the pairing", []string{"X", "Y"}, []string{a, "not-an-id"}, nil},
{"empty values are ignored", []string{"X"}, []string{" ", a}, []string{a}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := pairArtistIDs(tt.artists, tt.ids)
if len(got) != len(tt.want) {
t.Fatalf("ids = %v, want %v", got, tt.want)
}
for i := range got {
if got[i] != tt.want[i] {
t.Errorf("ids = %v, want %v", got, tt.want)
}
}
})
}
if id := singleID([]string{a}); id != a {
t.Errorf("singleID = %q", id)
}
if id := singleID([]string{a, b}); id != "" {
t.Errorf("two ids for the album artist: %q, want none", id)
}
if id := singleID([]string{"nope"}); id != "" {
t.Errorf("a bad id: %q, want none", id)
}
}
func TestTaglibReaderReadsMusicBrainzArtistIDs(t *testing.T) {
const a = "11111111-2222-4333-8444-555555555555"
const aa = "66666666-7777-4888-8999-aaaaaaaaaaaa"
p := makeAudio(t, t.TempDir(), "song.flac", map[string]string{
"title": "T", "artist": "Synthetic Band", "album_artist": "Synthetic Band",
"MUSICBRAINZ_ARTISTID": a, "MUSICBRAINZ_ALBUMARTISTID": aa,
})
m, err := TaglibReader{}.Read(p)
if err != nil {
t.Fatal(err)
}
if len(m.ArtistMBIDs) != 1 || m.ArtistMBIDs[0] != a || m.AlbumArtistMBID != aa {
t.Errorf("ids = %v / %q", m.ArtistMBIDs, m.AlbumArtistMBID)
}
// A file without them has none, and a file with no artist tag has no pairing.
p = makeAudio(t, t.TempDir(), "plain.flac", map[string]string{"title": "T", "MUSICBRAINZ_ARTISTID": a})
m, _ = TaglibReader{}.Read(p)
if len(m.ArtistMBIDs) != 0 {
t.Errorf("an id for an artist nobody named: %v", m.ArtistMBIDs)
}
}

View file

@ -13,13 +13,19 @@ import (
var ErrUnreadable = errors.New("unreadable audio file")
// Tags are the descriptive tags of a track. Fields the file lacks are empty.
//
// The MusicBrainz ids are what the file itself says, read offline. ArtistMBIDs is
// either empty or has one id per entry of Artists, in the same order: ids that
// cannot be paired with their artists with certainty are not kept.
type Tags struct {
Title string
Album string
AlbumArtist string
Artists []string
TrackNo int
DiscNo int
Title string
Album string
AlbumArtist string
Artists []string
ArtistMBIDs []string
AlbumArtistMBID string
TrackNo int
DiscNo int
}
// Properties are the technical properties of an audio file.

View file

@ -17,8 +17,10 @@ import (
"errors"
"fmt"
"net/url"
"sort"
"strconv"
"strings"
"unicode/utf8"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
@ -114,6 +116,90 @@ func (c *Client) SimilarArtists(ctx context.Context, key string, a Artist, limit
return decodeSimilar(doc.Similar.Artist)
}
// Tag is one tag of an artist. Count is Last.fm's own number between 0 and 100,
// relative to the artist (the top tag is usually 100).
type Tag struct {
Name string // lower case
Count int
}
// maxTagLen bounds a tag name; longer ones are not tags, they are noise.
const maxTagLen = 100
// TopTags returns the top tags of an artist, most voted first. The name is never
// autocorrected, and the MusicBrainz id wins over the name when known.
func (c *Client) TopTags(ctx context.Context, key string, a Artist) ([]Tag, error) {
name, mbid := strings.TrimSpace(a.Name), strings.TrimSpace(a.MBID)
if name == "" && mbid == "" {
return nil, ErrBadRequest
}
q := url.Values{"method": {"artist.gettoptags"}, "autocorrect": {"0"}}
if mbid != "" {
q.Set("mbid", mbid)
} else {
q.Set("artist", name)
}
body, err := c.call(ctx, key, q)
if err != nil {
return nil, err
}
var doc struct {
Top *struct {
Tag json.RawMessage `json:"tag"`
} `json:"toptags"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.Top == nil {
return nil, ErrBadResponse
}
return decodeTags(doc.Top.Tag)
}
// decodeTags reads the tag list, which Last.fm sends as an array or, with one
// tag, as an object. Names are lower-cased and kept once with their highest
// count; empty names, long names and counts of 0 or less are dropped.
func decodeTags(raw json.RawMessage) ([]Tag, error) {
type item struct {
Name string `json:"name"`
Count json.RawMessage `json:"count"`
}
var items []item
switch {
case len(raw) == 0 || string(raw) == "null":
case raw[0] == '[':
if err := json.Unmarshal(raw, &items); err != nil {
return nil, ErrBadResponse
}
default:
var one item
if err := json.Unmarshal(raw, &one); err != nil {
return nil, ErrBadResponse
}
items = []item{one}
}
best := map[string]int{}
for _, it := range items {
name := strings.ToLower(strings.TrimSpace(it.Name))
count, err := strconv.ParseFloat(strings.Trim(string(it.Count), `"`), 64)
if name == "" || utf8.RuneCountInString(name) > maxTagLen || err != nil || count < 1 || count > 100 {
continue
}
if int(count) > best[name] {
best[name] = int(count)
}
}
out := make([]Tag, 0, len(best))
for name, count := range best {
out = append(out, Tag{Name: name, Count: count})
}
sort.Slice(out, func(i, j int) bool {
if out[i].Count != out[j].Count {
return out[i].Count > out[j].Count
}
return out[i].Name < out[j].Name
})
return out, nil
}
// Check asks one cheap question to find out whether the key works.
func (c *Client) Check(ctx context.Context, key string) error {
body, err := c.call(ctx, key, url.Values{"method": {"chart.gettopartists"}, "limit": {"1"}})

View file

@ -282,3 +282,72 @@ func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
t.Errorf("an unreadable answer: err = %v", err)
}
}
// ---- tags ----
const tagsBody = `{"toptags":{"tag":[
{"name":"Rock","count":100,"url":"https://example.invalid/tag/rock"},
{"name":"alternative rock","count":"54","url":"https://example.invalid/x"},
{"name":"rock","count":10},
{"name":"","count":9},
{"name":"zero","count":0}
],"@attr":{"artist":"Source"}}}`
func TestTopTagsDecodesTheAnswer(t *testing.T) {
s := newServer(t, 200, tagsBody)
got, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "Source"})
if err != nil {
t.Fatal(err)
}
// Names are lower-cased and kept once (the higher count wins); empty names and
// a count of 0 are dropped; the count may come as a string.
if len(got) != 2 || got[0] != (Tag{Name: "rock", Count: 100}) || got[1] != (Tag{Name: "alternative rock", Count: 54}) {
t.Fatalf("tags = %+v", got)
}
q := s.last(t)
want := map[string]string{"method": "artist.gettoptags", "artist": "Source", "autocorrect": "0", "api_key": testKey, "format": "json"}
for k, v := range want {
if q.Get(k) != v {
t.Errorf("%s = %q, want %q", k, q.Get(k), v)
}
}
if len(q) != len(want) {
t.Errorf("query = %v", q)
}
}
func TestTopTagsPrefersTheMBIDAndToleratesShapes(t *testing.T) {
s := newServer(t, 200, `{"toptags":{"tag":{"name":"solo","count":7}}}`)
got, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "x", MBID: "22222222-2222-2222-2222-222222222222"})
if err != nil || len(got) != 1 || got[0].Name != "solo" {
t.Fatalf("tags = %+v err=%v", got, err)
}
if q := s.last(t); q.Get("mbid") != "22222222-2222-2222-2222-222222222222" || q.Has("artist") || q.Get("autocorrect") != "0" {
t.Errorf("query = %v", q)
}
empty := newServer(t, 200, `{"toptags":{"tag":[],"@attr":{"artist":"x"}}}`)
if got, err := empty.client().TopTags(context.Background(), testKey, Artist{Name: "x"}); err != nil || len(got) != 0 {
t.Errorf("no tags: %+v err=%v", got, err)
}
}
func TestTopTagsErrors(t *testing.T) {
if _, err := newServer(t, 200, tagsBody).client().TopTags(context.Background(), testKey, Artist{}); !errors.Is(err, ErrBadRequest) {
t.Errorf("nothing to look up: %v", err)
}
for _, tt := range []struct {
status int
body string
want error
}{
{200, `{"error":6,"message":"The artist you supplied could not be found"}`, ErrNotFound},
{403, `{"error":10,"message":"Invalid API key"}`, ErrInvalidKey},
{200, `{"hello":"world"}`, ErrBadResponse},
{500, `x`, ErrUnavailable},
} {
s := newServer(t, tt.status, tt.body)
if _, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "x"}); !errors.Is(err, tt.want) {
t.Errorf("%d %s: err = %v, want %v", tt.status, tt.body, err, tt.want)
}
}
}

View file

@ -0,0 +1,178 @@
// Package musicbrainz is a small read-only client for the MusicBrainz web
// service.
//
// Sonda only reads an artist whose MusicBrainz id it already knows. It never
// searches by name: a search returns look-alikes (a conductor named like a rock
// band) and picking one would be a guess. Only the id leaves the server.
//
// MusicBrainz asks every client to identify itself with a contact in the
// User-Agent and to stay at one request per second: both are the business of the
// outbound client this one sends its requests through. Errors never repeat the
// provider's own text.
package musicbrainz
import (
"context"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"unicode/utf8"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
// BaseURL is the MusicBrainz web service, version 2.
const BaseURL = "https://musicbrainz.org/ws/2/"
// probeID is a public artist used to check that the service answers.
const probeID = "a74b1b7f-71a5-4011-9441-d0b5e4122711"
// maxTagLen bounds a tag name; longer ones are not tags, they are noise.
const maxTagLen = 100
// Errors a call can return, to be matched with errors.Is.
var (
ErrBadRequest = errors.New("musicbrainz: not a MusicBrainz id")
ErrNotFound = errors.New("musicbrainz: unknown artist")
ErrRateLimited = errors.New("musicbrainz: too many requests")
ErrUnavailable = errors.New("musicbrainz: service unavailable")
ErrBadResponse = errors.New("musicbrainz: unreadable answer")
)
var idRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
// Getter performs a GET. *outbound.Client is the one used in production.
type Getter interface {
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
}
// Client talks to MusicBrainz. It is safe for concurrent use.
type Client struct {
http Getter
base string
}
// New builds a Client that sends its requests through g to base (BaseURL in
// production).
func New(g Getter, base string) *Client { return &Client{http: g, base: base} }
// Tag is one tag of an artist. Count is the number of people who voted for it, an
// absolute number (unlike Last.fm's, which is relative). Genre says that
// MusicBrainz also lists it among its curated genres.
type Tag struct {
Name string // lower case
Count int
Genre bool
}
// ArtistTags returns the tags of the artist with this MusicBrainz id, most voted
// first. An artist nobody tagged is an empty list, not an error.
func (c *Client) ArtistTags(ctx context.Context, id string) ([]Tag, error) {
if !idRe.MatchString(id) {
return nil, ErrBadRequest
}
body, err := c.get(ctx, c.base+"artist/"+strings.ToLower(id)+"?inc=tags+genres&fmt=json")
if err != nil {
return nil, err
}
var doc struct {
ID string `json:"id"`
Tags []struct {
Name string `json:"name"`
Count int `json:"count"`
} `json:"tags"`
Genres []struct {
Name string `json:"name"`
Count int `json:"count"`
} `json:"genres"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.ID == "" {
return nil, ErrBadResponse
}
byName := map[string]*Tag{}
add := func(name string, count int, genre bool) {
name = strings.ToLower(strings.TrimSpace(name))
if name == "" || utf8.RuneCountInString(name) > maxTagLen || count <= 0 {
return
}
if t, ok := byName[name]; ok {
t.Genre = t.Genre || genre
if count > t.Count {
t.Count = count
}
return
}
byName[name] = &Tag{Name: name, Count: count, Genre: genre}
}
for _, t := range doc.Tags {
add(t.Name, t.Count, false)
}
for _, g := range doc.Genres {
add(g.Name, g.Count, true)
}
out := make([]Tag, 0, len(byName))
for _, t := range byName {
out = append(out, *t)
}
sort.Slice(out, func(i, j int) bool {
if out[i].Count != out[j].Count {
return out[i].Count > out[j].Count
}
return out[i].Name < out[j].Name
})
return out, nil
}
// Probe asks for one public artist, to find out whether the service answers. It
// is a providers.Checker, and answers with the errors the settings understand.
func (c *Client) Probe(ctx context.Context) error {
body, err := c.get(ctx, c.base+"artist/"+probeID+"?fmt=json")
switch {
case errors.Is(err, ErrRateLimited):
return providers.ErrRateLimited
case errors.Is(err, ErrUnavailable):
return providers.ErrUnreachable
case err != nil:
return err
}
var doc struct {
ID string `json:"id"`
}
if json.Unmarshal(body, &doc) != nil || doc.ID == "" {
return ErrBadResponse
}
return nil
}
// get sends one request and returns the body of a successful answer.
func (c *Client) get(ctx context.Context, rawURL string) ([]byte, error) {
resp, err := c.http.Get(ctx, rawURL)
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
var limited *outbound.RateLimitedError
if errors.As(err, &limited) {
return nil, ErrRateLimited
}
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err)
}
switch {
case resp.Status == 200:
return resp.Body, nil
case resp.Status == 404:
return nil, ErrNotFound
case resp.Status == 400:
return nil, ErrBadRequest
case resp.Status == 429:
return nil, ErrRateLimited
case resp.Status >= 500:
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
}
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
}

View file

@ -0,0 +1,161 @@
package musicbrainz
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
const mbid = "11111111-2222-4333-8444-555555555555"
type server struct {
*httptest.Server
mu sync.Mutex
paths []string
hdr http.Header
}
func newServer(t *testing.T, status int, body string) *server {
t.Helper()
s := &server{}
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
s.paths = append(s.paths, r.URL.RequestURI())
s.hdr = r.Header.Clone()
s.mu.Unlock()
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
}))
t.Cleanup(s.Close)
return s
}
func (s *server) client() *Client {
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries}), s.URL+"/ws/2/")
}
func (s *server) requests() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.paths...)
}
const artistBody = `{"id":"11111111-2222-4333-8444-555555555555","name":"Synthetic Band","type":"Group",
"tags":[{"name":"Alternative Rock","count":43},{"name":"british","count":2},{"name":"ambient pop","count":1},{"name":"zero","count":0},{"name":"","count":9}],
"genres":[{"id":"x","name":"alternative rock","count":43,"disambiguation":""},{"id":"y","name":"art pop","count":3,"disambiguation":""}]}`
func TestArtistTagsReadsTagsAndMarksTheGenres(t *testing.T) {
s := newServer(t, 200, artistBody)
got, err := s.client().ArtistTags(context.Background(), mbid)
if err != nil {
t.Fatal(err)
}
want := []Tag{
{Name: "alternative rock", Count: 43, Genre: true}, // names are compared in lower case
{Name: "art pop", Count: 3, Genre: true}, // a genre that is not among the tags is kept
{Name: "british", Count: 2},
{Name: "ambient pop", Count: 1},
}
if len(got) != len(want) {
t.Fatalf("tags = %+v, want %+v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Errorf("tag %d = %+v, want %+v", i, got[i], want[i])
}
}
}
func TestArtistTagsAsksOnlyForTheIdOfTheArtist(t *testing.T) {
s := newServer(t, 200, artistBody)
if _, err := s.client().ArtistTags(context.Background(), strings.ToUpper(mbid)); err != nil {
t.Fatal(err)
}
reqs := s.requests()
if len(reqs) != 1 || reqs[0] != "/ws/2/artist/"+mbid+"?inc=tags+genres&fmt=json" {
t.Errorf("requests = %v", reqs)
}
if s.hdr.Get("User-Agent") != "Sonda/test ( https://example.org )" {
t.Errorf("User-Agent = %q", s.hdr.Get("User-Agent"))
}
}
// Anything that is not a plain id never reaches the network: it would end up
// inside the path of the request.
func TestArtistTagsRefusesWhatIsNotAnID(t *testing.T) {
for _, bad := range []string{"", "not-a-uuid", mbid + "/../x", mbid + "?x=1", " " + mbid + "\n", "00000000-0000-0000-0000-00000000000g"} {
s := newServer(t, 200, artistBody)
if _, err := s.client().ArtistTags(context.Background(), bad); !errors.Is(err, ErrBadRequest) {
t.Errorf("%q: err = %v, want ErrBadRequest", bad, err)
}
if len(s.requests()) != 0 {
t.Errorf("%q: a request left", bad)
}
}
}
func TestArtistTagsErrors(t *testing.T) {
for _, tt := range []struct {
name string
status int
body string
want error
}{
{"unknown id", 404, `{"error":"Not Found"}`, ErrNotFound},
{"invalid id", 400, `{"error":"Invalid mbid."}`, ErrBadRequest},
{"too many requests", 429, `{"error":"slow down"}`, ErrRateLimited},
{"over the limit", 503, `{"error":"Your requests are being throttled"}`, ErrUnavailable},
{"server error", 500, `oops`, ErrUnavailable},
{"not json", 200, `<html>`, ErrBadResponse},
{"empty", 200, ``, ErrBadResponse},
{"json of another kind", 200, `{"hello":"world"}`, ErrBadResponse},
{"forbidden", 403, `{"error":"banned"}`, ErrBadResponse},
} {
t.Run(tt.name, func(t *testing.T) {
s := newServer(t, tt.status, tt.body)
if _, err := s.client().ArtistTags(context.Background(), mbid); !errors.Is(err, tt.want) {
t.Fatalf("err = %v, want %v", err, tt.want)
}
})
}
// An artist with no tags at all is an answer, not an error.
s := newServer(t, 200, `{"id":"`+mbid+`","name":"Quiet"}`)
got, err := s.client().ArtistTags(context.Background(), mbid)
if err != nil || len(got) != 0 {
t.Errorf("no tags: %+v err=%v", got, err)
}
}
func TestErrorsDoNotCopyTheProvidersText(t *testing.T) {
s := newServer(t, 400, `{"error":"secret-text-from-the-provider"}`)
_, err := s.client().ArtistTags(context.Background(), mbid)
if err == nil || strings.Contains(err.Error(), "secret-text") {
t.Errorf("err = %v", err)
}
}
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
ok := newServer(t, 200, `{"id":"a74b1b7f-71a5-4011-9441-d0b5e4122711","name":"Anyone"}`)
if err := ok.client().Probe(context.Background()); err != nil {
t.Fatal(err)
}
if reqs := ok.requests(); len(reqs) != 1 || strings.Contains(reqs[0], "inc=") {
t.Errorf("a test asks for one plain artist: %v", reqs)
}
down := newServer(t, 503, `x`)
if err := down.client().Probe(context.Background()); !errors.Is(err, providers.ErrUnreachable) {
t.Errorf("down: %v", err)
}
slow := newServer(t, 429, `x`)
if err := slow.client().Probe(context.Background()); !errors.Is(err, providers.ErrRateLimited) {
t.Errorf("rate limited: %v", err)
}
}

View file

@ -70,8 +70,9 @@ func (realClock) Sleep(ctx context.Context, d time.Duration) error {
// Options configure a Client. Zero values mean sensible defaults.
type Options struct {
UserAgent string
Timeout time.Duration // per attempt; default 10s
UserAgent string
UserAgentFunc func() string // when set, read for every request instead of UserAgent
Timeout time.Duration // per attempt; default 10s
MinInterval time.Duration // minimum time between requests; 0 = no limit
MaxBody int64 // largest accepted answer; default 4 MiB
@ -99,7 +100,7 @@ type Response struct {
// Client talks to one provider. It is safe for concurrent use.
type Client struct {
userAgent string
userAgent func() string
timeout time.Duration
minInterval time.Duration
maxBody int64
@ -130,7 +131,7 @@ func New(o Options) *Client {
return v
}
c := &Client{
userAgent: o.UserAgent,
userAgent: func() string { return o.UserAgent },
timeout: def(o.Timeout, 10*time.Second),
minInterval: o.MinInterval,
maxBody: o.MaxBody,
@ -143,6 +144,9 @@ func New(o Options) *Client {
clock: o.Clock,
jitter: o.Jitter,
}
if o.UserAgentFunc != nil {
c.userAgent = o.UserAgentFunc
}
if c.maxBody <= 0 {
c.maxBody = 4 << 20
}
@ -268,7 +272,7 @@ func (c *Client) once(ctx context.Context, rawURL string) (*Response, error) {
if err != nil {
return nil, ErrBadURL
}
req.Header.Set("User-Agent", c.userAgent)
req.Header.Set("User-Agent", c.userAgent())
req.Header.Set("Accept", "application/json")
resp, err := c.hc.Do(req)

Some files were not shown because too many files have changed in this diff Show more