Merge pull request 'feat(discovery): artist tags from Last.fm and MusicBrainz, and ids from my files' (#24) from feat/artist-tags into main
This commit is contained in:
commit
146e1b1bf0
29 changed files with 2422 additions and 77 deletions
17
README.md
17
README.md
|
|
@ -15,11 +15,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
|
||||
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
|
||||
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights. See [External providers](#external-providers).
|
||||
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
|
||||
|
||||
## What is not there yet
|
||||
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. MusicBrainz can be set up (see [External providers](#external-providers)) but nothing asks it anything yet.
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Genres from your own files, and the removal of provider data on request.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -176,7 +177,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
|
|||
|
||||
## External providers
|
||||
|
||||
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today only Last.fm is used, to find similar artists.
|
||||
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, and MusicBrainz for tags.
|
||||
|
||||
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
|
||||
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
|
||||
|
|
@ -188,15 +189,20 @@ curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"api_key":"<your 32 charac
|
|||
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"enabled":true}'
|
||||
curl -s $BASE/providers -H "$AUTH" | jq
|
||||
curl -s -X POST $BASE/providers/lastfm/test -H "$AUTH" | jq # does the key work?
|
||||
curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"contact":"https://example.org/me"}'
|
||||
```
|
||||
|
||||
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
|
||||
|
||||
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
|
||||
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
|
||||
|
||||
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
|
||||
|
||||
The name of the artist is sent exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
|
||||
**Tags.** `GET /api/v1/artists/{id}/tags` asks each provider that is on, at the same time, and keeps the answers with the same rules (7 days, 1 for an unknown artist, old data if a provider fails). Each provider has its own entry with its own weights: Last.fm's are relative to the artist (1 to 100), MusicBrainz's are counts of votes, and they are never mixed. MusicBrainz also marks the tags it lists as genres. Its tags and genres are CC BY-NC-SA, so show the `attribution` that comes with them.
|
||||
|
||||
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. The scan reads a file's tags when the file is new or changed, so to pick up ids for music that is already indexed, the files have to be touched or re-added.
|
||||
|
||||
The artist is sent to Last.fm by its MusicBrainz id when your files gave one (if Last.fm does not know the id, by name), otherwise by its name exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
|
||||
|
||||
## The API at a glance
|
||||
|
||||
|
|
@ -211,6 +217,7 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
|
|||
| anyone | `POST /auth/login`, `/auth/refresh`, `/auth/logout` | Open, renew and end a session. |
|
||||
| signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). |
|
||||
| signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. |
|
||||
| signed in | `GET /artists/{id}/tags` | What Last.fm and MusicBrainz say about the artist's kind of music, from each provider that is on. |
|
||||
| signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 50). |
|
||||
| signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. |
|
||||
| signed in | `POST /listens` | Record a `play`, `complete` or `skip`; the answer says whether it counts as a listen. |
|
||||
|
|
@ -266,7 +273,7 @@ internal/auth/ users, roles, passwords, access and refresh tokens
|
|||
internal/catalog/ browsing and searching artists, albums, tracks
|
||||
internal/library/ libraries, the scanner, the event hub, the tag reader
|
||||
internal/discovery/ similar artists: what the providers said, kept and refreshed
|
||||
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm client
|
||||
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm and MusicBrainz clients
|
||||
internal/listening/ listen events
|
||||
internal/storage/ SQLite and PostgreSQL through bun, the repositories
|
||||
internal/streaming/ opening a track's file safely
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ import (
|
|||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
|
|
@ -37,6 +38,9 @@ const version = "0.1.0"
|
|||
// lastfmMinInterval spaces the requests to Last.fm well below its limit.
|
||||
const lastfmMinInterval = 250 * time.Millisecond
|
||||
|
||||
// musicbrainzMinInterval keeps a little above MusicBrainz's rule of one request a second.
|
||||
const musicbrainzMinInterval = 1100 * time.Millisecond
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "sondad:", err)
|
||||
|
|
@ -114,7 +118,21 @@ func run() error {
|
|||
MinInterval: lastfmMinInterval,
|
||||
}), lastfm.BaseURL)
|
||||
providerSettings.SetChecker(providers.LastFM, lastfmClient.Probe)
|
||||
discoveries := discovery.NewService(storage.NewDiscoveryRepo(db), lastfmClient, providerSettings, logger)
|
||||
// MusicBrainz wants a contact in the User-Agent, which an admin can change: it
|
||||
// is read again for every request.
|
||||
musicbrainzClient := musicbrainz.New(outbound.New(outbound.Options{
|
||||
UserAgentFunc: providerSettings.UserAgentFor(providers.MusicBrainz, version),
|
||||
MinInterval: musicbrainzMinInterval,
|
||||
}), musicbrainz.BaseURL)
|
||||
providerSettings.SetChecker(providers.MusicBrainz, func(ctx context.Context, _ string) error {
|
||||
return musicbrainzClient.Probe(ctx)
|
||||
})
|
||||
discoveryRepo := storage.NewDiscoveryRepo(db)
|
||||
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
|
||||
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
|
||||
providers.LastFM: discovery.LastfmTags{Client: lastfmClient},
|
||||
providers.MusicBrainz: discovery.MusicBrainzTags{Client: musicbrainzClient},
|
||||
}, providerSettings, logger)
|
||||
|
||||
router := httpapi.NewRouter(httpapi.Deps{
|
||||
Logger: logger,
|
||||
|
|
@ -126,6 +144,7 @@ func run() error {
|
|||
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
|
||||
Providers: providerSettings,
|
||||
Discovery: discoveries,
|
||||
Tags: tags,
|
||||
Streams: streaming.NewService(libraryRepo),
|
||||
})
|
||||
|
||||
|
|
|
|||
|
|
@ -85,6 +85,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
|
||||
Providers: prov,
|
||||
Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()),
|
||||
Tags: discovery.NewTagService(storage.NewDiscoveryRepo(db), map[string]discovery.TagSource{}, prov, zap.NewNop()),
|
||||
Streams: streaming.NewService(noFiles{}),
|
||||
}
|
||||
mutate(&deps, db)
|
||||
|
|
|
|||
|
|
@ -98,12 +98,17 @@ type contract struct {
|
|||
func newContract(t *testing.T) *contract {
|
||||
t.Helper()
|
||||
reader := &contractReader{}
|
||||
tagsLF := &fakeTags{tags: []discovery.Tag{{Name: "rock", Weight: 100}}}
|
||||
tagsMB := &fakeTags{tags: []discovery.Tag{{Name: "grunge", Weight: 12, Genre: true}}}
|
||||
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
|
||||
d.Providers = prov
|
||||
prov.SetChecker(providers.MusicBrainz, func(context.Context, string) error { return nil })
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), lf, prov, zap.NewNop())
|
||||
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
|
||||
map[string]discovery.TagSource{providers.LastFM: tagsLF, providers.MusicBrainz: tagsMB}, prov, zap.NewNop())
|
||||
repo := storage.NewLibraryRepo(db)
|
||||
mgr := library.NewManager(library.NewScanner(repo, reader, library.NewHub(), 2), repo)
|
||||
libs := library.NewLibraries(repo)
|
||||
|
|
@ -391,7 +396,9 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/spotify", token, map[string]any{"enabled": true}), 404, "unknown provider", "provider_not_found")
|
||||
c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test without a key")
|
||||
c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 400, "test a provider that has no test", "invalid_input")
|
||||
if mb := c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 200, "test musicbrainz"); mb["ok"] != true {
|
||||
t.Fatalf("musicbrainz test = %v", mb)
|
||||
}
|
||||
c.expect(c.call("POST", "/providers/spotify/test", token, nil), 404, "test an unknown provider", "provider_not_found")
|
||||
c.expect(c.call("POST", "/providers/lastfm/test", "", nil), 401, "test without a token", "unauthorized")
|
||||
off := c.expect(c.call("GET", "/artists/"+artistID+"/similar", token, nil), 200, "similar artists with the provider off")
|
||||
|
|
@ -415,6 +422,12 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("GET", "/artists/999/similar", token, nil), 404, "similar to an unknown artist", "artist_not_found")
|
||||
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
|
||||
tags := c.expect(c.call("GET", "/artists/"+artistID+"/tags", token, nil), 200, "tags")["sources"].([]any)
|
||||
if len(tags) != 2 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" {
|
||||
t.Fatalf("tags = %v, want Last.fm ok and MusicBrainz without an id", tags)
|
||||
}
|
||||
c.expect(c.call("GET", "/artists/999/tags", token, nil), 404, "tags of an unknown artist", "artist_not_found")
|
||||
c.expect(c.call("GET", "/artists/abc/tags", token, nil), 400, "tags of a bad id", "invalid_input")
|
||||
|
||||
// ---- playback ----
|
||||
stream := "/tracks/" + trackID + "/stream"
|
||||
|
|
|
|||
|
|
@ -75,3 +75,54 @@ func (h *discoveryHandlers) similar(c *gin.Context) {
|
|||
c.JSON(http.StatusOK, toSimilarJSON(res))
|
||||
}
|
||||
}
|
||||
|
||||
// tagsHandlers serve the tags that providers gave to an artist.
|
||||
type tagsHandlers struct{ svc tagService }
|
||||
|
||||
type tagJSON struct {
|
||||
Name string `json:"name"`
|
||||
Weight float64 `json:"weight"`
|
||||
Genre bool `json:"genre"`
|
||||
}
|
||||
|
||||
type tagSourceJSON struct {
|
||||
Provider string `json:"provider"`
|
||||
ProviderStatus string `json:"provider_status"`
|
||||
Stale bool `json:"stale"`
|
||||
RetrievedAt *string `json:"retrieved_at"`
|
||||
Attribution string `json:"attribution"`
|
||||
Tags []tagJSON `json:"tags"`
|
||||
}
|
||||
|
||||
func (h *tagsHandlers) tags(c *gin.Context) {
|
||||
id, ok := pathID(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
res, err := h.svc.Tags(c.Request.Context(), id)
|
||||
switch {
|
||||
case errors.Is(err, discovery.ErrNotFound):
|
||||
writeError(c, http.StatusNotFound, "artist_not_found", "not found")
|
||||
case errors.Is(err, context.Canceled):
|
||||
c.AbortWithStatus(statusClientClosed)
|
||||
case err != nil:
|
||||
_ = c.Error(err)
|
||||
writeError(c, http.StatusInternalServerError, "internal", "internal error")
|
||||
default:
|
||||
sources := make([]tagSourceJSON, len(res.Sources))
|
||||
for i, s := range res.Sources {
|
||||
out := tagSourceJSON{
|
||||
Provider: s.Provider, ProviderStatus: s.Status, Stale: s.Stale,
|
||||
Attribution: s.Attribution, Tags: make([]tagJSON, len(s.Tags)),
|
||||
}
|
||||
if !s.RetrievedAt.IsZero() {
|
||||
out.RetrievedAt = rfc3339(s.RetrievedAt)
|
||||
}
|
||||
for j, t := range s.Tags {
|
||||
out.Tags[j] = tagJSON{Name: t.Name, Weight: t.Weight, Genre: t.Genre}
|
||||
}
|
||||
sources[i] = out
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"sources": sources})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -61,6 +61,10 @@ type providerService interface {
|
|||
Test(ctx context.Context, name string) (providers.TestResult, error)
|
||||
}
|
||||
|
||||
type tagService interface {
|
||||
Tags(ctx context.Context, artistID int64) (discovery.TagResult, error)
|
||||
}
|
||||
|
||||
type discoveryService interface {
|
||||
Similar(ctx context.Context, artistID int64, limit int) (discovery.Result, error)
|
||||
}
|
||||
|
|
@ -92,6 +96,7 @@ type Deps struct {
|
|||
Catalog catalogService
|
||||
Providers providerService
|
||||
Discovery discoveryService
|
||||
Tags tagService
|
||||
Streams streamService
|
||||
}
|
||||
|
||||
|
|
@ -128,6 +133,8 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
protected.GET("/artists/:id", cat.getArtist)
|
||||
disc := &discoveryHandlers{svc: d.Discovery}
|
||||
protected.GET("/artists/:id/similar", disc.similar)
|
||||
tg := &tagsHandlers{svc: d.Tags}
|
||||
protected.GET("/artists/:id/tags", tg.tags)
|
||||
protected.GET("/albums", cat.listAlbums)
|
||||
protected.GET("/albums/:id", cat.getAlbum)
|
||||
protected.GET("/tracks", cat.listTracks)
|
||||
|
|
|
|||
|
|
@ -67,6 +67,12 @@ type similarEnv struct {
|
|||
// database that holds two local artists, Alpha and Beta, with Last.fm switched
|
||||
// off and without a key.
|
||||
func newSimilarEnv(t *testing.T) *similarEnv {
|
||||
t.Helper()
|
||||
return newSimilarEnvWith(t, func(*Deps, *bun.DB, *providers.Service) {})
|
||||
}
|
||||
|
||||
// newSimilarEnvWith is newSimilarEnv that lets a test add more to Deps.
|
||||
func newSimilarEnvWith(t *testing.T, more func(*Deps, *bun.DB, *providers.Service)) *similarEnv {
|
||||
t.Helper()
|
||||
src := &fakeLastfm{result: []lastfm.Similar{
|
||||
{Name: "Beta", Match: 1},
|
||||
|
|
@ -78,9 +84,16 @@ func newSimilarEnv(t *testing.T) *similarEnv {
|
|||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
more(d, db, prov)
|
||||
})
|
||||
e.admin = registerAndLogin(t, e.h)
|
||||
e.seedLibrary(t)
|
||||
return e
|
||||
}
|
||||
|
||||
// seedLibrary adds a library with the local artists Alpha and Beta.
|
||||
func (e *similarEnv) seedLibrary(t *testing.T) {
|
||||
t.Helper()
|
||||
lib := storage.NewLibraryRepo(e.db)
|
||||
l, err := lib.CreateLibrary(context.Background(), "music", "/lib")
|
||||
if err != nil {
|
||||
|
|
@ -99,7 +112,6 @@ func newSimilarEnv(t *testing.T) *similarEnv {
|
|||
}
|
||||
}
|
||||
e.alpha, e.beta = e.artist(t, "Alpha"), e.artist(t, "Beta")
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *similarEnv) artist(t *testing.T, name string) int64 {
|
||||
|
|
|
|||
234
internal/api/http/tags_test.go
Normal file
234
internal/api/http/tags_test.go
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
)
|
||||
|
||||
const alphaMBID = "aaaaaaaa-0000-4000-8000-0000000000a1"
|
||||
|
||||
// fakeTags answers for a provider and remembers what it was asked.
|
||||
type fakeTags struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
artist discovery.Artist
|
||||
tags []discovery.Tag
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeTags) ArtistTags(_ context.Context, _ string, a discovery.Artist) ([]discovery.Tag, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls++
|
||||
f.artist = a
|
||||
return f.tags, f.err
|
||||
}
|
||||
|
||||
type tagsEnv struct {
|
||||
*similarEnv
|
||||
lf, mb *fakeTags
|
||||
}
|
||||
|
||||
func newTagsEnv(t *testing.T) *tagsEnv {
|
||||
t.Helper()
|
||||
e := &tagsEnv{
|
||||
lf: &fakeTags{tags: []discovery.Tag{{Name: "rock", Weight: 100}, {Name: "grunge", Weight: 40}}},
|
||||
mb: &fakeTags{tags: []discovery.Tag{{Name: "alternative rock", Weight: 43, Genre: true}}},
|
||||
}
|
||||
e.similarEnv = newSimilarEnvWith(t, func(d *Deps, db *bun.DB, prov *providers.Service) {
|
||||
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
|
||||
map[string]discovery.TagSource{providers.LastFM: e.lf, providers.MusicBrainz: e.mb}, prov, zap.NewNop())
|
||||
})
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *tagsEnv) path(id int64) string {
|
||||
return "/api/v1/artists/" + strconv.FormatInt(id, 10) + "/tags"
|
||||
}
|
||||
|
||||
func (e *tagsEnv) enableMusicBrainz(t *testing.T) {
|
||||
t.Helper()
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/musicbrainz", e.admin, map[string]any{"enabled": true}); r.Code != http.StatusOK {
|
||||
t.Fatalf("enable musicbrainz: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsNeedASignedInUserAndAnArtist(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
if r := do(t, e.h, "GET", e.path(e.alpha), "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("no token: %d", r.Code)
|
||||
}
|
||||
if r := do(t, e.h, "GET", e.path(99999), e.admin, nil); r.Code != http.StatusNotFound || errCode(r) != "artist_not_found" {
|
||||
t.Errorf("unknown artist: %d %v", r.Code, r.Body)
|
||||
}
|
||||
if r := do(t, e.h, "GET", "/api/v1/artists/abc/tags", e.admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
|
||||
t.Errorf("bad id: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsWithEveryProviderOffAskNobody(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
sameKeys(t, "tags", r.Body, "sources")
|
||||
sources := r.Body["sources"].([]any)
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("sources = %v", sources)
|
||||
}
|
||||
for i, name := range []string{"lastfm", "musicbrainz"} {
|
||||
s := sources[i].(map[string]any)
|
||||
sameKeys(t, "source", s, "provider", "provider_status", "stale", "retrieved_at", "attribution", "tags")
|
||||
if s["provider"] != name || s["provider_status"] != "disabled" || s["stale"] != false || len(s["tags"].([]any)) != 0 {
|
||||
t.Errorf("%s = %v", name, s)
|
||||
}
|
||||
if v, present := s["retrieved_at"]; !present || v != nil {
|
||||
t.Errorf("%s retrieved_at = %#v, want an explicit null", name, v)
|
||||
}
|
||||
}
|
||||
if e.lf.calls+e.mb.calls != 0 {
|
||||
t.Error("a provider that is off was contacted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsFromEachProvider(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
e.enableLastfm(t)
|
||||
e.enableMusicBrainz(t)
|
||||
|
||||
// Alpha has no id yet: Last.fm is asked by name, MusicBrainz cannot be asked.
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
lf, mb := r.Body["sources"].([]any)[0].(map[string]any), r.Body["sources"].([]any)[1].(map[string]any)
|
||||
if lf["provider_status"] != "ok" || lf["retrieved_at"] == nil || len(lf["tags"].([]any)) != 2 {
|
||||
t.Fatalf("lastfm = %v", lf)
|
||||
}
|
||||
first := lf["tags"].([]any)[0].(map[string]any)
|
||||
sameKeys(t, "tag", first, "name", "weight", "genre")
|
||||
if first["name"] != "rock" || first["weight"] != float64(100) || first["genre"] != false {
|
||||
t.Errorf("first tag = %v", first)
|
||||
}
|
||||
if mb["provider_status"] != "no_id" || len(mb["tags"].([]any)) != 0 || e.mb.calls != 0 {
|
||||
t.Errorf("musicbrainz = %v calls = %d: it must not be asked without an id", mb, e.mb.calls)
|
||||
}
|
||||
if !strings.Contains(lf["attribution"].(string), "Last.fm") || !strings.Contains(mb["attribution"].(string), "MusicBrainz") {
|
||||
t.Errorf("attributions = %v / %v", lf["attribution"], mb["attribution"])
|
||||
}
|
||||
|
||||
// Once the id is known (from the file tags), MusicBrainz is asked with it.
|
||||
if _, err := e.db.Exec("UPDATE artists SET mbid = ? WHERE id = ?", alphaMBID, e.alpha); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r = do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
mb = r.Body["sources"].([]any)[1].(map[string]any)
|
||||
tag := mb["tags"].([]any)[0].(map[string]any)
|
||||
if mb["provider_status"] != "ok" || tag["name"] != "alternative rock" || tag["genre"] != true || tag["weight"] != float64(43) {
|
||||
t.Fatalf("musicbrainz = %v", mb)
|
||||
}
|
||||
if e.mb.artist.MBID != alphaMBID {
|
||||
t.Errorf("musicbrainz was asked about %+v", e.mb.artist)
|
||||
}
|
||||
raw, _ := json.Marshal(r.Body)
|
||||
if strings.Contains(string(raw), lastfmKey) {
|
||||
t.Error("the answer contains the API key")
|
||||
}
|
||||
|
||||
// A second look is served from the database.
|
||||
callsBefore := e.lf.calls + e.mb.calls
|
||||
do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if e.lf.calls+e.mb.calls != callsBefore {
|
||||
t.Error("fresh tags were asked for again")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsWhenAProviderFailsAreStillAnAnswer(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
e.enableLastfm(t)
|
||||
e.lf.err = lastfm.ErrUnavailable
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["sources"].([]any)[0].(map[string]any)["provider_status"] != "error" {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// What leaves the server for MusicBrainz, seen from the other end: the id of the
|
||||
// artist and a User-Agent that carries the contact the admin chose, and nothing
|
||||
// from the library.
|
||||
func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
seen []string
|
||||
hdr http.Header
|
||||
)
|
||||
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
seen = append(seen, r.URL.RequestURI())
|
||||
hdr = r.Header.Clone()
|
||||
mu.Unlock()
|
||||
_, _ = w.Write([]byte(`{"id":"` + alphaMBID + `","name":"Alpha","tags":[{"name":"rock","count":3}]}`))
|
||||
}))
|
||||
defer remote.Close()
|
||||
|
||||
e := &similarEnv{}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
mbClient := musicbrainz.New(outbound.New(outbound.Options{UserAgentFunc: prov.UserAgentFor(providers.MusicBrainz, "1.2.3")}), remote.URL+"/ws/2/")
|
||||
d.Tags = discovery.NewTagService(storage.NewDiscoveryRepo(db),
|
||||
map[string]discovery.TagSource{providers.MusicBrainz: discovery.MusicBrainzTags{Client: mbClient}}, prov, zap.NewNop())
|
||||
})
|
||||
e.admin = registerAndLogin(t, e.h)
|
||||
e.seedLibrary(t)
|
||||
if _, err := e.db.Exec("UPDATE artists SET mbid = ? WHERE id = ?", alphaMBID, e.alpha); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/musicbrainz", e.admin, map[string]any{"enabled": true, "contact": "https://example.org/me"}); r.Code != http.StatusOK {
|
||||
t.Fatal(r.Body)
|
||||
}
|
||||
|
||||
r := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(e.alpha, 10)+"/tags", e.admin, nil)
|
||||
mb := r.Body["sources"].([]any)[1].(map[string]any)
|
||||
if r.Code != http.StatusOK || mb["provider_status"] != "ok" {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(seen) != 1 || seen[0] != "/ws/2/artist/"+alphaMBID+"?inc=tags+genres&fmt=json" {
|
||||
t.Fatalf("requests = %v", seen)
|
||||
}
|
||||
if hdr.Get("User-Agent") != "Sonda/1.2.3 ( https://example.org/me )" {
|
||||
t.Errorf("User-Agent = %q", hdr.Get("User-Agent"))
|
||||
}
|
||||
for name := range hdr {
|
||||
switch http.CanonicalHeaderKey(name) {
|
||||
case "User-Agent", "Accept", "Accept-Encoding":
|
||||
default:
|
||||
t.Errorf("unexpected header %s", name)
|
||||
}
|
||||
}
|
||||
everything := strings.ToLower(seen[0] + hdr.Get("User-Agent"))
|
||||
for _, secret := range []string{"alice", "alpha", "/lib", "flac"} {
|
||||
if strings.Contains(everything, secret) {
|
||||
t.Errorf("the request mentions %q: %s", secret, everything)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -597,6 +597,42 @@ paths:
|
|||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/artists/{id}/tags:
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Id'}
|
||||
get:
|
||||
tags: [Discovery]
|
||||
operationId: getArtistTags
|
||||
summary: Tags of an artist, from each provider
|
||||
description: |
|
||||
What each provider says kind of music this artist plays: Last.fm's top tags and
|
||||
MusicBrainz's tags and genres. There is always one entry for each, Last.fm first. The
|
||||
weights are the providers' own numbers, kept as they came and never mixed: Last.fm's is
|
||||
relative to the artist (1 to 100, the top tag is usually 100; a tag with no votes is dropped),
|
||||
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres.
|
||||
|
||||
A provider is only asked when an admin switched it on and its stored answer is missing or
|
||||
old (7 days; a "no such artist" answer is kept 1 day), with the same rules as
|
||||
`GET /artists/{id}/similar`. What is sent is the artist's name or MusicBrainz id to
|
||||
Last.fm (its id when known), and only the MusicBrainz id to MusicBrainz: Sonda never looks
|
||||
an id up by name. An artist's id comes from the tags in the music files, or from Last.fm
|
||||
for an artist it named that is not in the library; for any other artist the MusicBrainz
|
||||
entry says `no_id`. Show each `attribution` where you show the tags.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
description: One entry per provider.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/ArtistTags'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'404':
|
||||
description: No such artist (`artist_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/albums:
|
||||
get:
|
||||
tags: [Catalog]
|
||||
|
|
@ -883,7 +919,8 @@ paths:
|
|||
sends nothing but the key. A provider that cannot be reached or refuses the key is not an
|
||||
error here: the answer is `200` with `ok` false and the reason in `status`.
|
||||
|
||||
Only Last.fm can be tested so far; asking for MusicBrainz is a `400`.
|
||||
MusicBrainz has no key: its test only asks whether the service answers. A provider that
|
||||
has no test is a `400`.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
|
|
@ -1365,6 +1402,45 @@ components:
|
|||
`no_api_key`: none is stored, so nobody was asked. `key_refused`: the provider does not accept
|
||||
it. `rate_limited`: try again later. `unreachable`: the provider did not answer. `error`: anything else.
|
||||
|
||||
ArtistTags:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [sources]
|
||||
properties:
|
||||
sources:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [provider, provider_status, stale, retrieved_at, attribution, tags]
|
||||
properties:
|
||||
provider: {type: string, enum: [lastfm, musicbrainz]}
|
||||
provider_status:
|
||||
type: string
|
||||
enum: [ok, disabled, error, no_id]
|
||||
description: |
|
||||
`ok`: the data is as fresh as it needs to be. `disabled`: the provider is off or has no
|
||||
key, so it was not asked. `error`: it was asked and failed. `no_id`: it needs the
|
||||
MusicBrainz id of the artist, which is not known, so it was not asked.
|
||||
stale: {type: boolean, description: "True when `tags` are older than they should be because the provider could not be asked."}
|
||||
retrieved_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: When the provider was last read for this artist; null if never.
|
||||
attribution: {type: string, description: Credit to show where the tags are shown.}
|
||||
tags:
|
||||
type: array
|
||||
description: The heaviest tags first, at most 50.
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [name, weight, genre]
|
||||
properties:
|
||||
name: {type: string, description: Lower case.}
|
||||
weight: {type: number, description: "The provider's own number; see the operation."}
|
||||
genre: {type: boolean}
|
||||
|
||||
SimilarArtists:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
|
|||
|
|
@ -213,11 +213,16 @@ func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Resul
|
|||
}
|
||||
|
||||
func (s *Service) fresh(snap Snapshot) bool {
|
||||
return fresh(s.now(), snap.RetrievedAt, snap.NotFound)
|
||||
}
|
||||
|
||||
// fresh says whether what was read at retrievedAt is still believed.
|
||||
func fresh(now, retrievedAt time.Time, notFound bool) bool {
|
||||
ttl := PositiveTTL
|
||||
if snap.NotFound {
|
||||
if notFound {
|
||||
ttl = NegativeTTL
|
||||
}
|
||||
return s.now().Sub(snap.RetrievedAt) < ttl
|
||||
return now.Sub(retrievedAt) < ttl
|
||||
}
|
||||
|
||||
// refresh reads the provider and stores what it said.
|
||||
|
|
|
|||
255
internal/discovery/tags.go
Normal file
255
internal/discovery/tags.go
Normal file
|
|
@ -0,0 +1,255 @@
|
|||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"sort"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
"golang.org/x/sync/singleflight"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
|
||||
)
|
||||
|
||||
// MaxTags is how many tags are kept for an artist from one provider, the most
|
||||
// weighty ones. Beyond that they are noise.
|
||||
const MaxTags = 50
|
||||
|
||||
// tagFetchTimeout bounds the wait for one provider's tags, queueing behind its
|
||||
// rate limit included: a provider limited to one request a second must not keep
|
||||
// a page waiting for a minute because a client asked about fifty artists.
|
||||
const tagFetchTimeout = 20 * time.Second
|
||||
|
||||
// StatusNoID says a provider that needs the MusicBrainz id of the artist cannot
|
||||
// be asked, because the id is not known. Sonda never looks one up by name.
|
||||
const StatusNoID = "no_id"
|
||||
|
||||
// ErrSourceNotFound is what a TagSource returns when the provider does not know
|
||||
// the artist.
|
||||
var ErrSourceNotFound = errors.New("the provider does not know this artist")
|
||||
|
||||
// Tag is one tag of an artist. Weight is the provider's own number, kept as it
|
||||
// came: Last.fm's is relative to the artist (0 to 100), MusicBrainz's is an
|
||||
// absolute count of votes, so weights of different providers are never compared.
|
||||
// Genre is set when the provider also lists the tag among its curated genres.
|
||||
type Tag struct {
|
||||
Name string
|
||||
Weight float64
|
||||
Genre bool
|
||||
}
|
||||
|
||||
// TagSnapshot is what is stored for one artist and one provider.
|
||||
type TagSnapshot struct {
|
||||
Fetched bool
|
||||
NotFound bool
|
||||
RetrievedAt time.Time
|
||||
Tags []Tag
|
||||
}
|
||||
|
||||
// TagSourceResult is what one provider says about an artist. RetrievedAt is zero
|
||||
// when nothing is known.
|
||||
type TagSourceResult struct {
|
||||
Provider string
|
||||
Status string // StatusOK, StatusDisabled, StatusError or StatusNoID
|
||||
Stale bool
|
||||
RetrievedAt time.Time
|
||||
Attribution string
|
||||
Tags []Tag
|
||||
}
|
||||
|
||||
// TagResult has one entry per provider, always in the same order.
|
||||
type TagResult struct{ Sources []TagSourceResult }
|
||||
|
||||
// TagStore keeps the tags. Artist returns ErrNotFound for an id the catalog does
|
||||
// not know.
|
||||
type TagStore interface {
|
||||
Artist(ctx context.Context, id int64) (Artist, error)
|
||||
Tags(ctx context.Context, id int64, provider string) (TagSnapshot, error)
|
||||
SaveTags(ctx context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error
|
||||
}
|
||||
|
||||
// TagSource asks one provider for the tags of an artist. It returns
|
||||
// ErrSourceNotFound when the provider does not know the artist.
|
||||
type TagSource interface {
|
||||
ArtistTags(ctx context.Context, key string, a Artist) ([]Tag, error)
|
||||
}
|
||||
|
||||
type tagProvider struct {
|
||||
name string
|
||||
needsID bool
|
||||
attribution string
|
||||
}
|
||||
|
||||
// tagProviders is the fixed order the providers are listed in.
|
||||
var tagProviders = []tagProvider{
|
||||
{providers.LastFM, false, "Tags from Last.fm (https://www.last.fm), for non-commercial use."},
|
||||
{providers.MusicBrainz, true, "Tags and genres from MusicBrainz (https://musicbrainz.org), CC BY-NC-SA 3.0: non-commercial, with attribution."},
|
||||
}
|
||||
|
||||
// TagService answers "what kind of music does this artist play", from every
|
||||
// provider that is on, with the same rules as Service: only on demand, kept for
|
||||
// 7 days (1 for an unknown artist), old data served when a provider fails, and
|
||||
// one call for any number of simultaneous requests.
|
||||
type TagService struct {
|
||||
store TagStore
|
||||
sources map[string]TagSource
|
||||
settings Settings
|
||||
log *zap.Logger
|
||||
now func() time.Time
|
||||
flight singleflight.Group
|
||||
}
|
||||
|
||||
// NewTagService builds a TagService. A provider without an entry in sources is
|
||||
// reported as disabled.
|
||||
func NewTagService(store TagStore, sources map[string]TagSource, settings Settings, log *zap.Logger) *TagService {
|
||||
if log == nil {
|
||||
log = zap.NewNop()
|
||||
}
|
||||
return &TagService{store: store, sources: sources, settings: settings, log: log, now: time.Now}
|
||||
}
|
||||
|
||||
// Tags returns what each provider says about an artist. A provider that is off,
|
||||
// cannot be asked or fails is not an error: its entry says so. The providers are
|
||||
// asked at the same time.
|
||||
func (s *TagService) Tags(ctx context.Context, artistID int64) (TagResult, error) {
|
||||
if artistID <= 0 {
|
||||
return TagResult{}, ErrNotFound
|
||||
}
|
||||
artist, err := s.store.Artist(ctx, artistID)
|
||||
if err != nil {
|
||||
return TagResult{}, err
|
||||
}
|
||||
out := make([]TagSourceResult, len(tagProviders))
|
||||
errs := make([]error, len(tagProviders))
|
||||
var wg sync.WaitGroup
|
||||
for i, p := range tagProviders {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
out[i], errs[i] = s.one(ctx, artist, p)
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
for _, err := range errs {
|
||||
if err != nil {
|
||||
return TagResult{}, err
|
||||
}
|
||||
}
|
||||
return TagResult{Sources: out}, nil
|
||||
}
|
||||
|
||||
func (s *TagService) one(ctx context.Context, a Artist, p tagProvider) (TagSourceResult, error) {
|
||||
snap, err := s.store.Tags(ctx, a.ID, p.name)
|
||||
if err != nil {
|
||||
return TagSourceResult{}, err
|
||||
}
|
||||
res := func(status string, stale bool, snap TagSnapshot) TagSourceResult {
|
||||
r := TagSourceResult{Provider: p.name, Status: status, Stale: stale && snap.Fetched, Attribution: p.attribution, Tags: snap.Tags}
|
||||
if r.Tags == nil {
|
||||
r.Tags = []Tag{}
|
||||
}
|
||||
if snap.Fetched {
|
||||
r.RetrievedAt = snap.RetrievedAt
|
||||
}
|
||||
return r
|
||||
}
|
||||
if snap.Fetched && fresh(s.now(), snap.RetrievedAt, snap.NotFound) {
|
||||
return res(StatusOK, false, snap), nil
|
||||
}
|
||||
src := s.sources[p.name]
|
||||
cfg, active, err := s.settings.Active(ctx, p.name)
|
||||
if err != nil {
|
||||
return TagSourceResult{}, err
|
||||
}
|
||||
if !active || src == nil {
|
||||
return res(StatusDisabled, snap.Fetched, snap), nil
|
||||
}
|
||||
if p.needsID && a.MBID == "" {
|
||||
return res(StatusNoID, snap.Fetched, snap), nil
|
||||
}
|
||||
|
||||
done := s.flight.DoChan(fmt.Sprintf("tags/%s/%d", p.name, a.ID), func() (_ any, err error) {
|
||||
defer func() {
|
||||
if r := recover(); r != nil {
|
||||
err = fmt.Errorf("reading tags panicked: %v", r)
|
||||
}
|
||||
}()
|
||||
return nil, s.refresh(src, cfg.APIKey, a, p.name)
|
||||
})
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return TagSourceResult{}, ctx.Err()
|
||||
case r := <-done:
|
||||
if r.Err != nil {
|
||||
s.log.Warn("tags could not be read", zap.String("provider", p.name), zap.Error(r.Err))
|
||||
return res(StatusError, snap.Fetched, snap), nil
|
||||
}
|
||||
}
|
||||
snap, err = s.store.Tags(ctx, a.ID, p.name)
|
||||
if err != nil {
|
||||
return TagSourceResult{}, err
|
||||
}
|
||||
return res(StatusOK, false, snap), nil
|
||||
}
|
||||
|
||||
// refresh reads one provider and stores what it said.
|
||||
func (s *TagService) refresh(src TagSource, key string, a Artist, provider string) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), tagFetchTimeout)
|
||||
defer cancel()
|
||||
tags, err := src.ArtistTags(ctx, key, a)
|
||||
switch {
|
||||
case errors.Is(err, ErrSourceNotFound):
|
||||
tags = nil
|
||||
case err != nil:
|
||||
return err
|
||||
}
|
||||
sort.SliceStable(tags, func(i, j int) bool { return tags[i].Weight > tags[j].Weight })
|
||||
if len(tags) > MaxTags {
|
||||
tags = tags[:MaxTags]
|
||||
}
|
||||
return s.store.SaveTags(ctx, a.ID, provider, s.now(), len(tags) > 0, tags)
|
||||
}
|
||||
|
||||
// LastfmTags adapts the Last.fm client to a TagSource.
|
||||
type LastfmTags struct{ Client *lastfm.Client }
|
||||
|
||||
// ArtistTags implements TagSource.
|
||||
func (l LastfmTags) ArtistTags(ctx context.Context, key string, a Artist) ([]Tag, error) {
|
||||
got, err := l.Client.TopTags(ctx, key, lastfm.Artist{Name: a.Name, MBID: a.MBID})
|
||||
if errors.Is(err, lastfm.ErrNotFound) {
|
||||
return nil, ErrSourceNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]Tag, len(got))
|
||||
for i, t := range got {
|
||||
out[i] = Tag{Name: t.Name, Weight: float64(t.Count)}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// MusicBrainzTags adapts the MusicBrainz client to a TagSource. It only ever
|
||||
// asks by the id of the artist.
|
||||
type MusicBrainzTags struct{ Client *musicbrainz.Client }
|
||||
|
||||
// ArtistTags implements TagSource.
|
||||
func (m MusicBrainzTags) ArtistTags(ctx context.Context, _ string, a Artist) ([]Tag, error) {
|
||||
got, err := m.Client.ArtistTags(ctx, a.MBID)
|
||||
if errors.Is(err, musicbrainz.ErrNotFound) {
|
||||
return nil, ErrSourceNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]Tag, len(got))
|
||||
for i, t := range got {
|
||||
out[i] = Tag{Name: t.Name, Weight: float64(t.Count), Genre: t.Genre}
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
295
internal/discovery/tags_test.go
Normal file
295
internal/discovery/tags_test.go
Normal file
|
|
@ -0,0 +1,295 @@
|
|||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
)
|
||||
|
||||
type savedTags struct {
|
||||
artistID int64
|
||||
provider string
|
||||
at time.Time
|
||||
found bool
|
||||
tags []Tag
|
||||
}
|
||||
|
||||
// tagStore keeps one snapshot per artist and provider.
|
||||
type tagStore struct {
|
||||
mu sync.Mutex
|
||||
artists map[int64]Artist
|
||||
snaps map[string]TagSnapshot
|
||||
saves []savedTags
|
||||
}
|
||||
|
||||
func newTagStore() *tagStore {
|
||||
return &tagStore{
|
||||
artists: map[int64]Artist{
|
||||
1: {ID: 1, Name: "Source"},
|
||||
2: {ID: 2, Name: "With Id", MBID: "aaaaaaaa-0000-4000-8000-000000000001"},
|
||||
},
|
||||
snaps: map[string]TagSnapshot{},
|
||||
}
|
||||
}
|
||||
|
||||
func tkey(id int64, provider string) string { return provider + "/" + string(rune('0'+id)) }
|
||||
|
||||
func (f *tagStore) Artist(_ context.Context, id int64) (Artist, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
a, ok := f.artists[id]
|
||||
if !ok {
|
||||
return Artist{}, ErrNotFound
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func (f *tagStore) Tags(_ context.Context, id int64, provider string) (TagSnapshot, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.snaps[tkey(id, provider)], nil
|
||||
}
|
||||
|
||||
func (f *tagStore) SaveTags(_ context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.saves = append(f.saves, savedTags{id, provider, at, found, tags})
|
||||
f.snaps[tkey(id, provider)] = TagSnapshot{Fetched: true, NotFound: !found, RetrievedAt: at, Tags: tags}
|
||||
return nil
|
||||
}
|
||||
|
||||
type fakeTagSource struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
key string
|
||||
artist Artist
|
||||
tags []Tag
|
||||
err error
|
||||
release chan struct{}
|
||||
boom any
|
||||
}
|
||||
|
||||
func (s *fakeTagSource) ArtistTags(_ context.Context, key string, a Artist) ([]Tag, error) {
|
||||
s.mu.Lock()
|
||||
s.calls++
|
||||
s.key, s.artist = key, a
|
||||
release := s.release
|
||||
s.mu.Unlock()
|
||||
if release != nil {
|
||||
<-release
|
||||
}
|
||||
if s.boom != nil {
|
||||
panic(s.boom)
|
||||
}
|
||||
return s.tags, s.err
|
||||
}
|
||||
|
||||
func (s *fakeTagSource) count() int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.calls
|
||||
}
|
||||
|
||||
// perProvider answers Active for each provider on its own.
|
||||
type perProvider map[string]bool
|
||||
|
||||
func (p perProvider) Active(_ context.Context, name string) (providers.Settings, bool, error) {
|
||||
return providers.Settings{Provider: name, Enabled: p[name], APIKey: key}, p[name], nil
|
||||
}
|
||||
|
||||
type tagRig struct {
|
||||
svc *TagService
|
||||
store *tagStore
|
||||
lf *fakeTagSource
|
||||
mb *fakeTagSource
|
||||
on perProvider
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func newTagRig(lastfm, musicbrainz bool) *tagRig {
|
||||
r := &tagRig{
|
||||
store: newTagStore(), lf: &fakeTagSource{}, mb: &fakeTagSource{},
|
||||
on: perProvider{providers.LastFM: lastfm, providers.MusicBrainz: musicbrainz},
|
||||
now: t0,
|
||||
}
|
||||
r.lf.tags = []Tag{{Name: "rock", Weight: 100}, {Name: "grunge", Weight: 40}}
|
||||
r.mb.tags = []Tag{{Name: "alternative rock", Weight: 43, Genre: true}, {Name: "british", Weight: 2}}
|
||||
r.svc = NewTagService(r.store, map[string]TagSource{providers.LastFM: r.lf, providers.MusicBrainz: r.mb}, r.on, zap.NewNop())
|
||||
r.svc.now = func() time.Time { return r.now }
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *tagRig) tags(t *testing.T, id int64) TagResult {
|
||||
t.Helper()
|
||||
res, err := r.svc.Tags(context.Background(), id)
|
||||
if err != nil {
|
||||
t.Fatalf("Tags: %v", err)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func TestTagsOfAnUnknownArtistAreNotFoundAndNobodyIsAsked(t *testing.T) {
|
||||
r := newTagRig(true, true)
|
||||
for _, id := range []int64{0, -1, 99} {
|
||||
if _, err := r.svc.Tags(context.Background(), id); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("id %d: err = %v", id, err)
|
||||
}
|
||||
}
|
||||
if r.lf.count()+r.mb.count() != 0 {
|
||||
t.Error("a provider was asked about an artist that does not exist")
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsAlwaysListBothProvidersInTheSameOrder(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
res := r.tags(t, 1)
|
||||
if len(res.Sources) != 2 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz {
|
||||
t.Fatalf("sources = %+v", res.Sources)
|
||||
}
|
||||
for _, s := range res.Sources {
|
||||
if s.Status != StatusDisabled || len(s.Tags) != 0 || !s.RetrievedAt.IsZero() || s.Stale || s.Attribution == "" {
|
||||
t.Errorf("%s = %+v", s.Provider, s)
|
||||
}
|
||||
if s.Tags == nil {
|
||||
t.Errorf("%s: tags is nil, want an empty list", s.Provider)
|
||||
}
|
||||
}
|
||||
if r.lf.count()+r.mb.count() != 0 {
|
||||
t.Error("a provider that is off was contacted")
|
||||
}
|
||||
}
|
||||
|
||||
// MusicBrainz is only asked for an artist whose id is known; Last.fm can be
|
||||
// asked by name, and by id when there is one.
|
||||
func TestMusicBrainzIsNeverAskedWithoutAnID(t *testing.T) {
|
||||
r := newTagRig(true, true)
|
||||
res := r.tags(t, 1)
|
||||
if res.Sources[0].Status != StatusOK || len(res.Sources[0].Tags) != 2 || r.lf.artist.Name != "Source" || r.lf.artist.MBID != "" || r.lf.key != key {
|
||||
t.Errorf("lastfm = %+v asked about %+v with %q", res.Sources[0], r.lf.artist, r.lf.key)
|
||||
}
|
||||
if res.Sources[1].Status != StatusNoID || len(res.Sources[1].Tags) != 0 || r.mb.count() != 0 {
|
||||
t.Errorf("musicbrainz = %+v calls = %d", res.Sources[1], r.mb.count())
|
||||
}
|
||||
|
||||
res = r.tags(t, 2)
|
||||
if r.mb.count() != 1 || r.mb.artist.MBID != "aaaaaaaa-0000-4000-8000-000000000001" {
|
||||
t.Fatalf("musicbrainz was asked about %+v (%d calls)", r.mb.artist, r.mb.count())
|
||||
}
|
||||
if r.lf.artist.MBID != "aaaaaaaa-0000-4000-8000-000000000001" {
|
||||
t.Errorf("lastfm was not given the id: %+v", r.lf.artist)
|
||||
}
|
||||
mb := res.Sources[1]
|
||||
if mb.Status != StatusOK || len(mb.Tags) != 2 || !mb.Tags[0].Genre || mb.Tags[0].Name != "alternative rock" || mb.Tags[0].Weight != 43 || !mb.RetrievedAt.Equal(t0) {
|
||||
t.Errorf("musicbrainz = %+v", mb)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTagsOfOneProviderDoNotDependOnTheOther(t *testing.T) {
|
||||
r := newTagRig(true, true)
|
||||
r.lf.err = errors.New("last.fm is down")
|
||||
res := r.tags(t, 2)
|
||||
if res.Sources[0].Status != StatusError || len(res.Sources[0].Tags) != 0 {
|
||||
t.Errorf("lastfm = %+v", res.Sources[0])
|
||||
}
|
||||
if res.Sources[1].Status != StatusOK || len(res.Sources[1].Tags) != 2 {
|
||||
t.Errorf("musicbrainz = %+v: one provider failing must not hide the other", res.Sources[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreshTagsAreNotAskedAgainAndOldOnesAre(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.tags(t, 1)
|
||||
r.now = t0.Add(PositiveTTL - time.Second)
|
||||
res := r.tags(t, 1)
|
||||
if r.lf.count() != 1 || res.Sources[0].Stale || res.Sources[0].Status != StatusOK {
|
||||
t.Errorf("calls = %d result = %+v", r.lf.count(), res.Sources[0])
|
||||
}
|
||||
r.now = t0.Add(PositiveTTL)
|
||||
r.tags(t, 1)
|
||||
if r.lf.count() != 2 {
|
||||
t.Errorf("calls = %d, want 2", r.lf.count())
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnknownAtTheProviderIsRememberedForADayAndOldDataSurvivesFailure(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.lf.err = ErrSourceNotFound
|
||||
res := r.tags(t, 1)
|
||||
if res.Sources[0].Status != StatusOK || len(res.Sources[0].Tags) != 0 || len(r.store.saves) != 1 || r.store.saves[0].found {
|
||||
t.Fatalf("result = %+v saves = %+v", res.Sources[0], r.store.saves)
|
||||
}
|
||||
r.now = t0.Add(NegativeTTL - time.Second)
|
||||
r.tags(t, 1)
|
||||
if r.lf.count() != 1 {
|
||||
t.Errorf("calls = %d: a negative answer is kept for %v", r.lf.count(), NegativeTTL)
|
||||
}
|
||||
|
||||
// Old data is served when the provider fails.
|
||||
r2 := newTagRig(true, false)
|
||||
r2.tags(t, 1)
|
||||
r2.now = t0.Add(PositiveTTL + time.Hour)
|
||||
r2.lf.err = errors.New("down")
|
||||
res = r2.tags(t, 1)
|
||||
s := res.Sources[0]
|
||||
if !s.Stale || s.Status != StatusError || len(s.Tags) != 2 || !s.RetrievedAt.Equal(t0) || len(r2.store.saves) != 1 {
|
||||
t.Errorf("result = %+v saves = %d", s, len(r2.store.saves))
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnEmptyTagListIsRememberedLikeAnUnknownArtist(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.lf.tags = nil
|
||||
r.tags(t, 1)
|
||||
if len(r.store.saves) != 1 || r.store.saves[0].found {
|
||||
t.Errorf("saves = %+v", r.store.saves)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOnlyTheBestTagsAreKeptAndNamesAreClean(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
var many []Tag
|
||||
for i := 0; i < MaxTags+20; i++ {
|
||||
many = append(many, Tag{Name: "tag" + string(rune('a'+i%26)) + string(rune('a'+i/26)), Weight: float64(1000 - i)})
|
||||
}
|
||||
r.lf.tags = many
|
||||
res := r.tags(t, 1)
|
||||
if got := len(res.Sources[0].Tags); got != MaxTags || res.Sources[0].Tags[0].Weight != 1000 {
|
||||
t.Errorf("kept %d tags, first weight %v; want %d, best first", got, res.Sources[0].Tags[0].Weight, MaxTags)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPanicWhileReadingTagsIsAFailureNotACrash(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.lf.boom = "bug"
|
||||
res, err := r.svc.Tags(context.Background(), 1)
|
||||
if err != nil || res.Sources[0].Status != StatusError {
|
||||
t.Fatalf("result = %+v err = %v", res.Sources[0], err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConcurrentTagLooksShareOneCall(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.lf.release = make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 8; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if res, err := r.svc.Tags(context.Background(), 1); err != nil || len(res.Sources[0].Tags) != 2 {
|
||||
t.Errorf("result = %+v err = %v", res, err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
close(r.lf.release)
|
||||
wg.Wait()
|
||||
if r.lf.count() != 1 {
|
||||
t.Errorf("calls = %d, want 1", r.lf.count())
|
||||
}
|
||||
}
|
||||
|
|
@ -3,6 +3,7 @@ package library
|
|||
import (
|
||||
"fmt"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
|
|
@ -31,14 +32,24 @@ func (TaglibReader) Read(path string) (Metadata, error) {
|
|||
return Metadata{}, fmt.Errorf("%w: no audio stream found", ErrUnreadable)
|
||||
}
|
||||
|
||||
artists := nonEmpty(tags["ARTIST"])
|
||||
albumArtist := first(tags, "ALBUMARTIST")
|
||||
albumArtistID := ""
|
||||
if albumArtist != "" {
|
||||
// The id belongs to the album artist tag: when the tag is missing the name
|
||||
// below is only a fallback, and an id would land on the wrong artist.
|
||||
albumArtistID = singleID(tags["MUSICBRAINZ_ALBUMARTISTID"])
|
||||
}
|
||||
m := Metadata{
|
||||
Tags: Tags{
|
||||
Title: first(tags, "TITLE"),
|
||||
Album: first(tags, "ALBUM"),
|
||||
AlbumArtist: first(tags, "ALBUMARTIST"),
|
||||
Artists: nonEmpty(tags["ARTIST"]),
|
||||
TrackNo: leadingInt(first(tags, "TRACKNUMBER")),
|
||||
DiscNo: leadingInt(first(tags, "DISCNUMBER")),
|
||||
Title: first(tags, "TITLE"),
|
||||
Album: first(tags, "ALBUM"),
|
||||
AlbumArtist: albumArtist,
|
||||
Artists: artists,
|
||||
ArtistMBIDs: pairArtistIDs(artists, tags["MUSICBRAINZ_ARTISTID"]),
|
||||
AlbumArtistMBID: albumArtistID,
|
||||
TrackNo: leadingInt(first(tags, "TRACKNUMBER")),
|
||||
DiscNo: leadingInt(first(tags, "DISCNUMBER")),
|
||||
},
|
||||
Properties: Properties{
|
||||
Format: strings.ToLower(props.Format),
|
||||
|
|
@ -54,6 +65,7 @@ func (TaglibReader) Read(path string) (Metadata, error) {
|
|||
}
|
||||
if len(m.Artists) == 0 {
|
||||
m.Artists = []string{UnknownArtist}
|
||||
m.ArtistMBIDs = nil // an id for an artist the file never named is not ours to use
|
||||
}
|
||||
if m.AlbumArtist == "" {
|
||||
m.AlbumArtist = m.Artists[0]
|
||||
|
|
@ -61,6 +73,69 @@ func (TaglibReader) Read(path string) (Metadata, error) {
|
|||
return m, nil
|
||||
}
|
||||
|
||||
var mbidRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
|
||||
// splitIDs reads the MusicBrainz ids out of the values of one tag. Taggers write
|
||||
// several artists either as several values or as one value joined with "/" or ";".
|
||||
// ok is false when any piece is not a MusicBrainz id.
|
||||
func splitIDs(values []string) (ids []string, ok bool) {
|
||||
for _, v := range values {
|
||||
for _, piece := range strings.FieldsFunc(v, func(r rune) bool { return r == '/' || r == ';' || r == ',' }) {
|
||||
piece = strings.TrimSpace(piece)
|
||||
if piece == "" {
|
||||
continue
|
||||
}
|
||||
if !mbidRe.MatchString(piece) {
|
||||
return nil, false
|
||||
}
|
||||
ids = append(ids, strings.ToLower(piece))
|
||||
}
|
||||
}
|
||||
return ids, true
|
||||
}
|
||||
|
||||
// pairArtistIDs returns one id per artist, in order, or nothing. Fewer or more
|
||||
// ids than artists, or an id that is not one, means the pairing would be a guess,
|
||||
// and a wrong id on an artist is worse than none.
|
||||
func pairArtistIDs(artists, values []string) []string {
|
||||
ids, ok := splitIDs(values)
|
||||
if !ok || len(ids) == 0 || len(ids) != len(artists) {
|
||||
return nil
|
||||
}
|
||||
// One credit with one id ("X feat. Y" with the id of X) is the usual way a
|
||||
// tagger writes the id of the first artist only. Without a way to tell, no id.
|
||||
if len(artists) == 1 && looksLikeCredit(artists[0]) {
|
||||
return nil
|
||||
}
|
||||
return ids
|
||||
}
|
||||
|
||||
// creditWords are the separators of a credit that names several artists. A slash
|
||||
// is not one: AC/DC is a band.
|
||||
var creditWords = []string{" feat.", " feat ", " featuring ", " ft.", " ft ", " with ", " & ", " and ", " vs ", " vs. ", " x ", ",", ";"}
|
||||
|
||||
// looksLikeCredit says whether a name seems to credit more than one artist. It
|
||||
// errs on the side of "yes": a missing id costs a lookup, a wrong one a wrong artist.
|
||||
func looksLikeCredit(name string) bool {
|
||||
n := " " + strings.ToLower(name) + " "
|
||||
for _, w := range creditWords {
|
||||
// Text must stand on both sides: a band called "X" is not a credit.
|
||||
if i := strings.Index(n, w); i > 1 && i+len(w) < len(n)-1 {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// singleID returns the id when the tag holds exactly one valid id.
|
||||
func singleID(values []string) string {
|
||||
ids, ok := splitIDs(values)
|
||||
if !ok || len(ids) != 1 {
|
||||
return ""
|
||||
}
|
||||
return ids[0]
|
||||
}
|
||||
|
||||
func first(tags map[string][]string, key string) string {
|
||||
for _, v := range tags[key] {
|
||||
if v = strings.TrimSpace(v); v != "" {
|
||||
|
|
|
|||
|
|
@ -121,3 +121,117 @@ func TestScanWithRealFiles(t *testing.T) {
|
|||
t.Errorf("a.mp3 row = %+v", r)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- MusicBrainz ids in the file tags ----
|
||||
|
||||
func TestArtistIDsAreOnlyKeptWhenTheyCanBePairedWithTheirArtists(t *testing.T) {
|
||||
const a = "11111111-2222-4333-8444-555555555555"
|
||||
const b = "66666666-7777-4888-8999-aaaaaaaaaaaa"
|
||||
tests := []struct {
|
||||
name string
|
||||
artists []string
|
||||
ids []string
|
||||
want []string
|
||||
}{
|
||||
{"one and one", []string{"X"}, []string{a}, []string{a}},
|
||||
{"two and two, in order", []string{"X", "Y"}, []string{a, b}, []string{a, b}},
|
||||
{"upper case is lowered", []string{"X"}, []string{"11111111-2222-4333-8444-55555555555A"}, []string{"11111111-2222-4333-8444-55555555555a"}},
|
||||
{"ids joined in one value", []string{"X", "Y"}, []string{a + "/" + b}, []string{a, b}},
|
||||
{"ids joined with a semicolon", []string{"X", "Y"}, []string{a + "; " + b}, []string{a, b}},
|
||||
{"fewer ids than artists is a guess", []string{"X", "Y"}, []string{a}, nil},
|
||||
{"more ids than artists is a guess", []string{"X"}, []string{a, b}, nil},
|
||||
{"no ids", []string{"X"}, nil, nil},
|
||||
{"one bad id spoils the pairing", []string{"X", "Y"}, []string{a, "not-an-id"}, nil},
|
||||
{"empty values are ignored", []string{"X"}, []string{" ", a}, []string{a}},
|
||||
// A credit with one id is usually the id of the first artist only.
|
||||
{"a featuring credit with one id", []string{"X feat. Y"}, []string{a}, nil},
|
||||
{"ft. credit", []string{"X ft. Y"}, []string{a}, nil},
|
||||
{"with an ampersand", []string{"Simon & Garfunkel"}, []string{a}, nil},
|
||||
{"with a comma", []string{"X, Y"}, []string{a}, nil},
|
||||
{"a slash is part of a name", []string{"AC/DC"}, []string{a}, []string{a}},
|
||||
{"a plain name that contains 'and' inside a word", []string{"Sandra Bland"}, []string{a}, []string{a}},
|
||||
{"credits are fine when each artist has an id", []string{"X", "Y"}, []string{a, b}, []string{a, b}},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := pairArtistIDs(tt.artists, tt.ids)
|
||||
if len(got) != len(tt.want) {
|
||||
t.Fatalf("ids = %v, want %v", got, tt.want)
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.want[i] {
|
||||
t.Errorf("ids = %v, want %v", got, tt.want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
if id := singleID([]string{a}); id != a {
|
||||
t.Errorf("singleID = %q", id)
|
||||
}
|
||||
if id := singleID([]string{a, b}); id != "" {
|
||||
t.Errorf("two ids for the album artist: %q, want none", id)
|
||||
}
|
||||
if id := singleID([]string{"nope"}); id != "" {
|
||||
t.Errorf("a bad id: %q, want none", id)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaglibReaderReadsMusicBrainzArtistIDs(t *testing.T) {
|
||||
const a = "11111111-2222-4333-8444-555555555555"
|
||||
const aa = "66666666-7777-4888-8999-aaaaaaaaaaaa"
|
||||
p := makeAudio(t, t.TempDir(), "song.flac", map[string]string{
|
||||
"title": "T", "artist": "Synthetic Band", "album_artist": "Synthetic Band",
|
||||
"MUSICBRAINZ_ARTISTID": a, "MUSICBRAINZ_ALBUMARTISTID": aa,
|
||||
})
|
||||
m, err := TaglibReader{}.Read(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(m.ArtistMBIDs) != 1 || m.ArtistMBIDs[0] != a || m.AlbumArtistMBID != aa {
|
||||
t.Errorf("ids = %v / %q", m.ArtistMBIDs, m.AlbumArtistMBID)
|
||||
}
|
||||
|
||||
// A file without them has none, and a file with no artist tag has no pairing.
|
||||
p = makeAudio(t, t.TempDir(), "plain.flac", map[string]string{"title": "T", "MUSICBRAINZ_ARTISTID": a})
|
||||
m, _ = TaglibReader{}.Read(p)
|
||||
if len(m.ArtistMBIDs) != 0 {
|
||||
t.Errorf("an id for an artist nobody named: %v", m.ArtistMBIDs)
|
||||
}
|
||||
}
|
||||
|
||||
// The id of the album artist belongs to the name of the album artist tag. When
|
||||
// the tag is missing the name is only a fallback, and an id would land on the
|
||||
// wrong artist.
|
||||
func TestAlbumArtistIDNeedsAnAlbumArtistTag(t *testing.T) {
|
||||
const variousArtists = "89ad4ac3-39f7-470e-963a-56509c546377"
|
||||
p := makeAudio(t, t.TempDir(), "song.flac", map[string]string{
|
||||
"title": "T", "artist": "Foo", "MUSICBRAINZ_ALBUMARTISTID": variousArtists,
|
||||
})
|
||||
m, err := TaglibReader{}.Read(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if m.AlbumArtist != "Foo" || m.AlbumArtistMBID != "" {
|
||||
t.Errorf("album artist = %q with id %q: the id must not follow a fallback name", m.AlbumArtist, m.AlbumArtistMBID)
|
||||
}
|
||||
|
||||
p = makeAudio(t, t.TempDir(), "nothing.flac", map[string]string{"title": "T", "MUSICBRAINZ_ALBUMARTISTID": variousArtists})
|
||||
m, _ = TaglibReader{}.Read(p)
|
||||
if m.AlbumArtistMBID != "" {
|
||||
t.Errorf("an id for an album artist nobody named: %q", m.AlbumArtistMBID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTaglibReaderReadsMusicBrainzIDsFromMP3(t *testing.T) {
|
||||
const a = "11111111-2222-4333-8444-555555555555"
|
||||
p := makeAudio(t, t.TempDir(), "song.mp3", map[string]string{
|
||||
"title": "T", "artist": "Synthetic Band", "MusicBrainz Artist Id": a,
|
||||
})
|
||||
m, err := TaglibReader{}.Read(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(m.ArtistMBIDs) != 1 || m.ArtistMBIDs[0] != a {
|
||||
t.Errorf("ids = %v: an ID3 'MusicBrainz Artist Id' frame should be read", m.ArtistMBIDs)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,13 +13,19 @@ import (
|
|||
var ErrUnreadable = errors.New("unreadable audio file")
|
||||
|
||||
// Tags are the descriptive tags of a track. Fields the file lacks are empty.
|
||||
//
|
||||
// The MusicBrainz ids are what the file itself says, read offline. ArtistMBIDs is
|
||||
// either empty or has one id per entry of Artists, in the same order: ids that
|
||||
// cannot be paired with their artists with certainty are not kept.
|
||||
type Tags struct {
|
||||
Title string
|
||||
Album string
|
||||
AlbumArtist string
|
||||
Artists []string
|
||||
TrackNo int
|
||||
DiscNo int
|
||||
Title string
|
||||
Album string
|
||||
AlbumArtist string
|
||||
Artists []string
|
||||
ArtistMBIDs []string
|
||||
AlbumArtistMBID string
|
||||
TrackNo int
|
||||
DiscNo int
|
||||
}
|
||||
|
||||
// Properties are the technical properties of an audio file.
|
||||
|
|
|
|||
|
|
@ -17,8 +17,10 @@ import (
|
|||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
|
|
@ -92,14 +94,7 @@ func (c *Client) SimilarArtists(ctx context.Context, key string, a Artist, limit
|
|||
case limit > MaxLimit:
|
||||
limit = MaxLimit
|
||||
}
|
||||
q := url.Values{"method": {"artist.getsimilar"}, "autocorrect": {"0"}, "limit": {strconv.Itoa(limit)}}
|
||||
if mbid != "" {
|
||||
q.Set("mbid", mbid)
|
||||
} else {
|
||||
q.Set("artist", name)
|
||||
}
|
||||
|
||||
body, err := c.call(ctx, key, q)
|
||||
body, err := c.callFor(ctx, key, name, mbid, url.Values{"method": {"artist.getsimilar"}, "limit": {strconv.Itoa(limit)}})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -114,6 +109,84 @@ func (c *Client) SimilarArtists(ctx context.Context, key string, a Artist, limit
|
|||
return decodeSimilar(doc.Similar.Artist)
|
||||
}
|
||||
|
||||
// Tag is one tag of an artist. Count is Last.fm's own number between 0 and 100,
|
||||
// relative to the artist (the top tag is usually 100).
|
||||
type Tag struct {
|
||||
Name string // lower case
|
||||
Count int
|
||||
}
|
||||
|
||||
// maxTagLen bounds a tag name; longer ones are not tags, they are noise.
|
||||
const maxTagLen = 100
|
||||
|
||||
// TopTags returns the top tags of an artist, most voted first. The name is never
|
||||
// autocorrected, and the MusicBrainz id wins over the name when known.
|
||||
func (c *Client) TopTags(ctx context.Context, key string, a Artist) ([]Tag, error) {
|
||||
name, mbid := strings.TrimSpace(a.Name), strings.TrimSpace(a.MBID)
|
||||
if name == "" && mbid == "" {
|
||||
return nil, ErrBadRequest
|
||||
}
|
||||
body, err := c.callFor(ctx, key, name, mbid, url.Values{"method": {"artist.gettoptags"}})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc struct {
|
||||
Top *struct {
|
||||
Tag json.RawMessage `json:"tag"`
|
||||
} `json:"toptags"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || doc.Top == nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
return decodeTags(doc.Top.Tag)
|
||||
}
|
||||
|
||||
// decodeTags reads the tag list, which Last.fm sends as an array or, with one
|
||||
// tag, as an object. Names are lower-cased and kept once with their highest
|
||||
// count; empty names, long names and counts of 0 or less are dropped.
|
||||
func decodeTags(raw json.RawMessage) ([]Tag, error) {
|
||||
type item struct {
|
||||
Name string `json:"name"`
|
||||
Count json.RawMessage `json:"count"`
|
||||
}
|
||||
var items []item
|
||||
switch {
|
||||
case len(raw) == 0 || string(raw) == "null":
|
||||
case raw[0] == '[':
|
||||
if err := json.Unmarshal(raw, &items); err != nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
default:
|
||||
var one item
|
||||
if err := json.Unmarshal(raw, &one); err != nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
items = []item{one}
|
||||
}
|
||||
best := map[string]int{}
|
||||
for _, it := range items {
|
||||
name := strings.ToLower(strings.TrimSpace(it.Name))
|
||||
count, err := strconv.ParseFloat(strings.Trim(string(it.Count), `"`), 64)
|
||||
if name == "" || utf8.RuneCountInString(name) > maxTagLen || err != nil || count < 1 || count > 100 {
|
||||
continue
|
||||
}
|
||||
if int(count) > best[name] {
|
||||
best[name] = int(count)
|
||||
}
|
||||
}
|
||||
out := make([]Tag, 0, len(best))
|
||||
for name, count := range best {
|
||||
out = append(out, Tag{Name: name, Count: count})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Count != out[j].Count {
|
||||
return out[i].Count > out[j].Count
|
||||
}
|
||||
return out[i].Name < out[j].Name
|
||||
})
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Check asks one cheap question to find out whether the key works.
|
||||
func (c *Client) Check(ctx context.Context, key string) error {
|
||||
body, err := c.call(ctx, key, url.Values{"method": {"chart.gettopartists"}, "limit": {"1"}})
|
||||
|
|
@ -129,6 +202,35 @@ func (c *Client) Check(ctx context.Context, key string) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// callFor asks about one artist. The MusicBrainz id goes first, since it cannot be
|
||||
// confused with a namesake. Last.fm does not know every id: when it says it does
|
||||
// not know this one and there is a name, the name is tried, never autocorrected,
|
||||
// which is what was done before an id was known. Any other failure is final.
|
||||
func (c *Client) callFor(ctx context.Context, key, name, mbid string, q url.Values) ([]byte, error) {
|
||||
q.Set("autocorrect", "0")
|
||||
if mbid != "" {
|
||||
byID := cloneValues(q)
|
||||
byID.Set("mbid", mbid)
|
||||
body, err := c.call(ctx, key, byID)
|
||||
if !errors.Is(err, ErrNotFound) || name == "" {
|
||||
return body, err
|
||||
}
|
||||
}
|
||||
if name == "" {
|
||||
return nil, ErrNotFound
|
||||
}
|
||||
q.Set("artist", name)
|
||||
return c.call(ctx, key, q)
|
||||
}
|
||||
|
||||
func cloneValues(q url.Values) url.Values {
|
||||
out := make(url.Values, len(q)+1)
|
||||
for k, v := range q {
|
||||
out[k] = append([]string(nil), v...)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Probe is Check as a providers.Checker: it answers with the errors the provider
|
||||
// settings understand, so that a test does not depend on this client.
|
||||
func (c *Client) Probe(ctx context.Context, key string) error {
|
||||
|
|
|
|||
|
|
@ -282,3 +282,134 @@ func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
|
|||
t.Errorf("an unreadable answer: err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- tags ----
|
||||
|
||||
const tagsBody = `{"toptags":{"tag":[
|
||||
{"name":"Rock","count":100,"url":"https://example.invalid/tag/rock"},
|
||||
{"name":"alternative rock","count":"54","url":"https://example.invalid/x"},
|
||||
{"name":"rock","count":10},
|
||||
{"name":"","count":9},
|
||||
{"name":"zero","count":0}
|
||||
],"@attr":{"artist":"Source"}}}`
|
||||
|
||||
func TestTopTagsDecodesTheAnswer(t *testing.T) {
|
||||
s := newServer(t, 200, tagsBody)
|
||||
got, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "Source"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Names are lower-cased and kept once (the higher count wins); empty names and
|
||||
// a count of 0 are dropped; the count may come as a string.
|
||||
if len(got) != 2 || got[0] != (Tag{Name: "rock", Count: 100}) || got[1] != (Tag{Name: "alternative rock", Count: 54}) {
|
||||
t.Fatalf("tags = %+v", got)
|
||||
}
|
||||
q := s.last(t)
|
||||
want := map[string]string{"method": "artist.gettoptags", "artist": "Source", "autocorrect": "0", "api_key": testKey, "format": "json"}
|
||||
for k, v := range want {
|
||||
if q.Get(k) != v {
|
||||
t.Errorf("%s = %q, want %q", k, q.Get(k), v)
|
||||
}
|
||||
}
|
||||
if len(q) != len(want) {
|
||||
t.Errorf("query = %v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTopTagsPrefersTheMBIDAndToleratesShapes(t *testing.T) {
|
||||
s := newServer(t, 200, `{"toptags":{"tag":{"name":"solo","count":7}}}`)
|
||||
got, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "x", MBID: "22222222-2222-2222-2222-222222222222"})
|
||||
if err != nil || len(got) != 1 || got[0].Name != "solo" {
|
||||
t.Fatalf("tags = %+v err=%v", got, err)
|
||||
}
|
||||
if q := s.last(t); q.Get("mbid") != "22222222-2222-2222-2222-222222222222" || q.Has("artist") || q.Get("autocorrect") != "0" {
|
||||
t.Errorf("query = %v", q)
|
||||
}
|
||||
empty := newServer(t, 200, `{"toptags":{"tag":[],"@attr":{"artist":"x"}}}`)
|
||||
if got, err := empty.client().TopTags(context.Background(), testKey, Artist{Name: "x"}); err != nil || len(got) != 0 {
|
||||
t.Errorf("no tags: %+v err=%v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTopTagsErrors(t *testing.T) {
|
||||
if _, err := newServer(t, 200, tagsBody).client().TopTags(context.Background(), testKey, Artist{}); !errors.Is(err, ErrBadRequest) {
|
||||
t.Errorf("nothing to look up: %v", err)
|
||||
}
|
||||
for _, tt := range []struct {
|
||||
status int
|
||||
body string
|
||||
want error
|
||||
}{
|
||||
{200, `{"error":6,"message":"The artist you supplied could not be found"}`, ErrNotFound},
|
||||
{403, `{"error":10,"message":"Invalid API key"}`, ErrInvalidKey},
|
||||
{200, `{"hello":"world"}`, ErrBadResponse},
|
||||
{500, `x`, ErrUnavailable},
|
||||
} {
|
||||
s := newServer(t, tt.status, tt.body)
|
||||
if _, err := s.client().TopTags(context.Background(), testKey, Artist{Name: "x"}); !errors.Is(err, tt.want) {
|
||||
t.Errorf("%d %s: err = %v, want %v", tt.status, tt.body, err, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Last.fm does not know every MusicBrainz id. When it does not know this one,
|
||||
// the name is tried, which is what was done before the file gave an id. A name
|
||||
// is only tried then: never when the id worked or failed for another reason.
|
||||
func TestAnIDLastFMDoesNotKnowFallsBackToTheName(t *testing.T) {
|
||||
const id = "22222222-2222-2222-2222-222222222222"
|
||||
unknownThenKnown := func(t *testing.T) (*httptest.Server, *[]url.Values) {
|
||||
var mu sync.Mutex
|
||||
var seen []url.Values
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
seen = append(seen, r.URL.Query())
|
||||
n := len(seen)
|
||||
mu.Unlock()
|
||||
if r.URL.Query().Has("mbid") && n == 1 {
|
||||
_, _ = w.Write([]byte(`{"error":6,"message":"The artist you supplied could not be found"}`))
|
||||
return
|
||||
}
|
||||
if r.URL.Query().Get("method") == "artist.gettoptags" {
|
||||
_, _ = w.Write([]byte(`{"toptags":{"tag":[{"name":"rock","count":9}]}}`))
|
||||
return
|
||||
}
|
||||
_, _ = w.Write([]byte(similarBody))
|
||||
}))
|
||||
t.Cleanup(srv.Close)
|
||||
return srv, &seen
|
||||
}
|
||||
client := func(srv *httptest.Server) *Client {
|
||||
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( test )", Retries: outbound.NoRetries}), srv.URL)
|
||||
}
|
||||
|
||||
srv, seen := unknownThenKnown(t)
|
||||
got, err := client(srv).SimilarArtists(context.Background(), testKey, Artist{Name: "Source", MBID: id}, 10)
|
||||
if err != nil || len(got) != 2 || len(*seen) != 2 {
|
||||
t.Fatalf("similar: %v err=%v requests=%d", got, err, len(*seen))
|
||||
}
|
||||
if (*seen)[0].Get("mbid") != id || (*seen)[1].Get("artist") != "Source" || (*seen)[1].Has("mbid") || (*seen)[1].Get("autocorrect") != "0" {
|
||||
t.Errorf("requests = %v", *seen)
|
||||
}
|
||||
|
||||
srv, seen = unknownThenKnown(t)
|
||||
tags, err := client(srv).TopTags(context.Background(), testKey, Artist{Name: "Source", MBID: id})
|
||||
if err != nil || len(tags) != 1 || len(*seen) != 2 || (*seen)[1].Get("artist") != "Source" {
|
||||
t.Fatalf("tags: %v err=%v requests=%v", tags, err, *seen)
|
||||
}
|
||||
|
||||
// Without a name there is nothing to try.
|
||||
srv, seen = unknownThenKnown(t)
|
||||
if _, err := client(srv).SimilarArtists(context.Background(), testKey, Artist{MBID: id}, 10); !errors.Is(err, ErrNotFound) || len(*seen) != 1 {
|
||||
t.Errorf("no name: err=%v requests=%d", err, len(*seen))
|
||||
}
|
||||
// Another failure is not an unknown artist.
|
||||
bad := newServer(t, 403, `{"error":10,"message":"Invalid API key"}`)
|
||||
if _, err := bad.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x", MBID: id}, 10); !errors.Is(err, ErrInvalidKey) || len(bad.queries) != 1 {
|
||||
t.Errorf("invalid key: err=%v requests=%d", err, len(bad.queries))
|
||||
}
|
||||
// An id that works is the only request.
|
||||
ok := newServer(t, 200, similarBody)
|
||||
if _, err := ok.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x", MBID: id}, 10); err != nil || len(ok.queries) != 1 {
|
||||
t.Errorf("working id: err=%v requests=%d", err, len(ok.queries))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
178
internal/providers/musicbrainz/musicbrainz.go
Normal file
178
internal/providers/musicbrainz/musicbrainz.go
Normal file
|
|
@ -0,0 +1,178 @@
|
|||
// Package musicbrainz is a small read-only client for the MusicBrainz web
|
||||
// service.
|
||||
//
|
||||
// Sonda only reads an artist whose MusicBrainz id it already knows. It never
|
||||
// searches by name: a search returns look-alikes (a conductor named like a rock
|
||||
// band) and picking one would be a guess. Only the id leaves the server.
|
||||
//
|
||||
// MusicBrainz asks every client to identify itself with a contact in the
|
||||
// User-Agent and to stay at one request per second: both are the business of the
|
||||
// outbound client this one sends its requests through. Errors never repeat the
|
||||
// provider's own text.
|
||||
package musicbrainz
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
)
|
||||
|
||||
// BaseURL is the MusicBrainz web service, version 2.
|
||||
const BaseURL = "https://musicbrainz.org/ws/2/"
|
||||
|
||||
// probeID is a public artist used to check that the service answers.
|
||||
const probeID = "a74b1b7f-71a5-4011-9441-d0b5e4122711"
|
||||
|
||||
// maxTagLen bounds a tag name; longer ones are not tags, they are noise.
|
||||
const maxTagLen = 100
|
||||
|
||||
// Errors a call can return, to be matched with errors.Is.
|
||||
var (
|
||||
ErrBadRequest = errors.New("musicbrainz: not a MusicBrainz id")
|
||||
ErrNotFound = errors.New("musicbrainz: unknown artist")
|
||||
ErrRateLimited = errors.New("musicbrainz: too many requests")
|
||||
ErrUnavailable = errors.New("musicbrainz: service unavailable")
|
||||
ErrBadResponse = errors.New("musicbrainz: unreadable answer")
|
||||
)
|
||||
|
||||
var idRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
|
||||
// Getter performs a GET. *outbound.Client is the one used in production.
|
||||
type Getter interface {
|
||||
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
|
||||
}
|
||||
|
||||
// Client talks to MusicBrainz. It is safe for concurrent use.
|
||||
type Client struct {
|
||||
http Getter
|
||||
base string
|
||||
}
|
||||
|
||||
// New builds a Client that sends its requests through g to base (BaseURL in
|
||||
// production).
|
||||
func New(g Getter, base string) *Client { return &Client{http: g, base: base} }
|
||||
|
||||
// Tag is one tag of an artist. Count is the number of people who voted for it, an
|
||||
// absolute number (unlike Last.fm's, which is relative). Genre says that
|
||||
// MusicBrainz also lists it among its curated genres.
|
||||
type Tag struct {
|
||||
Name string // lower case
|
||||
Count int
|
||||
Genre bool
|
||||
}
|
||||
|
||||
// ArtistTags returns the tags of the artist with this MusicBrainz id, most voted
|
||||
// first. An artist nobody tagged is an empty list, not an error.
|
||||
func (c *Client) ArtistTags(ctx context.Context, id string) ([]Tag, error) {
|
||||
if !idRe.MatchString(id) {
|
||||
return nil, ErrBadRequest
|
||||
}
|
||||
body, err := c.get(ctx, c.base+"artist/"+strings.ToLower(id)+"?inc=tags+genres&fmt=json")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc struct {
|
||||
ID string `json:"id"`
|
||||
Tags []struct {
|
||||
Name string `json:"name"`
|
||||
Count int `json:"count"`
|
||||
} `json:"tags"`
|
||||
Genres []struct {
|
||||
Name string `json:"name"`
|
||||
Count int `json:"count"`
|
||||
} `json:"genres"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || doc.ID == "" {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
|
||||
byName := map[string]*Tag{}
|
||||
add := func(name string, count int, genre bool) {
|
||||
name = strings.ToLower(strings.TrimSpace(name))
|
||||
if name == "" || utf8.RuneCountInString(name) > maxTagLen || count <= 0 {
|
||||
return
|
||||
}
|
||||
if t, ok := byName[name]; ok {
|
||||
t.Genre = t.Genre || genre
|
||||
if count > t.Count {
|
||||
t.Count = count
|
||||
}
|
||||
return
|
||||
}
|
||||
byName[name] = &Tag{Name: name, Count: count, Genre: genre}
|
||||
}
|
||||
for _, t := range doc.Tags {
|
||||
add(t.Name, t.Count, false)
|
||||
}
|
||||
for _, g := range doc.Genres {
|
||||
add(g.Name, g.Count, true)
|
||||
}
|
||||
out := make([]Tag, 0, len(byName))
|
||||
for _, t := range byName {
|
||||
out = append(out, *t)
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
if out[i].Count != out[j].Count {
|
||||
return out[i].Count > out[j].Count
|
||||
}
|
||||
return out[i].Name < out[j].Name
|
||||
})
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// Probe asks for one public artist, to find out whether the service answers. It
|
||||
// is a providers.Checker, and answers with the errors the settings understand.
|
||||
func (c *Client) Probe(ctx context.Context) error {
|
||||
body, err := c.get(ctx, c.base+"artist/"+probeID+"?fmt=json")
|
||||
switch {
|
||||
case errors.Is(err, ErrRateLimited):
|
||||
return providers.ErrRateLimited
|
||||
case errors.Is(err, ErrUnavailable):
|
||||
return providers.ErrUnreachable
|
||||
case err != nil:
|
||||
return err
|
||||
}
|
||||
var doc struct {
|
||||
ID string `json:"id"`
|
||||
}
|
||||
if json.Unmarshal(body, &doc) != nil || doc.ID == "" {
|
||||
return ErrBadResponse
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// get sends one request and returns the body of a successful answer.
|
||||
func (c *Client) get(ctx context.Context, rawURL string) ([]byte, error) {
|
||||
resp, err := c.http.Get(ctx, rawURL)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
var limited *outbound.RateLimitedError
|
||||
if errors.As(err, &limited) {
|
||||
return nil, ErrRateLimited
|
||||
}
|
||||
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err)
|
||||
}
|
||||
switch {
|
||||
case resp.Status == 200:
|
||||
return resp.Body, nil
|
||||
case resp.Status == 404:
|
||||
return nil, ErrNotFound
|
||||
case resp.Status == 400:
|
||||
return nil, ErrBadRequest
|
||||
case resp.Status == 429:
|
||||
return nil, ErrRateLimited
|
||||
case resp.Status >= 500:
|
||||
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
|
||||
}
|
||||
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
|
||||
}
|
||||
161
internal/providers/musicbrainz/musicbrainz_test.go
Normal file
161
internal/providers/musicbrainz/musicbrainz_test.go
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
package musicbrainz
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
)
|
||||
|
||||
const mbid = "11111111-2222-4333-8444-555555555555"
|
||||
|
||||
type server struct {
|
||||
*httptest.Server
|
||||
mu sync.Mutex
|
||||
paths []string
|
||||
hdr http.Header
|
||||
}
|
||||
|
||||
func newServer(t *testing.T, status int, body string) *server {
|
||||
t.Helper()
|
||||
s := &server{}
|
||||
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
s.mu.Lock()
|
||||
s.paths = append(s.paths, r.URL.RequestURI())
|
||||
s.hdr = r.Header.Clone()
|
||||
s.mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write([]byte(body))
|
||||
}))
|
||||
t.Cleanup(s.Close)
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *server) client() *Client {
|
||||
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries}), s.URL+"/ws/2/")
|
||||
}
|
||||
|
||||
func (s *server) requests() []string {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return append([]string(nil), s.paths...)
|
||||
}
|
||||
|
||||
const artistBody = `{"id":"11111111-2222-4333-8444-555555555555","name":"Synthetic Band","type":"Group",
|
||||
"tags":[{"name":"Alternative Rock","count":43},{"name":"british","count":2},{"name":"ambient pop","count":1},{"name":"zero","count":0},{"name":"","count":9}],
|
||||
"genres":[{"id":"x","name":"alternative rock","count":43,"disambiguation":""},{"id":"y","name":"art pop","count":3,"disambiguation":""}]}`
|
||||
|
||||
func TestArtistTagsReadsTagsAndMarksTheGenres(t *testing.T) {
|
||||
s := newServer(t, 200, artistBody)
|
||||
got, err := s.client().ArtistTags(context.Background(), mbid)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
want := []Tag{
|
||||
{Name: "alternative rock", Count: 43, Genre: true}, // names are compared in lower case
|
||||
{Name: "art pop", Count: 3, Genre: true}, // a genre that is not among the tags is kept
|
||||
{Name: "british", Count: 2},
|
||||
{Name: "ambient pop", Count: 1},
|
||||
}
|
||||
if len(got) != len(want) {
|
||||
t.Fatalf("tags = %+v, want %+v", got, want)
|
||||
}
|
||||
for i := range want {
|
||||
if got[i] != want[i] {
|
||||
t.Errorf("tag %d = %+v, want %+v", i, got[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtistTagsAsksOnlyForTheIdOfTheArtist(t *testing.T) {
|
||||
s := newServer(t, 200, artistBody)
|
||||
if _, err := s.client().ArtistTags(context.Background(), strings.ToUpper(mbid)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
reqs := s.requests()
|
||||
if len(reqs) != 1 || reqs[0] != "/ws/2/artist/"+mbid+"?inc=tags+genres&fmt=json" {
|
||||
t.Errorf("requests = %v", reqs)
|
||||
}
|
||||
if s.hdr.Get("User-Agent") != "Sonda/test ( https://example.org )" {
|
||||
t.Errorf("User-Agent = %q", s.hdr.Get("User-Agent"))
|
||||
}
|
||||
}
|
||||
|
||||
// Anything that is not a plain id never reaches the network: it would end up
|
||||
// inside the path of the request.
|
||||
func TestArtistTagsRefusesWhatIsNotAnID(t *testing.T) {
|
||||
for _, bad := range []string{"", "not-a-uuid", mbid + "/../x", mbid + "?x=1", " " + mbid + "\n", "00000000-0000-0000-0000-00000000000g"} {
|
||||
s := newServer(t, 200, artistBody)
|
||||
if _, err := s.client().ArtistTags(context.Background(), bad); !errors.Is(err, ErrBadRequest) {
|
||||
t.Errorf("%q: err = %v, want ErrBadRequest", bad, err)
|
||||
}
|
||||
if len(s.requests()) != 0 {
|
||||
t.Errorf("%q: a request left", bad)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestArtistTagsErrors(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
want error
|
||||
}{
|
||||
{"unknown id", 404, `{"error":"Not Found"}`, ErrNotFound},
|
||||
{"invalid id", 400, `{"error":"Invalid mbid."}`, ErrBadRequest},
|
||||
{"too many requests", 429, `{"error":"slow down"}`, ErrRateLimited},
|
||||
{"over the limit", 503, `{"error":"Your requests are being throttled"}`, ErrUnavailable},
|
||||
{"server error", 500, `oops`, ErrUnavailable},
|
||||
{"not json", 200, `<html>`, ErrBadResponse},
|
||||
{"empty", 200, ``, ErrBadResponse},
|
||||
{"json of another kind", 200, `{"hello":"world"}`, ErrBadResponse},
|
||||
{"forbidden", 403, `{"error":"banned"}`, ErrBadResponse},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newServer(t, tt.status, tt.body)
|
||||
if _, err := s.client().ArtistTags(context.Background(), mbid); !errors.Is(err, tt.want) {
|
||||
t.Fatalf("err = %v, want %v", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
// An artist with no tags at all is an answer, not an error.
|
||||
s := newServer(t, 200, `{"id":"`+mbid+`","name":"Quiet"}`)
|
||||
got, err := s.client().ArtistTags(context.Background(), mbid)
|
||||
if err != nil || len(got) != 0 {
|
||||
t.Errorf("no tags: %+v err=%v", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestErrorsDoNotCopyTheProvidersText(t *testing.T) {
|
||||
s := newServer(t, 400, `{"error":"secret-text-from-the-provider"}`)
|
||||
_, err := s.client().ArtistTags(context.Background(), mbid)
|
||||
if err == nil || strings.Contains(err.Error(), "secret-text") {
|
||||
t.Errorf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
|
||||
ok := newServer(t, 200, `{"id":"a74b1b7f-71a5-4011-9441-d0b5e4122711","name":"Anyone"}`)
|
||||
if err := ok.client().Probe(context.Background()); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if reqs := ok.requests(); len(reqs) != 1 || strings.Contains(reqs[0], "inc=") {
|
||||
t.Errorf("a test asks for one plain artist: %v", reqs)
|
||||
}
|
||||
down := newServer(t, 503, `x`)
|
||||
if err := down.client().Probe(context.Background()); !errors.Is(err, providers.ErrUnreachable) {
|
||||
t.Errorf("down: %v", err)
|
||||
}
|
||||
slow := newServer(t, 429, `x`)
|
||||
if err := slow.client().Probe(context.Background()); !errors.Is(err, providers.ErrRateLimited) {
|
||||
t.Errorf("rate limited: %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -70,8 +70,9 @@ func (realClock) Sleep(ctx context.Context, d time.Duration) error {
|
|||
|
||||
// Options configure a Client. Zero values mean sensible defaults.
|
||||
type Options struct {
|
||||
UserAgent string
|
||||
Timeout time.Duration // per attempt; default 10s
|
||||
UserAgent string
|
||||
UserAgentFunc func() string // when set, read for every request instead of UserAgent
|
||||
Timeout time.Duration // per attempt; default 10s
|
||||
|
||||
MinInterval time.Duration // minimum time between requests; 0 = no limit
|
||||
MaxBody int64 // largest accepted answer; default 4 MiB
|
||||
|
|
@ -99,7 +100,7 @@ type Response struct {
|
|||
|
||||
// Client talks to one provider. It is safe for concurrent use.
|
||||
type Client struct {
|
||||
userAgent string
|
||||
userAgent func() string
|
||||
timeout time.Duration
|
||||
minInterval time.Duration
|
||||
maxBody int64
|
||||
|
|
@ -130,7 +131,7 @@ func New(o Options) *Client {
|
|||
return v
|
||||
}
|
||||
c := &Client{
|
||||
userAgent: o.UserAgent,
|
||||
userAgent: func() string { return o.UserAgent },
|
||||
timeout: def(o.Timeout, 10*time.Second),
|
||||
minInterval: o.MinInterval,
|
||||
maxBody: o.MaxBody,
|
||||
|
|
@ -143,6 +144,9 @@ func New(o Options) *Client {
|
|||
clock: o.Clock,
|
||||
jitter: o.Jitter,
|
||||
}
|
||||
if o.UserAgentFunc != nil {
|
||||
c.userAgent = o.UserAgentFunc
|
||||
}
|
||||
if c.maxBody <= 0 {
|
||||
c.maxBody = 4 << 20
|
||||
}
|
||||
|
|
@ -268,7 +272,7 @@ func (c *Client) once(ctx context.Context, rawURL string) (*Response, error) {
|
|||
if err != nil {
|
||||
return nil, ErrBadURL
|
||||
}
|
||||
req.Header.Set("User-Agent", c.userAgent)
|
||||
req.Header.Set("User-Agent", c.userAgent())
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := c.hc.Do(req)
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue