feat(discovery): similar artists from Last.fm

- feat: GET /api/v1/artists/{id}/similar lists the artists Last.fm finds similar
  to one in the library, best first, each with the provider's own score and
  rank and a flag saying whether it is in the library. It answers 200 when the
  provider is off or failing, with provider_status, stale and retrieved_at
- feat: a thin Last.fm client on top of the outbound client. It always sends
  autocorrect=0 so a misspelled local name cannot turn into another artist,
  prefers the MusicBrainz id when known, always sets a limit, and reads the
  error field even when the status is 200. Errors never carry the URL or the
  provider's own message, because the key travels in the query string
- feat: results are stored in artist_similar with a freshness ledger in
  provider_fetches: 7 days for an answer, 1 day for "unknown artist". Old data
  is served when the provider fails, and concurrent requests for the same
  artist share one call that does not depend on the first caller staying
- feat: artists named by a provider but without files become artists without
  tracks. GET /artists/{id} can show them (track_count 0), the lists of the
  library never include them, and a later scan adopts them by name. A provider
  id is never written onto an existing artist; names match only when at least
  one side has no id, and different ids always mean different artists
- feat: POST /providers/{name}/test (admin) checks a stored key with one cheap
  request, whether the provider is on or off, and reports ok, no_api_key,
  key_refused, rate_limited, unreachable or error
- test: the whole request is checked from the receiving end, with a user, a
  listen, a title and a library path in the database: only the artist and the
  key leave, with a User-Agent that names the project
- docs: README and OpenAPI updated; the contract test calls the new operations
This commit is contained in:
tgiachi 2026-11-11 16:36:16 +01:00
commit aafb2dd906
24 changed files with 2651 additions and 19 deletions

View file

@ -14,11 +14,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Catalog.** Browse and search artists, albums and tracks, with paging, by library, artist or album. Only tracks you can actually play are listed.
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere. Today Sonda makes no outbound network requests at all.
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to one of yours, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
## What is not there yet
Similar-artist discovery, recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Last.fm and MusicBrainz can already be set up (see [External providers](#external-providers)), but nothing asks them anything yet.
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. MusicBrainz can be set up (see [External providers](#external-providers)) but nothing asks it anything yet.
## Requirements
@ -175,7 +176,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
## External providers
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today a provider can be set up but nothing uses it yet.
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today only Last.fm is used, to find similar artists.
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
@ -186,11 +187,16 @@ Set them up with `PUT /api/v1/providers/{name}` (admin), for example:
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"api_key":"<your 32 character key>"}'
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"enabled":true}'
curl -s $BASE/providers -H "$AUTH" | jq
curl -s -X POST $BASE/providers/lastfm/test -H "$AUTH" | jq # does the key work?
```
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down.
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to one of yours and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the first is always 1), so compare artists only inside one answer.
The name of the artist is sent exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
## The API at a glance
@ -205,13 +211,14 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
| anyone | `POST /auth/login`, `/auth/refresh`, `/auth/logout` | Open, renew and end a session. |
| signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). |
| signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. |
| signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 100). |
| signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. |
| signed in | `POST /listens` | Record a `play`, `complete` or `skip`; the answer says whether it counts as a listen. |
| signed in | `GET /listens` | Your own history, newest first, with `counted` per event. Query: `limit`, `offset`, `counted=true`. |
| signed in | `GET /library/scan` | Scan status and the last result. |
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
| admin | `GET /providers`, `PUT /providers/{name}` | See and set up Last.fm and MusicBrainz. The API key is write-only. |
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test` | See, set up and check Last.fm and MusicBrainz. The API key is write-only. |
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.
@ -258,7 +265,8 @@ internal/api/openapi/ the OpenAPI spec, embedded in the binary
internal/auth/ users, roles, passwords, access and refresh tokens
internal/catalog/ browsing and searching artists, albums, tracks
internal/library/ libraries, the scanner, the event hub, the tag reader
internal/providers/ external providers: settings, and the careful outbound HTTP client
internal/discovery/ similar artists: what the providers said, kept and refreshed
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm client
internal/listening/ listen events
internal/storage/ SQLite and PostgreSQL through bun, the repositories
internal/streaming/ opening a track's file safely

View file

@ -18,9 +18,12 @@ import (
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/config"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/outbound"
"codeberg.org/squidd/sonda/internal/storage"
"codeberg.org/squidd/sonda/internal/streaming"
"codeberg.org/squidd/sonda/migrations"
@ -28,6 +31,12 @@ import (
const defaultConfigFile = "sonda.toml"
// version goes into the User-Agent sent to providers.
const version = "0.1.0"
// lastfmMinInterval spaces the requests to Last.fm well below its limit.
const lastfmMinInterval = 250 * time.Millisecond
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "sondad:", err)
@ -98,6 +107,15 @@ func run() error {
libraries.SetBusy(func() bool { return scans.Status().Running })
warnAboutLibraries(ctx, logger, libraries)
// External providers: nothing is contacted unless an admin switched one on.
providerSettings := providers.NewService(storage.NewProviderRepo(db))
lastfmClient := lastfm.New(outbound.New(outbound.Options{
UserAgent: providers.UserAgent(version, ""),
MinInterval: lastfmMinInterval,
}), lastfm.BaseURL)
providerSettings.SetChecker(providers.LastFM, lastfmClient.Probe)
discoveries := discovery.NewService(storage.NewDiscoveryRepo(db), lastfmClient, providerSettings, logger)
router := httpapi.NewRouter(httpapi.Deps{
Logger: logger,
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()),
@ -106,7 +124,8 @@ func run() error {
Scan: scanControl{ctx: ctx, m: scans},
Libraries: libraries,
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
Providers: providers.NewService(storage.NewProviderRepo(db)),
Providers: providerSettings,
Discovery: discoveries,
Streams: streaming.NewService(libraryRepo),
})

2
go.mod
View file

@ -15,6 +15,7 @@ require (
go.senan.xyz/taglib v0.15.0
go.uber.org/zap v1.28.0
golang.org/x/crypto v0.58.0
golang.org/x/sync v0.24.0
modernc.org/sqlite v1.62.0
)
@ -62,7 +63,6 @@ require (
go.yaml.in/yaml/v4 v4.0.0 // indirect
golang.org/x/arch v0.22.0 // indirect
golang.org/x/net v0.60.0 // indirect
golang.org/x/sync v0.24.0 // indirect
golang.org/x/sys v0.49.0 // indirect
golang.org/x/text v0.43.0 // indirect
google.golang.org/protobuf v1.36.10 // indirect

2
go.sum
View file

@ -52,6 +52,8 @@ github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50 h1:PlNWJRxu5APSV0pWqf
github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50/go.mod h1:yUo1AMMpsPIDFGuwDK8nLcf8+vvAd5l2O6JDZ6hpwrM=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=

View file

@ -16,6 +16,7 @@ import (
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
@ -74,6 +75,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
}
tokens := auth.NewTokenIssuer(secret, time.Hour)
prov := providers.NewService(storage.NewProviderRepo(db))
deps := Deps{
Logger: zap.NewNop(),
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, 24*time.Hour),
@ -81,7 +83,8 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
Listening: listening.NewService(storage.NewListeningRepo(db), 20*time.Second),
Scan: &fakeScan{},
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
Providers: providers.NewService(storage.NewProviderRepo(db)),
Providers: prov,
Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()),
Streams: streaming.NewService(noFiles{}),
}
mutate(&deps, db)

View file

@ -19,9 +19,13 @@ import (
"github.com/getkin/kin-openapi/routers"
"github.com/getkin/kin-openapi/routers/legacy"
"github.com/uptrace/bun"
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/storage"
"codeberg.org/squidd/sonda/internal/streaming"
)
@ -81,6 +85,7 @@ func (a scanAdapter) StartLibrary(id int64) error { return a.m.StartLibrary(cont
func (a scanAdapter) Status() library.Status { return a.m.Status() }
type contract struct {
lastfm *fakeLastfm
t *testing.T
h http.Handler
db *bun.DB
@ -93,7 +98,12 @@ type contract struct {
func newContract(t *testing.T) *contract {
t.Helper()
reader := &contractReader{}
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
d.Providers = prov
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), lf, prov, zap.NewNop())
repo := storage.NewLibraryRepo(db)
mgr := library.NewManager(library.NewScanner(repo, reader, library.NewHub(), 2), repo)
libs := library.NewLibraries(repo)
@ -115,7 +125,7 @@ func newContract(t *testing.T) *contract {
t.Fatal(err)
}
}
return &contract{t: t, h: h, db: db, reader: reader, router: router, seen: map[string]bool{}, music: music}
return &contract{lastfm: lf, t: t, h: h, db: db, reader: reader, router: router, seen: map[string]bool{}, music: music}
}
// call sends a request to /api/v1<path>, checks it and its answer against the
@ -380,6 +390,14 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
c.expect(c.call("PUT", "/providers/spotify", token, map[string]any{"enabled": true}), 404, "unknown provider", "provider_not_found")
c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test without a key")
c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 400, "test a provider that has no test", "invalid_input")
c.expect(c.call("POST", "/providers/spotify/test", token, nil), 404, "test an unknown provider", "provider_not_found")
c.expect(c.call("POST", "/providers/lastfm/test", "", nil), 401, "test without a token", "unauthorized")
off := c.expect(c.call("GET", "/artists/"+artistID+"/similar", token, nil), 200, "similar artists with the provider off")
if off["provider_status"] != "disabled" || c.lastfm.count() != 0 {
t.Fatalf("similar with the provider off = %v, calls %d", off, c.lastfm.count())
}
saved := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "0123456789abcdef0123456789abcdef"}), 200, "save a key")
if saved["has_api_key"] != true || saved["updated_at"] == nil {
t.Fatalf("saved = %v", saved)
@ -387,6 +405,15 @@ func TestResponsesMatchTheSpec(t *testing.T) {
if on := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 200, "enable"); on["enabled"] != true {
t.Fatalf("enabled = %v", on)
}
if ok := c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test with a key"); ok["ok"] != true {
t.Fatalf("test = %v", ok)
}
similar := c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=5", token, nil), 200, "similar artists")
if items := similar["items"].([]any); similar["provider_status"] != "ok" || len(items) != 1 {
t.Fatalf("similar = %v", similar)
}
c.expect(c.call("GET", "/artists/999/similar", token, nil), 404, "similar to an unknown artist", "artist_not_found")
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
// ---- playback ----

View file

@ -39,6 +39,8 @@ func writeProviderError(c *gin.Context, err error) {
switch {
case errors.Is(err, providers.ErrUnknownProvider):
writeError(c, http.StatusNotFound, "provider_not_found", "no such provider")
case errors.Is(err, providers.ErrNoTest):
writeError(c, http.StatusBadRequest, "invalid_input", "this provider has no test")
case errors.Is(err, providers.ErrInvalidSettings):
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
default:
@ -79,3 +81,12 @@ func (h *providersHandlers) update(c *gin.Context) {
}
c.JSON(http.StatusOK, toProviderJSON(info))
}
func (h *providersHandlers) test(c *gin.Context) {
res, err := h.svc.Test(c.Request.Context(), c.Param("name"))
if err != nil {
writeProviderError(c, err)
return
}
c.JSON(http.StatusOK, gin.H{"ok": res.OK, "status": res.Status})
}

View file

@ -0,0 +1,69 @@
package httpapi
import (
"errors"
"net/http"
"github.com/gin-gonic/gin"
"codeberg.org/squidd/sonda/internal/discovery"
)
// discoveryHandlers serve what providers say about artists outside the library.
type discoveryHandlers struct{ svc discoveryService }
type similarItemJSON struct {
Artist artistRefJSON `json:"artist"`
InLibrary bool `json:"in_library"`
Score float64 `json:"score"`
Rank int `json:"rank"`
}
type similarJSON struct {
Provider string `json:"provider"`
ProviderStatus string `json:"provider_status"`
Stale bool `json:"stale"`
RetrievedAt *string `json:"retrieved_at"`
Attribution string `json:"attribution"`
Items []similarItemJSON `json:"items"`
}
func toSimilarJSON(r discovery.Result) similarJSON {
out := similarJSON{
Provider: r.Provider, ProviderStatus: r.ProviderStatus, Stale: r.Stale,
Attribution: discovery.Attribution, Items: make([]similarItemJSON, len(r.Items)),
}
if !r.RetrievedAt.IsZero() {
out.RetrievedAt = rfc3339(r.RetrievedAt)
}
for i, it := range r.Items {
out.Items[i] = similarItemJSON{
Artist: artistRefJSON{ID: it.Artist.ID, Name: it.Artist.Name},
InLibrary: it.InLibrary, Score: it.Score, Rank: it.Rank,
}
}
return out
}
func (h *discoveryHandlers) similar(c *gin.Context) {
id, ok := pathID(c)
if !ok {
return
}
limit, ok := queryInt(c, "limit")
if !ok {
return
}
res, err := h.svc.Similar(c.Request.Context(), id, clampInt(limit))
switch {
case errors.Is(err, discovery.ErrNotFound):
writeError(c, http.StatusNotFound, "artist_not_found", "not found")
case errors.Is(err, discovery.ErrInvalidQuery):
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
case err != nil:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
default:
c.JSON(http.StatusOK, toSimilarJSON(res))
}
}

View file

@ -1,7 +1,9 @@
package httpapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"strings"
"testing"
@ -9,6 +11,9 @@ import (
"github.com/uptrace/bun"
"go.uber.org/zap"
"go.uber.org/zap/zaptest/observer"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/storage"
)
const lastfmKey = "0123456789abcdef0123456789abcdef"
@ -169,3 +174,60 @@ func TestTheKeyNeverReachesTheLog(t *testing.T) {
}
}
}
// ---- testing a provider ----
func TestProviderTestIsAdminOnly(t *testing.T) {
h := newTestAPI(t)
admin := registerAndLogin(t, h)
_, bob := addUser(t, h, admin, "bob", false)
if r := do(t, h, "POST", "/api/v1/providers/lastfm/test", "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("no token: %d", r.Code)
}
if r := do(t, h, "POST", "/api/v1/providers/lastfm/test", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
t.Errorf("normal user: %d %v", r.Code, r.Body)
}
}
func TestProviderTest(t *testing.T) {
var asked string
var answer error
h, _ := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov.SetChecker(providers.LastFM, func(_ context.Context, key string) error { asked = key; return answer })
d.Providers = prov
})
admin := registerAndLogin(t, h)
// Without a key nobody is asked.
r := do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
if r.Code != http.StatusOK || r.Body["ok"] != false || r.Body["status"] != "no_api_key" || asked != "" {
t.Fatalf("no key: %d %v asked=%q", r.Code, r.Body, asked)
}
sameKeys(t, "test result", r.Body, "ok", "status")
do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": lastfmKey})
r = do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
if r.Code != http.StatusOK || r.Body["ok"] != true || r.Body["status"] != "ok" || asked != lastfmKey {
t.Fatalf("working key: %d %v", r.Code, r.Body)
}
for want, err := range map[string]error{
"key_refused": providers.ErrKeyRefused, "rate_limited": providers.ErrRateLimited,
"unreachable": providers.ErrUnreachable, "error": errors.New("boom " + lastfmKey),
} {
answer = err
r = do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
raw, _ := json.Marshal(r.Body)
if r.Code != http.StatusOK || r.Body["ok"] != false || r.Body["status"] != want || strings.Contains(string(raw), lastfmKey) {
t.Errorf("%s: %d %v", want, r.Code, r.Body)
}
}
if r = do(t, h, "POST", "/api/v1/providers/spotify/test", admin, nil); r.Code != http.StatusNotFound || errCode(r) != "provider_not_found" {
t.Errorf("unknown provider: %d %v", r.Code, r.Body)
}
if r = do(t, h, "POST", "/api/v1/providers/musicbrainz/test", admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
t.Errorf("a provider without a test: %d %v", r.Code, r.Body)
}
}

View file

@ -12,6 +12,7 @@ import (
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
@ -57,6 +58,11 @@ type scanService interface {
type providerService interface {
List(ctx context.Context) ([]providers.Info, error)
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
Test(ctx context.Context, name string) (providers.TestResult, error)
}
type discoveryService interface {
Similar(ctx context.Context, artistID int64, limit int) (discovery.Result, error)
}
type catalogService interface {
@ -85,6 +91,7 @@ type Deps struct {
Libraries libraryService
Catalog catalogService
Providers providerService
Discovery discoveryService
Streams streamService
}
@ -119,6 +126,8 @@ func NewRouter(d Deps) *gin.Engine {
cat := &catalogHandlers{svc: d.Catalog}
protected.GET("/artists", cat.listArtists)
protected.GET("/artists/:id", cat.getArtist)
disc := &discoveryHandlers{svc: d.Discovery}
protected.GET("/artists/:id/similar", disc.similar)
protected.GET("/albums", cat.listAlbums)
protected.GET("/albums/:id", cat.getAlbum)
protected.GET("/tracks", cat.listTracks)
@ -143,6 +152,7 @@ func NewRouter(d Deps) *gin.Engine {
prov := &providersHandlers{svc: d.Providers}
admin.GET("/providers", prov.list)
admin.PUT("/providers/:name", prov.update)
admin.POST("/providers/:name/test", prov.test)
u := &usersHandlers{svc: d.Auth}
admin.GET("/users", u.list)
admin.POST("/users", u.create)

View file

@ -0,0 +1,349 @@
package httpapi
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"net/url"
"reflect"
"strconv"
"strings"
"sync"
"testing"
"github.com/uptrace/bun"
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/outbound"
"codeberg.org/squidd/sonda/internal/storage"
)
// fakeLastfm answers for Last.fm and counts the questions.
type fakeLastfm struct {
mu sync.Mutex
calls int
key string
result []lastfm.Similar
err error
}
func (f *fakeLastfm) SimilarArtists(_ context.Context, key string, _ lastfm.Artist, _ int) ([]lastfm.Similar, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.calls++
f.key = key
return f.result, f.err
}
func (f *fakeLastfm) count() int {
f.mu.Lock()
defer f.mu.Unlock()
return f.calls
}
type similarEnv struct {
h http.Handler
db *bun.DB
admin string
source *fakeLastfm
alpha int64 // a local artist
beta int64 // another local artist
}
// newSimilarEnv wires the real catalog, discovery and provider services on a
// database that holds two local artists, Alpha and Beta, with Last.fm switched
// off and without a key.
func newSimilarEnv(t *testing.T) *similarEnv {
t.Helper()
src := &fakeLastfm{result: []lastfm.Similar{
{Name: "Beta", Match: 1},
{Name: "Gamma", MBID: "99999999-0000-0000-0000-000000000001", Match: 0.5},
}}
e := &similarEnv{source: src}
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
d.Providers = prov
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
})
e.admin = registerAndLogin(t, e.h)
lib := storage.NewLibraryRepo(e.db)
l, err := lib.CreateLibrary(context.Background(), "music", "/lib")
if err != nil {
t.Fatal(err)
}
for i, name := range []string{"Alpha", "Beta"} {
f := library.ParsedFile{
FileInfo: library.FileInfo{RelPath: name + ".flac", Size: 1, MTimeUnix: 1},
Metadata: library.Metadata{
Tags: library.Tags{Title: name + " song", Artists: []string{name}},
Properties: library.Properties{Format: "flac", DurationMs: 1000},
},
}
if _, err := lib.SaveFile(context.Background(), l.ID, f, 0); err != nil {
t.Fatalf("seed %d: %v", i, err)
}
}
e.alpha, e.beta = e.artist(t, "Alpha"), e.artist(t, "Beta")
return e
}
func (e *similarEnv) artist(t *testing.T, name string) int64 {
t.Helper()
var id int64
if err := e.db.QueryRow("SELECT id FROM artists WHERE name = ?", name).Scan(&id); err != nil {
t.Fatalf("artist %q: %v", name, err)
}
return id
}
func (e *similarEnv) enableLastfm(t *testing.T) {
t.Helper()
for _, body := range []map[string]any{{"api_key": lastfmKey}, {"enabled": true}} {
if r := do(t, e.h, "PUT", "/api/v1/providers/lastfm", e.admin, body); r.Code != http.StatusOK {
t.Fatalf("set up lastfm: %d %v", r.Code, r.Body)
}
}
}
func (e *similarEnv) path(id int64) string {
return "/api/v1/artists/" + strconv.FormatInt(id, 10) + "/similar"
}
func TestSimilarNeedsASignedInUser(t *testing.T) {
e := newSimilarEnv(t)
if r := do(t, e.h, "GET", e.path(e.alpha), "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("no token: %d", r.Code)
}
}
func TestSimilarRejectsWhatIsNotAnArtistOrALimit(t *testing.T) {
e := newSimilarEnv(t)
e.enableLastfm(t)
for name, tt := range map[string]struct {
path string
code int
err string
}{
"unknown artist": {e.path(99999), 404, "artist_not_found"},
"not a number": {"/api/v1/artists/abc/similar", 400, "invalid_input"},
"artist with no files and no discovery": {e.path(e.artistWithoutFiles(t)), 404, "artist_not_found"},
"limit not a number": {e.path(e.alpha) + "?limit=lots", 400, "invalid_input"},
"negative limit": {e.path(e.alpha) + "?limit=-1", 400, "invalid_input"},
} {
r := do(t, e.h, "GET", tt.path, e.admin, nil)
if r.Code != tt.code || errCode(r) != tt.err {
t.Errorf("%s: %d %v, want %d %s", name, r.Code, r.Body, tt.code, tt.err)
}
}
if e.source.count() != 0 {
t.Error("Last.fm was asked about something that is not an artist")
}
}
func (e *similarEnv) artistWithoutFiles(t *testing.T) int64 {
t.Helper()
var id int64
if err := e.db.QueryRow("INSERT INTO artists (name, name_key) VALUES ('Nobody', 'nobody') RETURNING id").Scan(&id); err != nil {
t.Fatal(err)
}
return id
}
func TestSimilarWithTheProviderOffAsksNobody(t *testing.T) {
e := newSimilarEnv(t)
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if r.Code != http.StatusOK {
t.Fatalf("%d %v", r.Code, r.Body)
}
sameKeys(t, "similar", r.Body, "provider", "provider_status", "stale", "retrieved_at", "attribution", "items")
items, ok := r.Body["items"].([]any)
if r.Body["provider"] != "lastfm" || r.Body["provider_status"] != "disabled" || r.Body["stale"] != false ||
!ok || len(items) != 0 {
t.Errorf("answer = %v", r.Body)
}
if v, present := r.Body["retrieved_at"]; !present || v != nil {
t.Errorf("retrieved_at = %#v, want an explicit null when nothing is known", v)
}
if e.source.count() != 0 {
t.Error("a provider that is off was contacted")
}
}
func TestSimilarArtists(t *testing.T) {
e := newSimilarEnv(t)
e.enableLastfm(t)
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if r.Code != http.StatusOK {
t.Fatalf("%d %v", r.Code, r.Body)
}
sameKeys(t, "similar", r.Body, "provider", "provider_status", "stale", "retrieved_at", "attribution", "items")
if r.Body["provider_status"] != "ok" || r.Body["stale"] != false || r.Body["retrieved_at"] == nil {
t.Errorf("answer = %v", r.Body)
}
if !strings.Contains(r.Body["attribution"].(string), "Last.fm") {
t.Errorf("attribution = %v", r.Body["attribution"])
}
items := r.Body["items"].([]any)
if len(items) != 2 {
t.Fatalf("items = %v", items)
}
first, second := items[0].(map[string]any), items[1].(map[string]any)
sameKeys(t, "item", first, "artist", "in_library", "score", "rank")
sameSet := first["artist"].(map[string]any)
if sameSet["name"] != "Beta" || sameSet["id"] != float64(e.beta) || first["in_library"] != true || first["rank"] != float64(1) || first["score"] != float64(1) {
t.Errorf("first = %v", first)
}
if second["artist"].(map[string]any)["name"] != "Gamma" || second["in_library"] != false || second["rank"] != float64(2) || second["score"] != 0.5 {
t.Errorf("second = %v", second)
}
if e.source.key != lastfmKey {
t.Error("the stored key was not the one used")
}
raw, _ := json.Marshal(r.Body)
if strings.Contains(string(raw), lastfmKey) {
t.Fatal("the answer contains the API key")
}
// A second look comes from the database; the page size trims it.
again := do(t, e.h, "GET", e.path(e.alpha)+"?limit=1", e.admin, nil)
if again.Code != http.StatusOK || len(again.Body["items"].([]any)) != 1 || e.source.count() != 1 {
t.Errorf("second look: %d items=%v calls=%d", again.Code, again.Body["items"], e.source.count())
}
}
func TestSimilarWhenLastfmFailsIsAnAnswerNotAnError(t *testing.T) {
e := newSimilarEnv(t)
e.enableLastfm(t)
e.source.err = lastfm.ErrUnavailable
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if r.Code != http.StatusOK || r.Body["provider_status"] != "error" || len(r.Body["items"].([]any)) != 0 {
t.Fatalf("%d %v", r.Code, r.Body)
}
}
// Artists that only a provider named can be shown, and never appear in the
// lists of the library.
func TestADiscoveredArtistCanBeOpenedButIsNotListed(t *testing.T) {
e := newSimilarEnv(t)
e.enableLastfm(t)
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
gamma := int64(r.Body["items"].([]any)[1].(map[string]any)["artist"].(map[string]any)["id"].(float64))
a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil)
if a.Code != http.StatusOK || a.Body["name"] != "Gamma" || a.Body["track_count"] != float64(0) || a.Body["album_count"] != float64(0) {
t.Fatalf("GetArtist: %d %v", a.Code, a.Body)
}
list := do(t, e.h, "GET", "/api/v1/artists?q=gamma", e.admin, nil)
if list.Body["total"] != float64(0) {
t.Errorf("a discovered artist is listed: %v", list.Body)
}
// And it can be asked about in turn.
e.source.result = []lastfm.Similar{{Name: "Delta", Match: 1}}
next := do(t, e.h, "GET", e.path(gamma), e.admin, nil)
if next.Code != http.StatusOK || len(next.Body["items"].([]any)) != 1 {
t.Errorf("similar to a discovered artist: %d %v", next.Code, next.Body)
}
}
// What leaves the server, seen from the other end: a real Last.fm client and
// outbound client talk to a local server that records the whole request. The
// database holds a user, a listen, a track title and a library path; none of
// them may be in it.
func TestWhatLeavesTheServerIsTheArtistAndTheKeyOnly(t *testing.T) {
var (
mu sync.Mutex
seen []string
hdr http.Header
)
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
seen = append(seen, r.Method+" "+r.URL.String())
hdr = r.Header.Clone()
mu.Unlock()
_, _ = w.Write([]byte(`{"similarartists":{"artist":[{"name":"Gamma","mbid":"","match":"1"}]}}`))
}))
defer remote.Close()
var alpha int64
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
d.Providers = prov
client := lastfm.New(outbound.New(outbound.Options{UserAgent: providers.UserAgent("1.2.3", "")}), remote.URL)
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), client, prov, zap.NewNop())
})
admin := registerAndLogin(t, h)
l, err := storage.NewLibraryRepo(db).CreateLibrary(context.Background(), "Private Collection", "/home/alice/secret-music")
if err != nil {
t.Fatal(err)
}
f := library.ParsedFile{
FileInfo: library.FileInfo{RelPath: "private/embarrassing-song.flac", Size: 1, MTimeUnix: 1},
Metadata: library.Metadata{
Tags: library.Tags{Title: "Embarrassing Song", Album: "Guilty Pleasures", Artists: []string{"Alpha"}},
Properties: library.Properties{Format: "flac", DurationMs: 1000},
},
}
if _, err := storage.NewLibraryRepo(db).SaveFile(context.Background(), l.ID, f, 0); err != nil {
t.Fatal(err)
}
var track int64
_ = db.QueryRow("SELECT id FROM tracks WHERE title = 'Embarrassing Song'").Scan(&track)
do(t, h, "POST", "/api/v1/listens", admin, map[string]any{
"track_id": track, "event_type": "complete", "idempotency_key": "k1"})
_ = db.QueryRow("SELECT id FROM artists WHERE name = 'Alpha'").Scan(&alpha)
for _, body := range []map[string]any{{"api_key": lastfmKey}, {"enabled": true}} {
if r := do(t, h, "PUT", "/api/v1/providers/lastfm", admin, body); r.Code != http.StatusOK {
t.Fatal(r.Body)
}
}
r := do(t, h, "GET", "/api/v1/artists/"+strconv.FormatInt(alpha, 10)+"/similar", admin, nil)
if r.Code != http.StatusOK || r.Body["provider_status"] != "ok" {
t.Fatalf("%d %v", r.Code, r.Body)
}
mu.Lock()
defer mu.Unlock()
if len(seen) != 1 {
t.Fatalf("requests = %v, want exactly one", seen)
}
u, _ := url.Parse(strings.SplitN(seen[0], " ", 2)[1])
want := map[string]string{
"method": "artist.getsimilar", "artist": "Alpha", "autocorrect": "0", "limit": "50",
"api_key": lastfmKey, "format": "json",
}
got := map[string]string{}
for k, v := range u.Query() {
got[k] = v[0]
}
if !reflect.DeepEqual(got, want) {
t.Errorf("query = %v, want %v", got, want)
}
// Only a User-Agent that says who Sonda is, and nothing that says who uses it.
if hdr.Get("User-Agent") != "Sonda/1.2.3 ( "+providers.ProjectURL+" )" {
t.Errorf("User-Agent = %q", hdr.Get("User-Agent"))
}
for name := range hdr {
switch http.CanonicalHeaderKey(name) {
case "User-Agent", "Accept", "Accept-Encoding":
default:
t.Errorf("unexpected header %s: %v", name, hdr[name])
}
}
everything := seen[0] + " " + hdr.Get("User-Agent")
for _, secret := range []string{"alice", "Embarrassing", "Guilty", "Private Collection", "secret-music", "flac"} {
if strings.Contains(strings.ToLower(everything), strings.ToLower(secret)) {
t.Errorf("the request mentions %q: %s", secret, everything)
}
}
}

View file

@ -33,6 +33,8 @@ tags:
description: Browsing and searching what can be played.
- name: Playback
description: Streaming and listening history.
- name: Discovery
description: Music outside the library, from external providers an admin switched on.
- name: Providers
description: External services Sonda can ask about artists, set up by an admin.
@ -543,7 +545,54 @@ paths:
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: No such artist, or nothing of it can be played (`artist_not_found`).
description: |
No such artist (`artist_not_found`). An artist with nothing to play is not found either,
unless a provider named it as similar to another artist: those can be shown, with
`track_count` and `album_count` of 0, but they are never part of the lists.
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/artists/{id}/similar:
parameters:
- {$ref: '#/components/parameters/Id'}
get:
tags: [Discovery]
operationId: getSimilarArtists
summary: Artists similar to one
description: |
Artists an external provider (Last.fm) finds similar to this one, best first. Each says
whether it is in the library, and `id` works with `GET /artists/{id}`; one that is not in the
library has no files.
The provider is only asked when an admin switched it on and the stored answer is missing or
old (7 days; a "no such artist" answer is kept 1 day). What is sent is the artist's name or
MusicBrainz id and nothing about any user. Several requests for the same artist at once
make one call.
A provider that is off or failing is not an error: the answer is `200` and `provider_status`
says why. Then `items` holds whatever was stored before, if anything, and `stale` is true
when it is older than it should be.
`score` is the provider's own number between 0 and 1. It is relative to the list it came
in (the first is always 1), so it only compares artists inside one answer. `rank` is the
position. Show `attribution` where you show the data.
security: [{bearerAuth: []}]
parameters:
- name: limit
in: query
description: How many to return. 20 when absent or 0; values above 100 are lowered to 100.
schema: {type: integer, minimum: 0, default: 20}
responses:
'200':
description: The similar artists.
content:
application/json:
schema: {$ref: '#/components/schemas/SimilarArtists'}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: No such artist (`artist_not_found`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
@ -817,6 +866,40 @@ paths:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/providers/{name}/test:
parameters:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz]}
post:
tags: [Providers]
operationId: testProvider
summary: Check a provider's API key
description: |
Admin only. Asks the provider one cheap question with the stored key, to find out whether
it works. It works whether the provider is on or off, so you can check before switching it on.
It is the only request Sonda makes at an admin's command rather than for a listener, and it
sends nothing but the key. A provider that cannot be reached or refuses the key is not an
error here: the answer is `200` with `ok` false and the reason in `status`.
Only Last.fm can be tested so far; asking for MusicBrainz is a `400`.
security: [{bearerAuth: []}]
responses:
'200':
description: The outcome of the check.
content:
application/json:
schema: {$ref: '#/components/schemas/ProviderTest'}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'403': {$ref: '#/components/responses/AdminRequired'}
'404':
description: No such provider (`provider_not_found`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/listens:
get:
tags: [Playback]
@ -1269,6 +1352,50 @@ components:
maxLength: 200
description: A URL or e-mail address, without parentheses. Empty restores the default.
ProviderTest:
type: object
additionalProperties: false
required: [ok, status]
properties:
ok: {type: boolean}
status:
type: string
enum: [ok, no_api_key, key_refused, rate_limited, unreachable, error]
description: |
`no_api_key`: none is stored, so nobody was asked. `key_refused`: the provider does not accept
it. `rate_limited`: try again later. `unreachable`: the provider did not answer. `error`: anything else.
SimilarArtists:
type: object
additionalProperties: false
required: [provider, provider_status, stale, retrieved_at, attribution, items]
properties:
provider: {type: string, enum: [lastfm]}
provider_status:
type: string
enum: [ok, disabled, error]
description: |
`ok`: the data is as fresh as it needs to be. `disabled`: the provider is off or has no key,
so it was not asked. `error`: it was asked and failed.
stale: {type: boolean, description: "True when `items` are older than they should be because the provider could not be asked."}
retrieved_at:
type: string
format: date-time
nullable: true
description: When the provider was last read for this artist; null if never.
attribution: {type: string, description: Credit to show where the data is shown.}
items:
type: array
items:
type: object
additionalProperties: false
required: [artist, in_library, score, rank]
properties:
artist: {$ref: '#/components/schemas/ArtistRef'}
in_library: {type: boolean, description: Whether the artist has something playable in the library.}
score: {type: number, minimum: 0, maximum: 1, description: "The provider's own number, relative to this list."}
rank: {type: integer, minimum: 1, description: "Position in the provider's list, 1 first."}
ListenResult:
type: object
additionalProperties: false

View file

@ -0,0 +1,256 @@
// Package discovery finds music outside the library: artists similar to one the
// user already has.
//
// Data comes from an external provider (Last.fm today), only when the admin
// switched that provider on, and only when someone asks. What was read is kept
// in the database, so a second look costs nothing and nothing leaves the server
// again until the data is old. The raw numbers of the provider are stored as
// they are: turning them into a score is a later step.
package discovery
import (
"context"
"errors"
"fmt"
"strings"
"time"
"go.uber.org/zap"
"golang.org/x/sync/singleflight"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
)
// Provider is the provider similar artists come from.
const Provider = providers.LastFM
// Attribution is what has to be shown with the data: Last.fm asks for credit and
// allows non-commercial use only.
const Attribution = "Similar artists from Last.fm (https://www.last.fm), for non-commercial use."
// How long what was read is believed. Last.fm sends no cache headers, so these
// are our own choice: similarity moves slowly, and a "no such artist" answer is
// remembered shortly in case it was a typo that was since fixed.
const (
PositiveTTL = 7 * 24 * time.Hour
NegativeTTL = 24 * time.Hour
)
// FetchLimit is how many similar artists are read from the provider each time,
// whatever size of page was asked for, so that one answer serves every page.
const FetchLimit = 50
// Bounds of the number of items returned.
const (
DefaultLimit = 20
MaxLimit = 100
)
// fetchTimeout bounds a shared provider call, which does not depend on whoever
// started it still waiting.
const fetchTimeout = time.Minute
// Errors returned by the service.
var (
ErrNotFound = errors.New("artist not found")
ErrInvalidQuery = errors.New("invalid query")
)
// Provider states reported with an answer.
const (
StatusOK = "ok" // the data is as fresh as it needs to be
StatusDisabled = "disabled" // the provider is off or has no key, so it was not asked
StatusError = "error" // the provider was asked and failed
)
// Artist is who is being looked up.
type Artist struct {
ID int64
Name string
MBID string // empty when unknown
}
// ArtistRef is the short form of an artist.
type ArtistRef struct {
ID int64
Name string
}
// Item is one similar artist. Score is the provider's own number, relative to
// the list it came in, and Rank is the position in that list, starting at 1.
type Item struct {
Artist ArtistRef
InLibrary bool // has something playable in this library
Score float64
Rank int
}
// Snapshot is what is stored for one artist and one provider. Fetched is false
// when the provider was never asked; NotFound when it said it does not know the
// artist (or knew nothing similar).
type Snapshot struct {
Fetched bool
NotFound bool
RetrievedAt time.Time
Items []Item
}
// Result is the answer to Similar. RetrievedAt is zero when nothing is known.
// Stale means the items are older than they should be because the provider could
// not be asked.
type Result struct {
Provider string
ProviderStatus string
Stale bool
RetrievedAt time.Time
Items []Item
}
// Store keeps what the providers said.
type Store interface {
// Artist returns ErrNotFound for an id the catalog does not know.
Artist(ctx context.Context, id int64) (Artist, error)
// Similar reads the stored items (at most limit, best first) and when they were read.
Similar(ctx context.Context, id int64, provider string, limit int) (Snapshot, error)
// SaveSimilar replaces what was stored for the artist. The candidates become
// artists without files unless one is already known.
SaveSimilar(ctx context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error
}
// Source asks the provider.
type Source interface {
SimilarArtists(ctx context.Context, key string, a lastfm.Artist, limit int) ([]lastfm.Similar, error)
}
// Settings says whether a provider may be used right now.
type Settings interface {
Active(ctx context.Context, name string) (providers.Settings, bool, error)
}
// Service answers "who is similar to this artist".
type Service struct {
store Store
source Source
settings Settings
log *zap.Logger
now func() time.Time
flight singleflight.Group
}
// NewService builds a Service.
func NewService(store Store, source Source, settings Settings, log *zap.Logger) *Service {
if log == nil {
log = zap.NewNop()
}
return &Service{store: store, source: source, settings: settings, log: log, now: time.Now}
}
// Similar returns the artists similar to artistID, at most limit of them
// (DefaultLimit for 0). It asks the provider only when the stored data is
// missing or old and the provider is on. A provider that fails is not an error
// for the caller: the answer says so, and carries what was stored, if anything.
func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Result, error) {
switch {
case limit < 0:
return Result{}, fmt.Errorf("%w: limit cannot be negative", ErrInvalidQuery)
case limit == 0:
limit = DefaultLimit
case limit > MaxLimit:
limit = MaxLimit
}
if artistID <= 0 {
return Result{}, ErrNotFound
}
artist, err := s.store.Artist(ctx, artistID)
if err != nil {
return Result{}, err
}
snap, err := s.store.Similar(ctx, artistID, Provider, limit)
if err != nil {
return Result{}, err
}
if snap.Fetched && s.fresh(snap) {
return result(snap, StatusOK, false), nil
}
cfg, active, err := s.settings.Active(ctx, Provider)
if err != nil {
return Result{}, err
}
if !active {
return result(snap, StatusDisabled, snap.Fetched), nil
}
// One call per artist, however many ask at once. It runs on its own context,
// so the first caller leaving does not fail the others; each caller still
// waits only as long as its own context allows.
done := s.flight.DoChan(fmt.Sprint(artistID), func() (any, error) {
return nil, s.refresh(artist, cfg.APIKey)
})
select {
case <-ctx.Done():
return Result{}, ctx.Err()
case r := <-done:
if r.Err != nil {
s.log.Warn("similar artists could not be read", zap.String("provider", Provider), zap.Error(r.Err))
return result(snap, StatusError, snap.Fetched), nil
}
}
snap, err = s.store.Similar(ctx, artistID, Provider, limit)
if err != nil {
return Result{}, err
}
return result(snap, StatusOK, false), nil
}
func (s *Service) fresh(snap Snapshot) bool {
ttl := PositiveTTL
if snap.NotFound {
ttl = NegativeTTL
}
return s.now().Sub(snap.RetrievedAt) < ttl
}
// refresh reads the provider and stores what it said.
func (s *Service) refresh(a Artist, key string) error {
ctx, cancel := context.WithTimeout(context.Background(), fetchTimeout)
defer cancel()
got, err := s.source.SimilarArtists(ctx, key, lastfm.Artist{Name: a.Name, MBID: a.MBID}, FetchLimit)
switch {
case errors.Is(err, lastfm.ErrNotFound):
got = nil
case err != nil:
return err
}
cands := withoutArtist(got, a)
return s.store.SaveSimilar(ctx, a.ID, Provider, s.now(), len(cands) > 0, cands)
}
// withoutArtist drops the artist itself, which providers sometimes list under
// another spelling or the same id.
func withoutArtist(in []lastfm.Similar, a Artist) []lastfm.Similar {
name := strings.ToLower(strings.TrimSpace(a.Name))
out := make([]lastfm.Similar, 0, len(in))
for _, c := range in {
if strings.ToLower(strings.TrimSpace(c.Name)) == name ||
(a.MBID != "" && strings.EqualFold(c.MBID, a.MBID)) {
continue
}
out = append(out, c)
}
return out
}
func result(snap Snapshot, status string, stale bool) Result {
items := snap.Items
if items == nil {
items = []Item{}
}
r := Result{Provider: Provider, ProviderStatus: status, Stale: stale && snap.Fetched, Items: items}
if snap.Fetched {
r.RetrievedAt = snap.RetrievedAt
}
return r
}

View file

@ -0,0 +1,389 @@
package discovery
import (
"context"
"errors"
"sync"
"sync/atomic"
"testing"
"time"
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
)
const key = "0123456789abcdef0123456789abcdef"
var t0 = time.Unix(1_700_000_000, 0)
type saved struct {
artistID int64
provider string
at time.Time
found bool
cands []lastfm.Similar
}
// fakeStore keeps one snapshot per artist, replaced by every save.
type fakeStore struct {
mu sync.Mutex
artists map[int64]Artist
snaps map[int64]Snapshot
saves []saved
lastLim int
}
func newStore() *fakeStore {
return &fakeStore{
artists: map[int64]Artist{1: {ID: 1, Name: "Source", MBID: "aaaaaaaa-0000-0000-0000-000000000001"}},
snaps: map[int64]Snapshot{},
}
}
func (f *fakeStore) Artist(_ context.Context, id int64) (Artist, error) {
f.mu.Lock()
defer f.mu.Unlock()
a, ok := f.artists[id]
if !ok {
return Artist{}, ErrNotFound
}
return a, nil
}
func (f *fakeStore) Similar(_ context.Context, id int64, _ string, limit int) (Snapshot, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.lastLim = limit
s := f.snaps[id]
if len(s.Items) > limit {
s.Items = s.Items[:limit]
}
return s, nil
}
func (f *fakeStore) SaveSimilar(_ context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
f.mu.Lock()
defer f.mu.Unlock()
f.saves = append(f.saves, saved{id, provider, at, found, cands})
snap := Snapshot{Fetched: true, NotFound: !found, RetrievedAt: at}
for i, c := range cands {
snap.Items = append(snap.Items, Item{Artist: ArtistRef{ID: int64(100 + i), Name: c.Name}, Score: c.Match, Rank: i + 1})
}
f.snaps[id] = snap
return nil
}
func (f *fakeStore) put(id int64, s Snapshot) {
f.mu.Lock()
f.snaps[id] = s
f.mu.Unlock()
}
type fakeSource struct {
mu sync.Mutex
calls int
key string
artist lastfm.Artist
limit int
result []lastfm.Similar
err error
started chan struct{} // closed on the first call, when set
release chan struct{} // the call waits for it, when set
ctxErr error
}
func (s *fakeSource) SimilarArtists(ctx context.Context, k string, a lastfm.Artist, limit int) ([]lastfm.Similar, error) {
s.mu.Lock()
s.calls++
s.key, s.artist, s.limit = k, a, limit
first := s.calls == 1
started, release := s.started, s.release
s.mu.Unlock()
if started != nil && first {
close(started)
}
if release != nil {
<-release
}
s.mu.Lock()
s.ctxErr = ctx.Err()
s.mu.Unlock()
return s.result, s.err
}
func (s *fakeSource) count() int {
s.mu.Lock()
defer s.mu.Unlock()
return s.calls
}
type fakeSettings struct {
active bool
err error
}
func (f fakeSettings) Active(_ context.Context, name string) (providers.Settings, bool, error) {
return providers.Settings{Provider: name, Enabled: f.active, APIKey: key}, f.active, f.err
}
type rig struct {
svc *Service
store *fakeStore
src *fakeSource
now time.Time
}
func newRig(active bool) *rig {
r := &rig{store: newStore(), src: &fakeSource{}, now: t0}
r.src.result = []lastfm.Similar{{Name: "Alpha", Match: 1}, {Name: "Beta", MBID: "bbbbbbbb-0000-0000-0000-000000000002", Match: 0.5}}
r.svc = NewService(r.store, r.src, fakeSettings{active: active}, zap.NewNop())
r.svc.now = func() time.Time { return r.now }
return r
}
func (r *rig) similar(t *testing.T, limit int) Result {
t.Helper()
res, err := r.svc.Similar(context.Background(), 1, limit)
if err != nil {
t.Fatalf("Similar: %v", err)
}
return res
}
func TestAnUnknownArtistIsNotFoundAndNothingIsAsked(t *testing.T) {
r := newRig(true)
if _, err := r.svc.Similar(context.Background(), 999, 10); !errors.Is(err, ErrNotFound) {
t.Fatalf("err = %v, want ErrNotFound", err)
}
if _, err := r.svc.Similar(context.Background(), 0, 10); !errors.Is(err, ErrNotFound) {
t.Fatalf("id 0: err = %v, want ErrNotFound", err)
}
if r.src.count() != 0 {
t.Error("the provider was asked about an artist that does not exist")
}
}
func TestLimitIsCheckedAndDefaulted(t *testing.T) {
r := newRig(true)
if _, err := r.svc.Similar(context.Background(), 1, -1); !errors.Is(err, ErrInvalidQuery) {
t.Errorf("negative limit: err = %v", err)
}
for _, tt := range []struct{ in, want int }{{0, DefaultLimit}, {7, 7}, {MaxLimit, MaxLimit}, {MaxLimit + 50, MaxLimit}} {
r.similar(t, tt.in)
if r.store.lastLim != tt.want {
t.Errorf("limit %d reached the store as %d, want %d", tt.in, r.store.lastLim, tt.want)
}
}
}
func TestAProviderThatIsOffIsNeverAsked(t *testing.T) {
r := newRig(false)
res := r.similar(t, 10)
if res.ProviderStatus != StatusDisabled || len(res.Items) != 0 || !res.RetrievedAt.IsZero() || res.Stale {
t.Errorf("result = %+v", res)
}
if r.src.count() != 0 {
t.Error("a disabled provider was contacted")
}
}
func TestFirstLookAsksTheProviderAndStoresTheAnswer(t *testing.T) {
r := newRig(true)
res := r.similar(t, 10)
if r.src.count() != 1 {
t.Fatalf("calls = %d, want 1", r.src.count())
}
if r.src.key != key || r.src.artist.Name != "Source" || r.src.artist.MBID != "aaaaaaaa-0000-0000-0000-000000000001" {
t.Errorf("the provider was asked %q about %+v", r.src.key, r.src.artist)
}
// Always the same number, whatever the page asked for, so one answer serves any page size.
if r.src.limit != FetchLimit {
t.Errorf("fetch limit = %d, want %d", r.src.limit, FetchLimit)
}
if len(r.store.saves) != 1 || !r.store.saves[0].found || r.store.saves[0].provider != Provider || !r.store.saves[0].at.Equal(t0) {
t.Fatalf("saves = %+v", r.store.saves)
}
if res.ProviderStatus != StatusOK || res.Stale || !res.RetrievedAt.Equal(t0) || len(res.Items) != 2 || res.Provider != Provider {
t.Errorf("result = %+v", res)
}
if res.Items[0].Artist.Name != "Alpha" || res.Items[0].Rank != 1 || res.Items[1].Score != 0.5 {
t.Errorf("items = %+v", res.Items)
}
}
func TestAnArtistIsNeverSimilarToItself(t *testing.T) {
r := newRig(true)
r.src.result = []lastfm.Similar{
{Name: " SOURCE ", Match: 1},
{Name: "Other Spelling", MBID: "AAAAAAAA-0000-0000-0000-000000000001", Match: 0.9},
{Name: "Real One", Match: 0.8},
}
r.similar(t, 10)
got := r.store.saves[0].cands
if len(got) != 1 || got[0].Name != "Real One" {
t.Errorf("stored candidates = %+v", got)
}
}
func TestFreshDataIsNotAskedAgain(t *testing.T) {
r := newRig(true)
r.similar(t, 10)
r.now = t0.Add(PositiveTTL - time.Second)
res := r.similar(t, 10)
if r.src.count() != 1 {
t.Errorf("calls = %d, want 1: data younger than %v is fresh", r.src.count(), PositiveTTL)
}
if res.Stale || res.ProviderStatus != StatusOK || !res.RetrievedAt.Equal(t0) {
t.Errorf("result = %+v", res)
}
r.now = t0.Add(PositiveTTL)
r.similar(t, 10)
if r.src.count() != 2 {
t.Errorf("calls = %d, want 2: data of %v is old", r.src.count(), PositiveTTL)
}
}
func TestFreshDataIsServedEvenIfTheProviderIsSwitchedOff(t *testing.T) {
r := newRig(true)
r.similar(t, 10)
r.svc.settings = fakeSettings{active: false}
res := r.similar(t, 10)
if len(res.Items) != 2 || res.Stale {
t.Errorf("result = %+v", res)
}
r.now = t0.Add(PositiveTTL + time.Hour)
res = r.similar(t, 10)
if !res.Stale || res.ProviderStatus != StatusDisabled || len(res.Items) != 2 || r.src.count() != 1 {
t.Errorf("old data with the provider off: %+v calls=%d", res, r.src.count())
}
}
func TestAnUnknownArtistAtTheProviderIsRememberedForADay(t *testing.T) {
r := newRig(true)
r.src.err = lastfm.ErrNotFound
res := r.similar(t, 10)
if res.ProviderStatus != StatusOK || len(res.Items) != 0 || res.Stale {
t.Errorf("result = %+v", res)
}
if len(r.store.saves) != 1 || r.store.saves[0].found {
t.Fatalf("saves = %+v, want one 'not found'", r.store.saves)
}
r.now = t0.Add(NegativeTTL - time.Second)
r.similar(t, 10)
if r.src.count() != 1 {
t.Errorf("calls = %d: a negative answer is kept for %v", r.src.count(), NegativeTTL)
}
r.now = t0.Add(NegativeTTL)
r.similar(t, 10)
if r.src.count() != 2 {
t.Errorf("calls = %d: a negative answer expires after %v", r.src.count(), NegativeTTL)
}
}
func TestOldDataIsServedWhenTheProviderFails(t *testing.T) {
r := newRig(true)
r.similar(t, 10)
r.now = t0.Add(PositiveTTL + time.Hour)
r.src.err = lastfm.ErrUnavailable
res := r.similar(t, 10)
if !res.Stale || res.ProviderStatus != StatusError || len(res.Items) != 2 || !res.RetrievedAt.Equal(t0) {
t.Errorf("result = %+v", res)
}
if len(r.store.saves) != 1 {
t.Errorf("a failed call changed the stored data: %+v", r.store.saves)
}
}
func TestAFailureWithNothingStoredIsAnEmptyAnswerNotAnError(t *testing.T) {
for _, e := range []error{lastfm.ErrUnavailable, lastfm.ErrInvalidKey, lastfm.ErrRateLimited, lastfm.ErrBadResponse, errors.New("boom")} {
r := newRig(true)
r.src.err = e
res, err := r.svc.Similar(context.Background(), 1, 10)
if err != nil {
t.Fatalf("%v: err = %v", e, err)
}
if res.ProviderStatus != StatusError || len(res.Items) != 0 || res.Stale || !res.RetrievedAt.IsZero() {
t.Errorf("%v: result = %+v", e, res)
}
if len(r.store.saves) != 0 {
t.Errorf("%v: a failure was stored", e)
}
}
}
func TestSettingsErrorsAreReal(t *testing.T) {
r := newRig(true)
boom := errors.New("db down")
r.svc.settings = fakeSettings{err: boom}
if _, err := r.svc.Similar(context.Background(), 1, 10); !errors.Is(err, boom) {
t.Fatalf("err = %v, want the settings error", err)
}
}
// Ten people opening the same artist at once cost one call, and one of them
// walking away does not cancel it for the others.
func TestConcurrentLooksShareOneCall(t *testing.T) {
r := newRig(true)
r.src.started = make(chan struct{})
r.src.release = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
var failed atomic.Int32
results := make(chan Result, 10)
run := func(c context.Context) {
defer wg.Done()
res, err := r.svc.Similar(c, 1, 10)
if err != nil {
failed.Add(1)
return
}
results <- res
}
wg.Add(1)
go run(ctx)
<-r.src.started
for i := 0; i < 9; i++ {
wg.Add(1)
go run(context.Background())
}
time.Sleep(50 * time.Millisecond) // let the others join the flight
cancel()
time.Sleep(20 * time.Millisecond)
close(r.src.release)
wg.Wait()
close(results)
if r.src.count() != 1 {
t.Errorf("calls = %d, want 1", r.src.count())
}
if r.src.ctxErr != nil {
t.Errorf("the shared call was cancelled with its first caller: %v", r.src.ctxErr)
}
if failed.Load() != 1 {
t.Errorf("failed = %d, want only the caller that gave up", failed.Load())
}
n := 0
for res := range results {
n++
if len(res.Items) != 2 {
t.Errorf("result = %+v", res)
}
}
if n != 9 {
t.Errorf("%d callers got an answer, want 9", n)
}
}
// "No such artist" and "an empty list" both mean there is nothing to show, and
// both are remembered for a short time only.
func TestAnEmptyListIsRememberedLikeAnUnknownArtist(t *testing.T) {
r := newRig(true)
r.src.result = nil
res := r.similar(t, 10)
if len(res.Items) != 0 || len(r.store.saves) != 1 || r.store.saves[0].found {
t.Fatalf("result = %+v saves = %+v", res, r.store.saves)
}
}

View file

@ -0,0 +1,246 @@
// Package lastfm is a small read-only client for the Last.fm API.
//
// It only does what Sonda needs, with one decision made for it: nothing about
// the people using Sonda is ever sent, only the name or MusicBrainz id of the
// artist being looked up. Calls are unsigned, so the shared secret is never
// needed and never held.
//
// Last.fm reports most failures with HTTP 200 and an "error" field in the JSON,
// so the body is always read for it. Errors never repeat the provider's own
// message or the request URL: the API key travels in the query string and the
// message can echo it.
package lastfm
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"strconv"
"strings"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
// BaseURL is the Last.fm API endpoint.
const BaseURL = "https://ws.audioscrobbler.com/2.0/"
// Bounds of the number of similar artists asked for. Without a limit Last.fm
// answers with 100, and up to 250 when asked for more: always say how many.
const (
DefaultLimit = 50
MaxLimit = 100
)
// Errors a call can return, to be matched with errors.Is.
var (
ErrBadRequest = errors.New("lastfm: nothing to look up")
ErrNotFound = errors.New("lastfm: unknown artist")
ErrInvalidKey = errors.New("lastfm: the API key was refused")
ErrRateLimited = errors.New("lastfm: rate limit exceeded")
ErrUnavailable = errors.New("lastfm: service unavailable")
ErrBadResponse = errors.New("lastfm: unreadable answer")
ErrAPI = errors.New("lastfm: the service reported an error")
)
// Getter performs a GET. *outbound.Client is the one used in production.
type Getter interface {
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
}
// Client talks to Last.fm. It is safe for concurrent use. The API key is given
// to each call, because an admin can change it while Sonda runs.
type Client struct {
http Getter
base string
}
// New builds a Client that sends its requests through g to base (BaseURL in
// production).
func New(g Getter, base string) *Client { return &Client{http: g, base: base} }
// Artist names an artist to look up. The MusicBrainz id, when known, wins over
// the name: it cannot be corrected into someone else or confused with a namesake.
type Artist struct {
Name string
MBID string
}
// Similar is one similar artist. Match is what Last.fm calls "match", between
// 0 and 1. It is relative to the list it came in (the first is always 1), so it
// says "how high in this list", not "how alike in general".
type Similar struct {
Name string
MBID string // empty when Last.fm has none
Match float64
}
// SimilarArtists returns the artists Last.fm finds similar to a, best first, at
// most limit of them (DefaultLimit when limit <= 0, MaxLimit at most). The name
// is never autocorrected: a misspelled local name must not turn into another
// artist.
func (c *Client) SimilarArtists(ctx context.Context, key string, a Artist, limit int) ([]Similar, error) {
name, mbid := strings.TrimSpace(a.Name), strings.TrimSpace(a.MBID)
if name == "" && mbid == "" {
return nil, ErrBadRequest
}
switch {
case limit <= 0:
limit = DefaultLimit
case limit > MaxLimit:
limit = MaxLimit
}
q := url.Values{"method": {"artist.getsimilar"}, "autocorrect": {"0"}, "limit": {strconv.Itoa(limit)}}
if mbid != "" {
q.Set("mbid", mbid)
} else {
q.Set("artist", name)
}
body, err := c.call(ctx, key, q)
if err != nil {
return nil, err
}
var doc struct {
Similar *struct {
Artist json.RawMessage `json:"artist"`
} `json:"similarartists"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.Similar == nil {
return nil, ErrBadResponse
}
return decodeSimilar(doc.Similar.Artist)
}
// Check asks one cheap question to find out whether the key works.
func (c *Client) Check(ctx context.Context, key string) error {
body, err := c.call(ctx, key, url.Values{"method": {"chart.gettopartists"}, "limit": {"1"}})
if err != nil {
return err
}
var doc struct {
Artists *json.RawMessage `json:"artists"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.Artists == nil {
return ErrBadResponse
}
return nil
}
// Probe is Check as a providers.Checker: it answers with the errors the provider
// settings understand, so that a test does not depend on this client.
func (c *Client) Probe(ctx context.Context, key string) error {
err := c.Check(ctx, key)
switch {
case err == nil:
return nil
case errors.Is(err, ErrInvalidKey):
return providers.ErrKeyRefused
case errors.Is(err, ErrRateLimited):
return providers.ErrRateLimited
case errors.Is(err, ErrUnavailable):
return providers.ErrUnreachable
}
return err
}
// call sends one request and returns the body of a successful answer.
func (c *Client) call(ctx context.Context, key string, q url.Values) ([]byte, error) {
q.Set("api_key", key)
q.Set("format", "json")
resp, err := c.http.Get(ctx, c.base+"?"+q.Encode())
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
var limited *outbound.RateLimitedError
if errors.As(err, &limited) {
return nil, ErrRateLimited
}
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err) // the outbound client already strips the query
}
// An "error" field means failure whatever the status was.
var probe struct {
Error int `json:"error"`
}
if json.Unmarshal(resp.Body, &probe) == nil && probe.Error != 0 {
return nil, apiError(probe.Error)
}
switch {
case resp.Status == 429:
return nil, ErrRateLimited
case resp.Status >= 500:
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
case resp.Status != 200:
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
}
return resp.Body, nil
}
// apiError maps a Last.fm error code. Only the code is kept, never the message.
func apiError(code int) error {
switch code {
case 6:
return ErrNotFound
case 10, 26:
return ErrInvalidKey
case 29:
return ErrRateLimited
case 8, 11, 16:
return ErrUnavailable
}
return fmt.Errorf("%w (code %d)", ErrAPI, code)
}
// decodeSimilar reads the list of artists. Last.fm sends an array, and has been
// seen to send a single object when there is one result. An artist without a
// name or with a match outside 0..1 is dropped, and each artist is kept once.
func decodeSimilar(raw json.RawMessage) ([]Similar, error) {
type item struct {
Name string `json:"name"`
MBID string `json:"mbid"`
Match json.RawMessage `json:"match"`
}
var items []item
switch {
case len(raw) == 0 || string(raw) == "null":
case raw[0] == '[':
if err := json.Unmarshal(raw, &items); err != nil {
return nil, ErrBadResponse
}
default:
var one item
if err := json.Unmarshal(raw, &one); err != nil {
return nil, ErrBadResponse
}
items = []item{one}
}
out := make([]Similar, 0, len(items))
seen := map[string]bool{}
for _, it := range items {
name := strings.TrimSpace(it.Name)
match, ok := parseMatch(it.Match)
key := strings.ToLower(name)
if name == "" || !ok || seen[key] {
continue
}
seen[key] = true
out = append(out, Similar{Name: name, MBID: strings.TrimSpace(it.MBID), Match: match})
}
return out, nil
}
// parseMatch reads a match sent as a string ("0.75", what Last.fm does) or as a
// number, and accepts only values between 0 and 1.
func parseMatch(raw json.RawMessage) (float64, bool) {
s := strings.Trim(string(raw), `"`)
m, err := strconv.ParseFloat(s, 64)
if err != nil || !(m >= 0 && m <= 1) {
return 0, false
}
return m, true
}

View file

@ -0,0 +1,284 @@
package lastfm
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
const testKey = "0123456789abcdef0123456789abcdef"
// server answers every request with the given status and body and remembers
// the queries it saw.
type server struct {
*httptest.Server
mu sync.Mutex
queries []url.Values
}
func newServer(t *testing.T, status int, body string) *server {
t.Helper()
s := &server{}
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
s.queries = append(s.queries, r.URL.Query())
s.mu.Unlock()
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
}))
t.Cleanup(s.Close)
return s
}
func (s *server) client() *Client {
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( test )", Retries: outbound.NoRetries}), s.URL)
}
func (s *server) last(t *testing.T) url.Values {
t.Helper()
s.mu.Lock()
defer s.mu.Unlock()
if len(s.queries) == 0 {
t.Fatal("the server saw no request")
}
return s.queries[len(s.queries)-1]
}
const similarBody = `{"similarartists":{"artist":[
{"name":"Alpha Band","mbid":"11111111-1111-1111-1111-111111111111","match":"1","url":"https://example.invalid/a","image":[{"#text":"https://example.invalid/i.png","size":"small"}],"streamable":"0"},
{"name":"Beta Band","mbid":"","match":"0.755421","url":"https://example.invalid/b","streamable":"0"}
],"@attr":{"artist":"Source"}}}`
func TestSimilarArtistsDecodesTheAnswer(t *testing.T) {
s := newServer(t, 200, similarBody)
got, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Source"}, 30)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("got %d artists, want 2: %+v", len(got), got)
}
if got[0].Name != "Alpha Band" || got[0].MBID != "11111111-1111-1111-1111-111111111111" || got[0].Match != 1 {
t.Errorf("first = %+v", got[0])
}
if got[1].Name != "Beta Band" || got[1].MBID != "" || got[1].Match != 0.755421 {
t.Errorf("second = %+v", got[1])
}
}
func TestSimilarArtistsAsksOnlyWhatItNeeds(t *testing.T) {
s := newServer(t, 200, similarBody)
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Sigur Rós"}, 30); err != nil {
t.Fatal(err)
}
q := s.last(t)
want := map[string]string{
"method": "artist.getsimilar", "artist": "Sigur Rós", "autocorrect": "0",
"limit": "30", "api_key": testKey, "format": "json",
}
for k, v := range want {
if q.Get(k) != v {
t.Errorf("%s = %q, want %q", k, q.Get(k), v)
}
}
if len(q) != len(want) {
t.Errorf("query has other parameters than %v: %v", want, q)
}
}
// The id names the artist without doubt, so it wins over the name; the name
// would be corrected or confused, the id cannot.
func TestSimilarArtistsPrefersTheMBID(t *testing.T) {
s := newServer(t, 200, similarBody)
mbid := "22222222-2222-2222-2222-222222222222"
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Source", MBID: mbid}, 10); err != nil {
t.Fatal(err)
}
q := s.last(t)
if q.Get("mbid") != mbid || q.Has("artist") || q.Get("autocorrect") != "0" {
t.Errorf("query = %v", q)
}
}
func TestSimilarArtistsBoundsTheLimit(t *testing.T) {
for _, tt := range []struct {
in int
want string
}{{0, "50"}, {-3, "50"}, {7, "7"}, {100, "100"}, {5000, "100"}} {
s := newServer(t, 200, similarBody)
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, tt.in); err != nil {
t.Fatal(err)
}
if got := s.last(t).Get("limit"); got != tt.want {
t.Errorf("limit %d sent as %q, want %q", tt.in, got, tt.want)
}
}
}
func TestSimilarArtistsRefusesAnEmptyArtistWithoutAsking(t *testing.T) {
s := newServer(t, 200, similarBody)
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: " "}, 10)
if !errors.Is(err, ErrBadRequest) {
t.Fatalf("err = %v, want ErrBadRequest", err)
}
if len(s.queries) != 0 {
t.Error("a request left for an empty artist")
}
}
func TestSimilarArtistsToleratesTheShapesLastFMUses(t *testing.T) {
tests := []struct {
name, body string
want int
}{
{"one artist as an object", `{"similarartists":{"artist":{"name":"Solo","mbid":"","match":"1"}}}`, 1},
{"no artists", `{"similarartists":{"artist":[],"@attr":{"artist":"x"}}}`, 0},
{"match as a number", `{"similarartists":{"artist":[{"name":"N","match":0.5}]}}`, 1},
{"an artist without a name is dropped", `{"similarartists":{"artist":[{"name":"","match":"1"},{"name":"Ok","match":"0.5"}]}}`, 1},
{"an unreadable match is dropped", `{"similarartists":{"artist":[{"name":"A","match":"high"},{"name":"B","match":"0.5"}]}}`, 1},
{"a match outside 0..1 is dropped", `{"similarartists":{"artist":[{"name":"A","match":"7"},{"name":"B","match":"-1"},{"name":"C","match":"0"}]}}`, 1},
{"the same artist twice is kept once", `{"similarartists":{"artist":[{"name":"Dup","match":"1"},{"name":" dup ","match":"0.5"}]}}`, 1},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := newServer(t, 200, tt.body)
got, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
if err != nil {
t.Fatal(err)
}
if len(got) != tt.want {
t.Errorf("got %d artists %+v, want %d", len(got), got, tt.want)
}
})
}
}
// Last.fm reports most failures with status 200 and an "error" field.
func TestErrorsAreReadEvenWithStatus200(t *testing.T) {
tests := []struct {
name string
status int
body string
want error
}{
{"unknown artist", 200, `{"error":6,"message":"The artist you supplied could not be found","links":[]}`, ErrNotFound},
{"invalid key", 403, `{"message":"Invalid API key - You must be granted a valid key by last.fm","error":10}`, ErrInvalidKey},
{"invalid key with 200", 200, `{"error":10,"message":"Invalid API key"}`, ErrInvalidKey},
{"suspended key", 403, `{"error":26,"message":"Suspended API key"}`, ErrInvalidKey},
{"rate limit", 200, `{"error":29,"message":"Rate limit exceeded"}`, ErrRateLimited},
{"rate limit status", 429, `{"error":29,"message":"Rate limit exceeded"}`, ErrRateLimited},
{"temporary error", 200, `{"error":11,"message":"Service Offline"}`, ErrUnavailable},
{"temporarily unavailable", 200, `{"error":16,"message":"temporarily unavailable"}`, ErrUnavailable},
{"operation failed", 200, `{"error":8,"message":"Operation failed"}`, ErrUnavailable},
{"server error", 500, `oops`, ErrUnavailable},
{"bad gateway", 502, `<html>`, ErrUnavailable},
{"not json", 200, `<html>nope</html>`, ErrBadResponse},
{"empty answer", 200, ``, ErrBadResponse},
{"json without the expected part", 200, `{"hello":"world"}`, ErrBadResponse},
{"another error code", 200, `{"error":99,"message":"Something new"}`, ErrAPI},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := newServer(t, tt.status, tt.body)
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
if !errors.Is(err, tt.want) {
t.Fatalf("err = %v, want %v", err, tt.want)
}
})
}
}
// The key travels in the query string: no error may repeat it, and neither the
// URL nor the provider's own words (which can echo it) are copied into one.
func TestErrorsNeverCarryTheKeyOrTheProvidersText(t *testing.T) {
bodies := []struct {
status int
body string
}{
{403, `{"error":10,"message":"Invalid API key ` + testKey + `"}`},
{200, `{"error":99,"message":"weird ` + testKey + `"}`},
{500, "boom " + testKey},
{200, "not json " + testKey},
}
for _, b := range bodies {
s := newServer(t, b.status, b.body)
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
if err == nil {
t.Fatal("no error")
}
if strings.Contains(err.Error(), testKey) {
t.Errorf("the key is in the error: %v", err)
}
}
// A refused connection puts the URL in the error text: the outbound client
// strips the query, and this must stay so.
dead := httptest.NewServer(http.NotFoundHandler())
base := dead.URL
dead.Close()
c := New(outbound.New(outbound.Options{Retries: outbound.NoRetries}), base)
_, err := c.SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
if err == nil || strings.Contains(err.Error(), testKey) {
t.Errorf("err = %v", err)
}
if !errors.Is(err, ErrUnavailable) {
t.Errorf("a refused connection should read as unavailable: %v", err)
}
}
func TestCheckAsksForOneCheapThing(t *testing.T) {
s := newServer(t, 200, `{"artists":{"artist":[{"name":"Someone"}]}}`)
if err := s.client().Check(context.Background(), testKey); err != nil {
t.Fatal(err)
}
q := s.last(t)
if q.Get("method") != "chart.gettopartists" || q.Get("limit") != "1" || q.Get("api_key") != testKey {
t.Errorf("query = %v", q)
}
bad := newServer(t, 403, `{"error":10,"message":"Invalid API key"}`)
if err := bad.client().Check(context.Background(), testKey); !errors.Is(err, ErrInvalidKey) {
t.Errorf("err = %v, want ErrInvalidKey", err)
}
}
// Probe is Check in the words the provider settings use, so that the answer of a
// test does not depend on this client.
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
for _, tt := range []struct {
name string
status int
body string
want error
}{
{"works", 200, `{"artists":{"artist":[]}}`, nil},
{"key refused", 403, `{"error":10,"message":"Invalid API key"}`, providers.ErrKeyRefused},
{"suspended key", 200, `{"error":26,"message":"Suspended"}`, providers.ErrKeyRefused},
{"rate limited", 200, `{"error":29,"message":"Rate limit exceeded"}`, providers.ErrRateLimited},
{"down", 503, `nope`, providers.ErrUnreachable},
{"temporary", 200, `{"error":11,"message":"offline"}`, providers.ErrUnreachable},
} {
t.Run(tt.name, func(t *testing.T) {
s := newServer(t, tt.status, tt.body)
err := s.client().Probe(context.Background(), testKey)
if tt.want == nil && err != nil || tt.want != nil && !errors.Is(err, tt.want) {
t.Fatalf("err = %v, want %v", err, tt.want)
}
})
}
// Anything else is not turned into one of them.
s := newServer(t, 200, `<html>`)
err := s.client().Probe(context.Background(), testKey)
if err == nil || errors.Is(err, providers.ErrKeyRefused) || errors.Is(err, providers.ErrUnreachable) || errors.Is(err, providers.ErrRateLimited) {
t.Errorf("an unreadable answer: err = %v", err)
}
}

View file

@ -27,6 +27,15 @@ const (
var (
ErrUnknownProvider = errors.New("unknown provider")
ErrInvalidSettings = errors.New("invalid provider settings")
ErrNoTest = errors.New("this provider has no test")
)
// What a checker says went wrong, so that the result of a test does not depend
// on the client of any one provider.
var (
ErrKeyRefused = errors.New("the provider refused the API key")
ErrRateLimited = errors.New("the provider's rate limit was reached")
ErrUnreachable = errors.New("the provider could not be reached")
)
const maxContactLen = 200
@ -108,9 +117,10 @@ func lookup(name string) (meta, bool) {
// Service reads and changes provider settings.
type Service struct {
store Store
now func() time.Time
mu sync.Mutex // an update reads then writes: keep two of them apart
store Store
now func() time.Time
mu sync.Mutex // an update reads then writes: keep two of them apart
checkers map[string]Checker
}
// NewService builds a Service.
@ -264,3 +274,66 @@ func UserAgent(version, contact string) string {
}
return fmt.Sprintf("Sonda/%s ( %s )", v, c)
}
// Checker asks a provider one cheap question with a key, to find out whether the
// key works. It returns nil, or one of ErrKeyRefused, ErrRateLimited and
// ErrUnreachable (any other error counts as an unknown failure).
type Checker func(ctx context.Context, key string) error
// SetChecker registers the test of a provider. Call it before the service is used.
func (s *Service) SetChecker(name string, c Checker) {
if s.checkers == nil {
s.checkers = map[string]Checker{}
}
s.checkers[name] = c
}
// Results of a test.
const (
TestOK = "ok"
TestNoKey = "no_api_key"
TestKeyRefused = "key_refused"
TestRateLimited = "rate_limited"
TestUnreachable = "unreachable"
TestFailed = "error"
)
// TestResult is the outcome of Test. Status is one of the Test constants.
type TestResult struct {
OK bool
Status string
}
// Test checks a provider with the stored key. It works whether the provider is
// on or off, because an admin wants to know before switching it on; it is the one
// request Sonda makes on demand of an admin and not of a listener, and it sends
// nothing but the key. The key is never part of the result.
func (s *Service) Test(ctx context.Context, name string) (TestResult, error) {
m, ok := lookup(name)
if !ok {
return TestResult{}, ErrUnknownProvider
}
check := s.checkers[name]
if check == nil {
return TestResult{}, ErrNoTest
}
cur, _, err := s.store.Get(ctx, name)
if err != nil {
return TestResult{}, fmt.Errorf("read provider settings: %w", err)
}
if m.needsKey && cur.APIKey == "" {
return TestResult{Status: TestNoKey}, nil
}
switch err := check(ctx, cur.APIKey); {
case err == nil:
return TestResult{OK: true, Status: TestOK}, nil
case errors.Is(err, ErrKeyRefused):
return TestResult{Status: TestKeyRefused}, nil
case errors.Is(err, ErrRateLimited):
return TestResult{Status: TestRateLimited}, nil
case errors.Is(err, ErrUnreachable):
return TestResult{Status: TestUnreachable}, nil
default:
return TestResult{Status: TestFailed}, nil
}
}

View file

@ -275,3 +275,74 @@ func TestAnEmptyKeyDoesNotTurnOffAProviderThatNeedsNone(t *testing.T) {
t.Error("musicbrainz stopped being active")
}
}
// ---- testing a provider ----
func TestAProviderCanBeTestedBeforeItIsSwitchedOn(t *testing.T) {
s, _ := newSvc()
ctx := context.Background()
var gotKey string
s.SetChecker(LastFM, func(_ context.Context, key string) error { gotKey = key; return nil })
// No key yet: say so, and ask nobody.
res, err := s.Test(ctx, LastFM)
if err != nil || res.OK || res.Status != TestNoKey || gotKey != "" {
t.Fatalf("without a key: %+v err=%v key=%q", res, err, gotKey)
}
if _, err := s.Update(ctx, LastFM, Patch{APIKey: strp(goodKey)}); err != nil {
t.Fatal(err)
}
res, err = s.Test(ctx, LastFM) // still off: an explicit test is allowed
if err != nil || !res.OK || res.Status != TestOK || gotKey != goodKey {
t.Fatalf("with a key: %+v err=%v key=%q", res, err, gotKey)
}
}
func TestTestMapsWhatWentWrong(t *testing.T) {
for _, tt := range []struct {
err error
want string
}{
{ErrKeyRefused, TestKeyRefused},
{ErrRateLimited, TestRateLimited},
{ErrUnreachable, TestUnreachable},
{errors.New("something else"), TestFailed},
} {
s, _ := newSvc()
_, _ = s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)})
s.SetChecker(LastFM, func(context.Context, string) error { return tt.err })
res, err := s.Test(context.Background(), LastFM)
if err != nil || res.OK || res.Status != tt.want {
t.Errorf("%v: %+v err=%v, want status %s", tt.err, res, err, tt.want)
}
}
}
func TestTestErrors(t *testing.T) {
s, st := newSvc()
ctx := context.Background()
if _, err := s.Test(ctx, "spotify"); !errors.Is(err, ErrUnknownProvider) {
t.Errorf("unknown provider: %v", err)
}
if _, err := s.Test(ctx, MusicBrainz); !errors.Is(err, ErrNoTest) {
t.Errorf("a provider without a test: %v", err)
}
s.SetChecker(LastFM, func(context.Context, string) error { return nil })
boom := errors.New("db down")
st.err = boom
if _, err := s.Test(ctx, LastFM); !errors.Is(err, boom) {
t.Errorf("a storage error: %v", err)
}
}
// A checker must be handed the stored key and nothing in the answer may carry it.
func TestTestResultNeverCarriesTheKey(t *testing.T) {
s, _ := newSvc()
_, _ = s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)})
s.SetChecker(LastFM, func(context.Context, string) error { return errors.New("failed with " + goodKey) })
res, _ := s.Test(context.Background(), LastFM)
if strings.Contains(res.Status, goodKey) {
t.Errorf("the result repeats the key: %+v", res)
}
}

View file

@ -3,6 +3,7 @@ package storage
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
@ -111,10 +112,21 @@ func (r *CatalogRepo) GetArtist(ctx context.Context, id int64) (catalog.Artist,
if err != nil {
return catalog.Artist{}, err
}
if len(p.Items) == 0 {
if len(p.Items) > 0 {
return p.Items[0], nil
}
// Not in the library, but maybe named by a discovery provider: such an artist
// can be shown, with nothing to play. The lists never include it.
a := catalog.Artist{}
err = r.db.QueryRowContext(ctx,
"SELECT a.id, a.name FROM artists a WHERE a.id = ? AND "+discoveredArtist("a.id"), id).Scan(&a.ID, &a.Name)
if errors.Is(err, sql.ErrNoRows) {
return catalog.Artist{}, catalog.ErrNotFound
}
return p.Items[0], nil
if err != nil {
return catalog.Artist{}, fmt.Errorf("find discovered artist: %w", err)
}
return a, nil
}
func (r *CatalogRepo) listArtists(ctx context.Context, f catalog.Filter, id int64) (catalog.Page[catalog.Artist], error) {

View file

@ -0,0 +1,249 @@
package storage
import (
"context"
"database/sql"
"errors"
"fmt"
"strings"
"time"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
)
// kindSimilar is what provider_fetches records for a "similar artists" lookup.
const kindSimilar = "similar"
// Answers recorded in provider_fetches.
const (
fetchOK = "ok"
fetchNotFound = "not_found"
)
// DiscoveryRepo keeps what the discovery providers said. It implements
// discovery.Store.
type DiscoveryRepo struct{ db *bun.DB }
// NewDiscoveryRepo builds a DiscoveryRepo.
func NewDiscoveryRepo(db *bun.DB) *DiscoveryRepo { return &DiscoveryRepo{db: db} }
// playableArtist is the condition "this artist can be played": credited on a
// track with a playable file, or the album artist of an album with one
// (compilations). expr is the artist id in the surrounding query.
func playableArtist(expr string) string {
vis, _ := visible("t", 0)
return `(EXISTS (SELECT 1 FROM track_artists ta JOIN tracks t ON t.id = ta.track_id WHERE ta.artist_id = ` + expr + ` AND ` + vis + `)
OR EXISTS (SELECT 1 FROM albums al JOIN tracks t ON t.album_id = al.id WHERE al.artist_id = ` + expr + ` AND ` + vis + `))`
}
// discoveredArtist is the condition "a provider named this artist as similar to
// another". Such an artist can have no files at all.
func discoveredArtist(expr string) string {
return "EXISTS (SELECT 1 FROM artist_similar s WHERE s.similar_id = " + expr + ")"
}
// Artist returns an artist that can be looked up: one with something playable,
// or one a provider already named. Anything else is discovery.ErrNotFound.
func (r *DiscoveryRepo) Artist(ctx context.Context, id int64) (discovery.Artist, error) {
var (
a discovery.Artist
mbid sql.NullString
)
err := r.db.QueryRowContext(ctx,
"SELECT a.id, a.name, a.mbid FROM artists a WHERE a.id = ? AND ("+playableArtist("a.id")+" OR "+discoveredArtist("a.id")+")", id).
Scan(&a.ID, &a.Name, &mbid)
if errors.Is(err, sql.ErrNoRows) {
return discovery.Artist{}, discovery.ErrNotFound
}
if err != nil {
return discovery.Artist{}, fmt.Errorf("find artist: %w", err)
}
a.MBID = mbid.String
return a, nil
}
// Similar reads what was stored about an artist, best first, at most limit items.
func (r *DiscoveryRepo) Similar(ctx context.Context, id int64, provider string, limit int) (discovery.Snapshot, error) {
var (
snap discovery.Snapshot
status string
when int64
)
err := r.db.QueryRowContext(ctx,
"SELECT status, retrieved_at FROM provider_fetches WHERE provider = ? AND kind = ? AND artist_id = ?",
provider, kindSimilar, id).Scan(&status, &when)
switch {
case errors.Is(err, sql.ErrNoRows):
return snap, nil
case err != nil:
return snap, fmt.Errorf("read fetch ledger: %w", err)
}
snap.Fetched, snap.NotFound, snap.RetrievedAt = true, status == fetchNotFound, time.Unix(when, 0)
rows, err := r.db.QueryContext(ctx,
`SELECT a.id, a.name, s.rank, s.score, `+playableArtist("a.id")+`
FROM artist_similar s JOIN artists a ON a.id = s.similar_id
WHERE s.artist_id = ? AND s.provider = ? ORDER BY s.rank, a.id LIMIT ?`,
id, provider, limit)
if err != nil {
return snap, fmt.Errorf("read similar artists: %w", err)
}
defer rows.Close()
snap.Items = []discovery.Item{}
for rows.Next() {
var it discovery.Item
if err := rows.Scan(&it.Artist.ID, &it.Artist.Name, &it.Rank, &it.Score, &it.InLibrary); err != nil {
return snap, err
}
snap.Items = append(snap.Items, it)
}
return snap, rows.Err()
}
// SaveSimilar replaces what is stored about an artist and records the answer.
//
// Each candidate is matched to an artist that is already known, or becomes a new
// artist without files. Two artists are the same when their MusicBrainz ids are
// equal, or when their names match and at least one side has no id: an id that
// is missing never says "different", and one that differs always does. An id
// from a provider is never written onto an existing artist, because nobody
// checked it. Artists created here are kept when the list is replaced later.
func (r *DiscoveryRepo) SaveSimilar(ctx context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
return r.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
ids, err := resolveArtists(ctx, tx, id, cands)
if err != nil {
return err
}
if _, err := tx.ExecContext(ctx, "DELETE FROM artist_similar WHERE artist_id = ? AND provider = ?", id, provider); err != nil {
return fmt.Errorf("clear similar artists: %w", err)
}
rank := 0
for i, c := range cands {
if ids[i] == 0 {
continue // the artist itself, or already listed above
}
rank++
if _, err := tx.ExecContext(ctx,
"INSERT INTO artist_similar (artist_id, provider, similar_id, rank, score) VALUES (?, ?, ?, ?, ?)",
id, provider, ids[i], rank, c.Match); err != nil {
return fmt.Errorf("insert similar artist: %w", err)
}
}
status := fetchOK
if !found {
status = fetchNotFound
}
if _, err := tx.ExecContext(ctx,
`INSERT INTO provider_fetches (provider, kind, artist_id, status, retrieved_at) VALUES (?, ?, ?, ?, ?)
ON CONFLICT (provider, kind, artist_id) DO UPDATE SET status = excluded.status, retrieved_at = excluded.retrieved_at`,
provider, kindSimilar, id, status, at.Unix()); err != nil {
return fmt.Errorf("record fetch: %w", err)
}
return nil
})
}
// resolveArtists returns the artist id of each candidate, creating the ones that
// are not known. A candidate that is the artist itself, or an artist already
// seen earlier in the list, gets 0.
func resolveArtists(ctx context.Context, tx bun.Tx, self int64, cands []lastfm.Similar) ([]int64, error) {
type known struct {
id int64
mbid string
}
var mbids, keys []string
for _, c := range cands {
if m := cleanMBID(c.MBID); m != "" {
mbids = append(mbids, m)
}
keys = append(keys, matchKey(c.Name))
}
byMBID := map[string]int64{}
byKey := map[string][]known{}
if len(cands) > 0 {
// Two conditions with an empty list would match everything or nothing,
// depending on the database: always send at least one value.
if len(mbids) == 0 {
mbids = []string{""}
}
rows, err := tx.QueryContext(ctx,
"SELECT id, mbid, name_key FROM artists WHERE mbid IN (?) OR name_key IN (?) ORDER BY id",
bun.In(mbids), bun.In(keys))
if err != nil {
return nil, fmt.Errorf("find artists: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
id int64
mbid sql.NullString
nameKey string
)
if err := rows.Scan(&id, &mbid, &nameKey); err != nil {
return nil, err
}
if mbid.String != "" {
byMBID[strings.ToLower(mbid.String)] = id
}
byKey[nameKey] = append(byKey[nameKey], known{id, strings.ToLower(mbid.String)})
}
if err := rows.Err(); err != nil {
return nil, err
}
}
out := make([]int64, len(cands))
used := map[int64]bool{self: true}
for i, c := range cands {
mbid, key := cleanMBID(c.MBID), matchKey(c.Name)
var id int64
if mbid != "" {
id = byMBID[mbid]
}
if id == 0 {
for _, k := range byKey[key] {
if mbid == "" || k.mbid == "" {
id = k.id
break
}
}
}
if id == 0 {
var err error
if id, err = insertDiscovered(ctx, tx, c.Name, mbid); err != nil {
return nil, err
}
if mbid != "" {
byMBID[mbid] = id
}
byKey[key] = append(byKey[key], known{id, mbid})
}
if used[id] {
continue
}
used[id] = true
out[i] = id
}
return out, nil
}
// insertDiscovered creates an artist that exists only because a provider named it.
func insertDiscovered(ctx context.Context, tx bun.Tx, name, mbid string) (int64, error) {
var m sql.NullString
if mbid != "" {
m = sql.NullString{String: mbid, Valid: true}
}
var id int64
err := tx.QueryRowContext(ctx,
"INSERT INTO artists (mbid, name, name_key) VALUES (?, ?, ?) RETURNING id", m, name, matchKey(name)).Scan(&id)
if err != nil {
return 0, fmt.Errorf("insert artist: %w", err)
}
return id, nil
}
// cleanMBID lower-cases an id the way it is stored.
func cleanMBID(s string) string { return strings.ToLower(strings.TrimSpace(s)) }

Some files were not shown because too many files have changed in this diff Show more