feat(discovery): similar artists from Last.fm
- feat: GET /api/v1/artists/{id}/similar lists the artists Last.fm finds similar
to one in the library, best first, each with the provider's own score and
rank and a flag saying whether it is in the library. It answers 200 when the
provider is off or failing, with provider_status, stale and retrieved_at
- feat: a thin Last.fm client on top of the outbound client. It always sends
autocorrect=0 so a misspelled local name cannot turn into another artist,
prefers the MusicBrainz id when known, always sets a limit, and reads the
error field even when the status is 200. Errors never carry the URL or the
provider's own message, because the key travels in the query string
- feat: results are stored in artist_similar with a freshness ledger in
provider_fetches: 7 days for an answer, 1 day for "unknown artist". Old data
is served when the provider fails, and concurrent requests for the same
artist share one call that does not depend on the first caller staying
- feat: artists named by a provider but without files become artists without
tracks. GET /artists/{id} can show them (track_count 0), the lists of the
library never include them, and a later scan adopts them by name. A provider
id is never written onto an existing artist; names match only when at least
one side has no id, and different ids always mean different artists
- feat: POST /providers/{name}/test (admin) checks a stored key with one cheap
request, whether the provider is on or off, and reports ok, no_api_key,
key_refused, rate_limited, unreachable or error
- test: the whole request is checked from the receiving end, with a user, a
listen, a title and a library path in the database: only the artist and the
key leave, with a User-Agent that names the project
- docs: README and OpenAPI updated; the contract test calls the new operations
This commit is contained in:
parent
0097b707f2
commit
aafb2dd906
24 changed files with 2651 additions and 19 deletions
20
README.md
20
README.md
|
|
@ -14,11 +14,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Catalog.** Browse and search artists, albums and tracks, with paging, by library, artist or album. Only tracks you can actually play are listed.
|
||||
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
|
||||
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere. Today Sonda makes no outbound network requests at all.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
|
||||
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to one of yours, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
|
||||
|
||||
## What is not there yet
|
||||
|
||||
Similar-artist discovery, recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Last.fm and MusicBrainz can already be set up (see [External providers](#external-providers)), but nothing asks them anything yet.
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. MusicBrainz can be set up (see [External providers](#external-providers)) but nothing asks it anything yet.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -175,7 +176,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
|
|||
|
||||
## External providers
|
||||
|
||||
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today a provider can be set up but nothing uses it yet.
|
||||
Sonda can ask Last.fm and MusicBrainz about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today only Last.fm is used, to find similar artists.
|
||||
|
||||
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
|
||||
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
|
||||
|
|
@ -186,11 +187,16 @@ Set them up with `PUT /api/v1/providers/{name}` (admin), for example:
|
|||
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"api_key":"<your 32 character key>"}'
|
||||
curl -s -X PUT $BASE/providers/lastfm -H "$AUTH" -d '{"enabled":true}'
|
||||
curl -s $BASE/providers -H "$AUTH" | jq
|
||||
curl -s -X POST $BASE/providers/lastfm/test -H "$AUTH" | jq # does the key work?
|
||||
```
|
||||
|
||||
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
|
||||
|
||||
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down.
|
||||
**What leaves your server, when a provider is on:** the artist name (or MusicBrainz id) being looked up, your API key for Last.fm, your IP address and the `User-Agent`. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
|
||||
|
||||
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to one of yours and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the first is always 1), so compare artists only inside one answer.
|
||||
|
||||
The name of the artist is sent exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
|
||||
|
||||
## The API at a glance
|
||||
|
||||
|
|
@ -205,13 +211,14 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
|
|||
| anyone | `POST /auth/login`, `/auth/refresh`, `/auth/logout` | Open, renew and end a session. |
|
||||
| signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). |
|
||||
| signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. |
|
||||
| signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 100). |
|
||||
| signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. |
|
||||
| signed in | `POST /listens` | Record a `play`, `complete` or `skip`; the answer says whether it counts as a listen. |
|
||||
| signed in | `GET /listens` | Your own history, newest first, with `counted` per event. Query: `limit`, `offset`, `counted=true`. |
|
||||
| signed in | `GET /library/scan` | Scan status and the last result. |
|
||||
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}` | See and set up Last.fm and MusicBrainz. The API key is write-only. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test` | See, set up and check Last.fm and MusicBrainz. The API key is write-only. |
|
||||
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
|
||||
|
||||
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.
|
||||
|
|
@ -258,7 +265,8 @@ internal/api/openapi/ the OpenAPI spec, embedded in the binary
|
|||
internal/auth/ users, roles, passwords, access and refresh tokens
|
||||
internal/catalog/ browsing and searching artists, albums, tracks
|
||||
internal/library/ libraries, the scanner, the event hub, the tag reader
|
||||
internal/providers/ external providers: settings, and the careful outbound HTTP client
|
||||
internal/discovery/ similar artists: what the providers said, kept and refreshed
|
||||
internal/providers/ external providers: settings, the outbound HTTP client, the Last.fm client
|
||||
internal/listening/ listen events
|
||||
internal/storage/ SQLite and PostgreSQL through bun, the repositories
|
||||
internal/streaming/ opening a track's file safely
|
||||
|
|
|
|||
|
|
@ -18,9 +18,12 @@ import (
|
|||
"codeberg.org/squidd/sonda/internal/auth"
|
||||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/config"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
"codeberg.org/squidd/sonda/migrations"
|
||||
|
|
@ -28,6 +31,12 @@ import (
|
|||
|
||||
const defaultConfigFile = "sonda.toml"
|
||||
|
||||
// version goes into the User-Agent sent to providers.
|
||||
const version = "0.1.0"
|
||||
|
||||
// lastfmMinInterval spaces the requests to Last.fm well below its limit.
|
||||
const lastfmMinInterval = 250 * time.Millisecond
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "sondad:", err)
|
||||
|
|
@ -98,6 +107,15 @@ func run() error {
|
|||
libraries.SetBusy(func() bool { return scans.Status().Running })
|
||||
warnAboutLibraries(ctx, logger, libraries)
|
||||
|
||||
// External providers: nothing is contacted unless an admin switched one on.
|
||||
providerSettings := providers.NewService(storage.NewProviderRepo(db))
|
||||
lastfmClient := lastfm.New(outbound.New(outbound.Options{
|
||||
UserAgent: providers.UserAgent(version, ""),
|
||||
MinInterval: lastfmMinInterval,
|
||||
}), lastfm.BaseURL)
|
||||
providerSettings.SetChecker(providers.LastFM, lastfmClient.Probe)
|
||||
discoveries := discovery.NewService(storage.NewDiscoveryRepo(db), lastfmClient, providerSettings, logger)
|
||||
|
||||
router := httpapi.NewRouter(httpapi.Deps{
|
||||
Logger: logger,
|
||||
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()),
|
||||
|
|
@ -106,7 +124,8 @@ func run() error {
|
|||
Scan: scanControl{ctx: ctx, m: scans},
|
||||
Libraries: libraries,
|
||||
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
|
||||
Providers: providers.NewService(storage.NewProviderRepo(db)),
|
||||
Providers: providerSettings,
|
||||
Discovery: discoveries,
|
||||
Streams: streaming.NewService(libraryRepo),
|
||||
})
|
||||
|
||||
|
|
|
|||
2
go.mod
2
go.mod
|
|
@ -15,6 +15,7 @@ require (
|
|||
go.senan.xyz/taglib v0.15.0
|
||||
go.uber.org/zap v1.28.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/sync v0.24.0
|
||||
modernc.org/sqlite v1.62.0
|
||||
)
|
||||
|
||||
|
|
@ -62,7 +63,6 @@ require (
|
|||
go.yaml.in/yaml/v4 v4.0.0 // indirect
|
||||
golang.org/x/arch v0.22.0 // indirect
|
||||
golang.org/x/net v0.60.0 // indirect
|
||||
golang.org/x/sync v0.24.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/text v0.43.0 // indirect
|
||||
google.golang.org/protobuf v1.36.10 // indirect
|
||||
|
|
|
|||
2
go.sum
2
go.sum
|
|
@ -52,6 +52,8 @@ github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50 h1:PlNWJRxu5APSV0pWqf
|
|||
github.com/google/pprof v0.0.0-20261016150608-1facf8f91b50/go.mod h1:yUo1AMMpsPIDFGuwDK8nLcf8+vvAd5l2O6JDZ6hpwrM=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/gorilla/mux v1.8.0 h1:i40aqfkR1h2SlN9hojwV5ZA91wcXFOvkdNIeFDP5koI=
|
||||
github.com/gorilla/mux v1.8.0/go.mod h1:DVbg23sWSpFRCP0SfiEN6jmj59UnW/n46BH5rLB71So=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
|
||||
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
|
|
|
|||
|
|
@ -16,6 +16,7 @@ import (
|
|||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/auth"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
|
|
@ -74,6 +75,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
}
|
||||
|
||||
tokens := auth.NewTokenIssuer(secret, time.Hour)
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
deps := Deps{
|
||||
Logger: zap.NewNop(),
|
||||
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, 24*time.Hour),
|
||||
|
|
@ -81,7 +83,8 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
Listening: listening.NewService(storage.NewListeningRepo(db), 20*time.Second),
|
||||
Scan: &fakeScan{},
|
||||
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
|
||||
Providers: providers.NewService(storage.NewProviderRepo(db)),
|
||||
Providers: prov,
|
||||
Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()),
|
||||
Streams: streaming.NewService(noFiles{}),
|
||||
}
|
||||
mutate(&deps, db)
|
||||
|
|
|
|||
|
|
@ -19,9 +19,13 @@ import (
|
|||
"github.com/getkin/kin-openapi/routers"
|
||||
"github.com/getkin/kin-openapi/routers/legacy"
|
||||
"github.com/uptrace/bun"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
"codeberg.org/squidd/sonda/internal/streaming"
|
||||
)
|
||||
|
|
@ -81,6 +85,7 @@ func (a scanAdapter) StartLibrary(id int64) error { return a.m.StartLibrary(cont
|
|||
func (a scanAdapter) Status() library.Status { return a.m.Status() }
|
||||
|
||||
type contract struct {
|
||||
lastfm *fakeLastfm
|
||||
t *testing.T
|
||||
h http.Handler
|
||||
db *bun.DB
|
||||
|
|
@ -93,7 +98,12 @@ type contract struct {
|
|||
func newContract(t *testing.T) *contract {
|
||||
t.Helper()
|
||||
reader := &contractReader{}
|
||||
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
|
||||
d.Providers = prov
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), lf, prov, zap.NewNop())
|
||||
repo := storage.NewLibraryRepo(db)
|
||||
mgr := library.NewManager(library.NewScanner(repo, reader, library.NewHub(), 2), repo)
|
||||
libs := library.NewLibraries(repo)
|
||||
|
|
@ -115,7 +125,7 @@ func newContract(t *testing.T) *contract {
|
|||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
return &contract{t: t, h: h, db: db, reader: reader, router: router, seen: map[string]bool{}, music: music}
|
||||
return &contract{lastfm: lf, t: t, h: h, db: db, reader: reader, router: router, seen: map[string]bool{}, music: music}
|
||||
}
|
||||
|
||||
// call sends a request to /api/v1<path>, checks it and its answer against the
|
||||
|
|
@ -380,6 +390,14 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/spotify", token, map[string]any{"enabled": true}), 404, "unknown provider", "provider_not_found")
|
||||
c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test without a key")
|
||||
c.expect(c.call("POST", "/providers/musicbrainz/test", token, nil), 400, "test a provider that has no test", "invalid_input")
|
||||
c.expect(c.call("POST", "/providers/spotify/test", token, nil), 404, "test an unknown provider", "provider_not_found")
|
||||
c.expect(c.call("POST", "/providers/lastfm/test", "", nil), 401, "test without a token", "unauthorized")
|
||||
off := c.expect(c.call("GET", "/artists/"+artistID+"/similar", token, nil), 200, "similar artists with the provider off")
|
||||
if off["provider_status"] != "disabled" || c.lastfm.count() != 0 {
|
||||
t.Fatalf("similar with the provider off = %v, calls %d", off, c.lastfm.count())
|
||||
}
|
||||
saved := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "0123456789abcdef0123456789abcdef"}), 200, "save a key")
|
||||
if saved["has_api_key"] != true || saved["updated_at"] == nil {
|
||||
t.Fatalf("saved = %v", saved)
|
||||
|
|
@ -387,6 +405,15 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
if on := c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 200, "enable"); on["enabled"] != true {
|
||||
t.Fatalf("enabled = %v", on)
|
||||
}
|
||||
if ok := c.expect(c.call("POST", "/providers/lastfm/test", token, nil), 200, "test with a key"); ok["ok"] != true {
|
||||
t.Fatalf("test = %v", ok)
|
||||
}
|
||||
similar := c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=5", token, nil), 200, "similar artists")
|
||||
if items := similar["items"].([]any); similar["provider_status"] != "ok" || len(items) != 1 {
|
||||
t.Fatalf("similar = %v", similar)
|
||||
}
|
||||
c.expect(c.call("GET", "/artists/999/similar", token, nil), 404, "similar to an unknown artist", "artist_not_found")
|
||||
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
|
||||
|
||||
// ---- playback ----
|
||||
|
|
|
|||
|
|
@ -39,6 +39,8 @@ func writeProviderError(c *gin.Context, err error) {
|
|||
switch {
|
||||
case errors.Is(err, providers.ErrUnknownProvider):
|
||||
writeError(c, http.StatusNotFound, "provider_not_found", "no such provider")
|
||||
case errors.Is(err, providers.ErrNoTest):
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", "this provider has no test")
|
||||
case errors.Is(err, providers.ErrInvalidSettings):
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
|
||||
default:
|
||||
|
|
@ -79,3 +81,12 @@ func (h *providersHandlers) update(c *gin.Context) {
|
|||
}
|
||||
c.JSON(http.StatusOK, toProviderJSON(info))
|
||||
}
|
||||
|
||||
func (h *providersHandlers) test(c *gin.Context) {
|
||||
res, err := h.svc.Test(c.Request.Context(), c.Param("name"))
|
||||
if err != nil {
|
||||
writeProviderError(c, err)
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"ok": res.OK, "status": res.Status})
|
||||
}
|
||||
|
|
|
|||
69
internal/api/http/handlers_similar.go
Normal file
69
internal/api/http/handlers_similar.go
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/gin-gonic/gin"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
)
|
||||
|
||||
// discoveryHandlers serve what providers say about artists outside the library.
|
||||
type discoveryHandlers struct{ svc discoveryService }
|
||||
|
||||
type similarItemJSON struct {
|
||||
Artist artistRefJSON `json:"artist"`
|
||||
InLibrary bool `json:"in_library"`
|
||||
Score float64 `json:"score"`
|
||||
Rank int `json:"rank"`
|
||||
}
|
||||
|
||||
type similarJSON struct {
|
||||
Provider string `json:"provider"`
|
||||
ProviderStatus string `json:"provider_status"`
|
||||
Stale bool `json:"stale"`
|
||||
RetrievedAt *string `json:"retrieved_at"`
|
||||
Attribution string `json:"attribution"`
|
||||
Items []similarItemJSON `json:"items"`
|
||||
}
|
||||
|
||||
func toSimilarJSON(r discovery.Result) similarJSON {
|
||||
out := similarJSON{
|
||||
Provider: r.Provider, ProviderStatus: r.ProviderStatus, Stale: r.Stale,
|
||||
Attribution: discovery.Attribution, Items: make([]similarItemJSON, len(r.Items)),
|
||||
}
|
||||
if !r.RetrievedAt.IsZero() {
|
||||
out.RetrievedAt = rfc3339(r.RetrievedAt)
|
||||
}
|
||||
for i, it := range r.Items {
|
||||
out.Items[i] = similarItemJSON{
|
||||
Artist: artistRefJSON{ID: it.Artist.ID, Name: it.Artist.Name},
|
||||
InLibrary: it.InLibrary, Score: it.Score, Rank: it.Rank,
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func (h *discoveryHandlers) similar(c *gin.Context) {
|
||||
id, ok := pathID(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
limit, ok := queryInt(c, "limit")
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
res, err := h.svc.Similar(c.Request.Context(), id, clampInt(limit))
|
||||
switch {
|
||||
case errors.Is(err, discovery.ErrNotFound):
|
||||
writeError(c, http.StatusNotFound, "artist_not_found", "not found")
|
||||
case errors.Is(err, discovery.ErrInvalidQuery):
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
|
||||
case err != nil:
|
||||
_ = c.Error(err)
|
||||
writeError(c, http.StatusInternalServerError, "internal", "internal error")
|
||||
default:
|
||||
c.JSON(http.StatusOK, toSimilarJSON(res))
|
||||
}
|
||||
}
|
||||
|
|
@ -1,7 +1,9 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
"testing"
|
||||
|
|
@ -9,6 +11,9 @@ import (
|
|||
"github.com/uptrace/bun"
|
||||
"go.uber.org/zap"
|
||||
"go.uber.org/zap/zaptest/observer"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
)
|
||||
|
||||
const lastfmKey = "0123456789abcdef0123456789abcdef"
|
||||
|
|
@ -169,3 +174,60 @@ func TestTheKeyNeverReachesTheLog(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ---- testing a provider ----
|
||||
|
||||
func TestProviderTestIsAdminOnly(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
_, bob := addUser(t, h, admin, "bob", false)
|
||||
if r := do(t, h, "POST", "/api/v1/providers/lastfm/test", "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("no token: %d", r.Code)
|
||||
}
|
||||
if r := do(t, h, "POST", "/api/v1/providers/lastfm/test", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
|
||||
t.Errorf("normal user: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProviderTest(t *testing.T) {
|
||||
var asked string
|
||||
var answer error
|
||||
h, _ := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov.SetChecker(providers.LastFM, func(_ context.Context, key string) error { asked = key; return answer })
|
||||
d.Providers = prov
|
||||
})
|
||||
admin := registerAndLogin(t, h)
|
||||
|
||||
// Without a key nobody is asked.
|
||||
r := do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["ok"] != false || r.Body["status"] != "no_api_key" || asked != "" {
|
||||
t.Fatalf("no key: %d %v asked=%q", r.Code, r.Body, asked)
|
||||
}
|
||||
sameKeys(t, "test result", r.Body, "ok", "status")
|
||||
|
||||
do(t, h, "PUT", "/api/v1/providers/lastfm", admin, map[string]any{"api_key": lastfmKey})
|
||||
r = do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["ok"] != true || r.Body["status"] != "ok" || asked != lastfmKey {
|
||||
t.Fatalf("working key: %d %v", r.Code, r.Body)
|
||||
}
|
||||
|
||||
for want, err := range map[string]error{
|
||||
"key_refused": providers.ErrKeyRefused, "rate_limited": providers.ErrRateLimited,
|
||||
"unreachable": providers.ErrUnreachable, "error": errors.New("boom " + lastfmKey),
|
||||
} {
|
||||
answer = err
|
||||
r = do(t, h, "POST", "/api/v1/providers/lastfm/test", admin, nil)
|
||||
raw, _ := json.Marshal(r.Body)
|
||||
if r.Code != http.StatusOK || r.Body["ok"] != false || r.Body["status"] != want || strings.Contains(string(raw), lastfmKey) {
|
||||
t.Errorf("%s: %d %v", want, r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
if r = do(t, h, "POST", "/api/v1/providers/spotify/test", admin, nil); r.Code != http.StatusNotFound || errCode(r) != "provider_not_found" {
|
||||
t.Errorf("unknown provider: %d %v", r.Code, r.Body)
|
||||
}
|
||||
if r = do(t, h, "POST", "/api/v1/providers/musicbrainz/test", admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
|
||||
t.Errorf("a provider without a test: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ import (
|
|||
|
||||
"codeberg.org/squidd/sonda/internal/auth"
|
||||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/listening"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
|
|
@ -57,6 +58,11 @@ type scanService interface {
|
|||
type providerService interface {
|
||||
List(ctx context.Context) ([]providers.Info, error)
|
||||
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
|
||||
Test(ctx context.Context, name string) (providers.TestResult, error)
|
||||
}
|
||||
|
||||
type discoveryService interface {
|
||||
Similar(ctx context.Context, artistID int64, limit int) (discovery.Result, error)
|
||||
}
|
||||
|
||||
type catalogService interface {
|
||||
|
|
@ -85,6 +91,7 @@ type Deps struct {
|
|||
Libraries libraryService
|
||||
Catalog catalogService
|
||||
Providers providerService
|
||||
Discovery discoveryService
|
||||
Streams streamService
|
||||
}
|
||||
|
||||
|
|
@ -119,6 +126,8 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
cat := &catalogHandlers{svc: d.Catalog}
|
||||
protected.GET("/artists", cat.listArtists)
|
||||
protected.GET("/artists/:id", cat.getArtist)
|
||||
disc := &discoveryHandlers{svc: d.Discovery}
|
||||
protected.GET("/artists/:id/similar", disc.similar)
|
||||
protected.GET("/albums", cat.listAlbums)
|
||||
protected.GET("/albums/:id", cat.getAlbum)
|
||||
protected.GET("/tracks", cat.listTracks)
|
||||
|
|
@ -143,6 +152,7 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
prov := &providersHandlers{svc: d.Providers}
|
||||
admin.GET("/providers", prov.list)
|
||||
admin.PUT("/providers/:name", prov.update)
|
||||
admin.POST("/providers/:name/test", prov.test)
|
||||
u := &usersHandlers{svc: d.Auth}
|
||||
admin.GET("/users", u.list)
|
||||
admin.POST("/users", u.create)
|
||||
|
|
|
|||
349
internal/api/http/similar_test.go
Normal file
349
internal/api/http/similar_test.go
Normal file
|
|
@ -0,0 +1,349 @@
|
|||
package httpapi
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/catalog"
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/library"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
)
|
||||
|
||||
// fakeLastfm answers for Last.fm and counts the questions.
|
||||
type fakeLastfm struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
key string
|
||||
result []lastfm.Similar
|
||||
err error
|
||||
}
|
||||
|
||||
func (f *fakeLastfm) SimilarArtists(_ context.Context, key string, _ lastfm.Artist, _ int) ([]lastfm.Similar, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls++
|
||||
f.key = key
|
||||
return f.result, f.err
|
||||
}
|
||||
|
||||
func (f *fakeLastfm) count() int {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.calls
|
||||
}
|
||||
|
||||
type similarEnv struct {
|
||||
h http.Handler
|
||||
db *bun.DB
|
||||
admin string
|
||||
source *fakeLastfm
|
||||
alpha int64 // a local artist
|
||||
beta int64 // another local artist
|
||||
}
|
||||
|
||||
// newSimilarEnv wires the real catalog, discovery and provider services on a
|
||||
// database that holds two local artists, Alpha and Beta, with Last.fm switched
|
||||
// off and without a key.
|
||||
func newSimilarEnv(t *testing.T) *similarEnv {
|
||||
t.Helper()
|
||||
src := &fakeLastfm{result: []lastfm.Similar{
|
||||
{Name: "Beta", Match: 1},
|
||||
{Name: "Gamma", MBID: "99999999-0000-0000-0000-000000000001", Match: 0.5},
|
||||
}}
|
||||
e := &similarEnv{source: src}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
})
|
||||
e.admin = registerAndLogin(t, e.h)
|
||||
|
||||
lib := storage.NewLibraryRepo(e.db)
|
||||
l, err := lib.CreateLibrary(context.Background(), "music", "/lib")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, name := range []string{"Alpha", "Beta"} {
|
||||
f := library.ParsedFile{
|
||||
FileInfo: library.FileInfo{RelPath: name + ".flac", Size: 1, MTimeUnix: 1},
|
||||
Metadata: library.Metadata{
|
||||
Tags: library.Tags{Title: name + " song", Artists: []string{name}},
|
||||
Properties: library.Properties{Format: "flac", DurationMs: 1000},
|
||||
},
|
||||
}
|
||||
if _, err := lib.SaveFile(context.Background(), l.ID, f, 0); err != nil {
|
||||
t.Fatalf("seed %d: %v", i, err)
|
||||
}
|
||||
}
|
||||
e.alpha, e.beta = e.artist(t, "Alpha"), e.artist(t, "Beta")
|
||||
return e
|
||||
}
|
||||
|
||||
func (e *similarEnv) artist(t *testing.T, name string) int64 {
|
||||
t.Helper()
|
||||
var id int64
|
||||
if err := e.db.QueryRow("SELECT id FROM artists WHERE name = ?", name).Scan(&id); err != nil {
|
||||
t.Fatalf("artist %q: %v", name, err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func (e *similarEnv) enableLastfm(t *testing.T) {
|
||||
t.Helper()
|
||||
for _, body := range []map[string]any{{"api_key": lastfmKey}, {"enabled": true}} {
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/lastfm", e.admin, body); r.Code != http.StatusOK {
|
||||
t.Fatalf("set up lastfm: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (e *similarEnv) path(id int64) string {
|
||||
return "/api/v1/artists/" + strconv.FormatInt(id, 10) + "/similar"
|
||||
}
|
||||
|
||||
func TestSimilarNeedsASignedInUser(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
if r := do(t, e.h, "GET", e.path(e.alpha), "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("no token: %d", r.Code)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarRejectsWhatIsNotAnArtistOrALimit(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
for name, tt := range map[string]struct {
|
||||
path string
|
||||
code int
|
||||
err string
|
||||
}{
|
||||
"unknown artist": {e.path(99999), 404, "artist_not_found"},
|
||||
"not a number": {"/api/v1/artists/abc/similar", 400, "invalid_input"},
|
||||
"artist with no files and no discovery": {e.path(e.artistWithoutFiles(t)), 404, "artist_not_found"},
|
||||
"limit not a number": {e.path(e.alpha) + "?limit=lots", 400, "invalid_input"},
|
||||
"negative limit": {e.path(e.alpha) + "?limit=-1", 400, "invalid_input"},
|
||||
} {
|
||||
r := do(t, e.h, "GET", tt.path, e.admin, nil)
|
||||
if r.Code != tt.code || errCode(r) != tt.err {
|
||||
t.Errorf("%s: %d %v, want %d %s", name, r.Code, r.Body, tt.code, tt.err)
|
||||
}
|
||||
}
|
||||
if e.source.count() != 0 {
|
||||
t.Error("Last.fm was asked about something that is not an artist")
|
||||
}
|
||||
}
|
||||
|
||||
func (e *similarEnv) artistWithoutFiles(t *testing.T) int64 {
|
||||
t.Helper()
|
||||
var id int64
|
||||
if err := e.db.QueryRow("INSERT INTO artists (name, name_key) VALUES ('Nobody', 'nobody') RETURNING id").Scan(&id); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return id
|
||||
}
|
||||
|
||||
func TestSimilarWithTheProviderOffAsksNobody(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
sameKeys(t, "similar", r.Body, "provider", "provider_status", "stale", "retrieved_at", "attribution", "items")
|
||||
items, ok := r.Body["items"].([]any)
|
||||
if r.Body["provider"] != "lastfm" || r.Body["provider_status"] != "disabled" || r.Body["stale"] != false ||
|
||||
!ok || len(items) != 0 {
|
||||
t.Errorf("answer = %v", r.Body)
|
||||
}
|
||||
if v, present := r.Body["retrieved_at"]; !present || v != nil {
|
||||
t.Errorf("retrieved_at = %#v, want an explicit null when nothing is known", v)
|
||||
}
|
||||
if e.source.count() != 0 {
|
||||
t.Error("a provider that is off was contacted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarArtists(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
sameKeys(t, "similar", r.Body, "provider", "provider_status", "stale", "retrieved_at", "attribution", "items")
|
||||
if r.Body["provider_status"] != "ok" || r.Body["stale"] != false || r.Body["retrieved_at"] == nil {
|
||||
t.Errorf("answer = %v", r.Body)
|
||||
}
|
||||
if !strings.Contains(r.Body["attribution"].(string), "Last.fm") {
|
||||
t.Errorf("attribution = %v", r.Body["attribution"])
|
||||
}
|
||||
items := r.Body["items"].([]any)
|
||||
if len(items) != 2 {
|
||||
t.Fatalf("items = %v", items)
|
||||
}
|
||||
first, second := items[0].(map[string]any), items[1].(map[string]any)
|
||||
sameKeys(t, "item", first, "artist", "in_library", "score", "rank")
|
||||
sameSet := first["artist"].(map[string]any)
|
||||
if sameSet["name"] != "Beta" || sameSet["id"] != float64(e.beta) || first["in_library"] != true || first["rank"] != float64(1) || first["score"] != float64(1) {
|
||||
t.Errorf("first = %v", first)
|
||||
}
|
||||
if second["artist"].(map[string]any)["name"] != "Gamma" || second["in_library"] != false || second["rank"] != float64(2) || second["score"] != 0.5 {
|
||||
t.Errorf("second = %v", second)
|
||||
}
|
||||
if e.source.key != lastfmKey {
|
||||
t.Error("the stored key was not the one used")
|
||||
}
|
||||
raw, _ := json.Marshal(r.Body)
|
||||
if strings.Contains(string(raw), lastfmKey) {
|
||||
t.Fatal("the answer contains the API key")
|
||||
}
|
||||
|
||||
// A second look comes from the database; the page size trims it.
|
||||
again := do(t, e.h, "GET", e.path(e.alpha)+"?limit=1", e.admin, nil)
|
||||
if again.Code != http.StatusOK || len(again.Body["items"].([]any)) != 1 || e.source.count() != 1 {
|
||||
t.Errorf("second look: %d items=%v calls=%d", again.Code, again.Body["items"], e.source.count())
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarWhenLastfmFailsIsAnAnswerNotAnError(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
e.source.err = lastfm.ErrUnavailable
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["provider_status"] != "error" || len(r.Body["items"].([]any)) != 0 {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// Artists that only a provider named can be shown, and never appear in the
|
||||
// lists of the library.
|
||||
func TestADiscoveredArtistCanBeOpenedButIsNotListed(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
gamma := int64(r.Body["items"].([]any)[1].(map[string]any)["artist"].(map[string]any)["id"].(float64))
|
||||
|
||||
a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil)
|
||||
if a.Code != http.StatusOK || a.Body["name"] != "Gamma" || a.Body["track_count"] != float64(0) || a.Body["album_count"] != float64(0) {
|
||||
t.Fatalf("GetArtist: %d %v", a.Code, a.Body)
|
||||
}
|
||||
list := do(t, e.h, "GET", "/api/v1/artists?q=gamma", e.admin, nil)
|
||||
if list.Body["total"] != float64(0) {
|
||||
t.Errorf("a discovered artist is listed: %v", list.Body)
|
||||
}
|
||||
|
||||
// And it can be asked about in turn.
|
||||
e.source.result = []lastfm.Similar{{Name: "Delta", Match: 1}}
|
||||
next := do(t, e.h, "GET", e.path(gamma), e.admin, nil)
|
||||
if next.Code != http.StatusOK || len(next.Body["items"].([]any)) != 1 {
|
||||
t.Errorf("similar to a discovered artist: %d %v", next.Code, next.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// What leaves the server, seen from the other end: a real Last.fm client and
|
||||
// outbound client talk to a local server that records the whole request. The
|
||||
// database holds a user, a listen, a track title and a library path; none of
|
||||
// them may be in it.
|
||||
func TestWhatLeavesTheServerIsTheArtistAndTheKeyOnly(t *testing.T) {
|
||||
var (
|
||||
mu sync.Mutex
|
||||
seen []string
|
||||
hdr http.Header
|
||||
)
|
||||
remote := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
mu.Lock()
|
||||
seen = append(seen, r.Method+" "+r.URL.String())
|
||||
hdr = r.Header.Clone()
|
||||
mu.Unlock()
|
||||
_, _ = w.Write([]byte(`{"similarartists":{"artist":[{"name":"Gamma","mbid":"","match":"1"}]}}`))
|
||||
}))
|
||||
defer remote.Close()
|
||||
|
||||
var alpha int64
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
d.Providers = prov
|
||||
client := lastfm.New(outbound.New(outbound.Options{UserAgent: providers.UserAgent("1.2.3", "")}), remote.URL)
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), client, prov, zap.NewNop())
|
||||
})
|
||||
admin := registerAndLogin(t, h)
|
||||
l, err := storage.NewLibraryRepo(db).CreateLibrary(context.Background(), "Private Collection", "/home/alice/secret-music")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
f := library.ParsedFile{
|
||||
FileInfo: library.FileInfo{RelPath: "private/embarrassing-song.flac", Size: 1, MTimeUnix: 1},
|
||||
Metadata: library.Metadata{
|
||||
Tags: library.Tags{Title: "Embarrassing Song", Album: "Guilty Pleasures", Artists: []string{"Alpha"}},
|
||||
Properties: library.Properties{Format: "flac", DurationMs: 1000},
|
||||
},
|
||||
}
|
||||
if _, err := storage.NewLibraryRepo(db).SaveFile(context.Background(), l.ID, f, 0); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var track int64
|
||||
_ = db.QueryRow("SELECT id FROM tracks WHERE title = 'Embarrassing Song'").Scan(&track)
|
||||
do(t, h, "POST", "/api/v1/listens", admin, map[string]any{
|
||||
"track_id": track, "event_type": "complete", "idempotency_key": "k1"})
|
||||
_ = db.QueryRow("SELECT id FROM artists WHERE name = 'Alpha'").Scan(&alpha)
|
||||
|
||||
for _, body := range []map[string]any{{"api_key": lastfmKey}, {"enabled": true}} {
|
||||
if r := do(t, h, "PUT", "/api/v1/providers/lastfm", admin, body); r.Code != http.StatusOK {
|
||||
t.Fatal(r.Body)
|
||||
}
|
||||
}
|
||||
r := do(t, h, "GET", "/api/v1/artists/"+strconv.FormatInt(alpha, 10)+"/similar", admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["provider_status"] != "ok" {
|
||||
t.Fatalf("%d %v", r.Code, r.Body)
|
||||
}
|
||||
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if len(seen) != 1 {
|
||||
t.Fatalf("requests = %v, want exactly one", seen)
|
||||
}
|
||||
u, _ := url.Parse(strings.SplitN(seen[0], " ", 2)[1])
|
||||
want := map[string]string{
|
||||
"method": "artist.getsimilar", "artist": "Alpha", "autocorrect": "0", "limit": "50",
|
||||
"api_key": lastfmKey, "format": "json",
|
||||
}
|
||||
got := map[string]string{}
|
||||
for k, v := range u.Query() {
|
||||
got[k] = v[0]
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("query = %v, want %v", got, want)
|
||||
}
|
||||
// Only a User-Agent that says who Sonda is, and nothing that says who uses it.
|
||||
if hdr.Get("User-Agent") != "Sonda/1.2.3 ( "+providers.ProjectURL+" )" {
|
||||
t.Errorf("User-Agent = %q", hdr.Get("User-Agent"))
|
||||
}
|
||||
for name := range hdr {
|
||||
switch http.CanonicalHeaderKey(name) {
|
||||
case "User-Agent", "Accept", "Accept-Encoding":
|
||||
default:
|
||||
t.Errorf("unexpected header %s: %v", name, hdr[name])
|
||||
}
|
||||
}
|
||||
everything := seen[0] + " " + hdr.Get("User-Agent")
|
||||
for _, secret := range []string{"alice", "Embarrassing", "Guilty", "Private Collection", "secret-music", "flac"} {
|
||||
if strings.Contains(strings.ToLower(everything), strings.ToLower(secret)) {
|
||||
t.Errorf("the request mentions %q: %s", secret, everything)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -33,6 +33,8 @@ tags:
|
|||
description: Browsing and searching what can be played.
|
||||
- name: Playback
|
||||
description: Streaming and listening history.
|
||||
- name: Discovery
|
||||
description: Music outside the library, from external providers an admin switched on.
|
||||
- name: Providers
|
||||
description: External services Sonda can ask about artists, set up by an admin.
|
||||
|
||||
|
|
@ -543,7 +545,54 @@ paths:
|
|||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'404':
|
||||
description: No such artist, or nothing of it can be played (`artist_not_found`).
|
||||
description: |
|
||||
No such artist (`artist_not_found`). An artist with nothing to play is not found either,
|
||||
unless a provider named it as similar to another artist: those can be shown, with
|
||||
`track_count` and `album_count` of 0, but they are never part of the lists.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/artists/{id}/similar:
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Id'}
|
||||
get:
|
||||
tags: [Discovery]
|
||||
operationId: getSimilarArtists
|
||||
summary: Artists similar to one
|
||||
description: |
|
||||
Artists an external provider (Last.fm) finds similar to this one, best first. Each says
|
||||
whether it is in the library, and `id` works with `GET /artists/{id}`; one that is not in the
|
||||
library has no files.
|
||||
|
||||
The provider is only asked when an admin switched it on and the stored answer is missing or
|
||||
old (7 days; a "no such artist" answer is kept 1 day). What is sent is the artist's name or
|
||||
MusicBrainz id and nothing about any user. Several requests for the same artist at once
|
||||
make one call.
|
||||
|
||||
A provider that is off or failing is not an error: the answer is `200` and `provider_status`
|
||||
says why. Then `items` holds whatever was stored before, if anything, and `stale` is true
|
||||
when it is older than it should be.
|
||||
|
||||
`score` is the provider's own number between 0 and 1. It is relative to the list it came
|
||||
in (the first is always 1), so it only compares artists inside one answer. `rank` is the
|
||||
position. Show `attribution` where you show the data.
|
||||
security: [{bearerAuth: []}]
|
||||
parameters:
|
||||
- name: limit
|
||||
in: query
|
||||
description: How many to return. 20 when absent or 0; values above 100 are lowered to 100.
|
||||
schema: {type: integer, minimum: 0, default: 20}
|
||||
responses:
|
||||
'200':
|
||||
description: The similar artists.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/SimilarArtists'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'404':
|
||||
description: No such artist (`artist_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
|
@ -817,6 +866,40 @@ paths:
|
|||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/providers/{name}/test:
|
||||
parameters:
|
||||
- name: name
|
||||
in: path
|
||||
required: true
|
||||
schema: {type: string, enum: [lastfm, musicbrainz]}
|
||||
post:
|
||||
tags: [Providers]
|
||||
operationId: testProvider
|
||||
summary: Check a provider's API key
|
||||
description: |
|
||||
Admin only. Asks the provider one cheap question with the stored key, to find out whether
|
||||
it works. It works whether the provider is on or off, so you can check before switching it on.
|
||||
It is the only request Sonda makes at an admin's command rather than for a listener, and it
|
||||
sends nothing but the key. A provider that cannot be reached or refuses the key is not an
|
||||
error here: the answer is `200` with `ok` false and the reason in `status`.
|
||||
|
||||
Only Last.fm can be tested so far; asking for MusicBrainz is a `400`.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
description: The outcome of the check.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/ProviderTest'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'404':
|
||||
description: No such provider (`provider_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/listens:
|
||||
get:
|
||||
tags: [Playback]
|
||||
|
|
@ -1269,6 +1352,50 @@ components:
|
|||
maxLength: 200
|
||||
description: A URL or e-mail address, without parentheses. Empty restores the default.
|
||||
|
||||
ProviderTest:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [ok, status]
|
||||
properties:
|
||||
ok: {type: boolean}
|
||||
status:
|
||||
type: string
|
||||
enum: [ok, no_api_key, key_refused, rate_limited, unreachable, error]
|
||||
description: |
|
||||
`no_api_key`: none is stored, so nobody was asked. `key_refused`: the provider does not accept
|
||||
it. `rate_limited`: try again later. `unreachable`: the provider did not answer. `error`: anything else.
|
||||
|
||||
SimilarArtists:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [provider, provider_status, stale, retrieved_at, attribution, items]
|
||||
properties:
|
||||
provider: {type: string, enum: [lastfm]}
|
||||
provider_status:
|
||||
type: string
|
||||
enum: [ok, disabled, error]
|
||||
description: |
|
||||
`ok`: the data is as fresh as it needs to be. `disabled`: the provider is off or has no key,
|
||||
so it was not asked. `error`: it was asked and failed.
|
||||
stale: {type: boolean, description: "True when `items` are older than they should be because the provider could not be asked."}
|
||||
retrieved_at:
|
||||
type: string
|
||||
format: date-time
|
||||
nullable: true
|
||||
description: When the provider was last read for this artist; null if never.
|
||||
attribution: {type: string, description: Credit to show where the data is shown.}
|
||||
items:
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [artist, in_library, score, rank]
|
||||
properties:
|
||||
artist: {$ref: '#/components/schemas/ArtistRef'}
|
||||
in_library: {type: boolean, description: Whether the artist has something playable in the library.}
|
||||
score: {type: number, minimum: 0, maximum: 1, description: "The provider's own number, relative to this list."}
|
||||
rank: {type: integer, minimum: 1, description: "Position in the provider's list, 1 first."}
|
||||
|
||||
ListenResult:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
|
|||
256
internal/discovery/discovery.go
Normal file
256
internal/discovery/discovery.go
Normal file
|
|
@ -0,0 +1,256 @@
|
|||
// Package discovery finds music outside the library: artists similar to one the
|
||||
// user already has.
|
||||
//
|
||||
// Data comes from an external provider (Last.fm today), only when the admin
|
||||
// switched that provider on, and only when someone asks. What was read is kept
|
||||
// in the database, so a second look costs nothing and nothing leaves the server
|
||||
// again until the data is old. The raw numbers of the provider are stored as
|
||||
// they are: turning them into a score is a later step.
|
||||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
"golang.org/x/sync/singleflight"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
)
|
||||
|
||||
// Provider is the provider similar artists come from.
|
||||
const Provider = providers.LastFM
|
||||
|
||||
// Attribution is what has to be shown with the data: Last.fm asks for credit and
|
||||
// allows non-commercial use only.
|
||||
const Attribution = "Similar artists from Last.fm (https://www.last.fm), for non-commercial use."
|
||||
|
||||
// How long what was read is believed. Last.fm sends no cache headers, so these
|
||||
// are our own choice: similarity moves slowly, and a "no such artist" answer is
|
||||
// remembered shortly in case it was a typo that was since fixed.
|
||||
const (
|
||||
PositiveTTL = 7 * 24 * time.Hour
|
||||
NegativeTTL = 24 * time.Hour
|
||||
)
|
||||
|
||||
// FetchLimit is how many similar artists are read from the provider each time,
|
||||
// whatever size of page was asked for, so that one answer serves every page.
|
||||
const FetchLimit = 50
|
||||
|
||||
// Bounds of the number of items returned.
|
||||
const (
|
||||
DefaultLimit = 20
|
||||
MaxLimit = 100
|
||||
)
|
||||
|
||||
// fetchTimeout bounds a shared provider call, which does not depend on whoever
|
||||
// started it still waiting.
|
||||
const fetchTimeout = time.Minute
|
||||
|
||||
// Errors returned by the service.
|
||||
var (
|
||||
ErrNotFound = errors.New("artist not found")
|
||||
ErrInvalidQuery = errors.New("invalid query")
|
||||
)
|
||||
|
||||
// Provider states reported with an answer.
|
||||
const (
|
||||
StatusOK = "ok" // the data is as fresh as it needs to be
|
||||
StatusDisabled = "disabled" // the provider is off or has no key, so it was not asked
|
||||
StatusError = "error" // the provider was asked and failed
|
||||
)
|
||||
|
||||
// Artist is who is being looked up.
|
||||
type Artist struct {
|
||||
ID int64
|
||||
Name string
|
||||
MBID string // empty when unknown
|
||||
}
|
||||
|
||||
// ArtistRef is the short form of an artist.
|
||||
type ArtistRef struct {
|
||||
ID int64
|
||||
Name string
|
||||
}
|
||||
|
||||
// Item is one similar artist. Score is the provider's own number, relative to
|
||||
// the list it came in, and Rank is the position in that list, starting at 1.
|
||||
type Item struct {
|
||||
Artist ArtistRef
|
||||
InLibrary bool // has something playable in this library
|
||||
Score float64
|
||||
Rank int
|
||||
}
|
||||
|
||||
// Snapshot is what is stored for one artist and one provider. Fetched is false
|
||||
// when the provider was never asked; NotFound when it said it does not know the
|
||||
// artist (or knew nothing similar).
|
||||
type Snapshot struct {
|
||||
Fetched bool
|
||||
NotFound bool
|
||||
RetrievedAt time.Time
|
||||
Items []Item
|
||||
}
|
||||
|
||||
// Result is the answer to Similar. RetrievedAt is zero when nothing is known.
|
||||
// Stale means the items are older than they should be because the provider could
|
||||
// not be asked.
|
||||
type Result struct {
|
||||
Provider string
|
||||
ProviderStatus string
|
||||
Stale bool
|
||||
RetrievedAt time.Time
|
||||
Items []Item
|
||||
}
|
||||
|
||||
// Store keeps what the providers said.
|
||||
type Store interface {
|
||||
// Artist returns ErrNotFound for an id the catalog does not know.
|
||||
Artist(ctx context.Context, id int64) (Artist, error)
|
||||
// Similar reads the stored items (at most limit, best first) and when they were read.
|
||||
Similar(ctx context.Context, id int64, provider string, limit int) (Snapshot, error)
|
||||
// SaveSimilar replaces what was stored for the artist. The candidates become
|
||||
// artists without files unless one is already known.
|
||||
SaveSimilar(ctx context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error
|
||||
}
|
||||
|
||||
// Source asks the provider.
|
||||
type Source interface {
|
||||
SimilarArtists(ctx context.Context, key string, a lastfm.Artist, limit int) ([]lastfm.Similar, error)
|
||||
}
|
||||
|
||||
// Settings says whether a provider may be used right now.
|
||||
type Settings interface {
|
||||
Active(ctx context.Context, name string) (providers.Settings, bool, error)
|
||||
}
|
||||
|
||||
// Service answers "who is similar to this artist".
|
||||
type Service struct {
|
||||
store Store
|
||||
source Source
|
||||
settings Settings
|
||||
log *zap.Logger
|
||||
now func() time.Time
|
||||
flight singleflight.Group
|
||||
}
|
||||
|
||||
// NewService builds a Service.
|
||||
func NewService(store Store, source Source, settings Settings, log *zap.Logger) *Service {
|
||||
if log == nil {
|
||||
log = zap.NewNop()
|
||||
}
|
||||
return &Service{store: store, source: source, settings: settings, log: log, now: time.Now}
|
||||
}
|
||||
|
||||
// Similar returns the artists similar to artistID, at most limit of them
|
||||
// (DefaultLimit for 0). It asks the provider only when the stored data is
|
||||
// missing or old and the provider is on. A provider that fails is not an error
|
||||
// for the caller: the answer says so, and carries what was stored, if anything.
|
||||
func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Result, error) {
|
||||
switch {
|
||||
case limit < 0:
|
||||
return Result{}, fmt.Errorf("%w: limit cannot be negative", ErrInvalidQuery)
|
||||
case limit == 0:
|
||||
limit = DefaultLimit
|
||||
case limit > MaxLimit:
|
||||
limit = MaxLimit
|
||||
}
|
||||
if artistID <= 0 {
|
||||
return Result{}, ErrNotFound
|
||||
}
|
||||
artist, err := s.store.Artist(ctx, artistID)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
snap, err := s.store.Similar(ctx, artistID, Provider, limit)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if snap.Fetched && s.fresh(snap) {
|
||||
return result(snap, StatusOK, false), nil
|
||||
}
|
||||
|
||||
cfg, active, err := s.settings.Active(ctx, Provider)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if !active {
|
||||
return result(snap, StatusDisabled, snap.Fetched), nil
|
||||
}
|
||||
|
||||
// One call per artist, however many ask at once. It runs on its own context,
|
||||
// so the first caller leaving does not fail the others; each caller still
|
||||
// waits only as long as its own context allows.
|
||||
done := s.flight.DoChan(fmt.Sprint(artistID), func() (any, error) {
|
||||
return nil, s.refresh(artist, cfg.APIKey)
|
||||
})
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return Result{}, ctx.Err()
|
||||
case r := <-done:
|
||||
if r.Err != nil {
|
||||
s.log.Warn("similar artists could not be read", zap.String("provider", Provider), zap.Error(r.Err))
|
||||
return result(snap, StatusError, snap.Fetched), nil
|
||||
}
|
||||
}
|
||||
snap, err = s.store.Similar(ctx, artistID, Provider, limit)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
return result(snap, StatusOK, false), nil
|
||||
}
|
||||
|
||||
func (s *Service) fresh(snap Snapshot) bool {
|
||||
ttl := PositiveTTL
|
||||
if snap.NotFound {
|
||||
ttl = NegativeTTL
|
||||
}
|
||||
return s.now().Sub(snap.RetrievedAt) < ttl
|
||||
}
|
||||
|
||||
// refresh reads the provider and stores what it said.
|
||||
func (s *Service) refresh(a Artist, key string) error {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), fetchTimeout)
|
||||
defer cancel()
|
||||
|
||||
got, err := s.source.SimilarArtists(ctx, key, lastfm.Artist{Name: a.Name, MBID: a.MBID}, FetchLimit)
|
||||
switch {
|
||||
case errors.Is(err, lastfm.ErrNotFound):
|
||||
got = nil
|
||||
case err != nil:
|
||||
return err
|
||||
}
|
||||
cands := withoutArtist(got, a)
|
||||
return s.store.SaveSimilar(ctx, a.ID, Provider, s.now(), len(cands) > 0, cands)
|
||||
}
|
||||
|
||||
// withoutArtist drops the artist itself, which providers sometimes list under
|
||||
// another spelling or the same id.
|
||||
func withoutArtist(in []lastfm.Similar, a Artist) []lastfm.Similar {
|
||||
name := strings.ToLower(strings.TrimSpace(a.Name))
|
||||
out := make([]lastfm.Similar, 0, len(in))
|
||||
for _, c := range in {
|
||||
if strings.ToLower(strings.TrimSpace(c.Name)) == name ||
|
||||
(a.MBID != "" && strings.EqualFold(c.MBID, a.MBID)) {
|
||||
continue
|
||||
}
|
||||
out = append(out, c)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func result(snap Snapshot, status string, stale bool) Result {
|
||||
items := snap.Items
|
||||
if items == nil {
|
||||
items = []Item{}
|
||||
}
|
||||
r := Result{Provider: Provider, ProviderStatus: status, Stale: stale && snap.Fetched, Items: items}
|
||||
if snap.Fetched {
|
||||
r.RetrievedAt = snap.RetrievedAt
|
||||
}
|
||||
return r
|
||||
}
|
||||
389
internal/discovery/discovery_test.go
Normal file
389
internal/discovery/discovery_test.go
Normal file
|
|
@ -0,0 +1,389 @@
|
|||
package discovery
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
)
|
||||
|
||||
const key = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
var t0 = time.Unix(1_700_000_000, 0)
|
||||
|
||||
type saved struct {
|
||||
artistID int64
|
||||
provider string
|
||||
at time.Time
|
||||
found bool
|
||||
cands []lastfm.Similar
|
||||
}
|
||||
|
||||
// fakeStore keeps one snapshot per artist, replaced by every save.
|
||||
type fakeStore struct {
|
||||
mu sync.Mutex
|
||||
artists map[int64]Artist
|
||||
snaps map[int64]Snapshot
|
||||
saves []saved
|
||||
lastLim int
|
||||
}
|
||||
|
||||
func newStore() *fakeStore {
|
||||
return &fakeStore{
|
||||
artists: map[int64]Artist{1: {ID: 1, Name: "Source", MBID: "aaaaaaaa-0000-0000-0000-000000000001"}},
|
||||
snaps: map[int64]Snapshot{},
|
||||
}
|
||||
}
|
||||
|
||||
func (f *fakeStore) Artist(_ context.Context, id int64) (Artist, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
a, ok := f.artists[id]
|
||||
if !ok {
|
||||
return Artist{}, ErrNotFound
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func (f *fakeStore) Similar(_ context.Context, id int64, _ string, limit int) (Snapshot, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.lastLim = limit
|
||||
s := f.snaps[id]
|
||||
if len(s.Items) > limit {
|
||||
s.Items = s.Items[:limit]
|
||||
}
|
||||
return s, nil
|
||||
}
|
||||
|
||||
func (f *fakeStore) SaveSimilar(_ context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.saves = append(f.saves, saved{id, provider, at, found, cands})
|
||||
snap := Snapshot{Fetched: true, NotFound: !found, RetrievedAt: at}
|
||||
for i, c := range cands {
|
||||
snap.Items = append(snap.Items, Item{Artist: ArtistRef{ID: int64(100 + i), Name: c.Name}, Score: c.Match, Rank: i + 1})
|
||||
}
|
||||
f.snaps[id] = snap
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeStore) put(id int64, s Snapshot) {
|
||||
f.mu.Lock()
|
||||
f.snaps[id] = s
|
||||
f.mu.Unlock()
|
||||
}
|
||||
|
||||
type fakeSource struct {
|
||||
mu sync.Mutex
|
||||
calls int
|
||||
key string
|
||||
artist lastfm.Artist
|
||||
limit int
|
||||
result []lastfm.Similar
|
||||
err error
|
||||
started chan struct{} // closed on the first call, when set
|
||||
release chan struct{} // the call waits for it, when set
|
||||
ctxErr error
|
||||
}
|
||||
|
||||
func (s *fakeSource) SimilarArtists(ctx context.Context, k string, a lastfm.Artist, limit int) ([]lastfm.Similar, error) {
|
||||
s.mu.Lock()
|
||||
s.calls++
|
||||
s.key, s.artist, s.limit = k, a, limit
|
||||
first := s.calls == 1
|
||||
started, release := s.started, s.release
|
||||
s.mu.Unlock()
|
||||
if started != nil && first {
|
||||
close(started)
|
||||
}
|
||||
if release != nil {
|
||||
<-release
|
||||
}
|
||||
s.mu.Lock()
|
||||
s.ctxErr = ctx.Err()
|
||||
s.mu.Unlock()
|
||||
return s.result, s.err
|
||||
}
|
||||
|
||||
func (s *fakeSource) count() int {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
return s.calls
|
||||
}
|
||||
|
||||
type fakeSettings struct {
|
||||
active bool
|
||||
err error
|
||||
}
|
||||
|
||||
func (f fakeSettings) Active(_ context.Context, name string) (providers.Settings, bool, error) {
|
||||
return providers.Settings{Provider: name, Enabled: f.active, APIKey: key}, f.active, f.err
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
svc *Service
|
||||
store *fakeStore
|
||||
src *fakeSource
|
||||
now time.Time
|
||||
}
|
||||
|
||||
func newRig(active bool) *rig {
|
||||
r := &rig{store: newStore(), src: &fakeSource{}, now: t0}
|
||||
r.src.result = []lastfm.Similar{{Name: "Alpha", Match: 1}, {Name: "Beta", MBID: "bbbbbbbb-0000-0000-0000-000000000002", Match: 0.5}}
|
||||
r.svc = NewService(r.store, r.src, fakeSettings{active: active}, zap.NewNop())
|
||||
r.svc.now = func() time.Time { return r.now }
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *rig) similar(t *testing.T, limit int) Result {
|
||||
t.Helper()
|
||||
res, err := r.svc.Similar(context.Background(), 1, limit)
|
||||
if err != nil {
|
||||
t.Fatalf("Similar: %v", err)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func TestAnUnknownArtistIsNotFoundAndNothingIsAsked(t *testing.T) {
|
||||
r := newRig(true)
|
||||
if _, err := r.svc.Similar(context.Background(), 999, 10); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("err = %v, want ErrNotFound", err)
|
||||
}
|
||||
if _, err := r.svc.Similar(context.Background(), 0, 10); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("id 0: err = %v, want ErrNotFound", err)
|
||||
}
|
||||
if r.src.count() != 0 {
|
||||
t.Error("the provider was asked about an artist that does not exist")
|
||||
}
|
||||
}
|
||||
|
||||
func TestLimitIsCheckedAndDefaulted(t *testing.T) {
|
||||
r := newRig(true)
|
||||
if _, err := r.svc.Similar(context.Background(), 1, -1); !errors.Is(err, ErrInvalidQuery) {
|
||||
t.Errorf("negative limit: err = %v", err)
|
||||
}
|
||||
for _, tt := range []struct{ in, want int }{{0, DefaultLimit}, {7, 7}, {MaxLimit, MaxLimit}, {MaxLimit + 50, MaxLimit}} {
|
||||
r.similar(t, tt.in)
|
||||
if r.store.lastLim != tt.want {
|
||||
t.Errorf("limit %d reached the store as %d, want %d", tt.in, r.store.lastLim, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAProviderThatIsOffIsNeverAsked(t *testing.T) {
|
||||
r := newRig(false)
|
||||
res := r.similar(t, 10)
|
||||
if res.ProviderStatus != StatusDisabled || len(res.Items) != 0 || !res.RetrievedAt.IsZero() || res.Stale {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
if r.src.count() != 0 {
|
||||
t.Error("a disabled provider was contacted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFirstLookAsksTheProviderAndStoresTheAnswer(t *testing.T) {
|
||||
r := newRig(true)
|
||||
res := r.similar(t, 10)
|
||||
if r.src.count() != 1 {
|
||||
t.Fatalf("calls = %d, want 1", r.src.count())
|
||||
}
|
||||
if r.src.key != key || r.src.artist.Name != "Source" || r.src.artist.MBID != "aaaaaaaa-0000-0000-0000-000000000001" {
|
||||
t.Errorf("the provider was asked %q about %+v", r.src.key, r.src.artist)
|
||||
}
|
||||
// Always the same number, whatever the page asked for, so one answer serves any page size.
|
||||
if r.src.limit != FetchLimit {
|
||||
t.Errorf("fetch limit = %d, want %d", r.src.limit, FetchLimit)
|
||||
}
|
||||
if len(r.store.saves) != 1 || !r.store.saves[0].found || r.store.saves[0].provider != Provider || !r.store.saves[0].at.Equal(t0) {
|
||||
t.Fatalf("saves = %+v", r.store.saves)
|
||||
}
|
||||
if res.ProviderStatus != StatusOK || res.Stale || !res.RetrievedAt.Equal(t0) || len(res.Items) != 2 || res.Provider != Provider {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
if res.Items[0].Artist.Name != "Alpha" || res.Items[0].Rank != 1 || res.Items[1].Score != 0.5 {
|
||||
t.Errorf("items = %+v", res.Items)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArtistIsNeverSimilarToItself(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.result = []lastfm.Similar{
|
||||
{Name: " SOURCE ", Match: 1},
|
||||
{Name: "Other Spelling", MBID: "AAAAAAAA-0000-0000-0000-000000000001", Match: 0.9},
|
||||
{Name: "Real One", Match: 0.8},
|
||||
}
|
||||
r.similar(t, 10)
|
||||
got := r.store.saves[0].cands
|
||||
if len(got) != 1 || got[0].Name != "Real One" {
|
||||
t.Errorf("stored candidates = %+v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreshDataIsNotAskedAgain(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.similar(t, 10)
|
||||
r.now = t0.Add(PositiveTTL - time.Second)
|
||||
res := r.similar(t, 10)
|
||||
if r.src.count() != 1 {
|
||||
t.Errorf("calls = %d, want 1: data younger than %v is fresh", r.src.count(), PositiveTTL)
|
||||
}
|
||||
if res.Stale || res.ProviderStatus != StatusOK || !res.RetrievedAt.Equal(t0) {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
r.now = t0.Add(PositiveTTL)
|
||||
r.similar(t, 10)
|
||||
if r.src.count() != 2 {
|
||||
t.Errorf("calls = %d, want 2: data of %v is old", r.src.count(), PositiveTTL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFreshDataIsServedEvenIfTheProviderIsSwitchedOff(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.similar(t, 10)
|
||||
r.svc.settings = fakeSettings{active: false}
|
||||
res := r.similar(t, 10)
|
||||
if len(res.Items) != 2 || res.Stale {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
r.now = t0.Add(PositiveTTL + time.Hour)
|
||||
res = r.similar(t, 10)
|
||||
if !res.Stale || res.ProviderStatus != StatusDisabled || len(res.Items) != 2 || r.src.count() != 1 {
|
||||
t.Errorf("old data with the provider off: %+v calls=%d", res, r.src.count())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnUnknownArtistAtTheProviderIsRememberedForADay(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.err = lastfm.ErrNotFound
|
||||
res := r.similar(t, 10)
|
||||
if res.ProviderStatus != StatusOK || len(res.Items) != 0 || res.Stale {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
if len(r.store.saves) != 1 || r.store.saves[0].found {
|
||||
t.Fatalf("saves = %+v, want one 'not found'", r.store.saves)
|
||||
}
|
||||
r.now = t0.Add(NegativeTTL - time.Second)
|
||||
r.similar(t, 10)
|
||||
if r.src.count() != 1 {
|
||||
t.Errorf("calls = %d: a negative answer is kept for %v", r.src.count(), NegativeTTL)
|
||||
}
|
||||
r.now = t0.Add(NegativeTTL)
|
||||
r.similar(t, 10)
|
||||
if r.src.count() != 2 {
|
||||
t.Errorf("calls = %d: a negative answer expires after %v", r.src.count(), NegativeTTL)
|
||||
}
|
||||
}
|
||||
|
||||
func TestOldDataIsServedWhenTheProviderFails(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.similar(t, 10)
|
||||
r.now = t0.Add(PositiveTTL + time.Hour)
|
||||
r.src.err = lastfm.ErrUnavailable
|
||||
res := r.similar(t, 10)
|
||||
if !res.Stale || res.ProviderStatus != StatusError || len(res.Items) != 2 || !res.RetrievedAt.Equal(t0) {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
if len(r.store.saves) != 1 {
|
||||
t.Errorf("a failed call changed the stored data: %+v", r.store.saves)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailureWithNothingStoredIsAnEmptyAnswerNotAnError(t *testing.T) {
|
||||
for _, e := range []error{lastfm.ErrUnavailable, lastfm.ErrInvalidKey, lastfm.ErrRateLimited, lastfm.ErrBadResponse, errors.New("boom")} {
|
||||
r := newRig(true)
|
||||
r.src.err = e
|
||||
res, err := r.svc.Similar(context.Background(), 1, 10)
|
||||
if err != nil {
|
||||
t.Fatalf("%v: err = %v", e, err)
|
||||
}
|
||||
if res.ProviderStatus != StatusError || len(res.Items) != 0 || res.Stale || !res.RetrievedAt.IsZero() {
|
||||
t.Errorf("%v: result = %+v", e, res)
|
||||
}
|
||||
if len(r.store.saves) != 0 {
|
||||
t.Errorf("%v: a failure was stored", e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSettingsErrorsAreReal(t *testing.T) {
|
||||
r := newRig(true)
|
||||
boom := errors.New("db down")
|
||||
r.svc.settings = fakeSettings{err: boom}
|
||||
if _, err := r.svc.Similar(context.Background(), 1, 10); !errors.Is(err, boom) {
|
||||
t.Fatalf("err = %v, want the settings error", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Ten people opening the same artist at once cost one call, and one of them
|
||||
// walking away does not cancel it for the others.
|
||||
func TestConcurrentLooksShareOneCall(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.started = make(chan struct{})
|
||||
r.src.release = make(chan struct{})
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
var wg sync.WaitGroup
|
||||
var failed atomic.Int32
|
||||
results := make(chan Result, 10)
|
||||
run := func(c context.Context) {
|
||||
defer wg.Done()
|
||||
res, err := r.svc.Similar(c, 1, 10)
|
||||
if err != nil {
|
||||
failed.Add(1)
|
||||
return
|
||||
}
|
||||
results <- res
|
||||
}
|
||||
wg.Add(1)
|
||||
go run(ctx)
|
||||
<-r.src.started
|
||||
for i := 0; i < 9; i++ {
|
||||
wg.Add(1)
|
||||
go run(context.Background())
|
||||
}
|
||||
time.Sleep(50 * time.Millisecond) // let the others join the flight
|
||||
cancel()
|
||||
time.Sleep(20 * time.Millisecond)
|
||||
close(r.src.release)
|
||||
wg.Wait()
|
||||
close(results)
|
||||
|
||||
if r.src.count() != 1 {
|
||||
t.Errorf("calls = %d, want 1", r.src.count())
|
||||
}
|
||||
if r.src.ctxErr != nil {
|
||||
t.Errorf("the shared call was cancelled with its first caller: %v", r.src.ctxErr)
|
||||
}
|
||||
if failed.Load() != 1 {
|
||||
t.Errorf("failed = %d, want only the caller that gave up", failed.Load())
|
||||
}
|
||||
n := 0
|
||||
for res := range results {
|
||||
n++
|
||||
if len(res.Items) != 2 {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
}
|
||||
if n != 9 {
|
||||
t.Errorf("%d callers got an answer, want 9", n)
|
||||
}
|
||||
}
|
||||
|
||||
// "No such artist" and "an empty list" both mean there is nothing to show, and
|
||||
// both are remembered for a short time only.
|
||||
func TestAnEmptyListIsRememberedLikeAnUnknownArtist(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.result = nil
|
||||
res := r.similar(t, 10)
|
||||
if len(res.Items) != 0 || len(r.store.saves) != 1 || r.store.saves[0].found {
|
||||
t.Fatalf("result = %+v saves = %+v", res, r.store.saves)
|
||||
}
|
||||
}
|
||||
246
internal/providers/lastfm/lastfm.go
Normal file
246
internal/providers/lastfm/lastfm.go
Normal file
|
|
@ -0,0 +1,246 @@
|
|||
// Package lastfm is a small read-only client for the Last.fm API.
|
||||
//
|
||||
// It only does what Sonda needs, with one decision made for it: nothing about
|
||||
// the people using Sonda is ever sent, only the name or MusicBrainz id of the
|
||||
// artist being looked up. Calls are unsigned, so the shared secret is never
|
||||
// needed and never held.
|
||||
//
|
||||
// Last.fm reports most failures with HTTP 200 and an "error" field in the JSON,
|
||||
// so the body is always read for it. Errors never repeat the provider's own
|
||||
// message or the request URL: the API key travels in the query string and the
|
||||
// message can echo it.
|
||||
package lastfm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
)
|
||||
|
||||
// BaseURL is the Last.fm API endpoint.
|
||||
const BaseURL = "https://ws.audioscrobbler.com/2.0/"
|
||||
|
||||
// Bounds of the number of similar artists asked for. Without a limit Last.fm
|
||||
// answers with 100, and up to 250 when asked for more: always say how many.
|
||||
const (
|
||||
DefaultLimit = 50
|
||||
MaxLimit = 100
|
||||
)
|
||||
|
||||
// Errors a call can return, to be matched with errors.Is.
|
||||
var (
|
||||
ErrBadRequest = errors.New("lastfm: nothing to look up")
|
||||
ErrNotFound = errors.New("lastfm: unknown artist")
|
||||
ErrInvalidKey = errors.New("lastfm: the API key was refused")
|
||||
ErrRateLimited = errors.New("lastfm: rate limit exceeded")
|
||||
ErrUnavailable = errors.New("lastfm: service unavailable")
|
||||
ErrBadResponse = errors.New("lastfm: unreadable answer")
|
||||
ErrAPI = errors.New("lastfm: the service reported an error")
|
||||
)
|
||||
|
||||
// Getter performs a GET. *outbound.Client is the one used in production.
|
||||
type Getter interface {
|
||||
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
|
||||
}
|
||||
|
||||
// Client talks to Last.fm. It is safe for concurrent use. The API key is given
|
||||
// to each call, because an admin can change it while Sonda runs.
|
||||
type Client struct {
|
||||
http Getter
|
||||
base string
|
||||
}
|
||||
|
||||
// New builds a Client that sends its requests through g to base (BaseURL in
|
||||
// production).
|
||||
func New(g Getter, base string) *Client { return &Client{http: g, base: base} }
|
||||
|
||||
// Artist names an artist to look up. The MusicBrainz id, when known, wins over
|
||||
// the name: it cannot be corrected into someone else or confused with a namesake.
|
||||
type Artist struct {
|
||||
Name string
|
||||
MBID string
|
||||
}
|
||||
|
||||
// Similar is one similar artist. Match is what Last.fm calls "match", between
|
||||
// 0 and 1. It is relative to the list it came in (the first is always 1), so it
|
||||
// says "how high in this list", not "how alike in general".
|
||||
type Similar struct {
|
||||
Name string
|
||||
MBID string // empty when Last.fm has none
|
||||
Match float64
|
||||
}
|
||||
|
||||
// SimilarArtists returns the artists Last.fm finds similar to a, best first, at
|
||||
// most limit of them (DefaultLimit when limit <= 0, MaxLimit at most). The name
|
||||
// is never autocorrected: a misspelled local name must not turn into another
|
||||
// artist.
|
||||
func (c *Client) SimilarArtists(ctx context.Context, key string, a Artist, limit int) ([]Similar, error) {
|
||||
name, mbid := strings.TrimSpace(a.Name), strings.TrimSpace(a.MBID)
|
||||
if name == "" && mbid == "" {
|
||||
return nil, ErrBadRequest
|
||||
}
|
||||
switch {
|
||||
case limit <= 0:
|
||||
limit = DefaultLimit
|
||||
case limit > MaxLimit:
|
||||
limit = MaxLimit
|
||||
}
|
||||
q := url.Values{"method": {"artist.getsimilar"}, "autocorrect": {"0"}, "limit": {strconv.Itoa(limit)}}
|
||||
if mbid != "" {
|
||||
q.Set("mbid", mbid)
|
||||
} else {
|
||||
q.Set("artist", name)
|
||||
}
|
||||
|
||||
body, err := c.call(ctx, key, q)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var doc struct {
|
||||
Similar *struct {
|
||||
Artist json.RawMessage `json:"artist"`
|
||||
} `json:"similarartists"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || doc.Similar == nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
return decodeSimilar(doc.Similar.Artist)
|
||||
}
|
||||
|
||||
// Check asks one cheap question to find out whether the key works.
|
||||
func (c *Client) Check(ctx context.Context, key string) error {
|
||||
body, err := c.call(ctx, key, url.Values{"method": {"chart.gettopartists"}, "limit": {"1"}})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var doc struct {
|
||||
Artists *json.RawMessage `json:"artists"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || doc.Artists == nil {
|
||||
return ErrBadResponse
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Probe is Check as a providers.Checker: it answers with the errors the provider
|
||||
// settings understand, so that a test does not depend on this client.
|
||||
func (c *Client) Probe(ctx context.Context, key string) error {
|
||||
err := c.Check(ctx, key)
|
||||
switch {
|
||||
case err == nil:
|
||||
return nil
|
||||
case errors.Is(err, ErrInvalidKey):
|
||||
return providers.ErrKeyRefused
|
||||
case errors.Is(err, ErrRateLimited):
|
||||
return providers.ErrRateLimited
|
||||
case errors.Is(err, ErrUnavailable):
|
||||
return providers.ErrUnreachable
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// call sends one request and returns the body of a successful answer.
|
||||
func (c *Client) call(ctx context.Context, key string, q url.Values) ([]byte, error) {
|
||||
q.Set("api_key", key)
|
||||
q.Set("format", "json")
|
||||
resp, err := c.http.Get(ctx, c.base+"?"+q.Encode())
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
var limited *outbound.RateLimitedError
|
||||
if errors.As(err, &limited) {
|
||||
return nil, ErrRateLimited
|
||||
}
|
||||
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err) // the outbound client already strips the query
|
||||
}
|
||||
|
||||
// An "error" field means failure whatever the status was.
|
||||
var probe struct {
|
||||
Error int `json:"error"`
|
||||
}
|
||||
if json.Unmarshal(resp.Body, &probe) == nil && probe.Error != 0 {
|
||||
return nil, apiError(probe.Error)
|
||||
}
|
||||
switch {
|
||||
case resp.Status == 429:
|
||||
return nil, ErrRateLimited
|
||||
case resp.Status >= 500:
|
||||
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
|
||||
case resp.Status != 200:
|
||||
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
|
||||
}
|
||||
return resp.Body, nil
|
||||
}
|
||||
|
||||
// apiError maps a Last.fm error code. Only the code is kept, never the message.
|
||||
func apiError(code int) error {
|
||||
switch code {
|
||||
case 6:
|
||||
return ErrNotFound
|
||||
case 10, 26:
|
||||
return ErrInvalidKey
|
||||
case 29:
|
||||
return ErrRateLimited
|
||||
case 8, 11, 16:
|
||||
return ErrUnavailable
|
||||
}
|
||||
return fmt.Errorf("%w (code %d)", ErrAPI, code)
|
||||
}
|
||||
|
||||
// decodeSimilar reads the list of artists. Last.fm sends an array, and has been
|
||||
// seen to send a single object when there is one result. An artist without a
|
||||
// name or with a match outside 0..1 is dropped, and each artist is kept once.
|
||||
func decodeSimilar(raw json.RawMessage) ([]Similar, error) {
|
||||
type item struct {
|
||||
Name string `json:"name"`
|
||||
MBID string `json:"mbid"`
|
||||
Match json.RawMessage `json:"match"`
|
||||
}
|
||||
var items []item
|
||||
switch {
|
||||
case len(raw) == 0 || string(raw) == "null":
|
||||
case raw[0] == '[':
|
||||
if err := json.Unmarshal(raw, &items); err != nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
default:
|
||||
var one item
|
||||
if err := json.Unmarshal(raw, &one); err != nil {
|
||||
return nil, ErrBadResponse
|
||||
}
|
||||
items = []item{one}
|
||||
}
|
||||
|
||||
out := make([]Similar, 0, len(items))
|
||||
seen := map[string]bool{}
|
||||
for _, it := range items {
|
||||
name := strings.TrimSpace(it.Name)
|
||||
match, ok := parseMatch(it.Match)
|
||||
key := strings.ToLower(name)
|
||||
if name == "" || !ok || seen[key] {
|
||||
continue
|
||||
}
|
||||
seen[key] = true
|
||||
out = append(out, Similar{Name: name, MBID: strings.TrimSpace(it.MBID), Match: match})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// parseMatch reads a match sent as a string ("0.75", what Last.fm does) or as a
|
||||
// number, and accepts only values between 0 and 1.
|
||||
func parseMatch(raw json.RawMessage) (float64, bool) {
|
||||
s := strings.Trim(string(raw), `"`)
|
||||
m, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || !(m >= 0 && m <= 1) {
|
||||
return 0, false
|
||||
}
|
||||
return m, true
|
||||
}
|
||||
284
internal/providers/lastfm/lastfm_test.go
Normal file
284
internal/providers/lastfm/lastfm_test.go
Normal file
|
|
@ -0,0 +1,284 @@
|
|||
package lastfm
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
)
|
||||
|
||||
const testKey = "0123456789abcdef0123456789abcdef"
|
||||
|
||||
// server answers every request with the given status and body and remembers
|
||||
// the queries it saw.
|
||||
type server struct {
|
||||
*httptest.Server
|
||||
mu sync.Mutex
|
||||
queries []url.Values
|
||||
}
|
||||
|
||||
func newServer(t *testing.T, status int, body string) *server {
|
||||
t.Helper()
|
||||
s := &server{}
|
||||
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
s.mu.Lock()
|
||||
s.queries = append(s.queries, r.URL.Query())
|
||||
s.mu.Unlock()
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(status)
|
||||
_, _ = w.Write([]byte(body))
|
||||
}))
|
||||
t.Cleanup(s.Close)
|
||||
return s
|
||||
}
|
||||
|
||||
func (s *server) client() *Client {
|
||||
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( test )", Retries: outbound.NoRetries}), s.URL)
|
||||
}
|
||||
|
||||
func (s *server) last(t *testing.T) url.Values {
|
||||
t.Helper()
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
if len(s.queries) == 0 {
|
||||
t.Fatal("the server saw no request")
|
||||
}
|
||||
return s.queries[len(s.queries)-1]
|
||||
}
|
||||
|
||||
const similarBody = `{"similarartists":{"artist":[
|
||||
{"name":"Alpha Band","mbid":"11111111-1111-1111-1111-111111111111","match":"1","url":"https://example.invalid/a","image":[{"#text":"https://example.invalid/i.png","size":"small"}],"streamable":"0"},
|
||||
{"name":"Beta Band","mbid":"","match":"0.755421","url":"https://example.invalid/b","streamable":"0"}
|
||||
],"@attr":{"artist":"Source"}}}`
|
||||
|
||||
func TestSimilarArtistsDecodesTheAnswer(t *testing.T) {
|
||||
s := newServer(t, 200, similarBody)
|
||||
got, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Source"}, 30)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("got %d artists, want 2: %+v", len(got), got)
|
||||
}
|
||||
if got[0].Name != "Alpha Band" || got[0].MBID != "11111111-1111-1111-1111-111111111111" || got[0].Match != 1 {
|
||||
t.Errorf("first = %+v", got[0])
|
||||
}
|
||||
if got[1].Name != "Beta Band" || got[1].MBID != "" || got[1].Match != 0.755421 {
|
||||
t.Errorf("second = %+v", got[1])
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarArtistsAsksOnlyWhatItNeeds(t *testing.T) {
|
||||
s := newServer(t, 200, similarBody)
|
||||
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Sigur Rós"}, 30); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
q := s.last(t)
|
||||
want := map[string]string{
|
||||
"method": "artist.getsimilar", "artist": "Sigur Rós", "autocorrect": "0",
|
||||
"limit": "30", "api_key": testKey, "format": "json",
|
||||
}
|
||||
for k, v := range want {
|
||||
if q.Get(k) != v {
|
||||
t.Errorf("%s = %q, want %q", k, q.Get(k), v)
|
||||
}
|
||||
}
|
||||
if len(q) != len(want) {
|
||||
t.Errorf("query has other parameters than %v: %v", want, q)
|
||||
}
|
||||
}
|
||||
|
||||
// The id names the artist without doubt, so it wins over the name; the name
|
||||
// would be corrected or confused, the id cannot.
|
||||
func TestSimilarArtistsPrefersTheMBID(t *testing.T) {
|
||||
s := newServer(t, 200, similarBody)
|
||||
mbid := "22222222-2222-2222-2222-222222222222"
|
||||
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "Source", MBID: mbid}, 10); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
q := s.last(t)
|
||||
if q.Get("mbid") != mbid || q.Has("artist") || q.Get("autocorrect") != "0" {
|
||||
t.Errorf("query = %v", q)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarArtistsBoundsTheLimit(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
in int
|
||||
want string
|
||||
}{{0, "50"}, {-3, "50"}, {7, "7"}, {100, "100"}, {5000, "100"}} {
|
||||
s := newServer(t, 200, similarBody)
|
||||
if _, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, tt.in); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := s.last(t).Get("limit"); got != tt.want {
|
||||
t.Errorf("limit %d sent as %q, want %q", tt.in, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarArtistsRefusesAnEmptyArtistWithoutAsking(t *testing.T) {
|
||||
s := newServer(t, 200, similarBody)
|
||||
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: " "}, 10)
|
||||
if !errors.Is(err, ErrBadRequest) {
|
||||
t.Fatalf("err = %v, want ErrBadRequest", err)
|
||||
}
|
||||
if len(s.queries) != 0 {
|
||||
t.Error("a request left for an empty artist")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSimilarArtistsToleratesTheShapesLastFMUses(t *testing.T) {
|
||||
tests := []struct {
|
||||
name, body string
|
||||
want int
|
||||
}{
|
||||
{"one artist as an object", `{"similarartists":{"artist":{"name":"Solo","mbid":"","match":"1"}}}`, 1},
|
||||
{"no artists", `{"similarartists":{"artist":[],"@attr":{"artist":"x"}}}`, 0},
|
||||
{"match as a number", `{"similarartists":{"artist":[{"name":"N","match":0.5}]}}`, 1},
|
||||
{"an artist without a name is dropped", `{"similarartists":{"artist":[{"name":"","match":"1"},{"name":"Ok","match":"0.5"}]}}`, 1},
|
||||
{"an unreadable match is dropped", `{"similarartists":{"artist":[{"name":"A","match":"high"},{"name":"B","match":"0.5"}]}}`, 1},
|
||||
{"a match outside 0..1 is dropped", `{"similarartists":{"artist":[{"name":"A","match":"7"},{"name":"B","match":"-1"},{"name":"C","match":"0"}]}}`, 1},
|
||||
{"the same artist twice is kept once", `{"similarartists":{"artist":[{"name":"Dup","match":"1"},{"name":" dup ","match":"0.5"}]}}`, 1},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newServer(t, 200, tt.body)
|
||||
got, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(got) != tt.want {
|
||||
t.Errorf("got %d artists %+v, want %d", len(got), got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Last.fm reports most failures with status 200 and an "error" field.
|
||||
func TestErrorsAreReadEvenWithStatus200(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
want error
|
||||
}{
|
||||
{"unknown artist", 200, `{"error":6,"message":"The artist you supplied could not be found","links":[]}`, ErrNotFound},
|
||||
{"invalid key", 403, `{"message":"Invalid API key - You must be granted a valid key by last.fm","error":10}`, ErrInvalidKey},
|
||||
{"invalid key with 200", 200, `{"error":10,"message":"Invalid API key"}`, ErrInvalidKey},
|
||||
{"suspended key", 403, `{"error":26,"message":"Suspended API key"}`, ErrInvalidKey},
|
||||
{"rate limit", 200, `{"error":29,"message":"Rate limit exceeded"}`, ErrRateLimited},
|
||||
{"rate limit status", 429, `{"error":29,"message":"Rate limit exceeded"}`, ErrRateLimited},
|
||||
{"temporary error", 200, `{"error":11,"message":"Service Offline"}`, ErrUnavailable},
|
||||
{"temporarily unavailable", 200, `{"error":16,"message":"temporarily unavailable"}`, ErrUnavailable},
|
||||
{"operation failed", 200, `{"error":8,"message":"Operation failed"}`, ErrUnavailable},
|
||||
{"server error", 500, `oops`, ErrUnavailable},
|
||||
{"bad gateway", 502, `<html>`, ErrUnavailable},
|
||||
{"not json", 200, `<html>nope</html>`, ErrBadResponse},
|
||||
{"empty answer", 200, ``, ErrBadResponse},
|
||||
{"json without the expected part", 200, `{"hello":"world"}`, ErrBadResponse},
|
||||
{"another error code", 200, `{"error":99,"message":"Something new"}`, ErrAPI},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newServer(t, tt.status, tt.body)
|
||||
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
|
||||
if !errors.Is(err, tt.want) {
|
||||
t.Fatalf("err = %v, want %v", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// The key travels in the query string: no error may repeat it, and neither the
|
||||
// URL nor the provider's own words (which can echo it) are copied into one.
|
||||
func TestErrorsNeverCarryTheKeyOrTheProvidersText(t *testing.T) {
|
||||
bodies := []struct {
|
||||
status int
|
||||
body string
|
||||
}{
|
||||
{403, `{"error":10,"message":"Invalid API key ` + testKey + `"}`},
|
||||
{200, `{"error":99,"message":"weird ` + testKey + `"}`},
|
||||
{500, "boom " + testKey},
|
||||
{200, "not json " + testKey},
|
||||
}
|
||||
for _, b := range bodies {
|
||||
s := newServer(t, b.status, b.body)
|
||||
_, err := s.client().SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
|
||||
if err == nil {
|
||||
t.Fatal("no error")
|
||||
}
|
||||
if strings.Contains(err.Error(), testKey) {
|
||||
t.Errorf("the key is in the error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A refused connection puts the URL in the error text: the outbound client
|
||||
// strips the query, and this must stay so.
|
||||
dead := httptest.NewServer(http.NotFoundHandler())
|
||||
base := dead.URL
|
||||
dead.Close()
|
||||
c := New(outbound.New(outbound.Options{Retries: outbound.NoRetries}), base)
|
||||
_, err := c.SimilarArtists(context.Background(), testKey, Artist{Name: "x"}, 10)
|
||||
if err == nil || strings.Contains(err.Error(), testKey) {
|
||||
t.Errorf("err = %v", err)
|
||||
}
|
||||
if !errors.Is(err, ErrUnavailable) {
|
||||
t.Errorf("a refused connection should read as unavailable: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCheckAsksForOneCheapThing(t *testing.T) {
|
||||
s := newServer(t, 200, `{"artists":{"artist":[{"name":"Someone"}]}}`)
|
||||
if err := s.client().Check(context.Background(), testKey); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
q := s.last(t)
|
||||
if q.Get("method") != "chart.gettopartists" || q.Get("limit") != "1" || q.Get("api_key") != testKey {
|
||||
t.Errorf("query = %v", q)
|
||||
}
|
||||
|
||||
bad := newServer(t, 403, `{"error":10,"message":"Invalid API key"}`)
|
||||
if err := bad.client().Check(context.Background(), testKey); !errors.Is(err, ErrInvalidKey) {
|
||||
t.Errorf("err = %v, want ErrInvalidKey", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Probe is Check in the words the provider settings use, so that the answer of a
|
||||
// test does not depend on this client.
|
||||
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
status int
|
||||
body string
|
||||
want error
|
||||
}{
|
||||
{"works", 200, `{"artists":{"artist":[]}}`, nil},
|
||||
{"key refused", 403, `{"error":10,"message":"Invalid API key"}`, providers.ErrKeyRefused},
|
||||
{"suspended key", 200, `{"error":26,"message":"Suspended"}`, providers.ErrKeyRefused},
|
||||
{"rate limited", 200, `{"error":29,"message":"Rate limit exceeded"}`, providers.ErrRateLimited},
|
||||
{"down", 503, `nope`, providers.ErrUnreachable},
|
||||
{"temporary", 200, `{"error":11,"message":"offline"}`, providers.ErrUnreachable},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
s := newServer(t, tt.status, tt.body)
|
||||
err := s.client().Probe(context.Background(), testKey)
|
||||
if tt.want == nil && err != nil || tt.want != nil && !errors.Is(err, tt.want) {
|
||||
t.Fatalf("err = %v, want %v", err, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
// Anything else is not turned into one of them.
|
||||
s := newServer(t, 200, `<html>`)
|
||||
err := s.client().Probe(context.Background(), testKey)
|
||||
if err == nil || errors.Is(err, providers.ErrKeyRefused) || errors.Is(err, providers.ErrUnreachable) || errors.Is(err, providers.ErrRateLimited) {
|
||||
t.Errorf("an unreadable answer: err = %v", err)
|
||||
}
|
||||
}
|
||||
|
|
@ -27,6 +27,15 @@ const (
|
|||
var (
|
||||
ErrUnknownProvider = errors.New("unknown provider")
|
||||
ErrInvalidSettings = errors.New("invalid provider settings")
|
||||
ErrNoTest = errors.New("this provider has no test")
|
||||
)
|
||||
|
||||
// What a checker says went wrong, so that the result of a test does not depend
|
||||
// on the client of any one provider.
|
||||
var (
|
||||
ErrKeyRefused = errors.New("the provider refused the API key")
|
||||
ErrRateLimited = errors.New("the provider's rate limit was reached")
|
||||
ErrUnreachable = errors.New("the provider could not be reached")
|
||||
)
|
||||
|
||||
const maxContactLen = 200
|
||||
|
|
@ -108,9 +117,10 @@ func lookup(name string) (meta, bool) {
|
|||
|
||||
// Service reads and changes provider settings.
|
||||
type Service struct {
|
||||
store Store
|
||||
now func() time.Time
|
||||
mu sync.Mutex // an update reads then writes: keep two of them apart
|
||||
store Store
|
||||
now func() time.Time
|
||||
mu sync.Mutex // an update reads then writes: keep two of them apart
|
||||
checkers map[string]Checker
|
||||
}
|
||||
|
||||
// NewService builds a Service.
|
||||
|
|
@ -264,3 +274,66 @@ func UserAgent(version, contact string) string {
|
|||
}
|
||||
return fmt.Sprintf("Sonda/%s ( %s )", v, c)
|
||||
}
|
||||
|
||||
// Checker asks a provider one cheap question with a key, to find out whether the
|
||||
// key works. It returns nil, or one of ErrKeyRefused, ErrRateLimited and
|
||||
// ErrUnreachable (any other error counts as an unknown failure).
|
||||
type Checker func(ctx context.Context, key string) error
|
||||
|
||||
// SetChecker registers the test of a provider. Call it before the service is used.
|
||||
func (s *Service) SetChecker(name string, c Checker) {
|
||||
if s.checkers == nil {
|
||||
s.checkers = map[string]Checker{}
|
||||
}
|
||||
s.checkers[name] = c
|
||||
}
|
||||
|
||||
// Results of a test.
|
||||
const (
|
||||
TestOK = "ok"
|
||||
TestNoKey = "no_api_key"
|
||||
TestKeyRefused = "key_refused"
|
||||
TestRateLimited = "rate_limited"
|
||||
TestUnreachable = "unreachable"
|
||||
TestFailed = "error"
|
||||
)
|
||||
|
||||
// TestResult is the outcome of Test. Status is one of the Test constants.
|
||||
type TestResult struct {
|
||||
OK bool
|
||||
Status string
|
||||
}
|
||||
|
||||
// Test checks a provider with the stored key. It works whether the provider is
|
||||
// on or off, because an admin wants to know before switching it on; it is the one
|
||||
// request Sonda makes on demand of an admin and not of a listener, and it sends
|
||||
// nothing but the key. The key is never part of the result.
|
||||
func (s *Service) Test(ctx context.Context, name string) (TestResult, error) {
|
||||
m, ok := lookup(name)
|
||||
if !ok {
|
||||
return TestResult{}, ErrUnknownProvider
|
||||
}
|
||||
check := s.checkers[name]
|
||||
if check == nil {
|
||||
return TestResult{}, ErrNoTest
|
||||
}
|
||||
cur, _, err := s.store.Get(ctx, name)
|
||||
if err != nil {
|
||||
return TestResult{}, fmt.Errorf("read provider settings: %w", err)
|
||||
}
|
||||
if m.needsKey && cur.APIKey == "" {
|
||||
return TestResult{Status: TestNoKey}, nil
|
||||
}
|
||||
switch err := check(ctx, cur.APIKey); {
|
||||
case err == nil:
|
||||
return TestResult{OK: true, Status: TestOK}, nil
|
||||
case errors.Is(err, ErrKeyRefused):
|
||||
return TestResult{Status: TestKeyRefused}, nil
|
||||
case errors.Is(err, ErrRateLimited):
|
||||
return TestResult{Status: TestRateLimited}, nil
|
||||
case errors.Is(err, ErrUnreachable):
|
||||
return TestResult{Status: TestUnreachable}, nil
|
||||
default:
|
||||
return TestResult{Status: TestFailed}, nil
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -275,3 +275,74 @@ func TestAnEmptyKeyDoesNotTurnOffAProviderThatNeedsNone(t *testing.T) {
|
|||
t.Error("musicbrainz stopped being active")
|
||||
}
|
||||
}
|
||||
|
||||
// ---- testing a provider ----
|
||||
|
||||
func TestAProviderCanBeTestedBeforeItIsSwitchedOn(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
ctx := context.Background()
|
||||
var gotKey string
|
||||
s.SetChecker(LastFM, func(_ context.Context, key string) error { gotKey = key; return nil })
|
||||
|
||||
// No key yet: say so, and ask nobody.
|
||||
res, err := s.Test(ctx, LastFM)
|
||||
if err != nil || res.OK || res.Status != TestNoKey || gotKey != "" {
|
||||
t.Fatalf("without a key: %+v err=%v key=%q", res, err, gotKey)
|
||||
}
|
||||
|
||||
if _, err := s.Update(ctx, LastFM, Patch{APIKey: strp(goodKey)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
res, err = s.Test(ctx, LastFM) // still off: an explicit test is allowed
|
||||
if err != nil || !res.OK || res.Status != TestOK || gotKey != goodKey {
|
||||
t.Fatalf("with a key: %+v err=%v key=%q", res, err, gotKey)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestMapsWhatWentWrong(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
err error
|
||||
want string
|
||||
}{
|
||||
{ErrKeyRefused, TestKeyRefused},
|
||||
{ErrRateLimited, TestRateLimited},
|
||||
{ErrUnreachable, TestUnreachable},
|
||||
{errors.New("something else"), TestFailed},
|
||||
} {
|
||||
s, _ := newSvc()
|
||||
_, _ = s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)})
|
||||
s.SetChecker(LastFM, func(context.Context, string) error { return tt.err })
|
||||
res, err := s.Test(context.Background(), LastFM)
|
||||
if err != nil || res.OK || res.Status != tt.want {
|
||||
t.Errorf("%v: %+v err=%v, want status %s", tt.err, res, err, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTestErrors(t *testing.T) {
|
||||
s, st := newSvc()
|
||||
ctx := context.Background()
|
||||
if _, err := s.Test(ctx, "spotify"); !errors.Is(err, ErrUnknownProvider) {
|
||||
t.Errorf("unknown provider: %v", err)
|
||||
}
|
||||
if _, err := s.Test(ctx, MusicBrainz); !errors.Is(err, ErrNoTest) {
|
||||
t.Errorf("a provider without a test: %v", err)
|
||||
}
|
||||
s.SetChecker(LastFM, func(context.Context, string) error { return nil })
|
||||
boom := errors.New("db down")
|
||||
st.err = boom
|
||||
if _, err := s.Test(ctx, LastFM); !errors.Is(err, boom) {
|
||||
t.Errorf("a storage error: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// A checker must be handed the stored key and nothing in the answer may carry it.
|
||||
func TestTestResultNeverCarriesTheKey(t *testing.T) {
|
||||
s, _ := newSvc()
|
||||
_, _ = s.Update(context.Background(), LastFM, Patch{APIKey: strp(goodKey)})
|
||||
s.SetChecker(LastFM, func(context.Context, string) error { return errors.New("failed with " + goodKey) })
|
||||
res, _ := s.Test(context.Background(), LastFM)
|
||||
if strings.Contains(res.Status, goodKey) {
|
||||
t.Errorf("the result repeats the key: %+v", res)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package storage
|
|||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
|
|
@ -111,10 +112,21 @@ func (r *CatalogRepo) GetArtist(ctx context.Context, id int64) (catalog.Artist,
|
|||
if err != nil {
|
||||
return catalog.Artist{}, err
|
||||
}
|
||||
if len(p.Items) == 0 {
|
||||
if len(p.Items) > 0 {
|
||||
return p.Items[0], nil
|
||||
}
|
||||
// Not in the library, but maybe named by a discovery provider: such an artist
|
||||
// can be shown, with nothing to play. The lists never include it.
|
||||
a := catalog.Artist{}
|
||||
err = r.db.QueryRowContext(ctx,
|
||||
"SELECT a.id, a.name FROM artists a WHERE a.id = ? AND "+discoveredArtist("a.id"), id).Scan(&a.ID, &a.Name)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return catalog.Artist{}, catalog.ErrNotFound
|
||||
}
|
||||
return p.Items[0], nil
|
||||
if err != nil {
|
||||
return catalog.Artist{}, fmt.Errorf("find discovered artist: %w", err)
|
||||
}
|
||||
return a, nil
|
||||
}
|
||||
|
||||
func (r *CatalogRepo) listArtists(ctx context.Context, f catalog.Filter, id int64) (catalog.Page[catalog.Artist], error) {
|
||||
|
|
|
|||
249
internal/storage/discovery.go
Normal file
249
internal/storage/discovery.go
Normal file
|
|
@ -0,0 +1,249 @@
|
|||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/uptrace/bun"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/providers/lastfm"
|
||||
)
|
||||
|
||||
// kindSimilar is what provider_fetches records for a "similar artists" lookup.
|
||||
const kindSimilar = "similar"
|
||||
|
||||
// Answers recorded in provider_fetches.
|
||||
const (
|
||||
fetchOK = "ok"
|
||||
fetchNotFound = "not_found"
|
||||
)
|
||||
|
||||
// DiscoveryRepo keeps what the discovery providers said. It implements
|
||||
// discovery.Store.
|
||||
type DiscoveryRepo struct{ db *bun.DB }
|
||||
|
||||
// NewDiscoveryRepo builds a DiscoveryRepo.
|
||||
func NewDiscoveryRepo(db *bun.DB) *DiscoveryRepo { return &DiscoveryRepo{db: db} }
|
||||
|
||||
// playableArtist is the condition "this artist can be played": credited on a
|
||||
// track with a playable file, or the album artist of an album with one
|
||||
// (compilations). expr is the artist id in the surrounding query.
|
||||
func playableArtist(expr string) string {
|
||||
vis, _ := visible("t", 0)
|
||||
return `(EXISTS (SELECT 1 FROM track_artists ta JOIN tracks t ON t.id = ta.track_id WHERE ta.artist_id = ` + expr + ` AND ` + vis + `)
|
||||
OR EXISTS (SELECT 1 FROM albums al JOIN tracks t ON t.album_id = al.id WHERE al.artist_id = ` + expr + ` AND ` + vis + `))`
|
||||
}
|
||||
|
||||
// discoveredArtist is the condition "a provider named this artist as similar to
|
||||
// another". Such an artist can have no files at all.
|
||||
func discoveredArtist(expr string) string {
|
||||
return "EXISTS (SELECT 1 FROM artist_similar s WHERE s.similar_id = " + expr + ")"
|
||||
}
|
||||
|
||||
// Artist returns an artist that can be looked up: one with something playable,
|
||||
// or one a provider already named. Anything else is discovery.ErrNotFound.
|
||||
func (r *DiscoveryRepo) Artist(ctx context.Context, id int64) (discovery.Artist, error) {
|
||||
var (
|
||||
a discovery.Artist
|
||||
mbid sql.NullString
|
||||
)
|
||||
err := r.db.QueryRowContext(ctx,
|
||||
"SELECT a.id, a.name, a.mbid FROM artists a WHERE a.id = ? AND ("+playableArtist("a.id")+" OR "+discoveredArtist("a.id")+")", id).
|
||||
Scan(&a.ID, &a.Name, &mbid)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return discovery.Artist{}, discovery.ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return discovery.Artist{}, fmt.Errorf("find artist: %w", err)
|
||||
}
|
||||
a.MBID = mbid.String
|
||||
return a, nil
|
||||
}
|
||||
|
||||
// Similar reads what was stored about an artist, best first, at most limit items.
|
||||
func (r *DiscoveryRepo) Similar(ctx context.Context, id int64, provider string, limit int) (discovery.Snapshot, error) {
|
||||
var (
|
||||
snap discovery.Snapshot
|
||||
status string
|
||||
when int64
|
||||
)
|
||||
err := r.db.QueryRowContext(ctx,
|
||||
"SELECT status, retrieved_at FROM provider_fetches WHERE provider = ? AND kind = ? AND artist_id = ?",
|
||||
provider, kindSimilar, id).Scan(&status, &when)
|
||||
switch {
|
||||
case errors.Is(err, sql.ErrNoRows):
|
||||
return snap, nil
|
||||
case err != nil:
|
||||
return snap, fmt.Errorf("read fetch ledger: %w", err)
|
||||
}
|
||||
snap.Fetched, snap.NotFound, snap.RetrievedAt = true, status == fetchNotFound, time.Unix(when, 0)
|
||||
|
||||
rows, err := r.db.QueryContext(ctx,
|
||||
`SELECT a.id, a.name, s.rank, s.score, `+playableArtist("a.id")+`
|
||||
FROM artist_similar s JOIN artists a ON a.id = s.similar_id
|
||||
WHERE s.artist_id = ? AND s.provider = ? ORDER BY s.rank, a.id LIMIT ?`,
|
||||
id, provider, limit)
|
||||
if err != nil {
|
||||
return snap, fmt.Errorf("read similar artists: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
snap.Items = []discovery.Item{}
|
||||
for rows.Next() {
|
||||
var it discovery.Item
|
||||
if err := rows.Scan(&it.Artist.ID, &it.Artist.Name, &it.Rank, &it.Score, &it.InLibrary); err != nil {
|
||||
return snap, err
|
||||
}
|
||||
snap.Items = append(snap.Items, it)
|
||||
}
|
||||
return snap, rows.Err()
|
||||
}
|
||||
|
||||
// SaveSimilar replaces what is stored about an artist and records the answer.
|
||||
//
|
||||
// Each candidate is matched to an artist that is already known, or becomes a new
|
||||
// artist without files. Two artists are the same when their MusicBrainz ids are
|
||||
// equal, or when their names match and at least one side has no id: an id that
|
||||
// is missing never says "different", and one that differs always does. An id
|
||||
// from a provider is never written onto an existing artist, because nobody
|
||||
// checked it. Artists created here are kept when the list is replaced later.
|
||||
func (r *DiscoveryRepo) SaveSimilar(ctx context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
|
||||
return r.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
|
||||
ids, err := resolveArtists(ctx, tx, id, cands)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx, "DELETE FROM artist_similar WHERE artist_id = ? AND provider = ?", id, provider); err != nil {
|
||||
return fmt.Errorf("clear similar artists: %w", err)
|
||||
}
|
||||
rank := 0
|
||||
for i, c := range cands {
|
||||
if ids[i] == 0 {
|
||||
continue // the artist itself, or already listed above
|
||||
}
|
||||
rank++
|
||||
if _, err := tx.ExecContext(ctx,
|
||||
"INSERT INTO artist_similar (artist_id, provider, similar_id, rank, score) VALUES (?, ?, ?, ?, ?)",
|
||||
id, provider, ids[i], rank, c.Match); err != nil {
|
||||
return fmt.Errorf("insert similar artist: %w", err)
|
||||
}
|
||||
}
|
||||
status := fetchOK
|
||||
if !found {
|
||||
status = fetchNotFound
|
||||
}
|
||||
if _, err := tx.ExecContext(ctx,
|
||||
`INSERT INTO provider_fetches (provider, kind, artist_id, status, retrieved_at) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT (provider, kind, artist_id) DO UPDATE SET status = excluded.status, retrieved_at = excluded.retrieved_at`,
|
||||
provider, kindSimilar, id, status, at.Unix()); err != nil {
|
||||
return fmt.Errorf("record fetch: %w", err)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
// resolveArtists returns the artist id of each candidate, creating the ones that
|
||||
// are not known. A candidate that is the artist itself, or an artist already
|
||||
// seen earlier in the list, gets 0.
|
||||
func resolveArtists(ctx context.Context, tx bun.Tx, self int64, cands []lastfm.Similar) ([]int64, error) {
|
||||
type known struct {
|
||||
id int64
|
||||
mbid string
|
||||
}
|
||||
var mbids, keys []string
|
||||
for _, c := range cands {
|
||||
if m := cleanMBID(c.MBID); m != "" {
|
||||
mbids = append(mbids, m)
|
||||
}
|
||||
keys = append(keys, matchKey(c.Name))
|
||||
}
|
||||
byMBID := map[string]int64{}
|
||||
byKey := map[string][]known{}
|
||||
if len(cands) > 0 {
|
||||
// Two conditions with an empty list would match everything or nothing,
|
||||
// depending on the database: always send at least one value.
|
||||
if len(mbids) == 0 {
|
||||
mbids = []string{""}
|
||||
}
|
||||
rows, err := tx.QueryContext(ctx,
|
||||
"SELECT id, mbid, name_key FROM artists WHERE mbid IN (?) OR name_key IN (?) ORDER BY id",
|
||||
bun.In(mbids), bun.In(keys))
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("find artists: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var (
|
||||
id int64
|
||||
mbid sql.NullString
|
||||
nameKey string
|
||||
)
|
||||
if err := rows.Scan(&id, &mbid, &nameKey); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if mbid.String != "" {
|
||||
byMBID[strings.ToLower(mbid.String)] = id
|
||||
}
|
||||
byKey[nameKey] = append(byKey[nameKey], known{id, strings.ToLower(mbid.String)})
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
|
||||
out := make([]int64, len(cands))
|
||||
used := map[int64]bool{self: true}
|
||||
for i, c := range cands {
|
||||
mbid, key := cleanMBID(c.MBID), matchKey(c.Name)
|
||||
var id int64
|
||||
if mbid != "" {
|
||||
id = byMBID[mbid]
|
||||
}
|
||||
if id == 0 {
|
||||
for _, k := range byKey[key] {
|
||||
if mbid == "" || k.mbid == "" {
|
||||
id = k.id
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
if id == 0 {
|
||||
var err error
|
||||
if id, err = insertDiscovered(ctx, tx, c.Name, mbid); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if mbid != "" {
|
||||
byMBID[mbid] = id
|
||||
}
|
||||
byKey[key] = append(byKey[key], known{id, mbid})
|
||||
}
|
||||
if used[id] {
|
||||
continue
|
||||
}
|
||||
used[id] = true
|
||||
out[i] = id
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// insertDiscovered creates an artist that exists only because a provider named it.
|
||||
func insertDiscovered(ctx context.Context, tx bun.Tx, name, mbid string) (int64, error) {
|
||||
var m sql.NullString
|
||||
if mbid != "" {
|
||||
m = sql.NullString{String: mbid, Valid: true}
|
||||
}
|
||||
var id int64
|
||||
err := tx.QueryRowContext(ctx,
|
||||
"INSERT INTO artists (mbid, name, name_key) VALUES (?, ?, ?) RETURNING id", m, name, matchKey(name)).Scan(&id)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("insert artist: %w", err)
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
|
||||
// cleanMBID lower-cases an id the way it is stored.
|
||||
func cleanMBID(s string) string { return strings.ToLower(strings.TrimSpace(s)) }
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue