Merge pull request 'feat(discovery): remove provider data, genres from my files, full rescans' (#26) from feat/provider-purge-and-genres into main

This commit is contained in:
squidd 2026-11-11 17:37:09 +01:00
commit 4da79601d2
32 changed files with 1187 additions and 57 deletions

View file

@ -15,12 +15,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights. See [External providers](#external-providers).
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
## What is not there yet
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Genres from your own files, and the removal of provider data on request.
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Recommendations from all this data (scoring) are next.
## Requirements
@ -200,7 +200,11 @@ The key is stored in the database, in clear text, and is never returned by the A
**Tags.** `GET /api/v1/artists/{id}/tags` asks each provider that is on, at the same time, and keeps the answers with the same rules (7 days, 1 for an unknown artist, old data if a provider fails). Each provider has its own entry with its own weights: Last.fm's are relative to the artist (1 to 100), MusicBrainz's are counts of votes, and they are never mixed. MusicBrainz also marks the tags it lists as genres. Its tags and genres are CC BY-NC-SA, so show the `attribution` that comes with them.
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. The scan reads a file's tags when the file is new or changed, so to pick up ids for music that is already indexed, the files have to be touched or re-added.
**Genres from your own files.** The scan also reads the genre tag of each file (several genres separated by `;`, at most five per file) and `GET /artists/{id}/tags` adds a `files` entry: the genres of the playable tracks the artist is credited on, each weighted by how many tracks have it. It needs no provider, asks nobody, and is always there.
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. A normal scan reads a file's tags when the file is new or changed. To pick up ids and genres for music that is already indexed, scan with `full=true` (`POST /library/scan?full=true`, or `POST /libraries/{id}/scan?full=true`): every file is read again, keeps its identity, and your listening history is untouched.
**Taking it back.** `DELETE /api/v1/providers/{name}/data` (admin) removes everything a provider gave: its similar artists, its tags and the records of when it was asked, and then the artists that only that data named and that you have no files for. Last.fm's terms ask for this when an agreement ends. Your own artists, music and listens stay, and the provider is not switched off. An artist that no list names any more is also removed when the list that named it is replaced, so they do not pile up.
The artist is sent to Last.fm by its MusicBrainz id when your files gave one (if Last.fm does not know the id, by name), otherwise by its name exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
@ -224,8 +228,8 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
| signed in | `GET /listens` | Your own history, newest first, with `counted` per event. Query: `limit`, `offset`, `counted=true`. |
| signed in | `GET /library/scan` | Scan status and the last result. |
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test` | See, set up and check Last.fm and MusicBrainz. The API key is write-only. |
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. `?full=true` reads every file again. |
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test`, `DELETE /providers/{name}/data` | See, set up and check Last.fm and MusicBrainz, and remove what they gave. The API key is write-only. |
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.

View file

@ -128,6 +128,9 @@ func run() error {
return musicbrainzClient.Probe(ctx)
})
discoveryRepo := storage.NewDiscoveryRepo(db)
providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
return discoveryRepo.Purge(ctx, name)
})
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
providers.LastFM: discovery.LastfmTags{Client: lastfmClient},

View file

@ -15,9 +15,20 @@ type scanControl struct {
m *library.Manager
}
func (c scanControl) Start() error { return c.m.Start(c.ctx) }
func (c scanControl) StartLibrary(id int64) error { return c.m.StartLibrary(c.ctx, id) }
func (c scanControl) Status() library.Status { return c.m.Status() }
func (c scanControl) Start(full bool) error { return c.m.Start(c.ctx, scanOptions(full)...) }
func (c scanControl) StartLibrary(id int64, full bool) error {
return c.m.StartLibrary(c.ctx, id, scanOptions(full)...)
}
func (c scanControl) Status() library.Status { return c.m.Status() }
func scanOptions(full bool) []library.ScanOption {
if full {
return []library.ScanOption{library.FullRescan}
}
return nil
}
// logScanResults writes one log line per finished scan. File paths are left
// out on purpose: audio paths are private.

View file

@ -75,7 +75,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
}
tokens := auth.NewTokenIssuer(secret, time.Hour)
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
deps := Deps{
Logger: zap.NewNop(),
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, 24*time.Hour),
@ -92,6 +92,16 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
return NewRouter(deps), db
}
// newProviders builds the provider service the way main does, with the real way
// of removing provider data.
func newProviders(db *bun.DB) *providers.Service {
svc := providers.NewService(storage.NewProviderRepo(db))
svc.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
return storage.NewDiscoveryRepo(db).Purge(ctx, name)
})
return svc
}
// noFiles is a streaming source with nothing to play.
type noFiles struct{}

View file

@ -80,9 +80,22 @@ func (r *contractReader) Read(path string) (library.Metadata, error) {
// scanAdapter lets the API start scans on a real manager.
type scanAdapter struct{ m *library.Manager }
func (a scanAdapter) Start() error { return a.m.Start(context.Background()) }
func (a scanAdapter) StartLibrary(id int64) error { return a.m.StartLibrary(context.Background(), id) }
func (a scanAdapter) Status() library.Status { return a.m.Status() }
func (a scanAdapter) Start(full bool) error {
return a.m.Start(context.Background(), scanOptions(full)...)
}
func (a scanAdapter) StartLibrary(id int64, full bool) error {
return a.m.StartLibrary(context.Background(), id, scanOptions(full)...)
}
func (a scanAdapter) Status() library.Status { return a.m.Status() }
func scanOptions(full bool) []library.ScanOption {
if full {
return []library.ScanOption{library.FullRescan}
}
return nil
}
type contract struct {
lastfm *fakeLastfm
@ -102,7 +115,7 @@ func newContract(t *testing.T) *contract {
tagsMB := &fakeTags{tags: []discovery.Tag{{Name: "grunge", Weight: 12, Genre: true}}}
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
d.Providers = prov
prov.SetChecker(providers.MusicBrainz, func(context.Context, string) error { return nil })
@ -339,6 +352,7 @@ func TestResponsesMatchTheSpec(t *testing.T) {
close(gate)
c.scanDone(token)
c.expect(c.call("POST", "/library/scan?full=maybe", token, nil), 400, "scan with a bad parameter", "invalid_input")
c.expect(c.call("POST", "/library/scan", token, nil), 202, "scan everything")
c.scanDone(token)
status := c.expect(c.call("GET", "/library/scan", token, nil), 200, "scan status")
@ -347,6 +361,12 @@ func TestResponsesMatchTheSpec(t *testing.T) {
t.Fatalf("second scan = %v, want 3 files in 2 libraries, all unchanged", last)
}
c.expect(c.call("POST", "/library/scan?full=true", token, nil), 202, "scan everything again, in full")
c.scanDone(token)
if full := c.expect(c.call("GET", "/library/scan", token, nil), 200, "status after a full scan")["last"].(map[string]any); full["updated"] != float64(3) || full["unchanged"] != float64(0) {
t.Fatalf("full scan = %v, want the 3 files read again", full)
}
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 204, "delete library")
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 404, "delete library twice", "library_not_found")
@ -423,10 +443,15 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
tags := c.expect(c.call("GET", "/artists/"+artistID+"/tags", token, nil), 200, "tags")["sources"].([]any)
if len(tags) != 2 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" {
t.Fatalf("tags = %v, want Last.fm ok and MusicBrainz without an id", tags)
if len(tags) != 3 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" || tags[2].(map[string]any)["provider"] != "files" {
t.Fatalf("tags = %v, want Last.fm ok, MusicBrainz without an id and the files", tags)
}
c.expect(c.call("GET", "/artists/999/tags", token, nil), 404, "tags of an unknown artist", "artist_not_found")
purged := c.expect(c.call("DELETE", "/providers/lastfm/data", token, nil), 200, "purge a provider's data")
if purged["tags"] != float64(1) || purged["similar_artists"] != float64(1) {
t.Fatalf("purge = %v", purged)
}
c.expect(c.call("DELETE", "/providers/spotify/data", token, nil), 404, "purge an unknown provider", "provider_not_found")
c.expect(c.call("GET", "/artists/abc/tags", token, nil), 400, "tags of a bad id", "invalid_input")
// ---- playback ----

View file

@ -35,9 +35,27 @@ type scanStatusJSON struct {
LastError string `json:"last_error,omitempty"`
}
// fullParam reads ?full=. Only "true" and "false" are accepted, and it answers
// 400 itself otherwise: a typo must not quietly start a scan of another kind.
func fullParam(c *gin.Context) (full, ok bool) {
values := c.QueryArray("full")
switch {
case len(values) == 0, len(values) == 1 && values[0] == "false":
return false, true
case len(values) == 1 && values[0] == "true":
return true, true
}
writeError(c, http.StatusBadRequest, "invalid_input", "full must be true or false")
return false, false
}
// startScan launches a background scan of the configured roots.
func (h *libraryHandlers) startScan(c *gin.Context) {
writeScanStarted(c, h.scan.Start())
full, ok := fullParam(c)
if !ok {
return
}
writeScanStarted(c, h.scan.Start(full))
}
// startLibraryScan launches a background scan of one library.
@ -46,7 +64,11 @@ func (h *libraryHandlers) startLibraryScan(c *gin.Context) {
if !ok {
return
}
writeScanStarted(c, h.scan.StartLibrary(id))
full, ok := fullParam(c)
if !ok {
return
}
writeScanStarted(c, h.scan.StartLibrary(id, full))
}
func writeScanStarted(c *gin.Context, err error) {

View file

@ -90,3 +90,12 @@ func (h *providersHandlers) test(c *gin.Context) {
}
c.JSON(http.StatusOK, gin.H{"ok": res.OK, "status": res.Status})
}
func (h *providersHandlers) purge(c *gin.Context) {
res, err := h.svc.Purge(c.Request.Context(), c.Param("name"))
if err != nil {
writeProviderError(c, err)
return
}
c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "artists": res.Artists})
}

View file

@ -13,10 +13,11 @@ type fakeScan struct {
noLibs bool
starts int
lastOnly int64 // library of the last StartLibrary call
lastFull bool // whether the last start asked for a full rescan
status library.Status
}
func (f *fakeScan) Start() error {
func (f *fakeScan) Start(full bool) error {
if f.noLibs {
return library.ErrNoLibraries
}
@ -25,10 +26,11 @@ func (f *fakeScan) Start() error {
}
f.running = true
f.starts++
f.lastFull = full
return nil
}
func (f *fakeScan) StartLibrary(id int64) error {
func (f *fakeScan) StartLibrary(id int64, full bool) error {
if id == 999 {
return library.ErrLibraryNotFound
}
@ -38,6 +40,7 @@ func (f *fakeScan) StartLibrary(id int64) error {
f.running = true
f.starts++
f.lastOnly = id
f.lastFull = full
return nil
}
@ -96,3 +99,37 @@ func TestScanStatus(t *testing.T) {
t.Errorf("last = %v", last)
}
}
// A full rescan is asked for explicitly; anything else in the parameter is a mistake.
func TestScanCanBeAskedToReadEveryFileAgain(t *testing.T) {
scan := &fakeScan{}
h := newTestAPIWithScan(t, scan)
tok := registerAndLogin(t, h)
for _, tt := range []struct {
query string
full bool
}{{"", false}, {"?full=false", false}, {"?full=true", true}} {
scan.running = false
if r := do(t, h, "POST", "/api/v1/library/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full {
t.Errorf("scan%s: %d full=%v, want %v", tt.query, r.Code, scan.lastFull, tt.full)
}
scan.running = false
if r := do(t, h, "POST", "/api/v1/libraries/7/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full || scan.lastOnly != 7 {
t.Errorf("library scan%s: %d full=%v only=%d", tt.query, r.Code, scan.lastFull, scan.lastOnly)
}
}
starts := scan.starts
scan.running = false
for _, q := range []string{"?full=maybe", "?full=1", "?full=TRUE", "?full=", "?full=true&full=bogus", "?full=false&full=true"} {
for _, path := range []string{"/api/v1/library/scan", "/api/v1/libraries/7/scan"} {
if r := do(t, h, "POST", path+q, tok, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
t.Errorf("%s%s: %d %v, want 400 invalid_input", path, q, r.Code, r.Body)
}
}
}
if scan.starts != starts {
t.Error("a scan started with a bad parameter")
}
}

View file

@ -5,6 +5,7 @@ import (
"encoding/json"
"errors"
"net/http"
"strconv"
"strings"
"testing"
@ -13,7 +14,6 @@ import (
"go.uber.org/zap/zaptest/observer"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/storage"
)
const lastfmKey = "0123456789abcdef0123456789abcdef"
@ -193,7 +193,7 @@ func TestProviderTest(t *testing.T) {
var asked string
var answer error
h, _ := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
prov.SetChecker(providers.LastFM, func(_ context.Context, key string) error { asked = key; return answer })
d.Providers = prov
})
@ -231,3 +231,64 @@ func TestProviderTest(t *testing.T) {
t.Errorf("a provider without a test: %d %v", r.Code, r.Body)
}
}
// ---- taking back a provider's data ----
func TestPurgeIsAdminOnly(t *testing.T) {
h := newTestAPI(t)
admin := registerAndLogin(t, h)
_, bob := addUser(t, h, admin, "bob", false)
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("no token: %d", r.Code)
}
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
t.Errorf("normal user: %d %v", r.Code, r.Body)
}
if r := do(t, h, "DELETE", "/api/v1/providers/spotify/data", admin, nil); r.Code != http.StatusNotFound || errCode(r) != "provider_not_found" {
t.Errorf("unknown provider: %d %v", r.Code, r.Body)
}
}
func TestPurgeRemovesWhatTheProviderGaveAndNothingOfMine(t *testing.T) {
e := newSimilarEnv(t)
e.enableLastfm(t)
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil) // names Beta (mine) and Gamma (discovered)
gamma := int64(r.Body["items"].([]any)[1].(map[string]any)["artist"].(map[string]any)["id"].(float64))
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusOK {
t.Fatalf("the discovered artist is not there to begin with: %d", a.Code)
}
p := do(t, e.h, "DELETE", "/api/v1/providers/lastfm/data", e.admin, nil)
if p.Code != http.StatusOK {
t.Fatalf("%d %v", p.Code, p.Body)
}
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "artists")
if p.Body["similar_artists"] != float64(2) || p.Body["tags"] != float64(0) || p.Body["artists"] != float64(1) {
t.Errorf("purge = %v, want 2 similar artists, no tags and the one artist Gamma", p.Body)
}
raw, _ := json.Marshal(p.Body)
if strings.Contains(string(raw), lastfmKey) {
t.Fatal("the answer contains the API key")
}
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusNotFound {
t.Errorf("the discovered artist survived: %d", a.Code)
}
for _, id := range []int64{e.alpha, e.beta} {
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(id, 10), e.admin, nil); a.Code != http.StatusOK {
t.Errorf("one of my artists was touched: %d", a.Code)
}
}
// The provider is as it was: still on, still with its key.
list := do(t, e.h, "GET", "/api/v1/providers", e.admin, nil).Body["providers"].([]any)[0].(map[string]any)
if list["enabled"] != true || list["has_api_key"] != true {
t.Errorf("the purge changed the provider: %v", list)
}
// And the next look asks again, since nothing is stored.
calls := e.source.count()
again := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
if again.Code != http.StatusOK || e.source.count() != calls+1 {
t.Errorf("after the purge: %d, calls %d -> %d", again.Code, calls, e.source.count())
}
}

View file

@ -50,8 +50,8 @@ type streamService interface {
}
type scanService interface {
Start() error
StartLibrary(id int64) error
Start(full bool) error
StartLibrary(id int64, full bool) error
Status() library.Status
}
@ -59,6 +59,7 @@ type providerService interface {
List(ctx context.Context) ([]providers.Info, error)
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
Test(ctx context.Context, name string) (providers.TestResult, error)
Purge(ctx context.Context, name string) (providers.Purged, error)
}
type tagService interface {
@ -160,6 +161,7 @@ func NewRouter(d Deps) *gin.Engine {
admin.GET("/providers", prov.list)
admin.PUT("/providers/:name", prov.update)
admin.POST("/providers/:name/test", prov.test)
admin.DELETE("/providers/:name/data", prov.purge)
u := &usersHandlers{svc: d.Auth}
admin.GET("/users", u.list)
admin.POST("/users", u.create)

View file

@ -80,7 +80,7 @@ func newSimilarEnvWith(t *testing.T, more func(*Deps, *bun.DB, *providers.Servic
}}
e := &similarEnv{source: src}
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
d.Providers = prov
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
@ -295,7 +295,7 @@ func TestWhatLeavesTheServerIsTheArtistAndTheKeyOnly(t *testing.T) {
var alpha int64
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
d.Providers = prov
client := lastfm.New(outbound.New(outbound.Options{UserAgent: providers.UserAgent("1.2.3", "")}), remote.URL)
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), client, prov, zap.NewNop())
@ -373,7 +373,7 @@ func TestALeavingClientIsNotAServerError(t *testing.T) {
core, logs := observer.New(zap.DebugLevel)
src := &fakeLastfm{release: make(chan struct{}), result: []lastfm.Similar{{Name: "Gamma", Match: 1}}}
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
d.Providers = prov
d.Logger = zap.New(core)
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())

View file

@ -91,8 +91,8 @@ func TestTagsWithEveryProviderOffAskNobody(t *testing.T) {
}
sameKeys(t, "tags", r.Body, "sources")
sources := r.Body["sources"].([]any)
if len(sources) != 2 {
t.Fatalf("sources = %v", sources)
if len(sources) != 3 {
t.Fatalf("sources = %v, want Last.fm, MusicBrainz and the files", sources)
}
for i, name := range []string{"lastfm", "musicbrainz"} {
s := sources[i].(map[string]any)
@ -188,7 +188,7 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
e := &similarEnv{}
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
prov := providers.NewService(storage.NewProviderRepo(db))
prov := newProviders(db)
d.Providers = prov
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
mbClient := musicbrainz.New(outbound.New(outbound.Options{UserAgentFunc: prov.UserAgentFor(providers.MusicBrainz, "1.2.3")}), remote.URL+"/ws/2/")
@ -232,3 +232,30 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
}
}
}
// The genres of my own files are always there, and come from nobody.
func TestTagsIncludeTheGenresOfMyOwnFiles(t *testing.T) {
e := newTagsEnv(t)
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'grunge' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
t.Fatal(err)
}
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'rock' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
t.Fatal(err)
}
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
files := r.Body["sources"].([]any)[2].(map[string]any)
sameKeys(t, "files", files, "provider", "provider_status", "stale", "retrieved_at", "attribution", "tags")
tags := files["tags"].([]any)
if files["provider"] != "files" || files["provider_status"] != "ok" || files["stale"] != false || len(tags) != 2 {
t.Fatalf("files = %v", files)
}
if first := tags[0].(map[string]any); first["name"] != "grunge" || first["weight"] != float64(1) || first["genre"] != true {
t.Errorf("first genre = %v", first)
}
if v, present := files["retrieved_at"]; !present || v != nil {
t.Errorf("retrieved_at = %#v, want an explicit null: nothing was read from anywhere", v)
}
if e.lf.calls+e.mb.calls != 0 {
t.Error("the genres of my files asked a provider")
}
}

View file

@ -458,8 +458,12 @@ paths:
tags: [Scan]
operationId: scanLibrary
summary: Scan one library
description: Admin only. Starts a scan in the background; follow it with `GET /library/scan`.
description: |
Admin only. Starts a scan in the background; follow it with `GET /library/scan`. With
`full=true` every file is read again, not only the new or changed ones.
security: [{bearerAuth: []}]
parameters:
- {$ref: '#/components/parameters/Full'}
responses:
'202': {$ref: '#/components/responses/ScanStarted'}
'400': {$ref: '#/components/responses/InvalidInput'}
@ -494,9 +498,16 @@ paths:
Admin only. Starts a scan in the background. Only MP3 and FLAC files are read; other audio
formats are counted as `unsupported`. Housekeeping folders of NAS boxes (`@Recycle`,
`#recycle`, `@eaDir`, `.@__thumb`, `$RECYCLE.BIN`) are skipped.
A scan reads the tags of files that are new or changed. Send `full=true` to read every file
again: that is how MusicBrainz ids and genres reach music that was indexed before they were
read. Files keep their identity, so listening history is kept.
security: [{bearerAuth: []}]
parameters:
- {$ref: '#/components/parameters/Full'}
responses:
'202': {$ref: '#/components/responses/ScanStarted'}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'403': {$ref: '#/components/responses/AdminRequired'}
'409':
@ -606,10 +617,14 @@ paths:
summary: Tags of an artist, from each provider
description: |
What each provider says kind of music this artist plays: Last.fm's top tags and
MusicBrainz's tags and genres. There is always one entry for each, Last.fm first. The
MusicBrainz's tags and genres, and last the genres of the artist's own music files
(`provider` is `files`: the genre tags the scan read, each weighted by how many of the
playable tracks the artist is credited on have it; it asks nobody, is always `ok` and has a null `retrieved_at`). There is
always one entry for each, Last.fm first. The
weights are the providers' own numbers, kept as they came and never mixed: Last.fm's is
relative to the artist (1 to 100, the top tag is usually 100; a tag with no votes is dropped),
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres.
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres, and for
every tag of the `files` entry, which are genres by definition.
A provider is only asked when an admin switched it on and its stored answer is missing or
old (7 days; a "no such artist" answer is kept 1 day), with the same rules as
@ -937,6 +952,39 @@ paths:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/providers/{name}/data:
parameters:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz]}
delete:
tags: [Providers]
operationId: purgeProviderData
summary: Remove what a provider gave
description: |
Admin only. Removes everything the provider told Sonda: its lists of similar artists, its
tags and the records of when it was asked, and then the artists that only that data named
and that nobody has files for. Last.fm's terms ask for exactly this when an agreement ends.
Your own artists, music, listens and settings are not touched, and the provider is not
switched off: if it is still on, it is asked again when someone asks. The answer counts what
was removed; doing it again removes nothing.
security: [{bearerAuth: []}]
responses:
'200':
description: What was removed.
content:
application/json:
schema: {$ref: '#/components/schemas/ProviderPurge'}
'401': {$ref: '#/components/responses/Unauthorized'}
'403': {$ref: '#/components/responses/AdminRequired'}
'404':
description: No such provider (`provider_not_found`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/listens:
get:
tags: [Playback]
@ -1035,6 +1083,11 @@ components:
Case-insensitive substring of the name or title. Accented letters are matched ignoring
case (`BJÖRK` finds `Björk`), but accents are kept (`bjork` does not). At most 200 characters.
schema: {type: string, maxLength: 200}
Full:
name: full
in: query
description: "`true` reads every file again, not only the new or changed ones. Only `true` and `false` are accepted."
schema: {type: boolean, default: false}
Limit:
name: limit
in: query
@ -1389,6 +1442,15 @@ components:
maxLength: 200
description: A URL or e-mail address, without parentheses. Empty restores the default.
ProviderPurge:
type: object
additionalProperties: false
required: [similar_artists, tags, artists]
properties:
similar_artists: {type: integer, description: Similar-artist links removed.}
tags: {type: integer, description: Tags removed.}
artists: {type: integer, description: "Artists removed because only the removed data named them and nobody has files for them."}
ProviderTest:
type: object
additionalProperties: false
@ -1414,7 +1476,7 @@ components:
additionalProperties: false
required: [provider, provider_status, stale, retrieved_at, attribution, tags]
properties:
provider: {type: string, enum: [lastfm, musicbrainz]}
provider: {type: string, enum: [lastfm, musicbrainz, files]}
provider_status:
type: string
enum: [ok, disabled, error, no_id]

View file

@ -200,6 +200,9 @@ func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Resul
case <-ctx.Done():
return Result{}, ctx.Err()
case r := <-done:
if errors.Is(r.Err, ErrNotFound) {
return Result{}, ErrNotFound // removed while it was being read
}
if r.Err != nil {
s.log.Warn("similar artists could not be read", zap.String("provider", Provider), zap.Error(r.Err))
return result(snap, StatusError, snap.Fetched), nil

View file

@ -33,6 +33,7 @@ type fakeStore struct {
snaps map[int64]Snapshot
saves []saved
lastLim int
saveErr error // returned by SaveSimilar when set
}
func newStore() *fakeStore {
@ -66,6 +67,9 @@ func (f *fakeStore) Similar(_ context.Context, id int64, _ string, limit int) (S
func (f *fakeStore) SaveSimilar(_ context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.saveErr != nil {
return f.saveErr
}
f.saves = append(f.saves, saved{id, provider, at, found, cands})
snap := Snapshot{Fetched: true, NotFound: !found, RetrievedAt: at}
for i, c := range cands {
@ -413,3 +417,13 @@ func TestNoPageCanAskForMoreThanIsStored(t *testing.T) {
t.Errorf("MaxLimit = %d is above FetchLimit = %d", MaxLimit, FetchLimit)
}
}
// An artist removed while its list was being read is not there any more: that is
// "not found", not a provider failure.
func TestAnArtistRemovedWhileItsListWasReadIsNotFound(t *testing.T) {
r := newRig(true)
r.store.saveErr = ErrNotFound
if _, err := r.svc.Similar(context.Background(), 1, 10); !errors.Is(err, ErrNotFound) {
t.Fatalf("err = %v, want ErrNotFound", err)
}
}

View file

@ -25,6 +25,12 @@ const MaxTags = 50
// a page waiting for a minute because a client asked about fifty artists.
const tagFetchTimeout = 20 * time.Second
// SourceFiles names the tags that come from the user's own music files. It is not
// a provider: it is always there, asks nobody and needs no settings.
const SourceFiles = "files"
const attributionFiles = "From the genre tags of your own music files."
// StatusNoID says a provider that needs the MusicBrainz id of the artist cannot
// be asked, because the id is not known. Sonda never looks one up by name.
const StatusNoID = "no_id"
@ -62,13 +68,16 @@ type TagSourceResult struct {
Tags []Tag
}
// TagResult has one entry per provider, always in the same order.
// TagResult has one entry per provider, always in the same order, and the tags
// of the user's own files last.
type TagResult struct{ Sources []TagSourceResult }
// TagStore keeps the tags. Artist returns ErrNotFound for an id the catalog does
// not know.
type TagStore interface {
Artist(ctx context.Context, id int64) (Artist, error)
// FileGenres counts the genres of the artist's own playable tracks, most first.
FileGenres(ctx context.Context, id int64) ([]Tag, error)
Tags(ctx context.Context, id int64, provider string) (TagSnapshot, error)
SaveTags(ctx context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error
}
@ -134,12 +143,17 @@ func (s *TagService) Tags(ctx context.Context, artistID int64) (TagResult, error
out[i], errs[i] = s.one(ctx, artist, p)
}()
}
genres, genErr := s.store.FileGenres(ctx, artistID)
wg.Wait()
for _, err := range errs {
for _, err := range append(errs, genErr) {
if err != nil {
return TagResult{}, err
}
}
if genres == nil {
genres = []Tag{}
}
out = append(out, TagSourceResult{Provider: SourceFiles, Status: StatusOK, Attribution: attributionFiles, Tags: genres})
return TagResult{Sources: out}, nil
}
@ -185,6 +199,9 @@ func (s *TagService) one(ctx context.Context, a Artist, p tagProvider) (TagSourc
case <-ctx.Done():
return TagSourceResult{}, ctx.Err()
case r := <-done:
if errors.Is(r.Err, ErrNotFound) {
return TagSourceResult{}, ErrNotFound // removed while it was being read
}
if r.Err != nil {
s.log.Warn("tags could not be read", zap.String("provider", p.name), zap.Error(r.Err))
return res(StatusError, snap.Fetched, snap), nil

View file

@ -26,6 +26,9 @@ type tagStore struct {
artists map[int64]Artist
snaps map[string]TagSnapshot
saves []savedTags
genres map[int64][]Tag // what the artist's own files say
genErr error
saveErr error // returned by SaveTags when set
}
func newTagStore() *tagStore {
@ -38,6 +41,12 @@ func newTagStore() *tagStore {
}
}
func (f *tagStore) FileGenres(_ context.Context, id int64) ([]Tag, error) {
f.mu.Lock()
defer f.mu.Unlock()
return f.genres[id], f.genErr
}
func tkey(id int64, provider string) string { return provider + "/" + string(rune('0'+id)) }
func (f *tagStore) Artist(_ context.Context, id int64) (Artist, error) {
@ -59,6 +68,9 @@ func (f *tagStore) Tags(_ context.Context, id int64, provider string) (TagSnapsh
func (f *tagStore) SaveTags(_ context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error {
f.mu.Lock()
defer f.mu.Unlock()
if f.saveErr != nil {
return f.saveErr
}
f.saves = append(f.saves, savedTags{id, provider, at, found, tags})
f.snaps[tkey(id, provider)] = TagSnapshot{Fetched: true, NotFound: !found, RetrievedAt: at, Tags: tags}
return nil
@ -149,10 +161,10 @@ func TestTagsOfAnUnknownArtistAreNotFoundAndNobodyIsAsked(t *testing.T) {
func TestTagsAlwaysListBothProvidersInTheSameOrder(t *testing.T) {
r := newTagRig(false, false)
res := r.tags(t, 1)
if len(res.Sources) != 2 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz {
if len(res.Sources) != 3 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz || res.Sources[2].Provider != SourceFiles {
t.Fatalf("sources = %+v", res.Sources)
}
for _, s := range res.Sources {
for _, s := range res.Sources[:2] {
if s.Status != StatusDisabled || len(s.Tags) != 0 || !s.RetrievedAt.IsZero() || s.Stale || s.Attribution == "" {
t.Errorf("%s = %+v", s.Provider, s)
}
@ -293,3 +305,39 @@ func TestConcurrentTagLooksShareOneCall(t *testing.T) {
t.Errorf("calls = %d, want 1", r.lf.count())
}
}
// The genres of the user's own files need no provider: they are always there,
// ask nobody, and are computed when asked.
func TestTheGenresOfTheFilesAreAlwaysThere(t *testing.T) {
r := newTagRig(false, false)
r.store.genres = map[int64][]Tag{1: {{Name: "rock", Weight: 3, Genre: true}, {Name: "grunge", Weight: 1, Genre: true}}}
res := r.tags(t, 1)
files := res.Sources[2]
if files.Provider != SourceFiles || files.Status != StatusOK || files.Stale || len(files.Tags) != 2 || files.Tags[0].Name != "rock" || files.Tags[0].Weight != 3 || !files.RetrievedAt.IsZero() || files.Attribution == "" {
t.Errorf("files = %+v", files)
}
if r.lf.count()+r.mb.count() != 0 {
t.Error("reading the genres of the files asked a provider")
}
// An artist without files says nothing, and it is not an error.
res = r.tags(t, 2)
if f := res.Sources[2]; f.Status != StatusOK || f.Tags == nil || len(f.Tags) != 0 {
t.Errorf("files of an artist without any = %+v", f)
}
}
func TestAFailureReadingFileGenresIsARealError(t *testing.T) {
r := newTagRig(false, false)
r.store.genErr = errors.New("db down")
if _, err := r.svc.Tags(context.Background(), 1); err == nil {
t.Fatal("the database failed and nobody was told")
}
}
func TestAnArtistRemovedWhileItsTagsWereReadIsNotFound(t *testing.T) {
r := newTagRig(true, false)
r.store.saveErr = ErrNotFound
if _, err := r.svc.Tags(context.Background(), 1); !errors.Is(err, ErrNotFound) {
t.Fatalf("err = %v, want ErrNotFound", err)
}
}

View file

@ -57,7 +57,7 @@ func (m *Manager) OnFinish(fn func(Summary, error)) {
// Start scans every library in the background. ctx bounds the scan, so pass a
// context that lives as long as the application, not an HTTP request.
// It returns ErrNoLibraries when none are defined.
func (m *Manager) Start(ctx context.Context) error {
func (m *Manager) Start(ctx context.Context, opts ...ScanOption) error {
libs, err := m.libraries.ListLibraries(ctx)
if err != nil {
return fmt.Errorf("list libraries: %w", err)
@ -65,20 +65,20 @@ func (m *Manager) Start(ctx context.Context) error {
if len(libs) == 0 {
return ErrNoLibraries
}
return m.start(ctx, libs)
return m.start(ctx, libs, opts)
}
// StartLibrary scans a single library in the background.
// It returns ErrLibraryNotFound for an unknown id.
func (m *Manager) StartLibrary(ctx context.Context, id int64) error {
func (m *Manager) StartLibrary(ctx context.Context, id int64, opts ...ScanOption) error {
lib, err := m.libraries.GetLibrary(ctx, id)
if err != nil {
return err
}
return m.start(ctx, []Library{lib})
return m.start(ctx, []Library{lib}, opts)
}
func (m *Manager) start(ctx context.Context, libs []Library) error {
func (m *Manager) start(ctx context.Context, libs []Library, opts []ScanOption) error {
m.mu.Lock()
defer m.mu.Unlock()
if m.status.Running {
@ -86,12 +86,12 @@ func (m *Manager) start(ctx context.Context, libs []Library) error {
}
m.status.Running = true
m.status.StartedAt = time.Now()
go m.run(ctx, libs)
go m.run(ctx, libs, opts)
return nil
}
func (m *Manager) run(ctx context.Context, libs []Library) {
sum, err := m.scanner.Scan(ctx, libs)
func (m *Manager) run(ctx context.Context, libs []Library, opts []ScanOption) {
sum, err := m.scanner.Scan(ctx, libs, opts...)
m.mu.Lock()
m.status.Running = false

View file

@ -12,7 +12,7 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
files: []FileInfo{{RelPath: "new.mp3", Size: 10, MTimeUnix: 5}}, // same size+mtime as flaky.mp3
failedFiles: map[string]bool{"flaky.mp3": true},
}
p := makePlan(w, existing)
p := makePlan(w, existing, scanOptions{})
for _, id := range p.missing {
if id == 1 {
@ -29,3 +29,43 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
t.Errorf("parse = %+v, want new.mp3 as a new file", p.parse)
}
}
// A full rescan reads every file again, whether or not it changed, so what the
// reader learned since (ids, genres) reaches files that are already indexed.
func TestFullPlanReadsEveryPresentFileAgain(t *testing.T) {
existing := []ExistingFile{
{ID: 1, FileInfo: FileInfo{RelPath: "same.mp3", Size: 10, MTimeUnix: 5}},
{ID: 2, FileInfo: FileInfo{RelPath: "changed.mp3", Size: 20, MTimeUnix: 6}},
{ID: 3, FileInfo: FileInfo{RelPath: "back.mp3", Size: 30, MTimeUnix: 7}, Missing: true},
{ID: 4, FileInfo: FileInfo{RelPath: "gone.mp3", Size: 40, MTimeUnix: 8}},
}
w := walkResult{files: []FileInfo{
{RelPath: "same.mp3", Size: 10, MTimeUnix: 5},
{RelPath: "changed.mp3", Size: 21, MTimeUnix: 6},
{RelPath: "back.mp3", Size: 30, MTimeUnix: 7},
{RelPath: "new.mp3", Size: 50, MTimeUnix: 9},
}}
normal := makePlan(w, existing, scanOptions{})
if len(normal.parse) != 2 || normal.unchanged != 1 || len(normal.restore) != 1 {
t.Fatalf("normal plan: parse=%d unchanged=%d restore=%d", len(normal.parse), normal.unchanged, len(normal.restore))
}
full := makePlan(w, existing, scanOptions{full: true})
ids := map[int64]bool{}
for _, j := range full.parse {
ids[j.existingID] = true
}
if len(full.parse) != 4 || !ids[1] || !ids[2] || !ids[3] || !ids[0] {
t.Errorf("full plan parses %+v, want the four present files (three known, one new)", full.parse)
}
// A file that came back is restored even if reading it fails this time, as in a
// normal scan, and it is read as well.
if full.unchanged != 0 || len(full.restore) != 1 || full.restore[0] != 3 {
t.Errorf("full plan: unchanged=%d restore=%v, want only back.mp3 restored", full.unchanged, full.restore)
}
// What is gone is still gone.
if len(full.missing) != 1 || full.missing[0] != 4 {
t.Errorf("missing = %v, want only gone.mp3", full.missing)
}
}

View file

@ -59,15 +59,35 @@ func NewScanner(store Store, reader TagReader, hub *Hub, workers int) *Scanner {
return &Scanner{store: store, reader: reader, hub: hub, workers: workers, now: time.Now}
}
// ScanOption changes how a scan runs.
type ScanOption func(*scanOptions)
type scanOptions struct{ full bool }
// FullRescan makes a scan read the tags of every file again, not only of the files
// that are new or changed. It is how what the reader learned since (MusicBrainz
// ids, genres) reaches music that was indexed before. Nothing else changes: files
// keep their identity, so listening history is kept.
func FullRescan(o *scanOptions) { o.full = true }
func buildOptions(opts []ScanOption) scanOptions {
var o scanOptions
for _, f := range opts {
f(&o)
}
return o
}
// Scan indexes every library in turn. An unreadable library is reported in the
// returned error and the other libraries are still scanned.
func (s *Scanner) Scan(ctx context.Context, libs []Library) (Summary, error) {
func (s *Scanner) Scan(ctx context.Context, libs []Library, opts ...ScanOption) (Summary, error) {
o := buildOptions(opts)
total := Summary{Libraries: len(libs)}
s.publish(Event{Type: ScanStarted})
var errs []error
for _, lib := range libs {
sum, err := s.scanLibrary(ctx, lib)
sum, err := s.scanLibrary(ctx, lib, o)
total.add(sum)
if err != nil {
errs = append(errs, fmt.Errorf("%s: %w", lib.Name, err))
@ -102,7 +122,7 @@ type parseResult struct {
err error
}
func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error) {
func (s *Scanner) scanLibrary(ctx context.Context, lib Library, o scanOptions) (Summary, error) {
var sum Summary
walked, err := s.walk(ctx, lib, &sum)
@ -126,7 +146,7 @@ func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error)
"refusing to mark them missing", n)
}
}
p := makePlan(walked, existing)
p := makePlan(walked, existing, o)
sum.Unchanged = p.unchanged
if err := s.parseAndSave(ctx, lib, p.parse, &sum); err != nil {
@ -348,7 +368,7 @@ type fileKey struct{ size, mtime int64 }
// A new file and a vanished file with the same size and modification time are
// treated as a rename only when each is the sole candidate for that pair, so
// look-alike files are never guessed.
func makePlan(w walkResult, existing []ExistingFile) plan {
func makePlan(w walkResult, existing []ExistingFile, o scanOptions) plan {
var p plan
byPath := make(map[string]ExistingFile, len(existing))
for _, e := range existing {
@ -364,9 +384,14 @@ func makePlan(w walkResult, existing []ExistingFile) plan {
continue
}
seen[f.RelPath] = true
changed := e.Size != f.Size || e.MTimeUnix != f.MTimeUnix
switch {
case e.Size != f.Size || e.MTimeUnix != f.MTimeUnix:
case changed || o.full:
p.parse = append(p.parse, parseJob{file: f, existingID: e.ID})
if e.Missing && !changed {
// Back, and read again. It is back even if reading fails this time.
p.restore = append(p.restore, e.ID)
}
case e.Missing:
p.restore = append(p.restore, e.ID)
default:

Some files were not shown because too many files have changed in this diff Show more