Merge pull request 'feat(discovery): remove provider data, genres from my files, full rescans' (#26) from feat/provider-purge-and-genres into main
This commit is contained in:
commit
4da79601d2
32 changed files with 1187 additions and 57 deletions
14
README.md
14
README.md
|
|
@ -15,12 +15,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
|
||||
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
|
||||
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights. See [External providers](#external-providers).
|
||||
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
|
||||
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
|
||||
|
||||
## What is not there yet
|
||||
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Genres from your own files, and the removal of provider data on request.
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Recommendations from all this data (scoring) are next.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -200,7 +200,11 @@ The key is stored in the database, in clear text, and is never returned by the A
|
|||
|
||||
**Tags.** `GET /api/v1/artists/{id}/tags` asks each provider that is on, at the same time, and keeps the answers with the same rules (7 days, 1 for an unknown artist, old data if a provider fails). Each provider has its own entry with its own weights: Last.fm's are relative to the artist (1 to 100), MusicBrainz's are counts of votes, and they are never mixed. MusicBrainz also marks the tags it lists as genres. Its tags and genres are CC BY-NC-SA, so show the `attribution` that comes with them.
|
||||
|
||||
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. The scan reads a file's tags when the file is new or changed, so to pick up ids for music that is already indexed, the files have to be touched or re-added.
|
||||
**Genres from your own files.** The scan also reads the genre tag of each file (several genres separated by `;`, at most five per file) and `GET /artists/{id}/tags` adds a `files` entry: the genres of the playable tracks the artist is credited on, each weighted by how many tracks have it. It needs no provider, asks nobody, and is always there.
|
||||
|
||||
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. A normal scan reads a file's tags when the file is new or changed. To pick up ids and genres for music that is already indexed, scan with `full=true` (`POST /library/scan?full=true`, or `POST /libraries/{id}/scan?full=true`): every file is read again, keeps its identity, and your listening history is untouched.
|
||||
|
||||
**Taking it back.** `DELETE /api/v1/providers/{name}/data` (admin) removes everything a provider gave: its similar artists, its tags and the records of when it was asked, and then the artists that only that data named and that you have no files for. Last.fm's terms ask for this when an agreement ends. Your own artists, music and listens stay, and the provider is not switched off. An artist that no list names any more is also removed when the list that named it is replaced, so they do not pile up.
|
||||
|
||||
The artist is sent to Last.fm by its MusicBrainz id when your files gave one (if Last.fm does not know the id, by name), otherwise by its name exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
|
||||
|
||||
|
|
@ -224,8 +228,8 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
|
|||
| signed in | `GET /listens` | Your own history, newest first, with `counted` per event. Query: `limit`, `offset`, `counted=true`. |
|
||||
| signed in | `GET /library/scan` | Scan status and the last result. |
|
||||
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test` | See, set up and check Last.fm and MusicBrainz. The API key is write-only. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. `?full=true` reads every file again. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test`, `DELETE /providers/{name}/data` | See, set up and check Last.fm and MusicBrainz, and remove what they gave. The API key is write-only. |
|
||||
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
|
||||
|
||||
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.
|
||||
|
|
|
|||
|
|
@ -128,6 +128,9 @@ func run() error {
|
|||
return musicbrainzClient.Probe(ctx)
|
||||
})
|
||||
discoveryRepo := storage.NewDiscoveryRepo(db)
|
||||
providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
|
||||
return discoveryRepo.Purge(ctx, name)
|
||||
})
|
||||
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
|
||||
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
|
||||
providers.LastFM: discovery.LastfmTags{Client: lastfmClient},
|
||||
|
|
|
|||
|
|
@ -15,9 +15,20 @@ type scanControl struct {
|
|||
m *library.Manager
|
||||
}
|
||||
|
||||
func (c scanControl) Start() error { return c.m.Start(c.ctx) }
|
||||
func (c scanControl) StartLibrary(id int64) error { return c.m.StartLibrary(c.ctx, id) }
|
||||
func (c scanControl) Status() library.Status { return c.m.Status() }
|
||||
func (c scanControl) Start(full bool) error { return c.m.Start(c.ctx, scanOptions(full)...) }
|
||||
|
||||
func (c scanControl) StartLibrary(id int64, full bool) error {
|
||||
return c.m.StartLibrary(c.ctx, id, scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (c scanControl) Status() library.Status { return c.m.Status() }
|
||||
|
||||
func scanOptions(full bool) []library.ScanOption {
|
||||
if full {
|
||||
return []library.ScanOption{library.FullRescan}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// logScanResults writes one log line per finished scan. File paths are left
|
||||
// out on purpose: audio paths are private.
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
}
|
||||
|
||||
tokens := auth.NewTokenIssuer(secret, time.Hour)
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
deps := Deps{
|
||||
Logger: zap.NewNop(),
|
||||
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, 24*time.Hour),
|
||||
|
|
@ -92,6 +92,16 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
return NewRouter(deps), db
|
||||
}
|
||||
|
||||
// newProviders builds the provider service the way main does, with the real way
|
||||
// of removing provider data.
|
||||
func newProviders(db *bun.DB) *providers.Service {
|
||||
svc := providers.NewService(storage.NewProviderRepo(db))
|
||||
svc.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
|
||||
return storage.NewDiscoveryRepo(db).Purge(ctx, name)
|
||||
})
|
||||
return svc
|
||||
}
|
||||
|
||||
// noFiles is a streaming source with nothing to play.
|
||||
type noFiles struct{}
|
||||
|
||||
|
|
|
|||
|
|
@ -80,9 +80,22 @@ func (r *contractReader) Read(path string) (library.Metadata, error) {
|
|||
// scanAdapter lets the API start scans on a real manager.
|
||||
type scanAdapter struct{ m *library.Manager }
|
||||
|
||||
func (a scanAdapter) Start() error { return a.m.Start(context.Background()) }
|
||||
func (a scanAdapter) StartLibrary(id int64) error { return a.m.StartLibrary(context.Background(), id) }
|
||||
func (a scanAdapter) Status() library.Status { return a.m.Status() }
|
||||
func (a scanAdapter) Start(full bool) error {
|
||||
return a.m.Start(context.Background(), scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (a scanAdapter) StartLibrary(id int64, full bool) error {
|
||||
return a.m.StartLibrary(context.Background(), id, scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (a scanAdapter) Status() library.Status { return a.m.Status() }
|
||||
|
||||
func scanOptions(full bool) []library.ScanOption {
|
||||
if full {
|
||||
return []library.ScanOption{library.FullRescan}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type contract struct {
|
||||
lastfm *fakeLastfm
|
||||
|
|
@ -102,7 +115,7 @@ func newContract(t *testing.T) *contract {
|
|||
tagsMB := &fakeTags{tags: []discovery.Tag{{Name: "grunge", Weight: 12, Genre: true}}}
|
||||
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
|
||||
d.Providers = prov
|
||||
prov.SetChecker(providers.MusicBrainz, func(context.Context, string) error { return nil })
|
||||
|
|
@ -339,6 +352,7 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
close(gate)
|
||||
c.scanDone(token)
|
||||
|
||||
c.expect(c.call("POST", "/library/scan?full=maybe", token, nil), 400, "scan with a bad parameter", "invalid_input")
|
||||
c.expect(c.call("POST", "/library/scan", token, nil), 202, "scan everything")
|
||||
c.scanDone(token)
|
||||
status := c.expect(c.call("GET", "/library/scan", token, nil), 200, "scan status")
|
||||
|
|
@ -347,6 +361,12 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
t.Fatalf("second scan = %v, want 3 files in 2 libraries, all unchanged", last)
|
||||
}
|
||||
|
||||
c.expect(c.call("POST", "/library/scan?full=true", token, nil), 202, "scan everything again, in full")
|
||||
c.scanDone(token)
|
||||
if full := c.expect(c.call("GET", "/library/scan", token, nil), 200, "status after a full scan")["last"].(map[string]any); full["updated"] != float64(3) || full["unchanged"] != float64(0) {
|
||||
t.Fatalf("full scan = %v, want the 3 files read again", full)
|
||||
}
|
||||
|
||||
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 204, "delete library")
|
||||
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 404, "delete library twice", "library_not_found")
|
||||
|
||||
|
|
@ -423,10 +443,15 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
|
||||
tags := c.expect(c.call("GET", "/artists/"+artistID+"/tags", token, nil), 200, "tags")["sources"].([]any)
|
||||
if len(tags) != 2 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" {
|
||||
t.Fatalf("tags = %v, want Last.fm ok and MusicBrainz without an id", tags)
|
||||
if len(tags) != 3 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" || tags[2].(map[string]any)["provider"] != "files" {
|
||||
t.Fatalf("tags = %v, want Last.fm ok, MusicBrainz without an id and the files", tags)
|
||||
}
|
||||
c.expect(c.call("GET", "/artists/999/tags", token, nil), 404, "tags of an unknown artist", "artist_not_found")
|
||||
purged := c.expect(c.call("DELETE", "/providers/lastfm/data", token, nil), 200, "purge a provider's data")
|
||||
if purged["tags"] != float64(1) || purged["similar_artists"] != float64(1) {
|
||||
t.Fatalf("purge = %v", purged)
|
||||
}
|
||||
c.expect(c.call("DELETE", "/providers/spotify/data", token, nil), 404, "purge an unknown provider", "provider_not_found")
|
||||
c.expect(c.call("GET", "/artists/abc/tags", token, nil), 400, "tags of a bad id", "invalid_input")
|
||||
|
||||
// ---- playback ----
|
||||
|
|
|
|||
|
|
@ -35,9 +35,27 @@ type scanStatusJSON struct {
|
|||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// fullParam reads ?full=. Only "true" and "false" are accepted, and it answers
|
||||
// 400 itself otherwise: a typo must not quietly start a scan of another kind.
|
||||
func fullParam(c *gin.Context) (full, ok bool) {
|
||||
values := c.QueryArray("full")
|
||||
switch {
|
||||
case len(values) == 0, len(values) == 1 && values[0] == "false":
|
||||
return false, true
|
||||
case len(values) == 1 && values[0] == "true":
|
||||
return true, true
|
||||
}
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", "full must be true or false")
|
||||
return false, false
|
||||
}
|
||||
|
||||
// startScan launches a background scan of the configured roots.
|
||||
func (h *libraryHandlers) startScan(c *gin.Context) {
|
||||
writeScanStarted(c, h.scan.Start())
|
||||
full, ok := fullParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.Start(full))
|
||||
}
|
||||
|
||||
// startLibraryScan launches a background scan of one library.
|
||||
|
|
@ -46,7 +64,11 @@ func (h *libraryHandlers) startLibraryScan(c *gin.Context) {
|
|||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.StartLibrary(id))
|
||||
full, ok := fullParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.StartLibrary(id, full))
|
||||
}
|
||||
|
||||
func writeScanStarted(c *gin.Context, err error) {
|
||||
|
|
|
|||
|
|
@ -90,3 +90,12 @@ func (h *providersHandlers) test(c *gin.Context) {
|
|||
}
|
||||
c.JSON(http.StatusOK, gin.H{"ok": res.OK, "status": res.Status})
|
||||
}
|
||||
|
||||
func (h *providersHandlers) purge(c *gin.Context) {
|
||||
res, err := h.svc.Purge(c.Request.Context(), c.Param("name"))
|
||||
if err != nil {
|
||||
writeProviderError(c, err)
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "artists": res.Artists})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,10 +13,11 @@ type fakeScan struct {
|
|||
noLibs bool
|
||||
starts int
|
||||
lastOnly int64 // library of the last StartLibrary call
|
||||
lastFull bool // whether the last start asked for a full rescan
|
||||
status library.Status
|
||||
}
|
||||
|
||||
func (f *fakeScan) Start() error {
|
||||
func (f *fakeScan) Start(full bool) error {
|
||||
if f.noLibs {
|
||||
return library.ErrNoLibraries
|
||||
}
|
||||
|
|
@ -25,10 +26,11 @@ func (f *fakeScan) Start() error {
|
|||
}
|
||||
f.running = true
|
||||
f.starts++
|
||||
f.lastFull = full
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeScan) StartLibrary(id int64) error {
|
||||
func (f *fakeScan) StartLibrary(id int64, full bool) error {
|
||||
if id == 999 {
|
||||
return library.ErrLibraryNotFound
|
||||
}
|
||||
|
|
@ -38,6 +40,7 @@ func (f *fakeScan) StartLibrary(id int64) error {
|
|||
f.running = true
|
||||
f.starts++
|
||||
f.lastOnly = id
|
||||
f.lastFull = full
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -96,3 +99,37 @@ func TestScanStatus(t *testing.T) {
|
|||
t.Errorf("last = %v", last)
|
||||
}
|
||||
}
|
||||
|
||||
// A full rescan is asked for explicitly; anything else in the parameter is a mistake.
|
||||
func TestScanCanBeAskedToReadEveryFileAgain(t *testing.T) {
|
||||
scan := &fakeScan{}
|
||||
h := newTestAPIWithScan(t, scan)
|
||||
tok := registerAndLogin(t, h)
|
||||
|
||||
for _, tt := range []struct {
|
||||
query string
|
||||
full bool
|
||||
}{{"", false}, {"?full=false", false}, {"?full=true", true}} {
|
||||
scan.running = false
|
||||
if r := do(t, h, "POST", "/api/v1/library/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full {
|
||||
t.Errorf("scan%s: %d full=%v, want %v", tt.query, r.Code, scan.lastFull, tt.full)
|
||||
}
|
||||
scan.running = false
|
||||
if r := do(t, h, "POST", "/api/v1/libraries/7/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full || scan.lastOnly != 7 {
|
||||
t.Errorf("library scan%s: %d full=%v only=%d", tt.query, r.Code, scan.lastFull, scan.lastOnly)
|
||||
}
|
||||
}
|
||||
|
||||
starts := scan.starts
|
||||
scan.running = false
|
||||
for _, q := range []string{"?full=maybe", "?full=1", "?full=TRUE", "?full=", "?full=true&full=bogus", "?full=false&full=true"} {
|
||||
for _, path := range []string{"/api/v1/library/scan", "/api/v1/libraries/7/scan"} {
|
||||
if r := do(t, h, "POST", path+q, tok, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
|
||||
t.Errorf("%s%s: %d %v, want 400 invalid_input", path, q, r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
if scan.starts != starts {
|
||||
t.Error("a scan started with a bad parameter")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
|
|
@ -13,7 +14,6 @@ import (
|
|||
"go.uber.org/zap/zaptest/observer"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
)
|
||||
|
||||
const lastfmKey = "0123456789abcdef0123456789abcdef"
|
||||
|
|
@ -193,7 +193,7 @@ func TestProviderTest(t *testing.T) {
|
|||
var asked string
|
||||
var answer error
|
||||
h, _ := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
prov.SetChecker(providers.LastFM, func(_ context.Context, key string) error { asked = key; return answer })
|
||||
d.Providers = prov
|
||||
})
|
||||
|
|
@ -231,3 +231,64 @@ func TestProviderTest(t *testing.T) {
|
|||
t.Errorf("a provider without a test: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- taking back a provider's data ----
|
||||
|
||||
func TestPurgeIsAdminOnly(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
_, bob := addUser(t, h, admin, "bob", false)
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("no token: %d", r.Code)
|
||||
}
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
|
||||
t.Errorf("normal user: %d %v", r.Code, r.Body)
|
||||
}
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/spotify/data", admin, nil); r.Code != http.StatusNotFound || errCode(r) != "provider_not_found" {
|
||||
t.Errorf("unknown provider: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPurgeRemovesWhatTheProviderGaveAndNothingOfMine(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil) // names Beta (mine) and Gamma (discovered)
|
||||
gamma := int64(r.Body["items"].([]any)[1].(map[string]any)["artist"].(map[string]any)["id"].(float64))
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusOK {
|
||||
t.Fatalf("the discovered artist is not there to begin with: %d", a.Code)
|
||||
}
|
||||
|
||||
p := do(t, e.h, "DELETE", "/api/v1/providers/lastfm/data", e.admin, nil)
|
||||
if p.Code != http.StatusOK {
|
||||
t.Fatalf("%d %v", p.Code, p.Body)
|
||||
}
|
||||
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "artists")
|
||||
if p.Body["similar_artists"] != float64(2) || p.Body["tags"] != float64(0) || p.Body["artists"] != float64(1) {
|
||||
t.Errorf("purge = %v, want 2 similar artists, no tags and the one artist Gamma", p.Body)
|
||||
}
|
||||
raw, _ := json.Marshal(p.Body)
|
||||
if strings.Contains(string(raw), lastfmKey) {
|
||||
t.Fatal("the answer contains the API key")
|
||||
}
|
||||
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusNotFound {
|
||||
t.Errorf("the discovered artist survived: %d", a.Code)
|
||||
}
|
||||
for _, id := range []int64{e.alpha, e.beta} {
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(id, 10), e.admin, nil); a.Code != http.StatusOK {
|
||||
t.Errorf("one of my artists was touched: %d", a.Code)
|
||||
}
|
||||
}
|
||||
// The provider is as it was: still on, still with its key.
|
||||
list := do(t, e.h, "GET", "/api/v1/providers", e.admin, nil).Body["providers"].([]any)[0].(map[string]any)
|
||||
if list["enabled"] != true || list["has_api_key"] != true {
|
||||
t.Errorf("the purge changed the provider: %v", list)
|
||||
}
|
||||
// And the next look asks again, since nothing is stored.
|
||||
calls := e.source.count()
|
||||
again := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if again.Code != http.StatusOK || e.source.count() != calls+1 {
|
||||
t.Errorf("after the purge: %d, calls %d -> %d", again.Code, calls, e.source.count())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -50,8 +50,8 @@ type streamService interface {
|
|||
}
|
||||
|
||||
type scanService interface {
|
||||
Start() error
|
||||
StartLibrary(id int64) error
|
||||
Start(full bool) error
|
||||
StartLibrary(id int64, full bool) error
|
||||
Status() library.Status
|
||||
}
|
||||
|
||||
|
|
@ -59,6 +59,7 @@ type providerService interface {
|
|||
List(ctx context.Context) ([]providers.Info, error)
|
||||
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
|
||||
Test(ctx context.Context, name string) (providers.TestResult, error)
|
||||
Purge(ctx context.Context, name string) (providers.Purged, error)
|
||||
}
|
||||
|
||||
type tagService interface {
|
||||
|
|
@ -160,6 +161,7 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
admin.GET("/providers", prov.list)
|
||||
admin.PUT("/providers/:name", prov.update)
|
||||
admin.POST("/providers/:name/test", prov.test)
|
||||
admin.DELETE("/providers/:name/data", prov.purge)
|
||||
u := &usersHandlers{svc: d.Auth}
|
||||
admin.GET("/users", u.list)
|
||||
admin.POST("/users", u.create)
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@ func newSimilarEnvWith(t *testing.T, more func(*Deps, *bun.DB, *providers.Servic
|
|||
}}
|
||||
e := &similarEnv{source: src}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
|
|
@ -295,7 +295,7 @@ func TestWhatLeavesTheServerIsTheArtistAndTheKeyOnly(t *testing.T) {
|
|||
|
||||
var alpha int64
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
client := lastfm.New(outbound.New(outbound.Options{UserAgent: providers.UserAgent("1.2.3", "")}), remote.URL)
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), client, prov, zap.NewNop())
|
||||
|
|
@ -373,7 +373,7 @@ func TestALeavingClientIsNotAServerError(t *testing.T) {
|
|||
core, logs := observer.New(zap.DebugLevel)
|
||||
src := &fakeLastfm{release: make(chan struct{}), result: []lastfm.Similar{{Name: "Gamma", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Logger = zap.New(core)
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
|
|
|
|||
|
|
@ -91,8 +91,8 @@ func TestTagsWithEveryProviderOffAskNobody(t *testing.T) {
|
|||
}
|
||||
sameKeys(t, "tags", r.Body, "sources")
|
||||
sources := r.Body["sources"].([]any)
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("sources = %v", sources)
|
||||
if len(sources) != 3 {
|
||||
t.Fatalf("sources = %v, want Last.fm, MusicBrainz and the files", sources)
|
||||
}
|
||||
for i, name := range []string{"lastfm", "musicbrainz"} {
|
||||
s := sources[i].(map[string]any)
|
||||
|
|
@ -188,7 +188,7 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
|
|||
|
||||
e := &similarEnv{}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
mbClient := musicbrainz.New(outbound.New(outbound.Options{UserAgentFunc: prov.UserAgentFor(providers.MusicBrainz, "1.2.3")}), remote.URL+"/ws/2/")
|
||||
|
|
@ -232,3 +232,30 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The genres of my own files are always there, and come from nobody.
|
||||
func TestTagsIncludeTheGenresOfMyOwnFiles(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'grunge' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'rock' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
files := r.Body["sources"].([]any)[2].(map[string]any)
|
||||
sameKeys(t, "files", files, "provider", "provider_status", "stale", "retrieved_at", "attribution", "tags")
|
||||
tags := files["tags"].([]any)
|
||||
if files["provider"] != "files" || files["provider_status"] != "ok" || files["stale"] != false || len(tags) != 2 {
|
||||
t.Fatalf("files = %v", files)
|
||||
}
|
||||
if first := tags[0].(map[string]any); first["name"] != "grunge" || first["weight"] != float64(1) || first["genre"] != true {
|
||||
t.Errorf("first genre = %v", first)
|
||||
}
|
||||
if v, present := files["retrieved_at"]; !present || v != nil {
|
||||
t.Errorf("retrieved_at = %#v, want an explicit null: nothing was read from anywhere", v)
|
||||
}
|
||||
if e.lf.calls+e.mb.calls != 0 {
|
||||
t.Error("the genres of my files asked a provider")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -458,8 +458,12 @@ paths:
|
|||
tags: [Scan]
|
||||
operationId: scanLibrary
|
||||
summary: Scan one library
|
||||
description: Admin only. Starts a scan in the background; follow it with `GET /library/scan`.
|
||||
description: |
|
||||
Admin only. Starts a scan in the background; follow it with `GET /library/scan`. With
|
||||
`full=true` every file is read again, not only the new or changed ones.
|
||||
security: [{bearerAuth: []}]
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Full'}
|
||||
responses:
|
||||
'202': {$ref: '#/components/responses/ScanStarted'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
|
|
@ -494,9 +498,16 @@ paths:
|
|||
Admin only. Starts a scan in the background. Only MP3 and FLAC files are read; other audio
|
||||
formats are counted as `unsupported`. Housekeeping folders of NAS boxes (`@Recycle`,
|
||||
`#recycle`, `@eaDir`, `.@__thumb`, `$RECYCLE.BIN`) are skipped.
|
||||
|
||||
A scan reads the tags of files that are new or changed. Send `full=true` to read every file
|
||||
again: that is how MusicBrainz ids and genres reach music that was indexed before they were
|
||||
read. Files keep their identity, so listening history is kept.
|
||||
security: [{bearerAuth: []}]
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Full'}
|
||||
responses:
|
||||
'202': {$ref: '#/components/responses/ScanStarted'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'409':
|
||||
|
|
@ -606,10 +617,14 @@ paths:
|
|||
summary: Tags of an artist, from each provider
|
||||
description: |
|
||||
What each provider says kind of music this artist plays: Last.fm's top tags and
|
||||
MusicBrainz's tags and genres. There is always one entry for each, Last.fm first. The
|
||||
MusicBrainz's tags and genres, and last the genres of the artist's own music files
|
||||
(`provider` is `files`: the genre tags the scan read, each weighted by how many of the
|
||||
playable tracks the artist is credited on have it; it asks nobody, is always `ok` and has a null `retrieved_at`). There is
|
||||
always one entry for each, Last.fm first. The
|
||||
weights are the providers' own numbers, kept as they came and never mixed: Last.fm's is
|
||||
relative to the artist (1 to 100, the top tag is usually 100; a tag with no votes is dropped),
|
||||
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres.
|
||||
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres, and for
|
||||
every tag of the `files` entry, which are genres by definition.
|
||||
|
||||
A provider is only asked when an admin switched it on and its stored answer is missing or
|
||||
old (7 days; a "no such artist" answer is kept 1 day), with the same rules as
|
||||
|
|
@ -937,6 +952,39 @@ paths:
|
|||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/providers/{name}/data:
|
||||
parameters:
|
||||
- name: name
|
||||
in: path
|
||||
required: true
|
||||
schema: {type: string, enum: [lastfm, musicbrainz]}
|
||||
delete:
|
||||
tags: [Providers]
|
||||
operationId: purgeProviderData
|
||||
summary: Remove what a provider gave
|
||||
description: |
|
||||
Admin only. Removes everything the provider told Sonda: its lists of similar artists, its
|
||||
tags and the records of when it was asked, and then the artists that only that data named
|
||||
and that nobody has files for. Last.fm's terms ask for exactly this when an agreement ends.
|
||||
|
||||
Your own artists, music, listens and settings are not touched, and the provider is not
|
||||
switched off: if it is still on, it is asked again when someone asks. The answer counts what
|
||||
was removed; doing it again removes nothing.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
description: What was removed.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/ProviderPurge'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'404':
|
||||
description: No such provider (`provider_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/listens:
|
||||
get:
|
||||
tags: [Playback]
|
||||
|
|
@ -1035,6 +1083,11 @@ components:
|
|||
Case-insensitive substring of the name or title. Accented letters are matched ignoring
|
||||
case (`BJÖRK` finds `Björk`), but accents are kept (`bjork` does not). At most 200 characters.
|
||||
schema: {type: string, maxLength: 200}
|
||||
Full:
|
||||
name: full
|
||||
in: query
|
||||
description: "`true` reads every file again, not only the new or changed ones. Only `true` and `false` are accepted."
|
||||
schema: {type: boolean, default: false}
|
||||
Limit:
|
||||
name: limit
|
||||
in: query
|
||||
|
|
@ -1389,6 +1442,15 @@ components:
|
|||
maxLength: 200
|
||||
description: A URL or e-mail address, without parentheses. Empty restores the default.
|
||||
|
||||
ProviderPurge:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [similar_artists, tags, artists]
|
||||
properties:
|
||||
similar_artists: {type: integer, description: Similar-artist links removed.}
|
||||
tags: {type: integer, description: Tags removed.}
|
||||
artists: {type: integer, description: "Artists removed because only the removed data named them and nobody has files for them."}
|
||||
|
||||
ProviderTest:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
@ -1414,7 +1476,7 @@ components:
|
|||
additionalProperties: false
|
||||
required: [provider, provider_status, stale, retrieved_at, attribution, tags]
|
||||
properties:
|
||||
provider: {type: string, enum: [lastfm, musicbrainz]}
|
||||
provider: {type: string, enum: [lastfm, musicbrainz, files]}
|
||||
provider_status:
|
||||
type: string
|
||||
enum: [ok, disabled, error, no_id]
|
||||
|
|
|
|||
|
|
@ -200,6 +200,9 @@ func (s *Service) Similar(ctx context.Context, artistID int64, limit int) (Resul
|
|||
case <-ctx.Done():
|
||||
return Result{}, ctx.Err()
|
||||
case r := <-done:
|
||||
if errors.Is(r.Err, ErrNotFound) {
|
||||
return Result{}, ErrNotFound // removed while it was being read
|
||||
}
|
||||
if r.Err != nil {
|
||||
s.log.Warn("similar artists could not be read", zap.String("provider", Provider), zap.Error(r.Err))
|
||||
return result(snap, StatusError, snap.Fetched), nil
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ type fakeStore struct {
|
|||
snaps map[int64]Snapshot
|
||||
saves []saved
|
||||
lastLim int
|
||||
saveErr error // returned by SaveSimilar when set
|
||||
}
|
||||
|
||||
func newStore() *fakeStore {
|
||||
|
|
@ -66,6 +67,9 @@ func (f *fakeStore) Similar(_ context.Context, id int64, _ string, limit int) (S
|
|||
func (f *fakeStore) SaveSimilar(_ context.Context, id int64, provider string, at time.Time, found bool, cands []lastfm.Similar) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.saveErr != nil {
|
||||
return f.saveErr
|
||||
}
|
||||
f.saves = append(f.saves, saved{id, provider, at, found, cands})
|
||||
snap := Snapshot{Fetched: true, NotFound: !found, RetrievedAt: at}
|
||||
for i, c := range cands {
|
||||
|
|
@ -413,3 +417,13 @@ func TestNoPageCanAskForMoreThanIsStored(t *testing.T) {
|
|||
t.Errorf("MaxLimit = %d is above FetchLimit = %d", MaxLimit, FetchLimit)
|
||||
}
|
||||
}
|
||||
|
||||
// An artist removed while its list was being read is not there any more: that is
|
||||
// "not found", not a provider failure.
|
||||
func TestAnArtistRemovedWhileItsListWasReadIsNotFound(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.store.saveErr = ErrNotFound
|
||||
if _, err := r.svc.Similar(context.Background(), 1, 10); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("err = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -25,6 +25,12 @@ const MaxTags = 50
|
|||
// a page waiting for a minute because a client asked about fifty artists.
|
||||
const tagFetchTimeout = 20 * time.Second
|
||||
|
||||
// SourceFiles names the tags that come from the user's own music files. It is not
|
||||
// a provider: it is always there, asks nobody and needs no settings.
|
||||
const SourceFiles = "files"
|
||||
|
||||
const attributionFiles = "From the genre tags of your own music files."
|
||||
|
||||
// StatusNoID says a provider that needs the MusicBrainz id of the artist cannot
|
||||
// be asked, because the id is not known. Sonda never looks one up by name.
|
||||
const StatusNoID = "no_id"
|
||||
|
|
@ -62,13 +68,16 @@ type TagSourceResult struct {
|
|||
Tags []Tag
|
||||
}
|
||||
|
||||
// TagResult has one entry per provider, always in the same order.
|
||||
// TagResult has one entry per provider, always in the same order, and the tags
|
||||
// of the user's own files last.
|
||||
type TagResult struct{ Sources []TagSourceResult }
|
||||
|
||||
// TagStore keeps the tags. Artist returns ErrNotFound for an id the catalog does
|
||||
// not know.
|
||||
type TagStore interface {
|
||||
Artist(ctx context.Context, id int64) (Artist, error)
|
||||
// FileGenres counts the genres of the artist's own playable tracks, most first.
|
||||
FileGenres(ctx context.Context, id int64) ([]Tag, error)
|
||||
Tags(ctx context.Context, id int64, provider string) (TagSnapshot, error)
|
||||
SaveTags(ctx context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error
|
||||
}
|
||||
|
|
@ -134,12 +143,17 @@ func (s *TagService) Tags(ctx context.Context, artistID int64) (TagResult, error
|
|||
out[i], errs[i] = s.one(ctx, artist, p)
|
||||
}()
|
||||
}
|
||||
genres, genErr := s.store.FileGenres(ctx, artistID)
|
||||
wg.Wait()
|
||||
for _, err := range errs {
|
||||
for _, err := range append(errs, genErr) {
|
||||
if err != nil {
|
||||
return TagResult{}, err
|
||||
}
|
||||
}
|
||||
if genres == nil {
|
||||
genres = []Tag{}
|
||||
}
|
||||
out = append(out, TagSourceResult{Provider: SourceFiles, Status: StatusOK, Attribution: attributionFiles, Tags: genres})
|
||||
return TagResult{Sources: out}, nil
|
||||
}
|
||||
|
||||
|
|
@ -185,6 +199,9 @@ func (s *TagService) one(ctx context.Context, a Artist, p tagProvider) (TagSourc
|
|||
case <-ctx.Done():
|
||||
return TagSourceResult{}, ctx.Err()
|
||||
case r := <-done:
|
||||
if errors.Is(r.Err, ErrNotFound) {
|
||||
return TagSourceResult{}, ErrNotFound // removed while it was being read
|
||||
}
|
||||
if r.Err != nil {
|
||||
s.log.Warn("tags could not be read", zap.String("provider", p.name), zap.Error(r.Err))
|
||||
return res(StatusError, snap.Fetched, snap), nil
|
||||
|
|
|
|||
|
|
@ -26,6 +26,9 @@ type tagStore struct {
|
|||
artists map[int64]Artist
|
||||
snaps map[string]TagSnapshot
|
||||
saves []savedTags
|
||||
genres map[int64][]Tag // what the artist's own files say
|
||||
genErr error
|
||||
saveErr error // returned by SaveTags when set
|
||||
}
|
||||
|
||||
func newTagStore() *tagStore {
|
||||
|
|
@ -38,6 +41,12 @@ func newTagStore() *tagStore {
|
|||
}
|
||||
}
|
||||
|
||||
func (f *tagStore) FileGenres(_ context.Context, id int64) ([]Tag, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.genres[id], f.genErr
|
||||
}
|
||||
|
||||
func tkey(id int64, provider string) string { return provider + "/" + string(rune('0'+id)) }
|
||||
|
||||
func (f *tagStore) Artist(_ context.Context, id int64) (Artist, error) {
|
||||
|
|
@ -59,6 +68,9 @@ func (f *tagStore) Tags(_ context.Context, id int64, provider string) (TagSnapsh
|
|||
func (f *tagStore) SaveTags(_ context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
if f.saveErr != nil {
|
||||
return f.saveErr
|
||||
}
|
||||
f.saves = append(f.saves, savedTags{id, provider, at, found, tags})
|
||||
f.snaps[tkey(id, provider)] = TagSnapshot{Fetched: true, NotFound: !found, RetrievedAt: at, Tags: tags}
|
||||
return nil
|
||||
|
|
@ -149,10 +161,10 @@ func TestTagsOfAnUnknownArtistAreNotFoundAndNobodyIsAsked(t *testing.T) {
|
|||
func TestTagsAlwaysListBothProvidersInTheSameOrder(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
res := r.tags(t, 1)
|
||||
if len(res.Sources) != 2 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz {
|
||||
if len(res.Sources) != 3 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz || res.Sources[2].Provider != SourceFiles {
|
||||
t.Fatalf("sources = %+v", res.Sources)
|
||||
}
|
||||
for _, s := range res.Sources {
|
||||
for _, s := range res.Sources[:2] {
|
||||
if s.Status != StatusDisabled || len(s.Tags) != 0 || !s.RetrievedAt.IsZero() || s.Stale || s.Attribution == "" {
|
||||
t.Errorf("%s = %+v", s.Provider, s)
|
||||
}
|
||||
|
|
@ -293,3 +305,39 @@ func TestConcurrentTagLooksShareOneCall(t *testing.T) {
|
|||
t.Errorf("calls = %d, want 1", r.lf.count())
|
||||
}
|
||||
}
|
||||
|
||||
// The genres of the user's own files need no provider: they are always there,
|
||||
// ask nobody, and are computed when asked.
|
||||
func TestTheGenresOfTheFilesAreAlwaysThere(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
r.store.genres = map[int64][]Tag{1: {{Name: "rock", Weight: 3, Genre: true}, {Name: "grunge", Weight: 1, Genre: true}}}
|
||||
res := r.tags(t, 1)
|
||||
files := res.Sources[2]
|
||||
if files.Provider != SourceFiles || files.Status != StatusOK || files.Stale || len(files.Tags) != 2 || files.Tags[0].Name != "rock" || files.Tags[0].Weight != 3 || !files.RetrievedAt.IsZero() || files.Attribution == "" {
|
||||
t.Errorf("files = %+v", files)
|
||||
}
|
||||
if r.lf.count()+r.mb.count() != 0 {
|
||||
t.Error("reading the genres of the files asked a provider")
|
||||
}
|
||||
// An artist without files says nothing, and it is not an error.
|
||||
res = r.tags(t, 2)
|
||||
if f := res.Sources[2]; f.Status != StatusOK || f.Tags == nil || len(f.Tags) != 0 {
|
||||
t.Errorf("files of an artist without any = %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailureReadingFileGenresIsARealError(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
r.store.genErr = errors.New("db down")
|
||||
if _, err := r.svc.Tags(context.Background(), 1); err == nil {
|
||||
t.Fatal("the database failed and nobody was told")
|
||||
}
|
||||
}
|
||||
|
||||
func TestAnArtistRemovedWhileItsTagsWereReadIsNotFound(t *testing.T) {
|
||||
r := newTagRig(true, false)
|
||||
r.store.saveErr = ErrNotFound
|
||||
if _, err := r.svc.Tags(context.Background(), 1); !errors.Is(err, ErrNotFound) {
|
||||
t.Fatalf("err = %v, want ErrNotFound", err)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ func (m *Manager) OnFinish(fn func(Summary, error)) {
|
|||
// Start scans every library in the background. ctx bounds the scan, so pass a
|
||||
// context that lives as long as the application, not an HTTP request.
|
||||
// It returns ErrNoLibraries when none are defined.
|
||||
func (m *Manager) Start(ctx context.Context) error {
|
||||
func (m *Manager) Start(ctx context.Context, opts ...ScanOption) error {
|
||||
libs, err := m.libraries.ListLibraries(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list libraries: %w", err)
|
||||
|
|
@ -65,20 +65,20 @@ func (m *Manager) Start(ctx context.Context) error {
|
|||
if len(libs) == 0 {
|
||||
return ErrNoLibraries
|
||||
}
|
||||
return m.start(ctx, libs)
|
||||
return m.start(ctx, libs, opts)
|
||||
}
|
||||
|
||||
// StartLibrary scans a single library in the background.
|
||||
// It returns ErrLibraryNotFound for an unknown id.
|
||||
func (m *Manager) StartLibrary(ctx context.Context, id int64) error {
|
||||
func (m *Manager) StartLibrary(ctx context.Context, id int64, opts ...ScanOption) error {
|
||||
lib, err := m.libraries.GetLibrary(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return m.start(ctx, []Library{lib})
|
||||
return m.start(ctx, []Library{lib}, opts)
|
||||
}
|
||||
|
||||
func (m *Manager) start(ctx context.Context, libs []Library) error {
|
||||
func (m *Manager) start(ctx context.Context, libs []Library, opts []ScanOption) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.status.Running {
|
||||
|
|
@ -86,12 +86,12 @@ func (m *Manager) start(ctx context.Context, libs []Library) error {
|
|||
}
|
||||
m.status.Running = true
|
||||
m.status.StartedAt = time.Now()
|
||||
go m.run(ctx, libs)
|
||||
go m.run(ctx, libs, opts)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, libs []Library) {
|
||||
sum, err := m.scanner.Scan(ctx, libs)
|
||||
func (m *Manager) run(ctx context.Context, libs []Library, opts []ScanOption) {
|
||||
sum, err := m.scanner.Scan(ctx, libs, opts...)
|
||||
|
||||
m.mu.Lock()
|
||||
m.status.Running = false
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
|
|||
files: []FileInfo{{RelPath: "new.mp3", Size: 10, MTimeUnix: 5}}, // same size+mtime as flaky.mp3
|
||||
failedFiles: map[string]bool{"flaky.mp3": true},
|
||||
}
|
||||
p := makePlan(w, existing)
|
||||
p := makePlan(w, existing, scanOptions{})
|
||||
|
||||
for _, id := range p.missing {
|
||||
if id == 1 {
|
||||
|
|
@ -29,3 +29,43 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
|
|||
t.Errorf("parse = %+v, want new.mp3 as a new file", p.parse)
|
||||
}
|
||||
}
|
||||
|
||||
// A full rescan reads every file again, whether or not it changed, so what the
|
||||
// reader learned since (ids, genres) reaches files that are already indexed.
|
||||
func TestFullPlanReadsEveryPresentFileAgain(t *testing.T) {
|
||||
existing := []ExistingFile{
|
||||
{ID: 1, FileInfo: FileInfo{RelPath: "same.mp3", Size: 10, MTimeUnix: 5}},
|
||||
{ID: 2, FileInfo: FileInfo{RelPath: "changed.mp3", Size: 20, MTimeUnix: 6}},
|
||||
{ID: 3, FileInfo: FileInfo{RelPath: "back.mp3", Size: 30, MTimeUnix: 7}, Missing: true},
|
||||
{ID: 4, FileInfo: FileInfo{RelPath: "gone.mp3", Size: 40, MTimeUnix: 8}},
|
||||
}
|
||||
w := walkResult{files: []FileInfo{
|
||||
{RelPath: "same.mp3", Size: 10, MTimeUnix: 5},
|
||||
{RelPath: "changed.mp3", Size: 21, MTimeUnix: 6},
|
||||
{RelPath: "back.mp3", Size: 30, MTimeUnix: 7},
|
||||
{RelPath: "new.mp3", Size: 50, MTimeUnix: 9},
|
||||
}}
|
||||
|
||||
normal := makePlan(w, existing, scanOptions{})
|
||||
if len(normal.parse) != 2 || normal.unchanged != 1 || len(normal.restore) != 1 {
|
||||
t.Fatalf("normal plan: parse=%d unchanged=%d restore=%d", len(normal.parse), normal.unchanged, len(normal.restore))
|
||||
}
|
||||
|
||||
full := makePlan(w, existing, scanOptions{full: true})
|
||||
ids := map[int64]bool{}
|
||||
for _, j := range full.parse {
|
||||
ids[j.existingID] = true
|
||||
}
|
||||
if len(full.parse) != 4 || !ids[1] || !ids[2] || !ids[3] || !ids[0] {
|
||||
t.Errorf("full plan parses %+v, want the four present files (three known, one new)", full.parse)
|
||||
}
|
||||
// A file that came back is restored even if reading it fails this time, as in a
|
||||
// normal scan, and it is read as well.
|
||||
if full.unchanged != 0 || len(full.restore) != 1 || full.restore[0] != 3 {
|
||||
t.Errorf("full plan: unchanged=%d restore=%v, want only back.mp3 restored", full.unchanged, full.restore)
|
||||
}
|
||||
// What is gone is still gone.
|
||||
if len(full.missing) != 1 || full.missing[0] != 4 {
|
||||
t.Errorf("missing = %v, want only gone.mp3", full.missing)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,15 +59,35 @@ func NewScanner(store Store, reader TagReader, hub *Hub, workers int) *Scanner {
|
|||
return &Scanner{store: store, reader: reader, hub: hub, workers: workers, now: time.Now}
|
||||
}
|
||||
|
||||
// ScanOption changes how a scan runs.
|
||||
type ScanOption func(*scanOptions)
|
||||
|
||||
type scanOptions struct{ full bool }
|
||||
|
||||
// FullRescan makes a scan read the tags of every file again, not only of the files
|
||||
// that are new or changed. It is how what the reader learned since (MusicBrainz
|
||||
// ids, genres) reaches music that was indexed before. Nothing else changes: files
|
||||
// keep their identity, so listening history is kept.
|
||||
func FullRescan(o *scanOptions) { o.full = true }
|
||||
|
||||
func buildOptions(opts []ScanOption) scanOptions {
|
||||
var o scanOptions
|
||||
for _, f := range opts {
|
||||
f(&o)
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Scan indexes every library in turn. An unreadable library is reported in the
|
||||
// returned error and the other libraries are still scanned.
|
||||
func (s *Scanner) Scan(ctx context.Context, libs []Library) (Summary, error) {
|
||||
func (s *Scanner) Scan(ctx context.Context, libs []Library, opts ...ScanOption) (Summary, error) {
|
||||
o := buildOptions(opts)
|
||||
total := Summary{Libraries: len(libs)}
|
||||
s.publish(Event{Type: ScanStarted})
|
||||
|
||||
var errs []error
|
||||
for _, lib := range libs {
|
||||
sum, err := s.scanLibrary(ctx, lib)
|
||||
sum, err := s.scanLibrary(ctx, lib, o)
|
||||
total.add(sum)
|
||||
if err != nil {
|
||||
errs = append(errs, fmt.Errorf("%s: %w", lib.Name, err))
|
||||
|
|
@ -102,7 +122,7 @@ type parseResult struct {
|
|||
err error
|
||||
}
|
||||
|
||||
func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error) {
|
||||
func (s *Scanner) scanLibrary(ctx context.Context, lib Library, o scanOptions) (Summary, error) {
|
||||
var sum Summary
|
||||
|
||||
walked, err := s.walk(ctx, lib, &sum)
|
||||
|
|
@ -126,7 +146,7 @@ func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error)
|
|||
"refusing to mark them missing", n)
|
||||
}
|
||||
}
|
||||
p := makePlan(walked, existing)
|
||||
p := makePlan(walked, existing, o)
|
||||
sum.Unchanged = p.unchanged
|
||||
|
||||
if err := s.parseAndSave(ctx, lib, p.parse, &sum); err != nil {
|
||||
|
|
@ -348,7 +368,7 @@ type fileKey struct{ size, mtime int64 }
|
|||
// A new file and a vanished file with the same size and modification time are
|
||||
// treated as a rename only when each is the sole candidate for that pair, so
|
||||
// look-alike files are never guessed.
|
||||
func makePlan(w walkResult, existing []ExistingFile) plan {
|
||||
func makePlan(w walkResult, existing []ExistingFile, o scanOptions) plan {
|
||||
var p plan
|
||||
byPath := make(map[string]ExistingFile, len(existing))
|
||||
for _, e := range existing {
|
||||
|
|
@ -364,9 +384,14 @@ func makePlan(w walkResult, existing []ExistingFile) plan {
|
|||
continue
|
||||
}
|
||||
seen[f.RelPath] = true
|
||||
changed := e.Size != f.Size || e.MTimeUnix != f.MTimeUnix
|
||||
switch {
|
||||
case e.Size != f.Size || e.MTimeUnix != f.MTimeUnix:
|
||||
case changed || o.full:
|
||||
p.parse = append(p.parse, parseJob{file: f, existingID: e.ID})
|
||||
if e.Missing && !changed {
|
||||
// Back, and read again. It is back even if reading fails this time.
|
||||
p.restore = append(p.restore, e.ID)
|
||||
}
|
||||
case e.Missing:
|
||||
p.restore = append(p.restore, e.ID)
|
||||
default:
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue