feat(discovery): remove provider data, genres from my files, full rescans
- feat: DELETE /api/v1/providers/{name}/data (admin) removes everything a
provider gave: its lists of similar artists, its tags and the records of
when it was asked, then the artists that only that data named and that
nobody has files for. Last.fm's terms ask for this when an agreement ends.
The user's own artists, music, listens and settings stay, the provider is not
switched off, and a second call removes nothing
- feat: a discovered artist that no list names any more is removed when the
list that named it is replaced, together with its tags, so they do not pile
up. Real artists, and any artist that another list still names, are kept
- feat: the scan reads the GENRE tag of each file, offline (several genres
separated by ";", lower case, at most five per file), and
GET /artists/{id}/tags gets a third entry, files: the genres of the artist's
own playable tracks, each weighted by how many tracks have it. It asks nobody
and is always there
- feat: POST /library/scan and POST /libraries/{id}/scan take full=true to read
every file again, not only the new or changed ones, so MusicBrainz ids and
genres reach music that was indexed before. Files keep their identity, so
listening history is kept. Anything but true or false is a 400
- test: the plan of a full scan, the scanner and the manager; the purge on both
databases, checked by removing the rule that spares real artists; the
cleanup of replaced lists; genres per file and per artist
- docs: README, OpenAPI and the todo list updated
This commit is contained in:
parent
146e1b1bf0
commit
1857bb9c53
30 changed files with 979 additions and 56 deletions
14
README.md
14
README.md
|
|
@ -15,12 +15,12 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Streaming.** `GET` and `HEAD` with `Range` support, so players can seek. Opening or prefetching a stream does not record a listen.
|
||||
- **API documentation.** The full API is described in OpenAPI and browsable with Swagger UI, served by Sonda itself. The UI is embedded in the binary, so opening it contacts no other server.
|
||||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
|
||||
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights. See [External providers](#external-providers).
|
||||
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
|
||||
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.
|
||||
|
||||
## What is not there yet
|
||||
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Genres from your own files, and the removal of provider data on request.
|
||||
Recommendations and radio queues, a web client, OpenSubsonic compatibility, transcoding, and the ListenBrainz integration. Recommendations from all this data (scoring) are next.
|
||||
|
||||
## Requirements
|
||||
|
||||
|
|
@ -200,7 +200,11 @@ The key is stored in the database, in clear text, and is never returned by the A
|
|||
|
||||
**Tags.** `GET /api/v1/artists/{id}/tags` asks each provider that is on, at the same time, and keeps the answers with the same rules (7 days, 1 for an unknown artist, old data if a provider fails). Each provider has its own entry with its own weights: Last.fm's are relative to the artist (1 to 100), MusicBrainz's are counts of votes, and they are never mixed. MusicBrainz also marks the tags it lists as genres. Its tags and genres are CC BY-NC-SA, so show the `attribution` that comes with them.
|
||||
|
||||
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. The scan reads a file's tags when the file is new or changed, so to pick up ids for music that is already indexed, the files have to be touched or re-added.
|
||||
**Genres from your own files.** The scan also reads the genre tag of each file (several genres separated by `;`, at most five per file) and `GET /artists/{id}/tags` adds a `files` entry: the genres of the artist's playable tracks, each weighted by how many tracks have it. It needs no provider, asks nobody, and is always there.
|
||||
|
||||
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. A normal scan reads a file's tags when the file is new or changed. To pick up ids and genres for music that is already indexed, scan with `full=true` (`POST /library/scan?full=true`, or `POST /libraries/{id}/scan?full=true`): every file is read again, keeps its identity, and your listening history is untouched.
|
||||
|
||||
**Taking it back.** `DELETE /api/v1/providers/{name}/data` (admin) removes everything a provider gave: its similar artists, its tags and the records of when it was asked, and then the artists that only that data named and that you have no files for. Last.fm's terms ask for this when an agreement ends. Your own artists, music and listens stay, and the provider is not switched off. An artist that no list names any more is also removed when the list that named it is replaced, so they do not pile up.
|
||||
|
||||
The artist is sent to Last.fm by its MusicBrainz id when your files gave one (if Last.fm does not know the id, by name), otherwise by its name exactly as it is written in your library, and Sonda never lets Last.fm correct it into another artist. Names and ids from Last.fm are never written onto your own artists.
|
||||
|
||||
|
|
@ -224,8 +228,8 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
|
|||
| signed in | `GET /listens` | Your own history, newest first, with `counted` per event. Query: `limit`, `offset`, `counted=true`. |
|
||||
| signed in | `GET /library/scan` | Scan status and the last result. |
|
||||
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test` | See, set up and check Last.fm and MusicBrainz. The API key is write-only. |
|
||||
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. `?full=true` reads every file again. |
|
||||
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test`, `DELETE /providers/{name}/data` | See, set up and check Last.fm and MusicBrainz, and remove what they gave. The API key is write-only. |
|
||||
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |
|
||||
|
||||
A library path must be an absolute, existing folder, never the filesystem root, and never inside or around another library. Search is a case-insensitive substring match: `BJÖRK` finds `Björk`, but accents are kept, so `bjork` does not.
|
||||
|
|
|
|||
|
|
@ -128,6 +128,9 @@ func run() error {
|
|||
return musicbrainzClient.Probe(ctx)
|
||||
})
|
||||
discoveryRepo := storage.NewDiscoveryRepo(db)
|
||||
providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
|
||||
return discoveryRepo.Purge(ctx, name)
|
||||
})
|
||||
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
|
||||
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
|
||||
providers.LastFM: discovery.LastfmTags{Client: lastfmClient},
|
||||
|
|
|
|||
|
|
@ -15,9 +15,20 @@ type scanControl struct {
|
|||
m *library.Manager
|
||||
}
|
||||
|
||||
func (c scanControl) Start() error { return c.m.Start(c.ctx) }
|
||||
func (c scanControl) StartLibrary(id int64) error { return c.m.StartLibrary(c.ctx, id) }
|
||||
func (c scanControl) Status() library.Status { return c.m.Status() }
|
||||
func (c scanControl) Start(full bool) error { return c.m.Start(c.ctx, scanOptions(full)...) }
|
||||
|
||||
func (c scanControl) StartLibrary(id int64, full bool) error {
|
||||
return c.m.StartLibrary(c.ctx, id, scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (c scanControl) Status() library.Status { return c.m.Status() }
|
||||
|
||||
func scanOptions(full bool) []library.ScanOption {
|
||||
if full {
|
||||
return []library.ScanOption{library.FullRescan}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// logScanResults writes one log line per finished scan. File paths are left
|
||||
// out on purpose: audio paths are private.
|
||||
|
|
|
|||
|
|
@ -75,7 +75,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
}
|
||||
|
||||
tokens := auth.NewTokenIssuer(secret, time.Hour)
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
deps := Deps{
|
||||
Logger: zap.NewNop(),
|
||||
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, 24*time.Hour),
|
||||
|
|
@ -92,6 +92,16 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
return NewRouter(deps), db
|
||||
}
|
||||
|
||||
// newProviders builds the provider service the way main does, with the real way
|
||||
// of removing provider data.
|
||||
func newProviders(db *bun.DB) *providers.Service {
|
||||
svc := providers.NewService(storage.NewProviderRepo(db))
|
||||
svc.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
|
||||
return storage.NewDiscoveryRepo(db).Purge(ctx, name)
|
||||
})
|
||||
return svc
|
||||
}
|
||||
|
||||
// noFiles is a streaming source with nothing to play.
|
||||
type noFiles struct{}
|
||||
|
||||
|
|
|
|||
|
|
@ -80,9 +80,22 @@ func (r *contractReader) Read(path string) (library.Metadata, error) {
|
|||
// scanAdapter lets the API start scans on a real manager.
|
||||
type scanAdapter struct{ m *library.Manager }
|
||||
|
||||
func (a scanAdapter) Start() error { return a.m.Start(context.Background()) }
|
||||
func (a scanAdapter) StartLibrary(id int64) error { return a.m.StartLibrary(context.Background(), id) }
|
||||
func (a scanAdapter) Status() library.Status { return a.m.Status() }
|
||||
func (a scanAdapter) Start(full bool) error {
|
||||
return a.m.Start(context.Background(), scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (a scanAdapter) StartLibrary(id int64, full bool) error {
|
||||
return a.m.StartLibrary(context.Background(), id, scanOptions(full)...)
|
||||
}
|
||||
|
||||
func (a scanAdapter) Status() library.Status { return a.m.Status() }
|
||||
|
||||
func scanOptions(full bool) []library.ScanOption {
|
||||
if full {
|
||||
return []library.ScanOption{library.FullRescan}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type contract struct {
|
||||
lastfm *fakeLastfm
|
||||
|
|
@ -102,7 +115,7 @@ func newContract(t *testing.T) *contract {
|
|||
tagsMB := &fakeTags{tags: []discovery.Tag{{Name: "grunge", Weight: 12, Genre: true}}}
|
||||
lf := &fakeLastfm{result: []lastfm.Similar{{Name: "Contract Neighbour", MBID: "99999999-0000-0000-0000-000000000009", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
prov.SetChecker(providers.LastFM, func(context.Context, string) error { return nil })
|
||||
d.Providers = prov
|
||||
prov.SetChecker(providers.MusicBrainz, func(context.Context, string) error { return nil })
|
||||
|
|
@ -339,6 +352,7 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
close(gate)
|
||||
c.scanDone(token)
|
||||
|
||||
c.expect(c.call("POST", "/library/scan?full=maybe", token, nil), 400, "scan with a bad parameter", "invalid_input")
|
||||
c.expect(c.call("POST", "/library/scan", token, nil), 202, "scan everything")
|
||||
c.scanDone(token)
|
||||
status := c.expect(c.call("GET", "/library/scan", token, nil), 200, "scan status")
|
||||
|
|
@ -347,6 +361,12 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
t.Fatalf("second scan = %v, want 3 files in 2 libraries, all unchanged", last)
|
||||
}
|
||||
|
||||
c.expect(c.call("POST", "/library/scan?full=true", token, nil), 202, "scan everything again, in full")
|
||||
c.scanDone(token)
|
||||
if full := c.expect(c.call("GET", "/library/scan", token, nil), 200, "status after a full scan")["last"].(map[string]any); full["updated"] != float64(3) || full["unchanged"] != float64(0) {
|
||||
t.Fatalf("full scan = %v, want the 3 files read again", full)
|
||||
}
|
||||
|
||||
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 204, "delete library")
|
||||
c.expect(c.call("DELETE", "/libraries/"+extraID, token, nil), 404, "delete library twice", "library_not_found")
|
||||
|
||||
|
|
@ -423,10 +443,15 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
c.expect(c.call("GET", "/artists/"+artistID+"/similar?limit=abc", token, nil), 400, "similar with a bad limit", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/musicbrainz", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "set up musicbrainz")
|
||||
tags := c.expect(c.call("GET", "/artists/"+artistID+"/tags", token, nil), 200, "tags")["sources"].([]any)
|
||||
if len(tags) != 2 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" {
|
||||
t.Fatalf("tags = %v, want Last.fm ok and MusicBrainz without an id", tags)
|
||||
if len(tags) != 3 || tags[0].(map[string]any)["provider_status"] != "ok" || tags[1].(map[string]any)["provider_status"] != "no_id" || tags[2].(map[string]any)["provider"] != "files" {
|
||||
t.Fatalf("tags = %v, want Last.fm ok, MusicBrainz without an id and the files", tags)
|
||||
}
|
||||
c.expect(c.call("GET", "/artists/999/tags", token, nil), 404, "tags of an unknown artist", "artist_not_found")
|
||||
purged := c.expect(c.call("DELETE", "/providers/lastfm/data", token, nil), 200, "purge a provider's data")
|
||||
if purged["tags"] != float64(1) || purged["similar_artists"] != float64(1) {
|
||||
t.Fatalf("purge = %v", purged)
|
||||
}
|
||||
c.expect(c.call("DELETE", "/providers/spotify/data", token, nil), 404, "purge an unknown provider", "provider_not_found")
|
||||
c.expect(c.call("GET", "/artists/abc/tags", token, nil), 400, "tags of a bad id", "invalid_input")
|
||||
|
||||
// ---- playback ----
|
||||
|
|
|
|||
|
|
@ -35,9 +35,27 @@ type scanStatusJSON struct {
|
|||
LastError string `json:"last_error,omitempty"`
|
||||
}
|
||||
|
||||
// fullParam reads ?full=. Only "true" and "false" are accepted, and it answers
|
||||
// 400 itself otherwise: a typo must not quietly start a scan of another kind.
|
||||
func fullParam(c *gin.Context) (full, ok bool) {
|
||||
raw, present := c.GetQuery("full")
|
||||
switch {
|
||||
case !present, raw == "false":
|
||||
return false, true
|
||||
case raw == "true":
|
||||
return true, true
|
||||
}
|
||||
writeError(c, http.StatusBadRequest, "invalid_input", "full must be true or false")
|
||||
return false, false
|
||||
}
|
||||
|
||||
// startScan launches a background scan of the configured roots.
|
||||
func (h *libraryHandlers) startScan(c *gin.Context) {
|
||||
writeScanStarted(c, h.scan.Start())
|
||||
full, ok := fullParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.Start(full))
|
||||
}
|
||||
|
||||
// startLibraryScan launches a background scan of one library.
|
||||
|
|
@ -46,7 +64,11 @@ func (h *libraryHandlers) startLibraryScan(c *gin.Context) {
|
|||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.StartLibrary(id))
|
||||
full, ok := fullParam(c)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
writeScanStarted(c, h.scan.StartLibrary(id, full))
|
||||
}
|
||||
|
||||
func writeScanStarted(c *gin.Context, err error) {
|
||||
|
|
|
|||
|
|
@ -90,3 +90,12 @@ func (h *providersHandlers) test(c *gin.Context) {
|
|||
}
|
||||
c.JSON(http.StatusOK, gin.H{"ok": res.OK, "status": res.Status})
|
||||
}
|
||||
|
||||
func (h *providersHandlers) purge(c *gin.Context) {
|
||||
res, err := h.svc.Purge(c.Request.Context(), c.Param("name"))
|
||||
if err != nil {
|
||||
writeProviderError(c, err)
|
||||
return
|
||||
}
|
||||
c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "artists": res.Artists})
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,10 +13,11 @@ type fakeScan struct {
|
|||
noLibs bool
|
||||
starts int
|
||||
lastOnly int64 // library of the last StartLibrary call
|
||||
lastFull bool // whether the last start asked for a full rescan
|
||||
status library.Status
|
||||
}
|
||||
|
||||
func (f *fakeScan) Start() error {
|
||||
func (f *fakeScan) Start(full bool) error {
|
||||
if f.noLibs {
|
||||
return library.ErrNoLibraries
|
||||
}
|
||||
|
|
@ -25,10 +26,11 @@ func (f *fakeScan) Start() error {
|
|||
}
|
||||
f.running = true
|
||||
f.starts++
|
||||
f.lastFull = full
|
||||
return nil
|
||||
}
|
||||
|
||||
func (f *fakeScan) StartLibrary(id int64) error {
|
||||
func (f *fakeScan) StartLibrary(id int64, full bool) error {
|
||||
if id == 999 {
|
||||
return library.ErrLibraryNotFound
|
||||
}
|
||||
|
|
@ -38,6 +40,7 @@ func (f *fakeScan) StartLibrary(id int64) error {
|
|||
f.running = true
|
||||
f.starts++
|
||||
f.lastOnly = id
|
||||
f.lastFull = full
|
||||
return nil
|
||||
}
|
||||
|
||||
|
|
@ -96,3 +99,37 @@ func TestScanStatus(t *testing.T) {
|
|||
t.Errorf("last = %v", last)
|
||||
}
|
||||
}
|
||||
|
||||
// A full rescan is asked for explicitly; anything else in the parameter is a mistake.
|
||||
func TestScanCanBeAskedToReadEveryFileAgain(t *testing.T) {
|
||||
scan := &fakeScan{}
|
||||
h := newTestAPIWithScan(t, scan)
|
||||
tok := registerAndLogin(t, h)
|
||||
|
||||
for _, tt := range []struct {
|
||||
query string
|
||||
full bool
|
||||
}{{"", false}, {"?full=false", false}, {"?full=true", true}} {
|
||||
scan.running = false
|
||||
if r := do(t, h, "POST", "/api/v1/library/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full {
|
||||
t.Errorf("scan%s: %d full=%v, want %v", tt.query, r.Code, scan.lastFull, tt.full)
|
||||
}
|
||||
scan.running = false
|
||||
if r := do(t, h, "POST", "/api/v1/libraries/7/scan"+tt.query, tok, nil); r.Code != http.StatusAccepted || scan.lastFull != tt.full || scan.lastOnly != 7 {
|
||||
t.Errorf("library scan%s: %d full=%v only=%d", tt.query, r.Code, scan.lastFull, scan.lastOnly)
|
||||
}
|
||||
}
|
||||
|
||||
starts := scan.starts
|
||||
scan.running = false
|
||||
for _, q := range []string{"?full=maybe", "?full=1", "?full=TRUE", "?full="} {
|
||||
for _, path := range []string{"/api/v1/library/scan", "/api/v1/libraries/7/scan"} {
|
||||
if r := do(t, h, "POST", path+q, tok, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
|
||||
t.Errorf("%s%s: %d %v, want 400 invalid_input", path, q, r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
if scan.starts != starts {
|
||||
t.Error("a scan started with a bad parameter")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ import (
|
|||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
|
|
@ -13,7 +14,6 @@ import (
|
|||
"go.uber.org/zap/zaptest/observer"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/storage"
|
||||
)
|
||||
|
||||
const lastfmKey = "0123456789abcdef0123456789abcdef"
|
||||
|
|
@ -193,7 +193,7 @@ func TestProviderTest(t *testing.T) {
|
|||
var asked string
|
||||
var answer error
|
||||
h, _ := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
prov.SetChecker(providers.LastFM, func(_ context.Context, key string) error { asked = key; return answer })
|
||||
d.Providers = prov
|
||||
})
|
||||
|
|
@ -231,3 +231,64 @@ func TestProviderTest(t *testing.T) {
|
|||
t.Errorf("a provider without a test: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
// ---- taking back a provider's data ----
|
||||
|
||||
func TestPurgeIsAdminOnly(t *testing.T) {
|
||||
h := newTestAPI(t)
|
||||
admin := registerAndLogin(t, h)
|
||||
_, bob := addUser(t, h, admin, "bob", false)
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", "", nil); r.Code != http.StatusUnauthorized {
|
||||
t.Errorf("no token: %d", r.Code)
|
||||
}
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/lastfm/data", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
|
||||
t.Errorf("normal user: %d %v", r.Code, r.Body)
|
||||
}
|
||||
if r := do(t, h, "DELETE", "/api/v1/providers/spotify/data", admin, nil); r.Code != http.StatusNotFound || errCode(r) != "provider_not_found" {
|
||||
t.Errorf("unknown provider: %d %v", r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPurgeRemovesWhatTheProviderGaveAndNothingOfMine(t *testing.T) {
|
||||
e := newSimilarEnv(t)
|
||||
e.enableLastfm(t)
|
||||
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil) // names Beta (mine) and Gamma (discovered)
|
||||
gamma := int64(r.Body["items"].([]any)[1].(map[string]any)["artist"].(map[string]any)["id"].(float64))
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusOK {
|
||||
t.Fatalf("the discovered artist is not there to begin with: %d", a.Code)
|
||||
}
|
||||
|
||||
p := do(t, e.h, "DELETE", "/api/v1/providers/lastfm/data", e.admin, nil)
|
||||
if p.Code != http.StatusOK {
|
||||
t.Fatalf("%d %v", p.Code, p.Body)
|
||||
}
|
||||
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "artists")
|
||||
if p.Body["similar_artists"] != float64(2) || p.Body["tags"] != float64(0) || p.Body["artists"] != float64(1) {
|
||||
t.Errorf("purge = %v, want 2 similar artists, no tags and the one artist Gamma", p.Body)
|
||||
}
|
||||
raw, _ := json.Marshal(p.Body)
|
||||
if strings.Contains(string(raw), lastfmKey) {
|
||||
t.Fatal("the answer contains the API key")
|
||||
}
|
||||
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(gamma, 10), e.admin, nil); a.Code != http.StatusNotFound {
|
||||
t.Errorf("the discovered artist survived: %d", a.Code)
|
||||
}
|
||||
for _, id := range []int64{e.alpha, e.beta} {
|
||||
if a := do(t, e.h, "GET", "/api/v1/artists/"+strconv.FormatInt(id, 10), e.admin, nil); a.Code != http.StatusOK {
|
||||
t.Errorf("one of my artists was touched: %d", a.Code)
|
||||
}
|
||||
}
|
||||
// The provider is as it was: still on, still with its key.
|
||||
list := do(t, e.h, "GET", "/api/v1/providers", e.admin, nil).Body["providers"].([]any)[0].(map[string]any)
|
||||
if list["enabled"] != true || list["has_api_key"] != true {
|
||||
t.Errorf("the purge changed the provider: %v", list)
|
||||
}
|
||||
// And the next look asks again, since nothing is stored.
|
||||
calls := e.source.count()
|
||||
again := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if again.Code != http.StatusOK || e.source.count() != calls+1 {
|
||||
t.Errorf("after the purge: %d, calls %d -> %d", again.Code, calls, e.source.count())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -50,8 +50,8 @@ type streamService interface {
|
|||
}
|
||||
|
||||
type scanService interface {
|
||||
Start() error
|
||||
StartLibrary(id int64) error
|
||||
Start(full bool) error
|
||||
StartLibrary(id int64, full bool) error
|
||||
Status() library.Status
|
||||
}
|
||||
|
||||
|
|
@ -59,6 +59,7 @@ type providerService interface {
|
|||
List(ctx context.Context) ([]providers.Info, error)
|
||||
Update(ctx context.Context, name string, p providers.Patch) (providers.Info, error)
|
||||
Test(ctx context.Context, name string) (providers.TestResult, error)
|
||||
Purge(ctx context.Context, name string) (providers.Purged, error)
|
||||
}
|
||||
|
||||
type tagService interface {
|
||||
|
|
@ -160,6 +161,7 @@ func NewRouter(d Deps) *gin.Engine {
|
|||
admin.GET("/providers", prov.list)
|
||||
admin.PUT("/providers/:name", prov.update)
|
||||
admin.POST("/providers/:name/test", prov.test)
|
||||
admin.DELETE("/providers/:name/data", prov.purge)
|
||||
u := &usersHandlers{svc: d.Auth}
|
||||
admin.GET("/users", u.list)
|
||||
admin.POST("/users", u.create)
|
||||
|
|
|
|||
|
|
@ -80,7 +80,7 @@ func newSimilarEnvWith(t *testing.T, more func(*Deps, *bun.DB, *providers.Servic
|
|||
}}
|
||||
e := &similarEnv{source: src}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
|
|
@ -295,7 +295,7 @@ func TestWhatLeavesTheServerIsTheArtistAndTheKeyOnly(t *testing.T) {
|
|||
|
||||
var alpha int64
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
client := lastfm.New(outbound.New(outbound.Options{UserAgent: providers.UserAgent("1.2.3", "")}), remote.URL)
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), client, prov, zap.NewNop())
|
||||
|
|
@ -373,7 +373,7 @@ func TestALeavingClientIsNotAServerError(t *testing.T) {
|
|||
core, logs := observer.New(zap.DebugLevel)
|
||||
src := &fakeLastfm{release: make(chan struct{}), result: []lastfm.Similar{{Name: "Gamma", Match: 1}}}
|
||||
h, db := newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Logger = zap.New(core)
|
||||
d.Discovery = discovery.NewService(storage.NewDiscoveryRepo(db), src, prov, zap.NewNop())
|
||||
|
|
|
|||
|
|
@ -91,8 +91,8 @@ func TestTagsWithEveryProviderOffAskNobody(t *testing.T) {
|
|||
}
|
||||
sameKeys(t, "tags", r.Body, "sources")
|
||||
sources := r.Body["sources"].([]any)
|
||||
if len(sources) != 2 {
|
||||
t.Fatalf("sources = %v", sources)
|
||||
if len(sources) != 3 {
|
||||
t.Fatalf("sources = %v, want Last.fm, MusicBrainz and the files", sources)
|
||||
}
|
||||
for i, name := range []string{"lastfm", "musicbrainz"} {
|
||||
s := sources[i].(map[string]any)
|
||||
|
|
@ -188,7 +188,7 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
|
|||
|
||||
e := &similarEnv{}
|
||||
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
|
||||
prov := providers.NewService(storage.NewProviderRepo(db))
|
||||
prov := newProviders(db)
|
||||
d.Providers = prov
|
||||
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
|
||||
mbClient := musicbrainz.New(outbound.New(outbound.Options{UserAgentFunc: prov.UserAgentFor(providers.MusicBrainz, "1.2.3")}), remote.URL+"/ws/2/")
|
||||
|
|
@ -232,3 +232,30 @@ func TestWhatLeavesTheServerForMusicBrainzIsTheIDAndTheContact(t *testing.T) {
|
|||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The genres of my own files are always there, and come from nobody.
|
||||
func TestTagsIncludeTheGenresOfMyOwnFiles(t *testing.T) {
|
||||
e := newTagsEnv(t)
|
||||
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'grunge' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := e.db.Exec("INSERT INTO track_genres (track_id, genre) SELECT t.id, 'rock' FROM tracks t WHERE t.title = 'Alpha song'"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
files := r.Body["sources"].([]any)[2].(map[string]any)
|
||||
sameKeys(t, "files", files, "provider", "provider_status", "stale", "retrieved_at", "attribution", "tags")
|
||||
tags := files["tags"].([]any)
|
||||
if files["provider"] != "files" || files["provider_status"] != "ok" || files["stale"] != false || len(tags) != 2 {
|
||||
t.Fatalf("files = %v", files)
|
||||
}
|
||||
if first := tags[0].(map[string]any); first["name"] != "grunge" || first["weight"] != float64(1) || first["genre"] != true {
|
||||
t.Errorf("first genre = %v", first)
|
||||
}
|
||||
if v, present := files["retrieved_at"]; !present || v != nil {
|
||||
t.Errorf("retrieved_at = %#v, want an explicit null: nothing was read from anywhere", v)
|
||||
}
|
||||
if e.lf.calls+e.mb.calls != 0 {
|
||||
t.Error("the genres of my files asked a provider")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -458,8 +458,12 @@ paths:
|
|||
tags: [Scan]
|
||||
operationId: scanLibrary
|
||||
summary: Scan one library
|
||||
description: Admin only. Starts a scan in the background; follow it with `GET /library/scan`.
|
||||
description: |
|
||||
Admin only. Starts a scan in the background; follow it with `GET /library/scan`. With
|
||||
`full=true` every file is read again, not only the new or changed ones.
|
||||
security: [{bearerAuth: []}]
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Full'}
|
||||
responses:
|
||||
'202': {$ref: '#/components/responses/ScanStarted'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
|
|
@ -494,9 +498,16 @@ paths:
|
|||
Admin only. Starts a scan in the background. Only MP3 and FLAC files are read; other audio
|
||||
formats are counted as `unsupported`. Housekeeping folders of NAS boxes (`@Recycle`,
|
||||
`#recycle`, `@eaDir`, `.@__thumb`, `$RECYCLE.BIN`) are skipped.
|
||||
|
||||
A scan reads the tags of files that are new or changed. Send `full=true` to read every file
|
||||
again: that is how MusicBrainz ids and genres reach music that was indexed before they were
|
||||
read. Files keep their identity, so listening history is kept.
|
||||
security: [{bearerAuth: []}]
|
||||
parameters:
|
||||
- {$ref: '#/components/parameters/Full'}
|
||||
responses:
|
||||
'202': {$ref: '#/components/responses/ScanStarted'}
|
||||
'400': {$ref: '#/components/responses/InvalidInput'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'409':
|
||||
|
|
@ -606,7 +617,10 @@ paths:
|
|||
summary: Tags of an artist, from each provider
|
||||
description: |
|
||||
What each provider says kind of music this artist plays: Last.fm's top tags and
|
||||
MusicBrainz's tags and genres. There is always one entry for each, Last.fm first. The
|
||||
MusicBrainz's tags and genres, and last the genres of the artist's own music files
|
||||
(`provider` is `files`: the genre tags the scan read, each weighted by how many playable
|
||||
tracks have it; it asks nobody, is always `ok` and has a null `retrieved_at`). There is
|
||||
always one entry for each, Last.fm first. The
|
||||
weights are the providers' own numbers, kept as they came and never mixed: Last.fm's is
|
||||
relative to the artist (1 to 100, the top tag is usually 100; a tag with no votes is dropped),
|
||||
MusicBrainz's is a count of votes. `genre` is true for a MusicBrainz tag that it also lists among its curated genres.
|
||||
|
|
@ -937,6 +951,39 @@ paths:
|
|||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/providers/{name}/data:
|
||||
parameters:
|
||||
- name: name
|
||||
in: path
|
||||
required: true
|
||||
schema: {type: string, enum: [lastfm, musicbrainz]}
|
||||
delete:
|
||||
tags: [Providers]
|
||||
operationId: purgeProviderData
|
||||
summary: Remove what a provider gave
|
||||
description: |
|
||||
Admin only. Removes everything the provider told Sonda: its lists of similar artists, its
|
||||
tags and the records of when it was asked, and then the artists that only that data named
|
||||
and that nobody has files for. Last.fm's terms ask for exactly this when an agreement ends.
|
||||
|
||||
Your own artists, music, listens and settings are not touched, and the provider is not
|
||||
switched off: if it is still on, it is asked again when someone asks. The answer counts what
|
||||
was removed; doing it again removes nothing.
|
||||
security: [{bearerAuth: []}]
|
||||
responses:
|
||||
'200':
|
||||
description: What was removed.
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/ProviderPurge'}
|
||||
'401': {$ref: '#/components/responses/Unauthorized'}
|
||||
'403': {$ref: '#/components/responses/AdminRequired'}
|
||||
'404':
|
||||
description: No such provider (`provider_not_found`).
|
||||
content:
|
||||
application/json:
|
||||
schema: {$ref: '#/components/schemas/Error'}
|
||||
|
||||
/listens:
|
||||
get:
|
||||
tags: [Playback]
|
||||
|
|
@ -1035,6 +1082,11 @@ components:
|
|||
Case-insensitive substring of the name or title. Accented letters are matched ignoring
|
||||
case (`BJÖRK` finds `Björk`), but accents are kept (`bjork` does not). At most 200 characters.
|
||||
schema: {type: string, maxLength: 200}
|
||||
Full:
|
||||
name: full
|
||||
in: query
|
||||
description: "`true` reads every file again, not only the new or changed ones. Only `true` and `false` are accepted."
|
||||
schema: {type: boolean, default: false}
|
||||
Limit:
|
||||
name: limit
|
||||
in: query
|
||||
|
|
@ -1389,6 +1441,15 @@ components:
|
|||
maxLength: 200
|
||||
description: A URL or e-mail address, without parentheses. Empty restores the default.
|
||||
|
||||
ProviderPurge:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
required: [similar_artists, tags, artists]
|
||||
properties:
|
||||
similar_artists: {type: integer, description: Similar-artist links removed.}
|
||||
tags: {type: integer, description: Tags removed.}
|
||||
artists: {type: integer, description: "Artists removed because only the removed data named them and nobody has files for them."}
|
||||
|
||||
ProviderTest:
|
||||
type: object
|
||||
additionalProperties: false
|
||||
|
|
@ -1414,7 +1475,7 @@ components:
|
|||
additionalProperties: false
|
||||
required: [provider, provider_status, stale, retrieved_at, attribution, tags]
|
||||
properties:
|
||||
provider: {type: string, enum: [lastfm, musicbrainz]}
|
||||
provider: {type: string, enum: [lastfm, musicbrainz, files]}
|
||||
provider_status:
|
||||
type: string
|
||||
enum: [ok, disabled, error, no_id]
|
||||
|
|
|
|||
|
|
@ -25,6 +25,12 @@ const MaxTags = 50
|
|||
// a page waiting for a minute because a client asked about fifty artists.
|
||||
const tagFetchTimeout = 20 * time.Second
|
||||
|
||||
// SourceFiles names the tags that come from the user's own music files. It is not
|
||||
// a provider: it is always there, asks nobody and needs no settings.
|
||||
const SourceFiles = "files"
|
||||
|
||||
const attributionFiles = "From the genre tags of your own music files."
|
||||
|
||||
// StatusNoID says a provider that needs the MusicBrainz id of the artist cannot
|
||||
// be asked, because the id is not known. Sonda never looks one up by name.
|
||||
const StatusNoID = "no_id"
|
||||
|
|
@ -62,13 +68,16 @@ type TagSourceResult struct {
|
|||
Tags []Tag
|
||||
}
|
||||
|
||||
// TagResult has one entry per provider, always in the same order.
|
||||
// TagResult has one entry per provider, always in the same order, and the tags
|
||||
// of the user's own files last.
|
||||
type TagResult struct{ Sources []TagSourceResult }
|
||||
|
||||
// TagStore keeps the tags. Artist returns ErrNotFound for an id the catalog does
|
||||
// not know.
|
||||
type TagStore interface {
|
||||
Artist(ctx context.Context, id int64) (Artist, error)
|
||||
// FileGenres counts the genres of the artist's own playable tracks, most first.
|
||||
FileGenres(ctx context.Context, id int64) ([]Tag, error)
|
||||
Tags(ctx context.Context, id int64, provider string) (TagSnapshot, error)
|
||||
SaveTags(ctx context.Context, id int64, provider string, at time.Time, found bool, tags []Tag) error
|
||||
}
|
||||
|
|
@ -134,12 +143,17 @@ func (s *TagService) Tags(ctx context.Context, artistID int64) (TagResult, error
|
|||
out[i], errs[i] = s.one(ctx, artist, p)
|
||||
}()
|
||||
}
|
||||
genres, genErr := s.store.FileGenres(ctx, artistID)
|
||||
wg.Wait()
|
||||
for _, err := range errs {
|
||||
for _, err := range append(errs, genErr) {
|
||||
if err != nil {
|
||||
return TagResult{}, err
|
||||
}
|
||||
}
|
||||
if genres == nil {
|
||||
genres = []Tag{}
|
||||
}
|
||||
out = append(out, TagSourceResult{Provider: SourceFiles, Status: StatusOK, Attribution: attributionFiles, Tags: genres})
|
||||
return TagResult{Sources: out}, nil
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -26,6 +26,8 @@ type tagStore struct {
|
|||
artists map[int64]Artist
|
||||
snaps map[string]TagSnapshot
|
||||
saves []savedTags
|
||||
genres map[int64][]Tag // what the artist's own files say
|
||||
genErr error
|
||||
}
|
||||
|
||||
func newTagStore() *tagStore {
|
||||
|
|
@ -38,6 +40,12 @@ func newTagStore() *tagStore {
|
|||
}
|
||||
}
|
||||
|
||||
func (f *tagStore) FileGenres(_ context.Context, id int64) ([]Tag, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
return f.genres[id], f.genErr
|
||||
}
|
||||
|
||||
func tkey(id int64, provider string) string { return provider + "/" + string(rune('0'+id)) }
|
||||
|
||||
func (f *tagStore) Artist(_ context.Context, id int64) (Artist, error) {
|
||||
|
|
@ -149,10 +157,10 @@ func TestTagsOfAnUnknownArtistAreNotFoundAndNobodyIsAsked(t *testing.T) {
|
|||
func TestTagsAlwaysListBothProvidersInTheSameOrder(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
res := r.tags(t, 1)
|
||||
if len(res.Sources) != 2 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz {
|
||||
if len(res.Sources) != 3 || res.Sources[0].Provider != providers.LastFM || res.Sources[1].Provider != providers.MusicBrainz || res.Sources[2].Provider != SourceFiles {
|
||||
t.Fatalf("sources = %+v", res.Sources)
|
||||
}
|
||||
for _, s := range res.Sources {
|
||||
for _, s := range res.Sources[:2] {
|
||||
if s.Status != StatusDisabled || len(s.Tags) != 0 || !s.RetrievedAt.IsZero() || s.Stale || s.Attribution == "" {
|
||||
t.Errorf("%s = %+v", s.Provider, s)
|
||||
}
|
||||
|
|
@ -293,3 +301,31 @@ func TestConcurrentTagLooksShareOneCall(t *testing.T) {
|
|||
t.Errorf("calls = %d, want 1", r.lf.count())
|
||||
}
|
||||
}
|
||||
|
||||
// The genres of the user's own files need no provider: they are always there,
|
||||
// ask nobody, and are computed when asked.
|
||||
func TestTheGenresOfTheFilesAreAlwaysThere(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
r.store.genres = map[int64][]Tag{1: {{Name: "rock", Weight: 3, Genre: true}, {Name: "grunge", Weight: 1, Genre: true}}}
|
||||
res := r.tags(t, 1)
|
||||
files := res.Sources[2]
|
||||
if files.Provider != SourceFiles || files.Status != StatusOK || files.Stale || len(files.Tags) != 2 || files.Tags[0].Name != "rock" || files.Tags[0].Weight != 3 || !files.RetrievedAt.IsZero() || files.Attribution == "" {
|
||||
t.Errorf("files = %+v", files)
|
||||
}
|
||||
if r.lf.count()+r.mb.count() != 0 {
|
||||
t.Error("reading the genres of the files asked a provider")
|
||||
}
|
||||
// An artist without files says nothing, and it is not an error.
|
||||
res = r.tags(t, 2)
|
||||
if f := res.Sources[2]; f.Status != StatusOK || f.Tags == nil || len(f.Tags) != 0 {
|
||||
t.Errorf("files of an artist without any = %+v", f)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAFailureReadingFileGenresIsARealError(t *testing.T) {
|
||||
r := newTagRig(false, false)
|
||||
r.store.genErr = errors.New("db down")
|
||||
if _, err := r.svc.Tags(context.Background(), 1); err == nil {
|
||||
t.Fatal("the database failed and nobody was told")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ func (m *Manager) OnFinish(fn func(Summary, error)) {
|
|||
// Start scans every library in the background. ctx bounds the scan, so pass a
|
||||
// context that lives as long as the application, not an HTTP request.
|
||||
// It returns ErrNoLibraries when none are defined.
|
||||
func (m *Manager) Start(ctx context.Context) error {
|
||||
func (m *Manager) Start(ctx context.Context, opts ...ScanOption) error {
|
||||
libs, err := m.libraries.ListLibraries(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("list libraries: %w", err)
|
||||
|
|
@ -65,20 +65,20 @@ func (m *Manager) Start(ctx context.Context) error {
|
|||
if len(libs) == 0 {
|
||||
return ErrNoLibraries
|
||||
}
|
||||
return m.start(ctx, libs)
|
||||
return m.start(ctx, libs, opts)
|
||||
}
|
||||
|
||||
// StartLibrary scans a single library in the background.
|
||||
// It returns ErrLibraryNotFound for an unknown id.
|
||||
func (m *Manager) StartLibrary(ctx context.Context, id int64) error {
|
||||
func (m *Manager) StartLibrary(ctx context.Context, id int64, opts ...ScanOption) error {
|
||||
lib, err := m.libraries.GetLibrary(ctx, id)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return m.start(ctx, []Library{lib})
|
||||
return m.start(ctx, []Library{lib}, opts)
|
||||
}
|
||||
|
||||
func (m *Manager) start(ctx context.Context, libs []Library) error {
|
||||
func (m *Manager) start(ctx context.Context, libs []Library, opts []ScanOption) error {
|
||||
m.mu.Lock()
|
||||
defer m.mu.Unlock()
|
||||
if m.status.Running {
|
||||
|
|
@ -86,12 +86,12 @@ func (m *Manager) start(ctx context.Context, libs []Library) error {
|
|||
}
|
||||
m.status.Running = true
|
||||
m.status.StartedAt = time.Now()
|
||||
go m.run(ctx, libs)
|
||||
go m.run(ctx, libs, opts)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *Manager) run(ctx context.Context, libs []Library) {
|
||||
sum, err := m.scanner.Scan(ctx, libs)
|
||||
func (m *Manager) run(ctx context.Context, libs []Library, opts []ScanOption) {
|
||||
sum, err := m.scanner.Scan(ctx, libs, opts...)
|
||||
|
||||
m.mu.Lock()
|
||||
m.status.Running = false
|
||||
|
|
|
|||
|
|
@ -12,7 +12,7 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
|
|||
files: []FileInfo{{RelPath: "new.mp3", Size: 10, MTimeUnix: 5}}, // same size+mtime as flaky.mp3
|
||||
failedFiles: map[string]bool{"flaky.mp3": true},
|
||||
}
|
||||
p := makePlan(w, existing)
|
||||
p := makePlan(w, existing, scanOptions{})
|
||||
|
||||
for _, id := range p.missing {
|
||||
if id == 1 {
|
||||
|
|
@ -29,3 +29,41 @@ func TestPlanDoesNotFlagFilesWithStatErrors(t *testing.T) {
|
|||
t.Errorf("parse = %+v, want new.mp3 as a new file", p.parse)
|
||||
}
|
||||
}
|
||||
|
||||
// A full rescan reads every file again, whether or not it changed, so what the
|
||||
// reader learned since (ids, genres) reaches files that are already indexed.
|
||||
func TestFullPlanReadsEveryPresentFileAgain(t *testing.T) {
|
||||
existing := []ExistingFile{
|
||||
{ID: 1, FileInfo: FileInfo{RelPath: "same.mp3", Size: 10, MTimeUnix: 5}},
|
||||
{ID: 2, FileInfo: FileInfo{RelPath: "changed.mp3", Size: 20, MTimeUnix: 6}},
|
||||
{ID: 3, FileInfo: FileInfo{RelPath: "back.mp3", Size: 30, MTimeUnix: 7}, Missing: true},
|
||||
{ID: 4, FileInfo: FileInfo{RelPath: "gone.mp3", Size: 40, MTimeUnix: 8}},
|
||||
}
|
||||
w := walkResult{files: []FileInfo{
|
||||
{RelPath: "same.mp3", Size: 10, MTimeUnix: 5},
|
||||
{RelPath: "changed.mp3", Size: 21, MTimeUnix: 6},
|
||||
{RelPath: "back.mp3", Size: 30, MTimeUnix: 7},
|
||||
{RelPath: "new.mp3", Size: 50, MTimeUnix: 9},
|
||||
}}
|
||||
|
||||
normal := makePlan(w, existing, scanOptions{})
|
||||
if len(normal.parse) != 2 || normal.unchanged != 1 || len(normal.restore) != 1 {
|
||||
t.Fatalf("normal plan: parse=%d unchanged=%d restore=%d", len(normal.parse), normal.unchanged, len(normal.restore))
|
||||
}
|
||||
|
||||
full := makePlan(w, existing, scanOptions{full: true})
|
||||
ids := map[int64]bool{}
|
||||
for _, j := range full.parse {
|
||||
ids[j.existingID] = true
|
||||
}
|
||||
if len(full.parse) != 4 || !ids[1] || !ids[2] || !ids[3] || !ids[0] {
|
||||
t.Errorf("full plan parses %+v, want the four present files (three known, one new)", full.parse)
|
||||
}
|
||||
if full.unchanged != 0 || len(full.restore) != 0 {
|
||||
t.Errorf("full plan: unchanged=%d restore=%v, want none: a re-read clears the missing flag itself", full.unchanged, full.restore)
|
||||
}
|
||||
// What is gone is still gone.
|
||||
if len(full.missing) != 1 || full.missing[0] != 4 {
|
||||
t.Errorf("missing = %v, want only gone.mp3", full.missing)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -59,15 +59,35 @@ func NewScanner(store Store, reader TagReader, hub *Hub, workers int) *Scanner {
|
|||
return &Scanner{store: store, reader: reader, hub: hub, workers: workers, now: time.Now}
|
||||
}
|
||||
|
||||
// ScanOption changes how a scan runs.
|
||||
type ScanOption func(*scanOptions)
|
||||
|
||||
type scanOptions struct{ full bool }
|
||||
|
||||
// FullRescan makes a scan read the tags of every file again, not only of the files
|
||||
// that are new or changed. It is how what the reader learned since (MusicBrainz
|
||||
// ids, genres) reaches music that was indexed before. Nothing else changes: files
|
||||
// keep their identity, so listening history is kept.
|
||||
func FullRescan(o *scanOptions) { o.full = true }
|
||||
|
||||
func buildOptions(opts []ScanOption) scanOptions {
|
||||
var o scanOptions
|
||||
for _, f := range opts {
|
||||
f(&o)
|
||||
}
|
||||
return o
|
||||
}
|
||||
|
||||
// Scan indexes every library in turn. An unreadable library is reported in the
|
||||
// returned error and the other libraries are still scanned.
|
||||
func (s *Scanner) Scan(ctx context.Context, libs []Library) (Summary, error) {
|
||||
func (s *Scanner) Scan(ctx context.Context, libs []Library, opts ...ScanOption) (Summary, error) {
|
||||
o := buildOptions(opts)
|
||||
total := Summary{Libraries: len(libs)}
|
||||
s.publish(Event{Type: ScanStarted})
|
||||
|
||||
var errs []error
|
||||
for _, lib := range libs {
|
||||
sum, err := s.scanLibrary(ctx, lib)
|
||||
sum, err := s.scanLibrary(ctx, lib, o)
|
||||
total.add(sum)
|
||||
if err != nil {
|
||||
errs = append(errs, fmt.Errorf("%s: %w", lib.Name, err))
|
||||
|
|
@ -102,7 +122,7 @@ type parseResult struct {
|
|||
err error
|
||||
}
|
||||
|
||||
func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error) {
|
||||
func (s *Scanner) scanLibrary(ctx context.Context, lib Library, o scanOptions) (Summary, error) {
|
||||
var sum Summary
|
||||
|
||||
walked, err := s.walk(ctx, lib, &sum)
|
||||
|
|
@ -126,7 +146,7 @@ func (s *Scanner) scanLibrary(ctx context.Context, lib Library) (Summary, error)
|
|||
"refusing to mark them missing", n)
|
||||
}
|
||||
}
|
||||
p := makePlan(walked, existing)
|
||||
p := makePlan(walked, existing, o)
|
||||
sum.Unchanged = p.unchanged
|
||||
|
||||
if err := s.parseAndSave(ctx, lib, p.parse, &sum); err != nil {
|
||||
|
|
@ -348,7 +368,7 @@ type fileKey struct{ size, mtime int64 }
|
|||
// A new file and a vanished file with the same size and modification time are
|
||||
// treated as a rename only when each is the sole candidate for that pair, so
|
||||
// look-alike files are never guessed.
|
||||
func makePlan(w walkResult, existing []ExistingFile) plan {
|
||||
func makePlan(w walkResult, existing []ExistingFile, o scanOptions) plan {
|
||||
var p plan
|
||||
byPath := make(map[string]ExistingFile, len(existing))
|
||||
for _, e := range existing {
|
||||
|
|
@ -365,7 +385,7 @@ func makePlan(w walkResult, existing []ExistingFile) plan {
|
|||
}
|
||||
seen[f.RelPath] = true
|
||||
switch {
|
||||
case e.Size != f.Size || e.MTimeUnix != f.MTimeUnix:
|
||||
case e.Size != f.Size || e.MTimeUnix != f.MTimeUnix || o.full:
|
||||
p.parse = append(p.parse, parseJob{file: f, existingID: e.ID})
|
||||
case e.Missing:
|
||||
p.restore = append(p.restore, e.ID)
|
||||
|
|
|
|||
|
|
@ -876,3 +876,58 @@ func TestScanCountsUnsupportedAudioFormats(t *testing.T) {
|
|||
t.Errorf("reader called %d times: unsupported files must not be opened", reader.calls.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestFullScanReadsUnchangedFilesAgain(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write(t, filepath.Join(root, "a.mp3"), "1")
|
||||
write(t, filepath.Join(root, "b.mp3"), "22")
|
||||
|
||||
store, reader := newFakeStore(), &fakeReader{}
|
||||
s, _ := newScanner(store, reader, 2)
|
||||
mustScan(t, s, root)
|
||||
idA := store.row(root, "a.mp3").ID
|
||||
before := reader.calls.Load()
|
||||
|
||||
sum, err := s.Scan(context.Background(), libsOf(root), FullRescan)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if sum.Updated != 2 || sum.Unchanged != 0 || sum.Added != 0 {
|
||||
t.Fatalf("full scan summary = %+v", sum)
|
||||
}
|
||||
if got := reader.calls.Load() - before; got != 2 {
|
||||
t.Errorf("files read = %d, want 2", got)
|
||||
}
|
||||
if store.row(root, "a.mp3").ID != idA {
|
||||
t.Error("a full scan changed the identity of a file")
|
||||
}
|
||||
// The next normal scan is back to skipping unchanged files.
|
||||
before = reader.calls.Load()
|
||||
if sum := mustScan(t, s, root); sum.Unchanged != 2 || reader.calls.Load() != before {
|
||||
t.Errorf("after a full scan: %+v", sum)
|
||||
}
|
||||
}
|
||||
|
||||
func TestManagerCanStartAFullScan(t *testing.T) {
|
||||
root := t.TempDir()
|
||||
write(t, filepath.Join(root, "a.mp3"), "1")
|
||||
store, reader := newFakeStore(), &fakeReader{}
|
||||
s, _ := newScanner(store, reader, 1)
|
||||
mustScan(t, s, root)
|
||||
before := reader.calls.Load()
|
||||
|
||||
m := NewManager(s, fakeSource{libs: libsOf(root)})
|
||||
done := make(chan Summary, 1)
|
||||
m.OnFinish(func(sum Summary, _ error) { done <- sum })
|
||||
if err := m.Start(context.Background(), FullRescan); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
select {
|
||||
case sum := <-done:
|
||||
if sum.Updated != 1 || reader.calls.Load() != before+1 {
|
||||
t.Errorf("summary = %+v, reads = %d", sum, reader.calls.Load()-before)
|
||||
}
|
||||
case <-time.After(5 * time.Second):
|
||||
t.Fatal("the scan did not finish")
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"go.senan.xyz/taglib"
|
||||
)
|
||||
|
|
@ -42,6 +43,7 @@ func (TaglibReader) Read(path string) (Metadata, error) {
|
|||
}
|
||||
m := Metadata{
|
||||
Tags: Tags{
|
||||
Genres: cleanGenres(tags["GENRE"]),
|
||||
Title: first(tags, "TITLE"),
|
||||
Album: first(tags, "ALBUM"),
|
||||
AlbumArtist: albumArtist,
|
||||
|
|
@ -73,6 +75,31 @@ func (TaglibReader) Read(path string) (Metadata, error) {
|
|||
return m, nil
|
||||
}
|
||||
|
||||
// maxGenreLen bounds one genre; a longer one is a sentence, not a genre.
|
||||
const maxGenreLen = 100
|
||||
|
||||
// cleanGenres turns the values of the GENRE tag into a short list of lower case
|
||||
// genres without repeats. A tagger may write several genres as several values or
|
||||
// as one value joined with ";". A slash is not a separator: "R&B/Soul" is one genre.
|
||||
func cleanGenres(values []string) []string {
|
||||
var out []string
|
||||
seen := map[string]bool{}
|
||||
for _, v := range values {
|
||||
for _, g := range strings.Split(v, ";") {
|
||||
g = strings.ToLower(strings.TrimSpace(g))
|
||||
if g == "" || utf8.RuneCountInString(g) > maxGenreLen || seen[g] {
|
||||
continue
|
||||
}
|
||||
seen[g] = true
|
||||
out = append(out, g)
|
||||
if len(out) == MaxGenresPerTrack {
|
||||
return out
|
||||
}
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
var mbidRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
|
||||
// splitIDs reads the MusicBrainz ids out of the values of one tag. Taggers write
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue