diff --git a/README.md b/README.md index 873606b..6f580d3 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr - **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers). - **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers). - **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers). -- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `` element can use it. +- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again). - **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it. - **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who. - **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers). @@ -205,7 +205,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends. -**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key. +**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the one exception is the covers job, which an admin starts on purpose with `POST /library/covers`: with the Cover Art Archive on, it sends the release group id of every album that has no picture in its files, one after the other, spaced out, and it stops when the archive keeps failing), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. Besides the covers job, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key. **Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer. @@ -245,8 +245,10 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er | signed in | `GET /discover/dismissed`, `POST /discover/{id}/dismiss`, `DELETE /discover/{id}/dismiss` | The artists you are not interested in: list them, dismiss one, bring it back. | | signed in | `GET /me/taste` | The artists and tags Sonda learned you like, from your own listening. Query: `limit` (default 20, max 50). | | signed in | `GET /me/settings`, `PATCH /me/settings` | Your own settings (today: `discover_from_listening`, off by default). | +| signed in | `GET /library/covers` | Progress of the job that builds the covers, and how the last one ended. | | signed in | `GET /library/scan` | Scan status and the last result. | | admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. | +| admin | `POST /library/covers`, `DELETE /library/covers` | Start, in the background, a job that builds the cover of every album (`?refresh=true` for all), or stop it. | | admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. `?full=true` reads every file again. | | admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test`, `DELETE /providers/{name}/data` | See, set up and check Last.fm and MusicBrainz, and remove what they gave. The API key is write-only. | | admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. | diff --git a/cmd/sondad/main.go b/cmd/sondad/main.go index 287162b..db47cf0 100644 --- a/cmd/sondad/main.go +++ b/cmd/sondad/main.go @@ -254,6 +254,7 @@ func run() error { Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights), Feedback: feedbacks, Images: pictures, + CoversJob: coversControl{ctx: ctx, job: images.NewJob(pictures, imageRepo)}, ArtistImages: pictures, About: about.NewService(discoveryRepo, about.WikipediaSource{MusicBrainz: musicbrainzClient, Wikipedia: wikipediaClient}, providerSettings, cfg.Wikipedia.Languages, logger), diff --git a/cmd/sondad/scan.go b/cmd/sondad/scan.go index 0679a67..0587b11 100644 --- a/cmd/sondad/scan.go +++ b/cmd/sondad/scan.go @@ -5,6 +5,7 @@ import ( "go.uber.org/zap" + "codeberg.org/squidd/sonda/internal/images" "codeberg.org/squidd/sonda/internal/library" ) @@ -61,3 +62,17 @@ func warnAboutLibraries(ctx context.Context, logger *zap.Logger, libs *library.L logger.Warn("library problem: review GET /api/v1/libraries", zap.String("problem", p)) } } + +// coversControl lets the HTTP layer start the covers job on a context that lives as long as +// the application, not as long as a single request. +type coversControl struct { + ctx context.Context + job *images.Job +} + +func (c coversControl) Start(refresh bool) error { + return c.job.Start(c.ctx, images.JobOptions{Refresh: refresh}) +} + +func (c coversControl) Stop() bool { return c.job.Stop() } +func (c coversControl) Status() images.JobStatus { return c.job.Status() } diff --git a/internal/api/http/api_test.go b/internal/api/http/api_test.go index 5cbf2ec..60cb8bb 100644 --- a/internal/api/http/api_test.go +++ b/internal/api/http/api_test.go @@ -17,6 +17,7 @@ import ( "codeberg.org/squidd/sonda/internal/about" "codeberg.org/squidd/sonda/internal/auth" + "codeberg.org/squidd/sonda/internal/catalog" "codeberg.org/squidd/sonda/internal/discover" "codeberg.org/squidd/sonda/internal/discovery" "codeberg.org/squidd/sonda/internal/feedback" @@ -89,6 +90,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun Listening: listening.NewService(storage.NewListeningRepo(db), 20*time.Second), Scan: &fakeScan{}, Libraries: library.NewLibraries(storage.NewLibraryRepo(db)), + Catalog: catalog.NewService(storage.NewCatalogRepo(db)), Providers: prov, Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()), Tags: discovery.NewTagService(storage.NewDiscoveryRepo(db), map[string]discovery.TagSource{}, prov, zap.NewNop()), @@ -101,6 +103,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun imageRepo := storage.NewImageRepo(db) imageSvc := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo) deps.Images, deps.ArtistImages = imageSvc, imageSvc + deps.CoversJob = &jobControl{ctx: context.Background(), job: images.NewJob(imageSvc, imageRepo)} mutate(&deps, db) if deps.About == nil { deps.About = about.NewService(storage.NewDiscoveryRepo(db), noWikipedia{}, prov, []string{"en"}, zap.NewNop()) @@ -114,6 +117,18 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun return NewRouter(deps), db } +// jobControl lets the API start the covers job on a context that outlives the request. +type jobControl struct { + ctx context.Context + job *images.Job +} + +func (c *jobControl) Start(refresh bool) error { + return c.job.Start(c.ctx, images.JobOptions{Refresh: refresh}) +} +func (c *jobControl) Stop() bool { return c.job.Stop() } +func (c *jobControl) Status() images.JobStatus { return c.job.Status() } + // noWikipedia is a Wikipedia that knows nothing, for tests that do not care about it. type noWikipedia struct{} diff --git a/internal/api/http/contract_test.go b/internal/api/http/contract_test.go index 0ff9bd7..b62cb90 100644 --- a/internal/api/http/contract_test.go +++ b/internal/api/http/contract_test.go @@ -556,6 +556,28 @@ func TestResponsesMatchTheSpec(t *testing.T) { c.expect(c.call("GET", "/artists/abc/description", token, nil), 400, "description of a bad id", "invalid_input") c.expect(c.call("GET", "/artists/"+artistID+"/description", "", nil), 401, "description without a token", "unauthorized") + // ---- the covers job ---- + if idle := c.expect(c.call("GET", "/library/covers", token, nil), 200, "the covers job before it ran"); idle["running"] != false { + t.Fatalf("covers job = %v", idle) + } + c.expect(c.call("POST", "/library/covers?refresh=maybe", token, nil), 400, "a covers job with a bad refresh", "invalid_input") + c.expect(c.call("POST", "/library/covers", "", nil), 401, "a covers job without a token", "unauthorized") + c.expect(c.call("POST", "/library/covers?refresh=true", token, nil), 202, "start the covers job") + for i := 0; ; i++ { + st := c.expect(c.call("GET", "/library/covers", token, nil), 200, "the covers job running") + if st["running"] == false { + if last, _ := st["last"].(map[string]any); last == nil || last["checked"] == nil { + t.Fatalf("covers job = %v, want a summary", st) + } + break + } + if i > 400 { + t.Fatal("the covers job did not finish") + } + time.Sleep(25 * time.Millisecond) + } + c.expect(c.call("DELETE", "/library/covers", token, nil), 204, "stop the covers job when none runs") + // ---- covers ---- cover := "/albums/" + albumID + "/cover" big := c.call("GET", cover, token, nil) diff --git a/internal/api/http/covers_job_test.go b/internal/api/http/covers_job_test.go new file mode 100644 index 0000000..928205c --- /dev/null +++ b/internal/api/http/covers_job_test.go @@ -0,0 +1,164 @@ +package httpapi + +import ( + "net/http" + "testing" + "time" + + "github.com/uptrace/bun" + + "codeberg.org/squidd/sonda/internal/images" + "codeberg.org/squidd/sonda/internal/storage" +) + +type fakeCoversJob struct { + started []bool + err error + running bool + stopped int + status images.JobStatus +} + +func (f *fakeCoversJob) Start(refresh bool) error { + if f.err != nil { + return f.err + } + f.started = append(f.started, refresh) + return nil +} + +func (f *fakeCoversJob) Stop() bool { f.stopped++; return f.running } +func (f *fakeCoversJob) Status() images.JobStatus { return f.status } + +func newJobAPI(t *testing.T) (http.Handler, *fakeCoversJob, string, string) { + t.Helper() + fake := &fakeCoversJob{} + h, _ := newTestEnvDB(t, func(d *Deps, _ *bun.DB) { d.CoversJob = fake }) + admin := registerAndLogin(t, h) + // A second user who is not an admin. + cred := map[string]any{"username": "bob", "password": "longenough"} + if r := do(t, h, "POST", "/api/v1/users", admin, cred); r.Code != http.StatusCreated { + t.Fatalf("create bob: %d %v", r.Code, r.Body) + } + bob, _ := do(t, h, "POST", "/api/v1/auth/login", "", cred).Body["access_token"].(string) + return h, fake, admin, bob +} + +func TestOnlyAnAdminStartsOrStopsTheCoversJob(t *testing.T) { + h, fake, admin, bob := newJobAPI(t) + for _, m := range []string{"POST", "DELETE"} { + if r := do(t, h, m, "/api/v1/library/covers", "", nil); r.Code != http.StatusUnauthorized { + t.Errorf("%s without a token: %d", m, r.Code) + } + if r := do(t, h, m, "/api/v1/library/covers", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" { + t.Errorf("%s as a user: %d %v", m, r.Code, r.Body) + } + } + if len(fake.started) != 0 || fake.stopped != 0 { + t.Error("something was started or stopped by someone who is not an admin") + } + if r := do(t, h, "GET", "/api/v1/library/covers", "", nil); r.Code != http.StatusUnauthorized { + t.Errorf("status without a token: %d", r.Code) + } + if r := do(t, h, "GET", "/api/v1/library/covers", bob, nil); r.Code != http.StatusOK { + t.Errorf("status for a user (like the scan status): %d", r.Code) + } + if r := do(t, h, "POST", "/api/v1/library/covers", admin, nil); r.Code != http.StatusAccepted || r.Body["status"] != "started" { + t.Errorf("start: %d %v", r.Code, r.Body) + } +} + +func TestTheCoversJobTakesAStrictRefreshParameter(t *testing.T) { + h, fake, admin, _ := newJobAPI(t) + for q, want := range map[string]bool{"": false, "?refresh=false": false, "?refresh=true": true} { + if r := do(t, h, "POST", "/api/v1/library/covers"+q, admin, nil); r.Code != http.StatusAccepted { + t.Fatalf("%q: %d %v", q, r.Code, r.Body) + } + if got := fake.started[len(fake.started)-1]; got != want { + t.Errorf("%q: refresh = %v, want %v", q, got, want) + } + } + n := len(fake.started) + for _, q := range []string{"?refresh=yes", "?refresh=1", "?refresh=true&refresh=false", "?refresh="} { + if r := do(t, h, "POST", "/api/v1/library/covers"+q, admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" { + t.Errorf("%q: %d %v, want 400", q, r.Code, r.Body) + } + } + if len(fake.started) != n { + t.Error("a request that was refused started a job") + } +} + +func TestTheCoversJobIsOneAtATime(t *testing.T) { + h, fake, admin, _ := newJobAPI(t) + fake.err = images.ErrJobRunning + if r := do(t, h, "POST", "/api/v1/library/covers", admin, nil); r.Code != http.StatusConflict || errCode(r) != "covers_job_running" { + t.Errorf("%d %v, want 409 covers_job_running", r.Code, r.Body) + } +} + +func TestStoppingTheCoversJobIsAlwaysFine(t *testing.T) { + h, fake, admin, _ := newJobAPI(t) + for _, running := range []bool{true, false} { + fake.running = running + if r := do(t, h, "DELETE", "/api/v1/library/covers", admin, nil); r.Code != http.StatusNoContent { + t.Errorf("running=%v: %d %v", running, r.Code, r.Body) + } + } + if fake.stopped != 2 { + t.Errorf("stopped %d times", fake.stopped) + } +} + +func TestTheCoversJobStatusSaysWhatItIsDoingAndHowTheLastOneEnded(t *testing.T) { + h, fake, admin, _ := newJobAPI(t) + r := do(t, h, "GET", "/api/v1/library/covers", admin, nil) + if r.Code != http.StatusOK || r.Body["running"] != false { + t.Fatalf("idle: %d %v", r.Code, r.Body) + } + sameKeys(t, "idle status", r.Body, "running") + + started := time.Date(2026, 11, 12, 10, 0, 0, 0, time.UTC) + fake.status = images.JobStatus{Running: true, StartedAt: started, Progress: &images.Progress{Done: 3, Total: 10}} + r = do(t, h, "GET", "/api/v1/library/covers", admin, nil) + sameKeys(t, "running status", r.Body, "running", "started_at", "progress") + p, _ := r.Body["progress"].(map[string]any) + if r.Body["running"] != true || r.Body["started_at"] != "2026-11-12T10:00:00Z" || p["done"] != float64(3) || p["total"] != float64(10) { + t.Errorf("running = %v", r.Body) + } + + fake.status = images.JobStatus{StartedAt: started, LastFinishedAt: started.Add(time.Minute), LastError: "gave up after 5 albums in a row failed", + Last: &images.JobSummary{Checked: 5, Failed: 5, Stopped: false}} + r = do(t, h, "GET", "/api/v1/library/covers", admin, nil) + sameKeys(t, "finished status", r.Body, "running", "started_at", "last_finished_at", "last", "last_error") + last, _ := r.Body["last"].(map[string]any) + sameKeys(t, "last", last, "checked", "from_files", "from_archive", "none", "skipped", "failed", "stopped") + if last["checked"] != float64(5) || last["failed"] != float64(5) || last["stopped"] != false || r.Body["last_error"] == "" { + t.Errorf("last = %v", r.Body) + } +} + +// With the real job on the real stack: the albums of a library get their covers. +func TestTheCoversJobBuildsTheCoversOfTheLibrary(t *testing.T) { + e := newCoversEnv(t) // "With Art" has a cover.jpg in its folder, "Without" has nothing + repo := storage.NewImageRepo(e.db) + job := images.NewJob(e.svc, repo) + if err := job.Start(t.Context(), images.JobOptions{}); err != nil { + t.Fatal(err) + } + deadline := time.Now().Add(20 * time.Second) + for job.Status().Running { + if time.Now().After(deadline) { + t.Fatal("the job did not finish") + } + time.Sleep(10 * time.Millisecond) + } + st := job.Status() + if st.Last == nil || st.Last.Checked != 2 || st.Last.FromFiles != 1 || st.Last.None != 1 { + t.Errorf("last = %+v, want two albums: one cover from the files, one with none", st.Last) + } + // The cover is there without anybody having asked for it. + if rec, ok, _ := repo.Get(t.Context(), "album", e.withArt); !ok || rec.Status != images.StatusOK { + t.Errorf("record of the album with a picture: %+v ok=%v", rec, ok) + } +} diff --git a/internal/api/http/covers_test.go b/internal/api/http/covers_test.go index 4a34867..ac950f4 100644 --- a/internal/api/http/covers_test.go +++ b/internal/api/http/covers_test.go @@ -33,6 +33,7 @@ func (f *fakeArchive) AlbumImage(context.Context, int64) (images.Found, error) { } type coversEnv struct { + svc *images.Service archive *fakeArchive h http.Handler db *bun.DB @@ -77,6 +78,7 @@ func newCoversEnv(t *testing.T) *coversEnv { repo := storage.NewImageRepo(db) svc := images.NewService(e.data, repo, repo) svc.SetAlbumRemote(e.archive) + e.svc = svc d.Images = svc }) e.token = registerAndLogin(t, e.h) diff --git a/internal/api/http/handlers_covers_job.go b/internal/api/http/handlers_covers_job.go new file mode 100644 index 0000000..18a0954 --- /dev/null +++ b/internal/api/http/handlers_covers_job.go @@ -0,0 +1,95 @@ +package httpapi + +import ( + "errors" + "net/http" + + "github.com/gin-gonic/gin" + + "codeberg.org/squidd/sonda/internal/images" +) + +type coversJobHandlers struct{ job coversJobService } + +type coversSummaryJSON struct { + Checked int `json:"checked"` + FromFiles int `json:"from_files"` + FromArchive int `json:"from_archive"` + None int `json:"none"` + Skipped int `json:"skipped"` + Failed int `json:"failed"` + Stopped bool `json:"stopped"` +} + +type progressJSON struct { + Done int `json:"done"` + Total int `json:"total"` +} + +type coversStatusJSON struct { + Running bool `json:"running"` + StartedAt *string `json:"started_at,omitempty"` + LastFinishedAt *string `json:"last_finished_at,omitempty"` + Progress *progressJSON `json:"progress,omitempty"` + Last *coversSummaryJSON `json:"last,omitempty"` + LastError string `json:"last_error,omitempty"` +} + +// refreshParam reads ?refresh=: only "true" and "false", once. It answers 400 itself otherwise: +// a typo must not quietly start a job of another kind. +func refreshParam(c *gin.Context) (refresh, ok bool) { + values := c.QueryArray("refresh") + switch { + case len(values) == 0, len(values) == 1 && values[0] == "false": + return false, true + case len(values) == 1 && values[0] == "true": + return true, true + } + writeError(c, http.StatusBadRequest, "invalid_input", "refresh must be true or false") + return false, false +} + +// start launches the background job that builds every cover. +func (h *coversJobHandlers) start(c *gin.Context) { + refresh, ok := refreshParam(c) + if !ok { + return + } + switch err := h.job.Start(refresh); { + case errors.Is(err, images.ErrJobRunning): + writeError(c, http.StatusConflict, "covers_job_running", err.Error()) + case err != nil: + _ = c.Error(err) + writeError(c, http.StatusInternalServerError, "internal", "internal error") + default: + c.JSON(http.StatusAccepted, gin.H{"status": "started"}) + } +} + +// stop asks a running job to end. It answers 204 whether or not one was running. +func (h *coversJobHandlers) stop(c *gin.Context) { + h.job.Stop() + c.Status(http.StatusNoContent) +} + +// status says whether the job is running, how far it is, and how the last one ended. +func (h *coversJobHandlers) status(c *gin.Context) { + st := h.job.Status() + out := coversStatusJSON{Running: st.Running, LastError: st.LastError} + if !st.StartedAt.IsZero() { + out.StartedAt = rfc3339(st.StartedAt) + } + if !st.LastFinishedAt.IsZero() { + out.LastFinishedAt = rfc3339(st.LastFinishedAt) + } + if st.Progress != nil { + out.Progress = &progressJSON{Done: st.Progress.Done, Total: st.Progress.Total} + } + if l := st.Last; l != nil { + out.Last = &coversSummaryJSON{ + Checked: l.Checked, FromFiles: l.FromFiles, FromArchive: l.FromArchive, None: l.None, + Skipped: l.Skipped, Failed: l.Failed, Stopped: l.Stopped, + } + } + c.JSON(http.StatusOK, out) +} diff --git a/internal/api/http/router.go b/internal/api/http/router.go index a2ff1d1..1835e99 100644 --- a/internal/api/http/router.go +++ b/internal/api/http/router.go @@ -6,6 +6,8 @@ package httpapi import ( "context" "net/http" + "reflect" + "strings" "github.com/gin-gonic/gin" "go.uber.org/zap" @@ -94,6 +96,12 @@ type aboutService interface { Description(ctx context.Context, artistID int64) (about.Result, error) } +type coversJobService interface { + Start(refresh bool) error + Stop() bool + Status() images.JobStatus +} + type coverService interface { AlbumCover(ctx context.Context, albumID int64, size images.Size) (*images.Cover, error) } @@ -139,14 +147,36 @@ type Deps struct { Discover discoverService Feedback feedbackService Images coverService + CoversJob coversJobService About aboutService ArtistImages artistImageService Settings userSettingsService Streams streamService } -// NewRouter builds the gin engine with all routes. +// missingDeps names the services of d that were not set. Every field but the logger is an +// interface that a handler calls, so one that is nil would panic on its first request. +func missingDeps(d Deps) []string { + var out []string + v := reflect.ValueOf(d) + for i := 0; i < v.NumField(); i++ { + f := v.Type().Field(i) + if f.Name == "Logger" { + continue + } + if v.Field(i).Kind() == reflect.Interface && v.Field(i).IsNil() { + out = append(out, f.Name) + } + } + return out +} + +// NewRouter builds the gin engine with all routes. It panics, at start-up and not at the +// first request, when a service was not wired. func NewRouter(d Deps) *gin.Engine { + if m := missingDeps(d); len(m) > 0 { + panic("httpapi: services not wired: " + strings.Join(m, ", ")) + } r := gin.New() _ = r.SetTrustedProxies(nil) // use the socket address; no forwarded-header trust yet r.Use(gin.Recovery(), limitBody(maxBodyBytes), requestLog(d.Logger)) @@ -187,6 +217,8 @@ func NewRouter(d Deps) *gin.Engine { lib := &libraryHandlers{scan: d.Scan} protected.GET("/library/scan", lib.scanStatus) + cj := &coversJobHandlers{job: d.CoversJob} + protected.GET("/library/covers", cj.status) cat := &catalogHandlers{svc: d.Catalog} protected.GET("/artists", cat.listArtists) @@ -220,6 +252,8 @@ func NewRouter(d Deps) *gin.Engine { // Admins only. admin := protected.Group("", requireAdmin(d.Auth)) admin.POST("/library/scan", lib.startScan) + admin.POST("/library/covers", cj.start) + admin.DELETE("/library/covers", cj.stop) libs := &librariesHandlers{svc: d.Libraries} admin.GET("/libraries", libs.list) admin.POST("/libraries", libs.create) diff --git a/internal/api/http/router_test.go b/internal/api/http/router_test.go new file mode 100644 index 0000000..8a353c9 --- /dev/null +++ b/internal/api/http/router_test.go @@ -0,0 +1,38 @@ +package httpapi + +import ( + "strings" + "testing" + + "go.uber.org/zap" +) + +// A route whose service was never wired panics on its first request, long after the server +// started. NewRouter refuses to build instead, naming what is missing. +func TestNewRouterRefusesAMissingDependency(t *testing.T) { + defer func() { + r := recover() + msg, _ := r.(string) + if !strings.Contains(msg, "CoversJob") { + t.Errorf("recovered %v, want a panic that names the missing CoversJob", r) + } + }() + d := Deps{Logger: zap.NewNop()} + NewRouter(d) +} + +func TestNewRouterNamesEveryMissingDependency(t *testing.T) { + missing := missingDeps(Deps{Logger: zap.NewNop()}) + for _, want := range []string{"Auth", "Tokens", "Listening", "CoversJob", "ArtistImages", "About", "Streams"} { + found := false + for _, m := range missing { + found = found || m == want + } + if !found { + t.Errorf("%s is not reported as missing: %v", want, missing) + } + } + if got := missingDeps(Deps{}); len(got) == 0 { + t.Error("an empty Deps has nothing missing") + } +} diff --git a/internal/api/openapi/openapi.yaml b/internal/api/openapi/openapi.yaml index 83c730d..4d1a3bb 100644 --- a/internal/api/openapi/openapi.yaml +++ b/internal/api/openapi/openapi.yaml @@ -478,6 +478,78 @@ paths: application/json: schema: {$ref: '#/components/schemas/Error'} + /library/covers: + get: + tags: [Scan] + operationId: getCoversJob + summary: Progress of the job that builds the covers + description: | + Whether the job is running, how far it is (`progress`, only while it runs), and how the last one + ended. The job is described at `POST /library/covers`. + security: [{bearerAuth: []}] + responses: + '200': + description: The job. + content: + application/json: + schema: {$ref: '#/components/schemas/CoversJobStatus'} + '401': {$ref: '#/components/responses/Unauthorized'} + post: + tags: [Scan] + operationId: startCoversJob + summary: Build all the covers in the background + description: | + Admin only. Starts, in the background and one at a time like a scan, a job that builds the + cover of every album that has something to play and no cover that is still good, instead of + waiting for each album to be opened; follow it with `GET /library/covers`. For each album it + does what `GET /albums/{id}/cover` does: the pictures in the user's own files first and, only + if the album has none and an admin switched the `coverartarchive` provider on, the Cover Art + Archive by the release group id of the album's tags. An album whose cover is still good (a + found one for a week, "no cover" for a day) is skipped; send `refresh=true` to search every + album again. + + **This is the one request that goes through the whole library.** With the archive on, it sends + the release group id of every album that has no picture in its files, one after the other and + spaced out. It gives up, and says why in `last_error`, when the provider fails for five albums + in a row. `DELETE /library/covers` stops it. + security: [{bearerAuth: []}] + parameters: + - name: refresh + in: query + description: "`true` searches every album again, also those that have a good cover. `false` when absent." + schema: {type: boolean, default: false} + responses: + '202': + description: The job was started in the background. + content: + application/json: + schema: + type: object + additionalProperties: false + required: [status] + properties: + status: {type: string, enum: [started]} + '400': {$ref: '#/components/responses/InvalidInput'} + '401': {$ref: '#/components/responses/Unauthorized'} + '403': {$ref: '#/components/responses/AdminRequired'} + '409': + description: The job is already running (`covers_job_running`). + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} + delete: + tags: [Scan] + operationId: stopCoversJob + summary: Stop the job that builds the covers + description: | + Admin only. Asks a running job to end after the album it is on; the covers already built stay. + It answers the same when no job is running. + security: [{bearerAuth: []}] + responses: + '204': {description: "Done, also when no job was running."} + '401': {$ref: '#/components/responses/Unauthorized'} + '403': {$ref: '#/components/responses/AdminRequired'} + /library/scan: get: tags: [Scan] @@ -2278,6 +2350,37 @@ components: artist: {$ref: '#/components/schemas/ArtistRef'} match: {type: number, minimum: 0, maximum: 1, description: "Last.fm's similarity of the suggestion to this seed, relative to the seed's own list."} + CoversJobStatus: + type: object + additionalProperties: false + required: [running] + properties: + running: {type: boolean} + started_at: {type: string, format: date-time, description: "When the last job started."} + last_finished_at: {type: string, format: date-time} + progress: + type: object + description: Only while the job runs. + additionalProperties: false + required: [done, total] + properties: + done: {type: integer} + total: {type: integer, description: "Albums the job set out to do."} + last: + type: object + description: How the last finished job ended. + additionalProperties: false + required: [checked, from_files, from_archive, none, skipped, failed, stopped] + properties: + checked: {type: integer, description: "Albums looked at."} + from_files: {type: integer, description: "Covers found in the user's own files."} + from_archive: {type: integer, description: "Covers from the Cover Art Archive."} + none: {type: integer, description: "Albums with no cover anywhere."} + skipped: {type: integer, description: "Albums that vanished while the job ran."} + failed: {type: integer, description: "Albums whose cover could not be built just then."} + stopped: {type: boolean, description: "It was stopped, or the application was."} + last_error: {type: string, description: "Why the last job ended before the end, for example that the provider kept failing."} + UserSettings: type: object additionalProperties: false diff --git a/internal/images/job.go b/internal/images/job.go new file mode 100644 index 0000000..901b7e9 --- /dev/null +++ b/internal/images/job.go @@ -0,0 +1,186 @@ +package images + +import ( + "context" + "errors" + "fmt" + "sync" + "time" +) + +// Errors returned by Start. +var ErrJobRunning = errors.New("the covers job is already running") + +// MaxConsecutiveFailures is how many albums in a row may fail before the job gives up: a +// provider that keeps failing is not asked a thousand more times. +const MaxConsecutiveFailures = 5 + +// jobPage is how many albums are read from the list at a time. +const jobPage = 100 + +// AlbumLister lists the albums that need a cover built, so the job does not read them all at once. +type AlbumLister interface { + // AlbumsToBuild returns up to limit ids greater than after, in order, of the albums that have + // something to play and no cover that is still good (or all of them with refresh). + AlbumsToBuild(ctx context.Context, after int64, limit int, refresh bool, now time.Time) ([]int64, error) + // CountAlbumsToBuild says how many there are. + CountAlbumsToBuild(ctx context.Context, refresh bool, now time.Time) (int, error) +} + +// JobOptions change what the job does. +type JobOptions struct { + Refresh bool // search every album again, even those with a good cover +} + +// JobSummary is the outcome of a job. +type JobSummary struct { + Checked int // albums looked at + FromFiles int // covers found in the user's own files + FromArchive int // covers from the Cover Art Archive + None int // albums with no cover anywhere + Skipped int // albums that vanished (nothing to play any more) + Failed int // albums whose cover could not be built now + Stopped bool // it was stopped, or the application was +} + +// Progress says how far a running job is. +type Progress struct{ Done, Total int } + +// JobStatus describes the job for the API. +type JobStatus struct { + Running bool + StartedAt time.Time + LastFinishedAt time.Time + Progress *Progress // only while it runs + Last *JobSummary // the last finished job + LastError string // why the last job ended early, if it did +} + +// Job builds the cover of every album in the background, one album at a time, the way a +// scan reads the library: it is started on purpose, runs once at a time and can be followed +// and stopped. It does for each album what opening its cover does (the user's own files first, +// then the provider of covers if one is on), and skips the albums that already have a good one. +type Job struct { + svc *Service + albums AlbumLister + + mu sync.Mutex + status JobStatus + cur JobSummary + total int + cancel context.CancelFunc +} + +// NewJob builds a Job. +func NewJob(svc *Service, albums AlbumLister) *Job { return &Job{svc: svc, albums: albums} } + +// Start begins the job in the background. ctx bounds it, so pass one that lives as long as the +// application, not an HTTP request. It returns ErrJobRunning when one is already running. +func (j *Job) Start(ctx context.Context, o JobOptions) error { + j.mu.Lock() + defer j.mu.Unlock() + if j.status.Running { + return ErrJobRunning + } + now := j.svc.now() + total, err := j.albums.CountAlbumsToBuild(ctx, o.Refresh, now) + if err != nil { + return fmt.Errorf("count the albums: %w", err) + } + runCtx, cancel := context.WithCancel(ctx) + j.cancel = cancel + j.cur, j.total = JobSummary{}, total + j.status.Running, j.status.StartedAt = true, now + go j.run(runCtx, cancel, o, now) + return nil +} + +// Stop asks a running job to end, and says whether there was one. +func (j *Job) Stop() bool { + j.mu.Lock() + defer j.mu.Unlock() + if !j.status.Running || j.cancel == nil { + return false + } + j.cancel() + return true +} + +// Status returns a snapshot of the job. +func (j *Job) Status() JobStatus { + j.mu.Lock() + defer j.mu.Unlock() + st := j.status + if st.Running { + st.Progress = &Progress{Done: j.cur.Checked, Total: j.total} + } + return st +} + +func (j *Job) count(f func(*JobSummary)) { + j.mu.Lock() + defer j.mu.Unlock() + f(&j.cur) +} + +func (j *Job) run(ctx context.Context, cancel context.CancelFunc, o JobOptions, started time.Time) { + defer cancel() // release the context when it ends by itself + reason := "" + failures := 0 +loop: + for after := int64(0); ; { + ids, err := j.albums.AlbumsToBuild(ctx, after, jobPage, o.Refresh, started) + if err != nil { + if ctx.Err() == nil { + reason = "could not list the albums: " + err.Error() + } + break + } + if len(ids) == 0 { + break + } + for _, id := range ids { + after = id + if ctx.Err() != nil { + break loop + } + source, err := j.svc.EnsureAlbumCover(ctx, id, o.Refresh) + switch { + case ctx.Err() != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)): + break loop // stopped while this one was being built: it does not count + case errors.Is(err, ErrAlbumNotFound): + failures = 0 + j.count(func(s *JobSummary) { s.Checked++; s.Skipped++ }) + case err != nil: + failures++ + j.count(func(s *JobSummary) { s.Checked++; s.Failed++ }) + if failures >= MaxConsecutiveFailures { + reason = fmt.Sprintf("gave up after %d albums in a row failed: %v", failures, err) + break loop + } + default: + failures = 0 + j.count(func(s *JobSummary) { + s.Checked++ + switch source { + case "": + s.None++ + case SourceCoverArt: + s.FromArchive++ + default: + s.FromFiles++ + } + }) + } + } + } + + j.mu.Lock() + defer j.mu.Unlock() + sum := j.cur + sum.Stopped = ctx.Err() != nil + j.status.Running = false + j.status.LastFinishedAt = j.svc.now() + j.status.Last = &sum + j.status.LastError = reason +} diff --git a/internal/images/job_test.go b/internal/images/job_test.go new file mode 100644 index 0000000..b253c7f --- /dev/null +++ b/internal/images/job_test.go @@ -0,0 +1,269 @@ +package images + +import ( + "context" + "errors" + "image/color" + "sync" + "sync/atomic" + "testing" + "time" +) + +type fakeLister struct { + mu sync.Mutex + ids []int64 + err error + refresh []bool + pages int +} + +func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, refresh bool, _ time.Time) ([]int64, error) { + f.mu.Lock() + defer f.mu.Unlock() + f.refresh = append(f.refresh, refresh) + f.pages++ + if f.err != nil { + return nil, f.err + } + var out []int64 + for _, id := range f.ids { + if id > after && len(out) < limit { + out = append(out, id) + } + } + return out, nil +} + +func (f *fakeLister) CountAlbumsToBuild(_ context.Context, _ bool, _ time.Time) (int, error) { + f.mu.Lock() + defer f.mu.Unlock() + return len(f.ids), f.err +} + +type jobRig struct { + *remoteRig + lister *fakeLister + job *Job +} + +// newJobRig has albums 1 to 6, all in the same folder with no picture, and an archive that +// has a cover for each. +func newJobRig(t *testing.T) *jobRig { + t.Helper() + r := &jobRig{remoteRig: newRemoteRig(t), lister: &fakeLister{}} + for id := int64(1); id <= 6; id++ { + r.src.files[id] = []File{{Root: r.root, RelPath: "Album One/01.mp3"}} + r.lister.ids = append(r.lister.ids, id) + } + r.job = NewJob(r.svc, r.lister) + return r +} + +func waitDone(t *testing.T, j *Job) JobStatus { + t.Helper() + deadline := time.Now().Add(20 * time.Second) + for { + if st := j.Status(); !st.Running { + return st + } + if time.Now().After(deadline) { + t.Fatal("the job did not finish") + } + time.Sleep(5 * time.Millisecond) + } +} + +func TestTheJobBuildsTheCoverOfEveryAlbumAndSaysWhatItFound(t *testing.T) { + r := newJobRig(t) + // Album 3 has a picture in its folder; the others are served by the archive, except 6. + r.src.files[3] = []File{{Root: r.root, RelPath: "Album One/01.mp3"}} + r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.White)) + calls := int32(0) + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Fatal(err) + } + st := waitDone(t, r.job) + _ = calls + if st.Last == nil || st.Last.Checked != 6 || st.LastError != "" { + t.Fatalf("status = %+v", st) + } + // Every album shares the folder with a cover.jpg, so the files serve all six. + if st.Last.FromFiles != 6 || st.Last.FromArchive != 0 || st.Last.None != 0 || st.Last.Failed != 0 { + t.Errorf("summary = %+v", *st.Last) + } + for id := 1; id <= 6; id++ { + if rec := r.store.recs[key("album", int64(id))]; rec.Status != StatusOK { + t.Errorf("album %d: record %+v", id, rec) + } + } + if st.StartedAt.IsZero() || st.LastFinishedAt.IsZero() || st.Running { + t.Errorf("times or running wrong: %+v", st) + } +} + +func TestTheJobFallsBackToTheArchiveAndCountsWhatHasNoCover(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil) + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Fatal(err) + } + st := waitDone(t, r.job) + if st.Last.FromArchive != 6 || st.Last.FromFiles != 0 { + t.Errorf("summary = %+v, want six from the archive", *st.Last) + } + r2 := newJobRig(t) + r2.remote.set(Found{}, ErrNoImage) + _ = r2.job.Start(context.Background(), JobOptions{}) + if st := waitDone(t, r2.job); st.Last.None != 6 { + t.Errorf("summary = %+v, want six without a cover", *st.Last) + } +} + +func TestAlbumsThatAreAlreadyDoneAreNotBuiltAgainUnlessRefreshIsAsked(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil) + _ = r.job.Start(context.Background(), JobOptions{}) + waitDone(t, r.job) + if n := atomic.LoadInt32(&r.remote.calls); n != 6 { + t.Fatalf("the archive was asked %d times, want 6", n) + } + // Run again: the lister (the database, in production) says nothing is left; the job asks for the + // albums that still need work, and a refresh asks for all of them. + _ = r.job.Start(context.Background(), JobOptions{}) + waitDone(t, r.job) + if n := atomic.LoadInt32(&r.remote.calls); n != 6 { + t.Errorf("a second run asked the archive again for covers it has: %d calls", n) + } + _ = r.job.Start(context.Background(), JobOptions{Refresh: true}) + st := waitDone(t, r.job) + if n := atomic.LoadInt32(&r.remote.calls); n != 12 { + t.Errorf("a refresh asked the archive %d times in all, want 12", n) + } + if st.Last.Checked != 6 { + t.Errorf("summary = %+v", *st.Last) + } + last := r.lister.refresh[len(r.lister.refresh)-1] + if !last { + t.Error("the refresh option was not given to the list of albums") + } +} + +func TestOnlyOneJobRunsAtATime(t *testing.T) { + r := newJobRig(t) + r.remote.gate = make(chan struct{}) + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Fatal(err) + } + time.Sleep(30 * time.Millisecond) + if err := r.job.Start(context.Background(), JobOptions{}); !errors.Is(err, ErrJobRunning) { + t.Errorf("a second job: err = %v, want ErrJobRunning", err) + } + st := r.job.Status() + if !st.Running || st.Progress == nil || st.Progress.Total != 6 || st.Progress.Done > 1 { + t.Errorf("progress = %+v", st) + } + close(r.remote.gate) + waitDone(t, r.job) + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Errorf("a job after the first finished: %v", err) + } + waitDone(t, r.job) +} + +func TestTheJobCanBeStoppedAndSaysSo(t *testing.T) { + r := newJobRig(t) + r.remote.gate = make(chan struct{}) + _ = r.job.Start(context.Background(), JobOptions{}) + time.Sleep(30 * time.Millisecond) + if !r.job.Stop() { + t.Fatal("Stop says there was nothing to stop") + } + close(r.remote.gate) + st := waitDone(t, r.job) + if st.Last == nil || !st.Last.Stopped || st.Last.Checked >= 6 { + t.Errorf("summary = %+v, want it stopped before the end", st.Last) + } + if r.job.Stop() { + t.Error("Stop says it stopped something when no job runs") + } +} + +func TestTheJobStopsWhenTheApplicationDoes(t *testing.T) { + r := newJobRig(t) + r.remote.gate = make(chan struct{}) + ctx, cancel := context.WithCancel(context.Background()) + _ = r.job.Start(ctx, JobOptions{}) + time.Sleep(30 * time.Millisecond) + cancel() + close(r.remote.gate) + if st := waitDone(t, r.job); st.Last == nil || !st.Last.Stopped { + t.Errorf("summary = %+v", st.Last) + } +} + +func TestTheJobGivesUpWhenTheArchiveKeepsFailing(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{}, ErrSourceUnavailable) + _ = r.job.Start(context.Background(), JobOptions{}) + st := waitDone(t, r.job) + if st.Last == nil || st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.Checked != MaxConsecutiveFailures { + t.Errorf("status = %+v, want it to give up after %d failures in a row with a reason", st, MaxConsecutiveFailures) + } + if n := atomic.LoadInt32(&r.remote.calls); n != int32(MaxConsecutiveFailures) { + t.Errorf("the archive was asked %d times after it kept failing", n) + } +} + +func TestAnOutageThatPassesDoesNotStopTheJob(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{}, ErrSourceUnavailable) + // It fails on the first album and works on the rest. + var seen int32 + r.svc.SetAlbumRemote(remoteFunc(func(ctx context.Context, id int64) (Found, error) { + if atomic.AddInt32(&seen, 1) == 1 { + return Found{}, ErrSourceUnavailable + } + return Found{Data: jpegBytes(t, 400, 400, color.White)}, nil + })) + _ = r.job.Start(context.Background(), JobOptions{}) + st := waitDone(t, r.job) + if st.Last.Checked != 6 || st.Last.Failed != 1 || st.Last.FromArchive != 5 || st.LastError != "" { + t.Errorf("status = %+v", st) + } +} + +type remoteFunc func(ctx context.Context, id int64) (Found, error) + +func (f remoteFunc) AlbumImage(ctx context.Context, id int64) (Found, error) { return f(ctx, id) } + +func TestAnAlbumThatWentIsSkippedNotCountedAsAFailure(t *testing.T) { + r := newJobRig(t) + delete(r.src.files, 2) // the album has nothing to play any more + _ = r.job.Start(context.Background(), JobOptions{}) + st := waitDone(t, r.job) + if st.Last.Skipped != 1 || st.Last.Failed != 0 || st.Last.Checked != 6 { + t.Errorf("summary = %+v", *st.Last) + } +} + +func TestAProblemListingTheAlbumsEndsTheJobWithItsReason(t *testing.T) { + r := newJobRig(t) + r.lister.err = errors.New("database is locked") + if err := r.job.Start(context.Background(), JobOptions{}); err == nil { + // The count is read before starting, so the error comes at once. + waitDone(t, r.job) + } + st := r.job.Status() + if st.Running { + t.Error("the job is still running after the list failed") + } +} + +func TestTheStatusOfAJobThatNeverRanIsEmpty(t *testing.T) { + r := newJobRig(t) + st := r.job.Status() + if st.Running || st.Last != nil || st.Progress != nil || !st.StartedAt.IsZero() || !st.LastFinishedAt.IsZero() { + t.Errorf("status = %+v", st) + } +} diff --git a/internal/images/service.go b/internal/images/service.go index d00fe5a..a6780e5 100644 --- a/internal/images/service.go +++ b/internal/images/service.go @@ -230,8 +230,26 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co } // the stored file is gone: build it again } - // One build per album, however many ask at once. It runs on its own context, so - // the first one to leave does not fail the others. + built, err := s.buildShared(ctx, albumID) + if err != nil { + // An older cover (from the provider) is better than none when it cannot be asked now. + if errors.Is(err, ErrSourceUnavailable) && ok && rec.Status == StatusOK { + if c, oerr := s.open(kindAlbum, albumID, size, rec); oerr == nil { + return c, nil + } + } + return nil, err + } + if built.Status == StatusNone { + return nil, ErrNoCover + } + return s.open(kindAlbum, albumID, size, built) +} + +// buildShared builds the cover of an album, one build per album however many ask at once. It +// runs on its own context, so the first one to leave does not fail the others; each caller +// waits only as long as its own context allows. +func (s *Service) buildShared(ctx context.Context, albumID int64) (Record, error) { done := s.flight.DoChan(strconv.FormatInt(albumID, 10), func() (v any, err error) { defer func() { if r := recover(); r != nil { @@ -240,26 +258,48 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co }() return s.build(albumID) }) - var built Record select { case <-ctx.Done(): - return nil, ctx.Err() + return Record{}, ctx.Err() case r := <-done: if r.Err != nil { - // An older cover (from the provider) is better than none when it cannot be asked now. - if errors.Is(r.Err, ErrSourceUnavailable) && ok && rec.Status == StatusOK { - if c, err := s.open(kindAlbum, albumID, size, rec); err == nil { - return c, nil - } - } - return nil, r.Err + return Record{}, r.Err } - built = r.Val.(Record) + return r.Val.(Record), nil } - if built.Status == StatusNone { - return nil, ErrNoCover +} + +// EnsureAlbumCover makes sure the cover of an album is built, as opening it would, and says +// where it came from (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album +// has none. With refresh the files are searched again even if a good cover is stored. It is +// what the job that builds every cover calls. +func (s *Service) EnsureAlbumCover(ctx context.Context, albumID int64, refresh bool) (string, error) { + if albumID <= 0 { + return "", ErrAlbumNotFound } - return s.open(kindAlbum, albumID, size, built) + if !refresh { + c, err := s.AlbumCover(ctx, albumID, Small) + switch { + case errors.Is(err, ErrNoCover): + return "", nil + case err != nil: + return "", err + } + _ = c.File.Close() + rec, ok, err := s.store.Get(ctx, kindAlbum, albumID) + if err != nil || !ok { + return "", err + } + return rec.Source, nil + } + rec, err := s.buildShared(ctx, albumID) + if err != nil { + return "", err + } + if rec.Status == StatusNone { + return "", nil + } + return rec.Source, nil } // folderOf is the folder (under images/) where the files of a picture are kept. The covers diff --git a/internal/providers/outbound/outbound_test.go b/internal/providers/outbound/outbound_test.go index d640c9a..875bad0 100644 --- a/internal/providers/outbound/outbound_test.go +++ b/internal/providers/outbound/outbound_test.go @@ -241,16 +241,18 @@ func TestSlotsAreNotWastedWhenTimePasses(t *testing.T) { // Many goroutines at once must still be spaced, with a real clock. func TestConcurrentRequestsStaySpaced(t *testing.T) { + // The moment is taken where the request leaves the client, after the spacing: how long the + // test server takes to start its handler on a busy machine says nothing about the spacing. var mu sync.Mutex var times []time.Time - srv, _ := server(t, func(w http.ResponseWriter, _ *http.Request) { + srv, _ := server(t, func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("{}")) }) + const gap = 20 * time.Millisecond + c := New(Options{UserAgent: "t", MinInterval: gap, Jitter: noJitter, Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) { mu.Lock() times = append(times, time.Now()) mu.Unlock() - _, _ = w.Write([]byte("{}")) - }) - const gap = 20 * time.Millisecond - c := New(Options{UserAgent: "t", MinInterval: gap, Jitter: noJitter}) + return http.DefaultTransport.RoundTrip(r) + })}) var wg sync.WaitGroup for i := 0; i < 10; i++ { @@ -785,3 +787,8 @@ func TestRedirectsAreCapped(t *testing.T) { t.Errorf("%d requests, want 4", n) } } + +// roundTripFunc is an http.RoundTripper made of a function. +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) } diff --git a/internal/providers/settings.go b/internal/providers/settings.go index a28ebac..5b523a2 100644 --- a/internal/providers/settings.go +++ b/internal/providers/settings.go @@ -130,7 +130,8 @@ var known = []meta{ }, { name: CoverArtArchive, - sends: "Only when an album has no picture in your own files: the MusicBrainz release group id that the tags of " + + sends: "Only when an album has no picture in your own files, when someone opens its cover or an admin starts the " + + "covers job (which goes through every album): the MusicBrainz release group id that the tags of " + "its files carry (never a title: Sonda does not search), your IP address and the User-Agent, which carries " + "the contact below. Redirects are followed only to archive.org, over https. Never your users, your listens, " + "track titles or file paths.", diff --git a/internal/storage/images.go b/internal/storage/images.go index 998f548..05925e5 100644 --- a/internal/storage/images.go +++ b/internal/storage/images.go @@ -127,3 +127,49 @@ func (r *ImageRepo) AlbumMBID(ctx context.Context, albumID int64) (string, error } return m.String, nil } + +// albumsToBuildWhere is the condition for "has something to play and no cover that is still +// good". A cover is good while it is younger than its time to live: a found one CoverTTL, "no +// cover" NoCoverTTL. A stale record (a file changed) and an album with no record are not good. +func albumsToBuildWhere(refresh bool, now time.Time) (string, []any) { + where := `EXISTS (SELECT 1 FROM tracks t JOIN audio_files f ON f.track_id = t.id WHERE t.album_id = a.id AND f.missing_at IS NULL)` + var args []any + if !refresh { + where += ` AND NOT EXISTS (SELECT 1 FROM images i WHERE i.kind = 'album' AND i.owner_id = a.id + AND ((i.status = 'ok' AND i.checked_at > ?) OR (i.status = 'none' AND i.checked_at > ?)))` + args = append(args, now.Add(-images.CoverTTL).Unix(), now.Add(-images.NoCoverTTL).Unix()) + } + return where, args +} + +// AlbumsToBuild returns up to limit ids greater than after, in order, of the albums that have +// something to play and no cover that is still good (all of those with refresh). +func (r *ImageRepo) AlbumsToBuild(ctx context.Context, after int64, limit int, refresh bool, now time.Time) ([]int64, error) { + where, args := albumsToBuildWhere(refresh, now) + rows, err := r.db.QueryContext(ctx, + "SELECT a.id FROM albums a WHERE a.id > ? AND "+where+" ORDER BY a.id LIMIT ?", + append(append([]any{after}, args...), limit)...) + if err != nil { + return nil, fmt.Errorf("list albums to build: %w", err) + } + defer rows.Close() + var out []int64 + for rows.Next() { + var id int64 + if err := rows.Scan(&id); err != nil { + return nil, err + } + out = append(out, id) + } + return out, rows.Err() +} + +// CountAlbumsToBuild says how many albums AlbumsToBuild would list in all. +func (r *ImageRepo) CountAlbumsToBuild(ctx context.Context, refresh bool, now time.Time) (int, error) { + where, args := albumsToBuildWhere(refresh, now) + var n int + if err := r.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM albums a WHERE "+where, args...).Scan(&n); err != nil { + return 0, fmt.Errorf("count albums to build: %w", err) + } + return n, nil +} diff --git a/internal/storage/images_test.go b/internal/storage/images_test.go index 10c65e9..ab10a8f 100644 --- a/internal/storage/images_test.go +++ b/internal/storage/images_test.go @@ -367,3 +367,94 @@ func TestPurgingTheArchiveForgetsOnlyTheCoversThatCameFromIt(t *testing.T) { } }) } + +func TestImageRepoListsTheAlbumsThatNeedACover(t *testing.T) { + forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) { + ctx := context.Background() + lib := NewLibraryRepo(db) + l, _ := lib.CreateLibrary(ctx, "m", "/lib/m") + repo := NewImageRepo(db) + now := time.Unix(1_800_000_000, 0) + fresh, oldOK, oldNone := now.Add(-time.Hour), now.Add(-images.CoverTTL-time.Hour), now.Add(-images.NoCoverTTL-time.Hour) + var ids = map[string]int64{} + for _, title := range []string{"A none yet", "B ok fresh", "C ok old", "D none fresh", "E none old", "F stale", "G missing files"} { + if _, err := lib.SaveFile(ctx, l.ID, pf("x/"+title+".flac", title, title, "Band", 1, 1, "flac", "Band"), 0); err != nil { + t.Fatal(err) + } + var id int64 + if err := db.QueryRow("SELECT id FROM albums WHERE title = ?", title).Scan(&id); err != nil { + t.Fatal(err) + } + ids[title] = id + } + save := func(title, status string, at time.Time) { + t.Helper() + // Saving a file marked the album stale at generation 1: a build saves at that one. + saved, err := repo.Save(ctx, "album", ids[title], images.Record{Status: status, Source: images.SourceFolder, Gen: 1, CheckedAt: at}) + if err != nil || !saved { + t.Fatalf("save %s: %v %v", title, saved, err) + } + } + save("B ok fresh", images.StatusOK, fresh) + save("C ok old", images.StatusOK, oldOK) + save("D none fresh", images.StatusNone, fresh) + save("E none old", images.StatusNone, oldNone) + // "A none yet" and "F stale" keep the stale record that saving their file left. + if _, err := db.Exec("UPDATE audio_files SET missing_at = ? WHERE rel_path = ?", now.Unix(), "x/G missing files.flac"); err != nil { + t.Fatal(err) + } + // An artist's picture with the same number must not count for an album. + if _, err := repo.Save(ctx, "artist", ids["A none yet"], images.Record{Status: images.StatusOK, CheckedAt: fresh}); err != nil { + t.Fatal(err) + } + + want := []int64{ids["A none yet"], ids["C ok old"], ids["E none old"], ids["F stale"]} + got, err := repo.AlbumsToBuild(ctx, 0, 100, false, now) + if err != nil || !sameIDs(got, want) { + t.Errorf("to build = %v err = %v, want %v", got, err, want) + } + if n, err := repo.CountAlbumsToBuild(ctx, false, now); err != nil || n != len(want) { + t.Errorf("count = %d err = %v, want %d", n, err, len(want)) + } + // With refresh every album that has something to play, but not the one whose files are gone. + all := []int64{ids["A none yet"], ids["B ok fresh"], ids["C ok old"], ids["D none fresh"], ids["E none old"], ids["F stale"]} + got, err = repo.AlbumsToBuild(ctx, 0, 100, true, now) + if err != nil || !sameIDs(got, all) { + t.Errorf("refresh = %v err = %v, want %v", got, err, all) + } + if n, _ := repo.CountAlbumsToBuild(ctx, true, now); n != len(all) { + t.Errorf("refresh count = %d, want %d", n, len(all)) + } + // Pages: after an id, with a limit, in order. + page, _ := repo.AlbumsToBuild(ctx, 0, 2, false, now) + if len(page) != 2 || page[0] != want[0] || page[1] != want[1] { + t.Errorf("first page = %v", page) + } + next, _ := repo.AlbumsToBuild(ctx, page[1], 2, false, now) + if len(next) != 2 || next[0] != want[2] || next[1] != want[3] { + t.Errorf("second page = %v", next) + } + if end, _ := repo.AlbumsToBuild(ctx, want[3], 2, false, now); len(end) != 0 { + t.Errorf("past the end = %v", end) + } + }) +} + +func sameIDs(a, b []int64) bool { + if len(a) != len(b) { + return false + } + seen := map[int64]int{} + for _, x := range a { + seen[x]++ + } + for _, x := range b { + seen[x]-- + } + for _, n := range seen { + if n != 0 { + return false + } + } + return true +}