feat(images): a background job that builds all the covers

- POST /library/covers (admin) starts, in the background and one at a time like a scan,
  a job that builds the cover of every album that has something to play and no cover that
  is still good; GET /library/covers shows the progress and how the last job ended;
  DELETE /library/covers stops it. ?refresh=true searches every album again.
- For each album it does what opening its cover does: the user's own files first, then the
  Cover Art Archive if the provider is on. It gives up, and says why, when the provider
  fails for five albums in a row; an album that vanished is skipped, not a failure.
- This is the one request that goes through the whole library, and the README and the
  provider text say so: with the archive on it sends the release group id of each album with
  no picture in its files, one after the other, spaced out.
- NewRouter now refuses to build when a service was not wired, naming it, instead of
  panicking at the first request (the wiring of this job in main had been missed, which no
  test could see).
- The outbound spacing test measures where the request leaves the client, so a busy
  machine cannot make it fail.
- Spec, contract and HTTP tests, README updated.

Closes #45
This commit is contained in:
tgiachi 2026-11-12 09:34:27 +01:00
commit 548ee03d83
18 changed files with 1155 additions and 24 deletions

View file

@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers).
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it.
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again).
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
@ -205,7 +205,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the one exception is the covers job, which an admin starts on purpose with `POST /library/covers`: with the Cover Art Archive on, it sends the release group id of every album that has no picture in its files, one after the other, spaced out, and it stops when the archive keeps failing), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. Besides the covers job, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
@ -245,8 +245,10 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
| signed in | `GET /discover/dismissed`, `POST /discover/{id}/dismiss`, `DELETE /discover/{id}/dismiss` | The artists you are not interested in: list them, dismiss one, bring it back. |
| signed in | `GET /me/taste` | The artists and tags Sonda learned you like, from your own listening. Query: `limit` (default 20, max 50). |
| signed in | `GET /me/settings`, `PATCH /me/settings` | Your own settings (today: `discover_from_listening`, off by default). |
| signed in | `GET /library/covers` | Progress of the job that builds the covers, and how the last one ended. |
| signed in | `GET /library/scan` | Scan status and the last result. |
| admin | `GET`, `POST /libraries`; `PATCH`, `DELETE /libraries/{id}` | Manage libraries. Deleting one removes its index entries, never your files. |
| admin | `POST /library/covers`, `DELETE /library/covers` | Start, in the background, a job that builds the cover of every album (`?refresh=true` for all), or stop it. |
| admin | `POST /library/scan`, `POST /libraries/{id}/scan` | Scan everything, or one library. `?full=true` reads every file again. |
| admin | `GET /providers`, `PUT /providers/{name}`, `POST /providers/{name}/test`, `DELETE /providers/{name}/data` | See, set up and check Last.fm and MusicBrainz, and remove what they gave. The API key is write-only. |
| admin | `GET`, `POST /users`; `PATCH`, `DELETE /users/{id}`; `PUT /users/{id}/password` | Manage users and the admin role. The last admin cannot be demoted or deleted. |

View file

@ -254,6 +254,7 @@ func run() error {
Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights),
Feedback: feedbacks,
Images: pictures,
CoversJob: coversControl{ctx: ctx, job: images.NewJob(pictures, imageRepo)},
ArtistImages: pictures,
About: about.NewService(discoveryRepo, about.WikipediaSource{MusicBrainz: musicbrainzClient, Wikipedia: wikipediaClient},
providerSettings, cfg.Wikipedia.Languages, logger),

View file

@ -5,6 +5,7 @@ import (
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/images"
"codeberg.org/squidd/sonda/internal/library"
)
@ -61,3 +62,17 @@ func warnAboutLibraries(ctx context.Context, logger *zap.Logger, libs *library.L
logger.Warn("library problem: review GET /api/v1/libraries", zap.String("problem", p))
}
}
// coversControl lets the HTTP layer start the covers job on a context that lives as long as
// the application, not as long as a single request.
type coversControl struct {
ctx context.Context
job *images.Job
}
func (c coversControl) Start(refresh bool) error {
return c.job.Start(c.ctx, images.JobOptions{Refresh: refresh})
}
func (c coversControl) Stop() bool { return c.job.Stop() }
func (c coversControl) Status() images.JobStatus { return c.job.Status() }

View file

@ -17,6 +17,7 @@ import (
"codeberg.org/squidd/sonda/internal/about"
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discover"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/feedback"
@ -89,6 +90,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
Listening: listening.NewService(storage.NewListeningRepo(db), 20*time.Second),
Scan: &fakeScan{},
Libraries: library.NewLibraries(storage.NewLibraryRepo(db)),
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
Providers: prov,
Discovery: discovery.NewService(storage.NewDiscoveryRepo(db), &fakeLastfm{}, prov, zap.NewNop()),
Tags: discovery.NewTagService(storage.NewDiscoveryRepo(db), map[string]discovery.TagSource{}, prov, zap.NewNop()),
@ -101,6 +103,7 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
imageRepo := storage.NewImageRepo(db)
imageSvc := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo)
deps.Images, deps.ArtistImages = imageSvc, imageSvc
deps.CoversJob = &jobControl{ctx: context.Background(), job: images.NewJob(imageSvc, imageRepo)}
mutate(&deps, db)
if deps.About == nil {
deps.About = about.NewService(storage.NewDiscoveryRepo(db), noWikipedia{}, prov, []string{"en"}, zap.NewNop())
@ -114,6 +117,18 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
return NewRouter(deps), db
}
// jobControl lets the API start the covers job on a context that outlives the request.
type jobControl struct {
ctx context.Context
job *images.Job
}
func (c *jobControl) Start(refresh bool) error {
return c.job.Start(c.ctx, images.JobOptions{Refresh: refresh})
}
func (c *jobControl) Stop() bool { return c.job.Stop() }
func (c *jobControl) Status() images.JobStatus { return c.job.Status() }
// noWikipedia is a Wikipedia that knows nothing, for tests that do not care about it.
type noWikipedia struct{}

View file

@ -556,6 +556,28 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.call("GET", "/artists/abc/description", token, nil), 400, "description of a bad id", "invalid_input")
c.expect(c.call("GET", "/artists/"+artistID+"/description", "", nil), 401, "description without a token", "unauthorized")
// ---- the covers job ----
if idle := c.expect(c.call("GET", "/library/covers", token, nil), 200, "the covers job before it ran"); idle["running"] != false {
t.Fatalf("covers job = %v", idle)
}
c.expect(c.call("POST", "/library/covers?refresh=maybe", token, nil), 400, "a covers job with a bad refresh", "invalid_input")
c.expect(c.call("POST", "/library/covers", "", nil), 401, "a covers job without a token", "unauthorized")
c.expect(c.call("POST", "/library/covers?refresh=true", token, nil), 202, "start the covers job")
for i := 0; ; i++ {
st := c.expect(c.call("GET", "/library/covers", token, nil), 200, "the covers job running")
if st["running"] == false {
if last, _ := st["last"].(map[string]any); last == nil || last["checked"] == nil {
t.Fatalf("covers job = %v, want a summary", st)
}
break
}
if i > 400 {
t.Fatal("the covers job did not finish")
}
time.Sleep(25 * time.Millisecond)
}
c.expect(c.call("DELETE", "/library/covers", token, nil), 204, "stop the covers job when none runs")
// ---- covers ----
cover := "/albums/" + albumID + "/cover"
big := c.call("GET", cover, token, nil)

View file

@ -0,0 +1,164 @@
package httpapi
import (
"net/http"
"testing"
"time"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/images"
"codeberg.org/squidd/sonda/internal/storage"
)
type fakeCoversJob struct {
started []bool
err error
running bool
stopped int
status images.JobStatus
}
func (f *fakeCoversJob) Start(refresh bool) error {
if f.err != nil {
return f.err
}
f.started = append(f.started, refresh)
return nil
}
func (f *fakeCoversJob) Stop() bool { f.stopped++; return f.running }
func (f *fakeCoversJob) Status() images.JobStatus { return f.status }
func newJobAPI(t *testing.T) (http.Handler, *fakeCoversJob, string, string) {
t.Helper()
fake := &fakeCoversJob{}
h, _ := newTestEnvDB(t, func(d *Deps, _ *bun.DB) { d.CoversJob = fake })
admin := registerAndLogin(t, h)
// A second user who is not an admin.
cred := map[string]any{"username": "bob", "password": "longenough"}
if r := do(t, h, "POST", "/api/v1/users", admin, cred); r.Code != http.StatusCreated {
t.Fatalf("create bob: %d %v", r.Code, r.Body)
}
bob, _ := do(t, h, "POST", "/api/v1/auth/login", "", cred).Body["access_token"].(string)
return h, fake, admin, bob
}
func TestOnlyAnAdminStartsOrStopsTheCoversJob(t *testing.T) {
h, fake, admin, bob := newJobAPI(t)
for _, m := range []string{"POST", "DELETE"} {
if r := do(t, h, m, "/api/v1/library/covers", "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("%s without a token: %d", m, r.Code)
}
if r := do(t, h, m, "/api/v1/library/covers", bob, nil); r.Code != http.StatusForbidden || errCode(r) != "admin_required" {
t.Errorf("%s as a user: %d %v", m, r.Code, r.Body)
}
}
if len(fake.started) != 0 || fake.stopped != 0 {
t.Error("something was started or stopped by someone who is not an admin")
}
if r := do(t, h, "GET", "/api/v1/library/covers", "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("status without a token: %d", r.Code)
}
if r := do(t, h, "GET", "/api/v1/library/covers", bob, nil); r.Code != http.StatusOK {
t.Errorf("status for a user (like the scan status): %d", r.Code)
}
if r := do(t, h, "POST", "/api/v1/library/covers", admin, nil); r.Code != http.StatusAccepted || r.Body["status"] != "started" {
t.Errorf("start: %d %v", r.Code, r.Body)
}
}
func TestTheCoversJobTakesAStrictRefreshParameter(t *testing.T) {
h, fake, admin, _ := newJobAPI(t)
for q, want := range map[string]bool{"": false, "?refresh=false": false, "?refresh=true": true} {
if r := do(t, h, "POST", "/api/v1/library/covers"+q, admin, nil); r.Code != http.StatusAccepted {
t.Fatalf("%q: %d %v", q, r.Code, r.Body)
}
if got := fake.started[len(fake.started)-1]; got != want {
t.Errorf("%q: refresh = %v, want %v", q, got, want)
}
}
n := len(fake.started)
for _, q := range []string{"?refresh=yes", "?refresh=1", "?refresh=true&refresh=false", "?refresh="} {
if r := do(t, h, "POST", "/api/v1/library/covers"+q, admin, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
t.Errorf("%q: %d %v, want 400", q, r.Code, r.Body)
}
}
if len(fake.started) != n {
t.Error("a request that was refused started a job")
}
}
func TestTheCoversJobIsOneAtATime(t *testing.T) {
h, fake, admin, _ := newJobAPI(t)
fake.err = images.ErrJobRunning
if r := do(t, h, "POST", "/api/v1/library/covers", admin, nil); r.Code != http.StatusConflict || errCode(r) != "covers_job_running" {
t.Errorf("%d %v, want 409 covers_job_running", r.Code, r.Body)
}
}
func TestStoppingTheCoversJobIsAlwaysFine(t *testing.T) {
h, fake, admin, _ := newJobAPI(t)
for _, running := range []bool{true, false} {
fake.running = running
if r := do(t, h, "DELETE", "/api/v1/library/covers", admin, nil); r.Code != http.StatusNoContent {
t.Errorf("running=%v: %d %v", running, r.Code, r.Body)
}
}
if fake.stopped != 2 {
t.Errorf("stopped %d times", fake.stopped)
}
}
func TestTheCoversJobStatusSaysWhatItIsDoingAndHowTheLastOneEnded(t *testing.T) {
h, fake, admin, _ := newJobAPI(t)
r := do(t, h, "GET", "/api/v1/library/covers", admin, nil)
if r.Code != http.StatusOK || r.Body["running"] != false {
t.Fatalf("idle: %d %v", r.Code, r.Body)
}
sameKeys(t, "idle status", r.Body, "running")
started := time.Date(2026, 11, 12, 10, 0, 0, 0, time.UTC)
fake.status = images.JobStatus{Running: true, StartedAt: started, Progress: &images.Progress{Done: 3, Total: 10}}
r = do(t, h, "GET", "/api/v1/library/covers", admin, nil)
sameKeys(t, "running status", r.Body, "running", "started_at", "progress")
p, _ := r.Body["progress"].(map[string]any)
if r.Body["running"] != true || r.Body["started_at"] != "2026-11-12T10:00:00Z" || p["done"] != float64(3) || p["total"] != float64(10) {
t.Errorf("running = %v", r.Body)
}
fake.status = images.JobStatus{StartedAt: started, LastFinishedAt: started.Add(time.Minute), LastError: "gave up after 5 albums in a row failed",
Last: &images.JobSummary{Checked: 5, Failed: 5, Stopped: false}}
r = do(t, h, "GET", "/api/v1/library/covers", admin, nil)
sameKeys(t, "finished status", r.Body, "running", "started_at", "last_finished_at", "last", "last_error")
last, _ := r.Body["last"].(map[string]any)
sameKeys(t, "last", last, "checked", "from_files", "from_archive", "none", "skipped", "failed", "stopped")
if last["checked"] != float64(5) || last["failed"] != float64(5) || last["stopped"] != false || r.Body["last_error"] == "" {
t.Errorf("last = %v", r.Body)
}
}
// With the real job on the real stack: the albums of a library get their covers.
func TestTheCoversJobBuildsTheCoversOfTheLibrary(t *testing.T) {
e := newCoversEnv(t) // "With Art" has a cover.jpg in its folder, "Without" has nothing
repo := storage.NewImageRepo(e.db)
job := images.NewJob(e.svc, repo)
if err := job.Start(t.Context(), images.JobOptions{}); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(20 * time.Second)
for job.Status().Running {
if time.Now().After(deadline) {
t.Fatal("the job did not finish")
}
time.Sleep(10 * time.Millisecond)
}
st := job.Status()
if st.Last == nil || st.Last.Checked != 2 || st.Last.FromFiles != 1 || st.Last.None != 1 {
t.Errorf("last = %+v, want two albums: one cover from the files, one with none", st.Last)
}
// The cover is there without anybody having asked for it.
if rec, ok, _ := repo.Get(t.Context(), "album", e.withArt); !ok || rec.Status != images.StatusOK {
t.Errorf("record of the album with a picture: %+v ok=%v", rec, ok)
}
}

View file

@ -33,6 +33,7 @@ func (f *fakeArchive) AlbumImage(context.Context, int64) (images.Found, error) {
}
type coversEnv struct {
svc *images.Service
archive *fakeArchive
h http.Handler
db *bun.DB
@ -77,6 +78,7 @@ func newCoversEnv(t *testing.T) *coversEnv {
repo := storage.NewImageRepo(db)
svc := images.NewService(e.data, repo, repo)
svc.SetAlbumRemote(e.archive)
e.svc = svc
d.Images = svc
})
e.token = registerAndLogin(t, e.h)

View file

@ -0,0 +1,95 @@
package httpapi
import (
"errors"
"net/http"
"github.com/gin-gonic/gin"
"codeberg.org/squidd/sonda/internal/images"
)
type coversJobHandlers struct{ job coversJobService }
type coversSummaryJSON struct {
Checked int `json:"checked"`
FromFiles int `json:"from_files"`
FromArchive int `json:"from_archive"`
None int `json:"none"`
Skipped int `json:"skipped"`
Failed int `json:"failed"`
Stopped bool `json:"stopped"`
}
type progressJSON struct {
Done int `json:"done"`
Total int `json:"total"`
}
type coversStatusJSON struct {
Running bool `json:"running"`
StartedAt *string `json:"started_at,omitempty"`
LastFinishedAt *string `json:"last_finished_at,omitempty"`
Progress *progressJSON `json:"progress,omitempty"`
Last *coversSummaryJSON `json:"last,omitempty"`
LastError string `json:"last_error,omitempty"`
}
// refreshParam reads ?refresh=: only "true" and "false", once. It answers 400 itself otherwise:
// a typo must not quietly start a job of another kind.
func refreshParam(c *gin.Context) (refresh, ok bool) {
values := c.QueryArray("refresh")
switch {
case len(values) == 0, len(values) == 1 && values[0] == "false":
return false, true
case len(values) == 1 && values[0] == "true":
return true, true
}
writeError(c, http.StatusBadRequest, "invalid_input", "refresh must be true or false")
return false, false
}
// start launches the background job that builds every cover.
func (h *coversJobHandlers) start(c *gin.Context) {
refresh, ok := refreshParam(c)
if !ok {
return
}
switch err := h.job.Start(refresh); {
case errors.Is(err, images.ErrJobRunning):
writeError(c, http.StatusConflict, "covers_job_running", err.Error())
case err != nil:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
default:
c.JSON(http.StatusAccepted, gin.H{"status": "started"})
}
}
// stop asks a running job to end. It answers 204 whether or not one was running.
func (h *coversJobHandlers) stop(c *gin.Context) {
h.job.Stop()
c.Status(http.StatusNoContent)
}
// status says whether the job is running, how far it is, and how the last one ended.
func (h *coversJobHandlers) status(c *gin.Context) {
st := h.job.Status()
out := coversStatusJSON{Running: st.Running, LastError: st.LastError}
if !st.StartedAt.IsZero() {
out.StartedAt = rfc3339(st.StartedAt)
}
if !st.LastFinishedAt.IsZero() {
out.LastFinishedAt = rfc3339(st.LastFinishedAt)
}
if st.Progress != nil {
out.Progress = &progressJSON{Done: st.Progress.Done, Total: st.Progress.Total}
}
if l := st.Last; l != nil {
out.Last = &coversSummaryJSON{
Checked: l.Checked, FromFiles: l.FromFiles, FromArchive: l.FromArchive, None: l.None,
Skipped: l.Skipped, Failed: l.Failed, Stopped: l.Stopped,
}
}
c.JSON(http.StatusOK, out)
}

View file

@ -6,6 +6,8 @@ package httpapi
import (
"context"
"net/http"
"reflect"
"strings"
"github.com/gin-gonic/gin"
"go.uber.org/zap"
@ -94,6 +96,12 @@ type aboutService interface {
Description(ctx context.Context, artistID int64) (about.Result, error)
}
type coversJobService interface {
Start(refresh bool) error
Stop() bool
Status() images.JobStatus
}
type coverService interface {
AlbumCover(ctx context.Context, albumID int64, size images.Size) (*images.Cover, error)
}
@ -139,14 +147,36 @@ type Deps struct {
Discover discoverService
Feedback feedbackService
Images coverService
CoversJob coversJobService
About aboutService
ArtistImages artistImageService
Settings userSettingsService
Streams streamService
}
// NewRouter builds the gin engine with all routes.
// missingDeps names the services of d that were not set. Every field but the logger is an
// interface that a handler calls, so one that is nil would panic on its first request.
func missingDeps(d Deps) []string {
var out []string
v := reflect.ValueOf(d)
for i := 0; i < v.NumField(); i++ {
f := v.Type().Field(i)
if f.Name == "Logger" {
continue
}
if v.Field(i).Kind() == reflect.Interface && v.Field(i).IsNil() {
out = append(out, f.Name)
}
}
return out
}
// NewRouter builds the gin engine with all routes. It panics, at start-up and not at the
// first request, when a service was not wired.
func NewRouter(d Deps) *gin.Engine {
if m := missingDeps(d); len(m) > 0 {
panic("httpapi: services not wired: " + strings.Join(m, ", "))
}
r := gin.New()
_ = r.SetTrustedProxies(nil) // use the socket address; no forwarded-header trust yet
r.Use(gin.Recovery(), limitBody(maxBodyBytes), requestLog(d.Logger))
@ -187,6 +217,8 @@ func NewRouter(d Deps) *gin.Engine {
lib := &libraryHandlers{scan: d.Scan}
protected.GET("/library/scan", lib.scanStatus)
cj := &coversJobHandlers{job: d.CoversJob}
protected.GET("/library/covers", cj.status)
cat := &catalogHandlers{svc: d.Catalog}
protected.GET("/artists", cat.listArtists)
@ -220,6 +252,8 @@ func NewRouter(d Deps) *gin.Engine {
// Admins only.
admin := protected.Group("", requireAdmin(d.Auth))
admin.POST("/library/scan", lib.startScan)
admin.POST("/library/covers", cj.start)
admin.DELETE("/library/covers", cj.stop)
libs := &librariesHandlers{svc: d.Libraries}
admin.GET("/libraries", libs.list)
admin.POST("/libraries", libs.create)

View file

@ -0,0 +1,38 @@
package httpapi
import (
"strings"
"testing"
"go.uber.org/zap"
)
// A route whose service was never wired panics on its first request, long after the server
// started. NewRouter refuses to build instead, naming what is missing.
func TestNewRouterRefusesAMissingDependency(t *testing.T) {
defer func() {
r := recover()
msg, _ := r.(string)
if !strings.Contains(msg, "CoversJob") {
t.Errorf("recovered %v, want a panic that names the missing CoversJob", r)
}
}()
d := Deps{Logger: zap.NewNop()}
NewRouter(d)
}
func TestNewRouterNamesEveryMissingDependency(t *testing.T) {
missing := missingDeps(Deps{Logger: zap.NewNop()})
for _, want := range []string{"Auth", "Tokens", "Listening", "CoversJob", "ArtistImages", "About", "Streams"} {
found := false
for _, m := range missing {
found = found || m == want
}
if !found {
t.Errorf("%s is not reported as missing: %v", want, missing)
}
}
if got := missingDeps(Deps{}); len(got) == 0 {
t.Error("an empty Deps has nothing missing")
}
}

View file

@ -478,6 +478,78 @@ paths:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/library/covers:
get:
tags: [Scan]
operationId: getCoversJob
summary: Progress of the job that builds the covers
description: |
Whether the job is running, how far it is (`progress`, only while it runs), and how the last one
ended. The job is described at `POST /library/covers`.
security: [{bearerAuth: []}]
responses:
'200':
description: The job.
content:
application/json:
schema: {$ref: '#/components/schemas/CoversJobStatus'}
'401': {$ref: '#/components/responses/Unauthorized'}
post:
tags: [Scan]
operationId: startCoversJob
summary: Build all the covers in the background
description: |
Admin only. Starts, in the background and one at a time like a scan, a job that builds the
cover of every album that has something to play and no cover that is still good, instead of
waiting for each album to be opened; follow it with `GET /library/covers`. For each album it
does what `GET /albums/{id}/cover` does: the pictures in the user's own files first and, only
if the album has none and an admin switched the `coverartarchive` provider on, the Cover Art
Archive by the release group id of the album's tags. An album whose cover is still good (a
found one for a week, "no cover" for a day) is skipped; send `refresh=true` to search every
album again.
**This is the one request that goes through the whole library.** With the archive on, it sends
the release group id of every album that has no picture in its files, one after the other and
spaced out. It gives up, and says why in `last_error`, when the provider fails for five albums
in a row. `DELETE /library/covers` stops it.
security: [{bearerAuth: []}]
parameters:
- name: refresh
in: query
description: "`true` searches every album again, also those that have a good cover. `false` when absent."
schema: {type: boolean, default: false}
responses:
'202':
description: The job was started in the background.
content:
application/json:
schema:
type: object
additionalProperties: false
required: [status]
properties:
status: {type: string, enum: [started]}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'403': {$ref: '#/components/responses/AdminRequired'}
'409':
description: The job is already running (`covers_job_running`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
delete:
tags: [Scan]
operationId: stopCoversJob
summary: Stop the job that builds the covers
description: |
Admin only. Asks a running job to end after the album it is on; the covers already built stay.
It answers the same when no job is running.
security: [{bearerAuth: []}]
responses:
'204': {description: "Done, also when no job was running."}
'401': {$ref: '#/components/responses/Unauthorized'}
'403': {$ref: '#/components/responses/AdminRequired'}
/library/scan:
get:
tags: [Scan]
@ -2278,6 +2350,37 @@ components:
artist: {$ref: '#/components/schemas/ArtistRef'}
match: {type: number, minimum: 0, maximum: 1, description: "Last.fm's similarity of the suggestion to this seed, relative to the seed's own list."}
CoversJobStatus:
type: object
additionalProperties: false
required: [running]
properties:
running: {type: boolean}
started_at: {type: string, format: date-time, description: "When the last job started."}
last_finished_at: {type: string, format: date-time}
progress:
type: object
description: Only while the job runs.
additionalProperties: false
required: [done, total]
properties:
done: {type: integer}
total: {type: integer, description: "Albums the job set out to do."}
last:
type: object
description: How the last finished job ended.
additionalProperties: false
required: [checked, from_files, from_archive, none, skipped, failed, stopped]
properties:
checked: {type: integer, description: "Albums looked at."}
from_files: {type: integer, description: "Covers found in the user's own files."}
from_archive: {type: integer, description: "Covers from the Cover Art Archive."}
none: {type: integer, description: "Albums with no cover anywhere."}
skipped: {type: integer, description: "Albums that vanished while the job ran."}
failed: {type: integer, description: "Albums whose cover could not be built just then."}
stopped: {type: boolean, description: "It was stopped, or the application was."}
last_error: {type: string, description: "Why the last job ended before the end, for example that the provider kept failing."}
UserSettings:
type: object
additionalProperties: false

186
internal/images/job.go Normal file
View file

@ -0,0 +1,186 @@
package images
import (
"context"
"errors"
"fmt"
"sync"
"time"
)
// Errors returned by Start.
var ErrJobRunning = errors.New("the covers job is already running")
// MaxConsecutiveFailures is how many albums in a row may fail before the job gives up: a
// provider that keeps failing is not asked a thousand more times.
const MaxConsecutiveFailures = 5
// jobPage is how many albums are read from the list at a time.
const jobPage = 100
// AlbumLister lists the albums that need a cover built, so the job does not read them all at once.
type AlbumLister interface {
// AlbumsToBuild returns up to limit ids greater than after, in order, of the albums that have
// something to play and no cover that is still good (or all of them with refresh).
AlbumsToBuild(ctx context.Context, after int64, limit int, refresh bool, now time.Time) ([]int64, error)
// CountAlbumsToBuild says how many there are.
CountAlbumsToBuild(ctx context.Context, refresh bool, now time.Time) (int, error)
}
// JobOptions change what the job does.
type JobOptions struct {
Refresh bool // search every album again, even those with a good cover
}
// JobSummary is the outcome of a job.
type JobSummary struct {
Checked int // albums looked at
FromFiles int // covers found in the user's own files
FromArchive int // covers from the Cover Art Archive
None int // albums with no cover anywhere
Skipped int // albums that vanished (nothing to play any more)
Failed int // albums whose cover could not be built now
Stopped bool // it was stopped, or the application was
}
// Progress says how far a running job is.
type Progress struct{ Done, Total int }
// JobStatus describes the job for the API.
type JobStatus struct {
Running bool
StartedAt time.Time
LastFinishedAt time.Time
Progress *Progress // only while it runs
Last *JobSummary // the last finished job
LastError string // why the last job ended early, if it did
}
// Job builds the cover of every album in the background, one album at a time, the way a
// scan reads the library: it is started on purpose, runs once at a time and can be followed
// and stopped. It does for each album what opening its cover does (the user's own files first,
// then the provider of covers if one is on), and skips the albums that already have a good one.
type Job struct {
svc *Service
albums AlbumLister
mu sync.Mutex
status JobStatus
cur JobSummary
total int
cancel context.CancelFunc
}
// NewJob builds a Job.
func NewJob(svc *Service, albums AlbumLister) *Job { return &Job{svc: svc, albums: albums} }
// Start begins the job in the background. ctx bounds it, so pass one that lives as long as the
// application, not an HTTP request. It returns ErrJobRunning when one is already running.
func (j *Job) Start(ctx context.Context, o JobOptions) error {
j.mu.Lock()
defer j.mu.Unlock()
if j.status.Running {
return ErrJobRunning
}
now := j.svc.now()
total, err := j.albums.CountAlbumsToBuild(ctx, o.Refresh, now)
if err != nil {
return fmt.Errorf("count the albums: %w", err)
}
runCtx, cancel := context.WithCancel(ctx)
j.cancel = cancel
j.cur, j.total = JobSummary{}, total
j.status.Running, j.status.StartedAt = true, now
go j.run(runCtx, cancel, o, now)
return nil
}
// Stop asks a running job to end, and says whether there was one.
func (j *Job) Stop() bool {
j.mu.Lock()
defer j.mu.Unlock()
if !j.status.Running || j.cancel == nil {
return false
}
j.cancel()
return true
}
// Status returns a snapshot of the job.
func (j *Job) Status() JobStatus {
j.mu.Lock()
defer j.mu.Unlock()
st := j.status
if st.Running {
st.Progress = &Progress{Done: j.cur.Checked, Total: j.total}
}
return st
}
func (j *Job) count(f func(*JobSummary)) {
j.mu.Lock()
defer j.mu.Unlock()
f(&j.cur)
}
func (j *Job) run(ctx context.Context, cancel context.CancelFunc, o JobOptions, started time.Time) {
defer cancel() // release the context when it ends by itself
reason := ""
failures := 0
loop:
for after := int64(0); ; {
ids, err := j.albums.AlbumsToBuild(ctx, after, jobPage, o.Refresh, started)
if err != nil {
if ctx.Err() == nil {
reason = "could not list the albums: " + err.Error()
}
break
}
if len(ids) == 0 {
break
}
for _, id := range ids {
after = id
if ctx.Err() != nil {
break loop
}
source, err := j.svc.EnsureAlbumCover(ctx, id, o.Refresh)
switch {
case ctx.Err() != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)):
break loop // stopped while this one was being built: it does not count
case errors.Is(err, ErrAlbumNotFound):
failures = 0
j.count(func(s *JobSummary) { s.Checked++; s.Skipped++ })
case err != nil:
failures++
j.count(func(s *JobSummary) { s.Checked++; s.Failed++ })
if failures >= MaxConsecutiveFailures {
reason = fmt.Sprintf("gave up after %d albums in a row failed: %v", failures, err)
break loop
}
default:
failures = 0
j.count(func(s *JobSummary) {
s.Checked++
switch source {
case "":
s.None++
case SourceCoverArt:
s.FromArchive++
default:
s.FromFiles++
}
})
}
}
}
j.mu.Lock()
defer j.mu.Unlock()
sum := j.cur
sum.Stopped = ctx.Err() != nil
j.status.Running = false
j.status.LastFinishedAt = j.svc.now()
j.status.Last = &sum
j.status.LastError = reason
}

269
internal/images/job_test.go Normal file
View file

@ -0,0 +1,269 @@
package images
import (
"context"
"errors"
"image/color"
"sync"
"sync/atomic"
"testing"
"time"
)
type fakeLister struct {
mu sync.Mutex
ids []int64
err error
refresh []bool
pages int
}
func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, refresh bool, _ time.Time) ([]int64, error) {
f.mu.Lock()
defer f.mu.Unlock()
f.refresh = append(f.refresh, refresh)
f.pages++
if f.err != nil {
return nil, f.err
}
var out []int64
for _, id := range f.ids {
if id > after && len(out) < limit {
out = append(out, id)
}
}
return out, nil
}
func (f *fakeLister) CountAlbumsToBuild(_ context.Context, _ bool, _ time.Time) (int, error) {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.ids), f.err
}
type jobRig struct {
*remoteRig
lister *fakeLister
job *Job
}
// newJobRig has albums 1 to 6, all in the same folder with no picture, and an archive that
// has a cover for each.
func newJobRig(t *testing.T) *jobRig {
t.Helper()
r := &jobRig{remoteRig: newRemoteRig(t), lister: &fakeLister{}}
for id := int64(1); id <= 6; id++ {
r.src.files[id] = []File{{Root: r.root, RelPath: "Album One/01.mp3"}}
r.lister.ids = append(r.lister.ids, id)
}
r.job = NewJob(r.svc, r.lister)
return r
}
func waitDone(t *testing.T, j *Job) JobStatus {
t.Helper()
deadline := time.Now().Add(20 * time.Second)
for {
if st := j.Status(); !st.Running {
return st
}
if time.Now().After(deadline) {
t.Fatal("the job did not finish")
}
time.Sleep(5 * time.Millisecond)
}
}
func TestTheJobBuildsTheCoverOfEveryAlbumAndSaysWhatItFound(t *testing.T) {
r := newJobRig(t)
// Album 3 has a picture in its folder; the others are served by the archive, except 6.
r.src.files[3] = []File{{Root: r.root, RelPath: "Album One/01.mp3"}}
r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.White))
calls := int32(0)
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Fatal(err)
}
st := waitDone(t, r.job)
_ = calls
if st.Last == nil || st.Last.Checked != 6 || st.LastError != "" {
t.Fatalf("status = %+v", st)
}
// Every album shares the folder with a cover.jpg, so the files serve all six.
if st.Last.FromFiles != 6 || st.Last.FromArchive != 0 || st.Last.None != 0 || st.Last.Failed != 0 {
t.Errorf("summary = %+v", *st.Last)
}
for id := 1; id <= 6; id++ {
if rec := r.store.recs[key("album", int64(id))]; rec.Status != StatusOK {
t.Errorf("album %d: record %+v", id, rec)
}
}
if st.StartedAt.IsZero() || st.LastFinishedAt.IsZero() || st.Running {
t.Errorf("times or running wrong: %+v", st)
}
}
func TestTheJobFallsBackToTheArchiveAndCountsWhatHasNoCover(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil)
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Fatal(err)
}
st := waitDone(t, r.job)
if st.Last.FromArchive != 6 || st.Last.FromFiles != 0 {
t.Errorf("summary = %+v, want six from the archive", *st.Last)
}
r2 := newJobRig(t)
r2.remote.set(Found{}, ErrNoImage)
_ = r2.job.Start(context.Background(), JobOptions{})
if st := waitDone(t, r2.job); st.Last.None != 6 {
t.Errorf("summary = %+v, want six without a cover", *st.Last)
}
}
func TestAlbumsThatAreAlreadyDoneAreNotBuiltAgainUnlessRefreshIsAsked(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil)
_ = r.job.Start(context.Background(), JobOptions{})
waitDone(t, r.job)
if n := atomic.LoadInt32(&r.remote.calls); n != 6 {
t.Fatalf("the archive was asked %d times, want 6", n)
}
// Run again: the lister (the database, in production) says nothing is left; the job asks for the
// albums that still need work, and a refresh asks for all of them.
_ = r.job.Start(context.Background(), JobOptions{})
waitDone(t, r.job)
if n := atomic.LoadInt32(&r.remote.calls); n != 6 {
t.Errorf("a second run asked the archive again for covers it has: %d calls", n)
}
_ = r.job.Start(context.Background(), JobOptions{Refresh: true})
st := waitDone(t, r.job)
if n := atomic.LoadInt32(&r.remote.calls); n != 12 {
t.Errorf("a refresh asked the archive %d times in all, want 12", n)
}
if st.Last.Checked != 6 {
t.Errorf("summary = %+v", *st.Last)
}
last := r.lister.refresh[len(r.lister.refresh)-1]
if !last {
t.Error("the refresh option was not given to the list of albums")
}
}
func TestOnlyOneJobRunsAtATime(t *testing.T) {
r := newJobRig(t)
r.remote.gate = make(chan struct{})
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Fatal(err)
}
time.Sleep(30 * time.Millisecond)
if err := r.job.Start(context.Background(), JobOptions{}); !errors.Is(err, ErrJobRunning) {
t.Errorf("a second job: err = %v, want ErrJobRunning", err)
}
st := r.job.Status()
if !st.Running || st.Progress == nil || st.Progress.Total != 6 || st.Progress.Done > 1 {
t.Errorf("progress = %+v", st)
}
close(r.remote.gate)
waitDone(t, r.job)
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Errorf("a job after the first finished: %v", err)
}
waitDone(t, r.job)
}
func TestTheJobCanBeStoppedAndSaysSo(t *testing.T) {
r := newJobRig(t)
r.remote.gate = make(chan struct{})
_ = r.job.Start(context.Background(), JobOptions{})
time.Sleep(30 * time.Millisecond)
if !r.job.Stop() {
t.Fatal("Stop says there was nothing to stop")
}
close(r.remote.gate)
st := waitDone(t, r.job)
if st.Last == nil || !st.Last.Stopped || st.Last.Checked >= 6 {
t.Errorf("summary = %+v, want it stopped before the end", st.Last)
}
if r.job.Stop() {
t.Error("Stop says it stopped something when no job runs")
}
}
func TestTheJobStopsWhenTheApplicationDoes(t *testing.T) {
r := newJobRig(t)
r.remote.gate = make(chan struct{})
ctx, cancel := context.WithCancel(context.Background())
_ = r.job.Start(ctx, JobOptions{})
time.Sleep(30 * time.Millisecond)
cancel()
close(r.remote.gate)
if st := waitDone(t, r.job); st.Last == nil || !st.Last.Stopped {
t.Errorf("summary = %+v", st.Last)
}
}
func TestTheJobGivesUpWhenTheArchiveKeepsFailing(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{}, ErrSourceUnavailable)
_ = r.job.Start(context.Background(), JobOptions{})
st := waitDone(t, r.job)
if st.Last == nil || st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.Checked != MaxConsecutiveFailures {
t.Errorf("status = %+v, want it to give up after %d failures in a row with a reason", st, MaxConsecutiveFailures)
}
if n := atomic.LoadInt32(&r.remote.calls); n != int32(MaxConsecutiveFailures) {
t.Errorf("the archive was asked %d times after it kept failing", n)
}
}
func TestAnOutageThatPassesDoesNotStopTheJob(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{}, ErrSourceUnavailable)
// It fails on the first album and works on the rest.
var seen int32
r.svc.SetAlbumRemote(remoteFunc(func(ctx context.Context, id int64) (Found, error) {
if atomic.AddInt32(&seen, 1) == 1 {
return Found{}, ErrSourceUnavailable
}
return Found{Data: jpegBytes(t, 400, 400, color.White)}, nil
}))
_ = r.job.Start(context.Background(), JobOptions{})
st := waitDone(t, r.job)
if st.Last.Checked != 6 || st.Last.Failed != 1 || st.Last.FromArchive != 5 || st.LastError != "" {
t.Errorf("status = %+v", st)
}
}
type remoteFunc func(ctx context.Context, id int64) (Found, error)
func (f remoteFunc) AlbumImage(ctx context.Context, id int64) (Found, error) { return f(ctx, id) }
func TestAnAlbumThatWentIsSkippedNotCountedAsAFailure(t *testing.T) {
r := newJobRig(t)
delete(r.src.files, 2) // the album has nothing to play any more
_ = r.job.Start(context.Background(), JobOptions{})
st := waitDone(t, r.job)
if st.Last.Skipped != 1 || st.Last.Failed != 0 || st.Last.Checked != 6 {
t.Errorf("summary = %+v", *st.Last)
}
}
func TestAProblemListingTheAlbumsEndsTheJobWithItsReason(t *testing.T) {
r := newJobRig(t)
r.lister.err = errors.New("database is locked")
if err := r.job.Start(context.Background(), JobOptions{}); err == nil {
// The count is read before starting, so the error comes at once.
waitDone(t, r.job)
}
st := r.job.Status()
if st.Running {
t.Error("the job is still running after the list failed")
}
}
func TestTheStatusOfAJobThatNeverRanIsEmpty(t *testing.T) {
r := newJobRig(t)
st := r.job.Status()
if st.Running || st.Last != nil || st.Progress != nil || !st.StartedAt.IsZero() || !st.LastFinishedAt.IsZero() {
t.Errorf("status = %+v", st)
}
}

View file

@ -230,8 +230,26 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co
} // the stored file is gone: build it again
}
// One build per album, however many ask at once. It runs on its own context, so
// the first one to leave does not fail the others.
built, err := s.buildShared(ctx, albumID)
if err != nil {
// An older cover (from the provider) is better than none when it cannot be asked now.
if errors.Is(err, ErrSourceUnavailable) && ok && rec.Status == StatusOK {
if c, oerr := s.open(kindAlbum, albumID, size, rec); oerr == nil {
return c, nil
}
}
return nil, err
}
if built.Status == StatusNone {
return nil, ErrNoCover
}
return s.open(kindAlbum, albumID, size, built)
}
// buildShared builds the cover of an album, one build per album however many ask at once. It
// runs on its own context, so the first one to leave does not fail the others; each caller
// waits only as long as its own context allows.
func (s *Service) buildShared(ctx context.Context, albumID int64) (Record, error) {
done := s.flight.DoChan(strconv.FormatInt(albumID, 10), func() (v any, err error) {
defer func() {
if r := recover(); r != nil {
@ -240,26 +258,48 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co
}()
return s.build(albumID)
})
var built Record
select {
case <-ctx.Done():
return nil, ctx.Err()
return Record{}, ctx.Err()
case r := <-done:
if r.Err != nil {
// An older cover (from the provider) is better than none when it cannot be asked now.
if errors.Is(r.Err, ErrSourceUnavailable) && ok && rec.Status == StatusOK {
if c, err := s.open(kindAlbum, albumID, size, rec); err == nil {
return c, nil
}
}
return nil, r.Err
return Record{}, r.Err
}
built = r.Val.(Record)
return r.Val.(Record), nil
}
if built.Status == StatusNone {
return nil, ErrNoCover
}
// EnsureAlbumCover makes sure the cover of an album is built, as opening it would, and says
// where it came from (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album
// has none. With refresh the files are searched again even if a good cover is stored. It is
// what the job that builds every cover calls.
func (s *Service) EnsureAlbumCover(ctx context.Context, albumID int64, refresh bool) (string, error) {
if albumID <= 0 {
return "", ErrAlbumNotFound
}
return s.open(kindAlbum, albumID, size, built)
if !refresh {
c, err := s.AlbumCover(ctx, albumID, Small)
switch {
case errors.Is(err, ErrNoCover):
return "", nil
case err != nil:
return "", err
}
_ = c.File.Close()
rec, ok, err := s.store.Get(ctx, kindAlbum, albumID)
if err != nil || !ok {
return "", err
}
return rec.Source, nil
}
rec, err := s.buildShared(ctx, albumID)
if err != nil {
return "", err
}
if rec.Status == StatusNone {
return "", nil
}
return rec.Source, nil
}
// folderOf is the folder (under images/) where the files of a picture are kept. The covers

View file

@ -241,16 +241,18 @@ func TestSlotsAreNotWastedWhenTimePasses(t *testing.T) {
// Many goroutines at once must still be spaced, with a real clock.
func TestConcurrentRequestsStaySpaced(t *testing.T) {
// The moment is taken where the request leaves the client, after the spacing: how long the
// test server takes to start its handler on a busy machine says nothing about the spacing.
var mu sync.Mutex
var times []time.Time
srv, _ := server(t, func(w http.ResponseWriter, _ *http.Request) {
srv, _ := server(t, func(w http.ResponseWriter, _ *http.Request) { _, _ = w.Write([]byte("{}")) })
const gap = 20 * time.Millisecond
c := New(Options{UserAgent: "t", MinInterval: gap, Jitter: noJitter, Transport: roundTripFunc(func(r *http.Request) (*http.Response, error) {
mu.Lock()
times = append(times, time.Now())
mu.Unlock()
_, _ = w.Write([]byte("{}"))
})
const gap = 20 * time.Millisecond
c := New(Options{UserAgent: "t", MinInterval: gap, Jitter: noJitter})
return http.DefaultTransport.RoundTrip(r)
})})
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
@ -785,3 +787,8 @@ func TestRedirectsAreCapped(t *testing.T) {
t.Errorf("%d requests, want 4", n)
}
}
// roundTripFunc is an http.RoundTripper made of a function.
type roundTripFunc func(*http.Request) (*http.Response, error)
func (f roundTripFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }

View file

@ -130,7 +130,8 @@ var known = []meta{
},
{
name: CoverArtArchive,
sends: "Only when an album has no picture in your own files: the MusicBrainz release group id that the tags of " +
sends: "Only when an album has no picture in your own files, when someone opens its cover or an admin starts the " +
"covers job (which goes through every album): the MusicBrainz release group id that the tags of " +
"its files carry (never a title: Sonda does not search), your IP address and the User-Agent, which carries " +
"the contact below. Redirects are followed only to archive.org, over https. Never your users, your listens, " +
"track titles or file paths.",

View file

@ -127,3 +127,49 @@ func (r *ImageRepo) AlbumMBID(ctx context.Context, albumID int64) (string, error
}
return m.String, nil
}
// albumsToBuildWhere is the condition for "has something to play and no cover that is still
// good". A cover is good while it is younger than its time to live: a found one CoverTTL, "no
// cover" NoCoverTTL. A stale record (a file changed) and an album with no record are not good.
func albumsToBuildWhere(refresh bool, now time.Time) (string, []any) {
where := `EXISTS (SELECT 1 FROM tracks t JOIN audio_files f ON f.track_id = t.id WHERE t.album_id = a.id AND f.missing_at IS NULL)`
var args []any
if !refresh {
where += ` AND NOT EXISTS (SELECT 1 FROM images i WHERE i.kind = 'album' AND i.owner_id = a.id
AND ((i.status = 'ok' AND i.checked_at > ?) OR (i.status = 'none' AND i.checked_at > ?)))`
args = append(args, now.Add(-images.CoverTTL).Unix(), now.Add(-images.NoCoverTTL).Unix())
}
return where, args
}
// AlbumsToBuild returns up to limit ids greater than after, in order, of the albums that have
// something to play and no cover that is still good (all of those with refresh).
func (r *ImageRepo) AlbumsToBuild(ctx context.Context, after int64, limit int, refresh bool, now time.Time) ([]int64, error) {
where, args := albumsToBuildWhere(refresh, now)
rows, err := r.db.QueryContext(ctx,
"SELECT a.id FROM albums a WHERE a.id > ? AND "+where+" ORDER BY a.id LIMIT ?",
append(append([]any{after}, args...), limit)...)
if err != nil {
return nil, fmt.Errorf("list albums to build: %w", err)
}
defer rows.Close()
var out []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, err
}
out = append(out, id)
}
return out, rows.Err()
}
// CountAlbumsToBuild says how many albums AlbumsToBuild would list in all.
func (r *ImageRepo) CountAlbumsToBuild(ctx context.Context, refresh bool, now time.Time) (int, error) {
where, args := albumsToBuildWhere(refresh, now)
var n int
if err := r.db.QueryRowContext(ctx, "SELECT COUNT(*) FROM albums a WHERE "+where, args...).Scan(&n); err != nil {
return 0, fmt.Errorf("count albums to build: %w", err)
}
return n, nil
}

View file

@ -367,3 +367,94 @@ func TestPurgingTheArchiveForgetsOnlyTheCoversThatCameFromIt(t *testing.T) {
}
})
}
func TestImageRepoListsTheAlbumsThatNeedACover(t *testing.T) {
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
ctx := context.Background()
lib := NewLibraryRepo(db)
l, _ := lib.CreateLibrary(ctx, "m", "/lib/m")
repo := NewImageRepo(db)
now := time.Unix(1_800_000_000, 0)
fresh, oldOK, oldNone := now.Add(-time.Hour), now.Add(-images.CoverTTL-time.Hour), now.Add(-images.NoCoverTTL-time.Hour)
var ids = map[string]int64{}
for _, title := range []string{"A none yet", "B ok fresh", "C ok old", "D none fresh", "E none old", "F stale", "G missing files"} {
if _, err := lib.SaveFile(ctx, l.ID, pf("x/"+title+".flac", title, title, "Band", 1, 1, "flac", "Band"), 0); err != nil {
t.Fatal(err)
}
var id int64
if err := db.QueryRow("SELECT id FROM albums WHERE title = ?", title).Scan(&id); err != nil {
t.Fatal(err)
}
ids[title] = id
}
save := func(title, status string, at time.Time) {
t.Helper()
// Saving a file marked the album stale at generation 1: a build saves at that one.
saved, err := repo.Save(ctx, "album", ids[title], images.Record{Status: status, Source: images.SourceFolder, Gen: 1, CheckedAt: at})
if err != nil || !saved {
t.Fatalf("save %s: %v %v", title, saved, err)
}
}
save("B ok fresh", images.StatusOK, fresh)
save("C ok old", images.StatusOK, oldOK)
save("D none fresh", images.StatusNone, fresh)
save("E none old", images.StatusNone, oldNone)
// "A none yet" and "F stale" keep the stale record that saving their file left.
if _, err := db.Exec("UPDATE audio_files SET missing_at = ? WHERE rel_path = ?", now.Unix(), "x/G missing files.flac"); err != nil {
t.Fatal(err)
}
// An artist's picture with the same number must not count for an album.
if _, err := repo.Save(ctx, "artist", ids["A none yet"], images.Record{Status: images.StatusOK, CheckedAt: fresh}); err != nil {
t.Fatal(err)
}
want := []int64{ids["A none yet"], ids["C ok old"], ids["E none old"], ids["F stale"]}
got, err := repo.AlbumsToBuild(ctx, 0, 100, false, now)
if err != nil || !sameIDs(got, want) {
t.Errorf("to build = %v err = %v, want %v", got, err, want)
}
if n, err := repo.CountAlbumsToBuild(ctx, false, now); err != nil || n != len(want) {
t.Errorf("count = %d err = %v, want %d", n, err, len(want))
}
// With refresh every album that has something to play, but not the one whose files are gone.
all := []int64{ids["A none yet"], ids["B ok fresh"], ids["C ok old"], ids["D none fresh"], ids["E none old"], ids["F stale"]}
got, err = repo.AlbumsToBuild(ctx, 0, 100, true, now)
if err != nil || !sameIDs(got, all) {
t.Errorf("refresh = %v err = %v, want %v", got, err, all)
}
if n, _ := repo.CountAlbumsToBuild(ctx, true, now); n != len(all) {
t.Errorf("refresh count = %d, want %d", n, len(all))
}
// Pages: after an id, with a limit, in order.
page, _ := repo.AlbumsToBuild(ctx, 0, 2, false, now)
if len(page) != 2 || page[0] != want[0] || page[1] != want[1] {
t.Errorf("first page = %v", page)
}
next, _ := repo.AlbumsToBuild(ctx, page[1], 2, false, now)
if len(next) != 2 || next[0] != want[2] || next[1] != want[3] {
t.Errorf("second page = %v", next)
}
if end, _ := repo.AlbumsToBuild(ctx, want[3], 2, false, now); len(end) != 0 {
t.Errorf("past the end = %v", end)
}
})
}
func sameIDs(a, b []int64) bool {
if len(a) != len(b) {
return false
}
seen := map[int64]int{}
for _, x := range a {
seen[x]++
}
for _, x := range b {
seen[x]--
}
for _, n := range seen {
if n != 0 {
return false
}
}
return true
}