From 345ad8eecbe2ea1b7dc350d67832bf1552bbb64f Mon Sep 17 00:00:00 2001 From: tgiachi Date: Thu, 12 Nov 2026 09:43:51 +0100 Subject: [PATCH] fix(images): a refresh never wipes covers, the job sees outages and starts cleanly - A refresh (or any rebuild) while the Cover Art Archive is switched off, but its data not taken back, no longer turns the covers it gave into "no cover" and deletes their files: the stored cover is kept and its date renewed, so the files are not searched again at every request. - When none of an album's files and folders can be looked at (an unmounted disk, a permission), that is ErrFilesUnreadable (503 files_unreadable), not "no picture": nothing is remembered, nothing built is touched, the archive is not asked, and the job counts it as a failure and gives up after five. A link that leaves the library is still just refused. - The job calls the build for every album that is not good yet and does not serve an older cover on an outage, so a dead provider shows as failures and trips the limit instead of being reported as covers from the archive. - Start marks the job as running before counting the albums and does not hold the lock meanwhile; a stop that comes back as a plain driver error is not a failure; a job that finished by itself is never marked stopped. - Spec and README say so; the tests cover each case, including a list that fails midway. --- README.md | 2 +- internal/api/http/covers_test.go | 18 +++ internal/api/http/handlers_covers.go | 4 + internal/api/openapi/openapi.yaml | 19 ++- internal/images/artist.go | 2 +- internal/images/job.go | 39 +++-- internal/images/job_test.go | 220 +++++++++++++++++++++++++-- internal/images/service.go | 153 ++++++++++++------- 8 files changed, 371 insertions(+), 86 deletions(-) diff --git a/README.md b/README.md index 6f580d3..b3880c1 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr - **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers). - **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers). - **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers). -- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again). +- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again; it never replaces a cover by "no cover" because a disk was not mounted, and keeps the covers that came from the archive if you switched it off). - **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it. - **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who. - **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers). diff --git a/internal/api/http/covers_test.go b/internal/api/http/covers_test.go index ac950f4..6e1100c 100644 --- a/internal/api/http/covers_test.go +++ b/internal/api/http/covers_test.go @@ -283,3 +283,21 @@ func TestAnOutageOfTheArchiveIsABadGatewayNotAnInternalError(t *testing.T) { t.Errorf("%d %s, want 502 provider_unavailable", rec.Code, rec.Body.String()) } } + +func TestACoverOfFilesThatCannotBeReadIsServiceUnavailableNotNoCover(t *testing.T) { + e := newCoversEnv(t) + if err := os.RemoveAll(e.root); err != nil { // the disk went away + t.Fatal(err) + } + rec := e.get("GET", "/albums/"+id(e.without)+"/cover", e.token) + var body struct { + Error struct{ Code string } `json:"error"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &body) + if rec.Code != http.StatusServiceUnavailable || body.Error.Code != "files_unreadable" { + t.Errorf("%d %s, want 503 files_unreadable", rec.Code, rec.Body.String()) + } + if n := atomic.LoadInt32(&e.archive.calls); n != 0 { + t.Errorf("the archive was asked %d times about files that could not be read", n) + } +} diff --git a/internal/api/http/handlers_covers.go b/internal/api/http/handlers_covers.go index c3eb16c..3546572 100644 --- a/internal/api/http/handlers_covers.go +++ b/internal/api/http/handlers_covers.go @@ -46,6 +46,10 @@ func (h *coverHandlers) album(c *gin.Context) { case errors.Is(err, images.ErrNoCover): writeError(c, http.StatusNotFound, "cover_not_found", "this album has no cover") return + case errors.Is(err, images.ErrFilesUnreadable): + _ = c.Error(err) + writeError(c, http.StatusServiceUnavailable, "files_unreadable", "the files of the album could not be read, try again later") + return case errors.Is(err, images.ErrSourceUnavailable): _ = c.Error(err) // keep the cause in the log writeError(c, http.StatusBadGateway, "provider_unavailable", "the cover provider could not be reached, try again later") diff --git a/internal/api/openapi/openapi.yaml b/internal/api/openapi/openapi.yaml index 4d1a3bb..3924dc0 100644 --- a/internal/api/openapi/openapi.yaml +++ b/internal/api/openapi/openapi.yaml @@ -511,7 +511,10 @@ paths: **This is the one request that goes through the whole library.** With the archive on, it sends the release group id of every album that has no picture in its files, one after the other and spaced out. It gives up, and says why in `last_error`, when the provider fails for five albums - in a row. `DELETE /library/covers` stops it. + in a row, and also when the files cannot be read (an unmounted disk): it never turns that into + "no cover" and never replaces a cover that was built. A cover that came from the archive is kept + when the provider was switched off since, even with `refresh=true`; only taking its data back + (`DELETE /providers/coverartarchive/data`) removes those. `DELETE /library/covers` stops it. security: [{bearerAuth: []}] parameters: - name: refresh @@ -537,6 +540,11 @@ paths: content: application/json: schema: {$ref: '#/components/schemas/Error'} + '500': + description: The albums could not be counted (`internal`). + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} delete: tags: [Scan] operationId: stopCoversJob @@ -1310,6 +1318,14 @@ paths: content: application/json: schema: {$ref: '#/components/schemas/Error'} + '503': + description: | + None of the album's files or folders could be read, for example because the disk is not + mounted (`files_unreadable`). That says nothing about whether there is a picture, so it is + not remembered as "no cover" and nothing already built is touched. Try again later. + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} head: tags: [Catalog] operationId: headAlbumCover @@ -1339,6 +1355,7 @@ paths: '401': {$ref: '#/components/responses/Unauthorized'} '404': {description: "No such album, or it has no cover."} '502': {description: "The cover provider could not be reached."} + '503': {description: "None of the album's files could be read."} /tracks: get: diff --git a/internal/images/artist.go b/internal/images/artist.go index 0a5aa38..a9c6165 100644 --- a/internal/images/artist.go +++ b/internal/images/artist.go @@ -119,7 +119,7 @@ func (s *Service) buildArtist(artistID int64) (Record, error) { return Record{}, err } rec.Gen = cur.Gen - saved, err := s.commit(ctx, kindArtist, artistID, rec, pic, epoch) + saved, err := s.commit(ctx, kindArtist, artistID, rec, pic, epoch, false) if errors.Is(err, errPurged) { purged = true continue diff --git a/internal/images/job.go b/internal/images/job.go index 901b7e9..831ac05 100644 --- a/internal/images/job.go +++ b/internal/images/job.go @@ -77,20 +77,32 @@ func NewJob(svc *Service, albums AlbumLister) *Job { return &Job{svc: svc, album // Start begins the job in the background. ctx bounds it, so pass one that lives as long as the // application, not an HTTP request. It returns ErrJobRunning when one is already running. func (j *Job) Start(ctx context.Context, o JobOptions) error { + now := j.svc.now() + runCtx, cancel := context.WithCancel(ctx) + // The job is marked as running before the albums are counted, which can take a while on a + // big library, so that the status can be read meanwhile and nobody starts a second one. j.mu.Lock() - defer j.mu.Unlock() if j.status.Running { + j.mu.Unlock() + cancel() return ErrJobRunning } - now := j.svc.now() - total, err := j.albums.CountAlbumsToBuild(ctx, o.Refresh, now) + j.cancel = cancel + j.cur, j.total = JobSummary{}, 0 + j.status.Running, j.status.StartedAt = true, now + j.mu.Unlock() + + total, err := j.albums.CountAlbumsToBuild(runCtx, o.Refresh, now) if err != nil { + j.mu.Lock() + j.status.Running = false + j.mu.Unlock() + cancel() return fmt.Errorf("count the albums: %w", err) } - runCtx, cancel := context.WithCancel(ctx) - j.cancel = cancel - j.cur, j.total = JobSummary{}, total - j.status.Running, j.status.StartedAt = true, now + j.mu.Lock() + j.total = total + j.mu.Unlock() go j.run(runCtx, cancel, o, now) return nil } @@ -127,11 +139,14 @@ func (j *Job) run(ctx context.Context, cancel context.CancelFunc, o JobOptions, defer cancel() // release the context when it ends by itself reason := "" failures := 0 + stopped := false loop: for after := int64(0); ; { ids, err := j.albums.AlbumsToBuild(ctx, after, jobPage, o.Refresh, started) if err != nil { - if ctx.Err() == nil { + if ctx.Err() != nil { + stopped = true + } else { reason = "could not list the albums: " + err.Error() } break @@ -142,12 +157,14 @@ loop: for _, id := range ids { after = id if ctx.Err() != nil { + stopped = true break loop } source, err := j.svc.EnsureAlbumCover(ctx, id, o.Refresh) switch { - case ctx.Err() != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)): - break loop // stopped while this one was being built: it does not count + case err != nil && ctx.Err() != nil: + stopped = true + break loop // stopped while this one was being built: whatever came back, it does not count case errors.Is(err, ErrAlbumNotFound): failures = 0 j.count(func(s *JobSummary) { s.Checked++; s.Skipped++ }) @@ -178,7 +195,7 @@ loop: j.mu.Lock() defer j.mu.Unlock() sum := j.cur - sum.Stopped = ctx.Err() != nil + sum.Stopped = stopped j.status.Running = false j.status.LastFinishedAt = j.svc.now() j.status.Last = &sum diff --git a/internal/images/job_test.go b/internal/images/job_test.go index b253c7f..9924bf7 100644 --- a/internal/images/job_test.go +++ b/internal/images/job_test.go @@ -4,6 +4,8 @@ import ( "context" "errors" "image/color" + "os" + "path/filepath" "sync" "sync/atomic" "testing" @@ -11,11 +13,14 @@ import ( ) type fakeLister struct { - mu sync.Mutex - ids []int64 - err error - refresh []bool - pages int + mu sync.Mutex + ids []int64 + err error + failOnPage int // fail from this page on (1 = the first); 0 = never, only err + countGate chan struct{} // when set, CountAlbumsToBuild waits for it to be closed + refresh []bool + pages int + pageSize int // 0 = as asked } func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, refresh bool, _ time.Time) ([]int64, error) { @@ -23,8 +28,11 @@ func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, re defer f.mu.Unlock() f.refresh = append(f.refresh, refresh) f.pages++ - if f.err != nil { - return nil, f.err + if f.err != nil || (f.failOnPage > 0 && f.pages >= f.failOnPage) { + return nil, errors.New("database is locked") + } + if f.pageSize > 0 && f.pageSize < limit { + limit = f.pageSize } var out []int64 for _, id := range f.ids { @@ -37,8 +45,12 @@ func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, re func (f *fakeLister) CountAlbumsToBuild(_ context.Context, _ bool, _ time.Time) (int, error) { f.mu.Lock() - defer f.mu.Unlock() - return len(f.ids), f.err + gate, n, err := f.countGate, len(f.ids), f.err + f.mu.Unlock() + if gate != nil { + <-gate + } + return n, err } type jobRig struct { @@ -247,16 +259,194 @@ func TestAnAlbumThatWentIsSkippedNotCountedAsAFailure(t *testing.T) { } } -func TestAProblemListingTheAlbumsEndsTheJobWithItsReason(t *testing.T) { +func TestAProblemListingTheAlbumsAtTheStartIsReportedAtOnce(t *testing.T) { r := newJobRig(t) r.lister.err = errors.New("database is locked") if err := r.job.Start(context.Background(), JobOptions{}); err == nil { - // The count is read before starting, so the error comes at once. - waitDone(t, r.job) + t.Fatal("the job started although the albums could not be counted") } - st := r.job.Status() - if st.Running { - t.Error("the job is still running after the list failed") + if st := r.job.Status(); st.Running { + t.Error("the job says it runs after it failed to start") + } + r.lister.err = nil + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Errorf("it cannot start again after the failure: %v", err) + } + waitDone(t, r.job) +} + +func TestAProblemListingTheAlbumsDuringTheJobEndsItWithItsReason(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{Data: jpegBytes(t, 400, 400, color.White)}, nil) + r.lister.pageSize, r.lister.failOnPage = 2, 2 // the first page of two albums works, the second fails + if err := r.job.Start(context.Background(), JobOptions{}); err != nil { + t.Fatal(err) + } + st := waitDone(t, r.job) + if st.Last == nil || st.Last.Checked != 2 || st.LastError == "" || st.Last.Stopped { + t.Errorf("status = %+v, want two albums done and the reason the list failed", st) + } +} + +func TestStartingDoesNotHoldTheStatusWhileTheAlbumsAreCounted(t *testing.T) { + r := newJobRig(t) + r.lister.countGate = make(chan struct{}) + started := make(chan error, 1) + go func() { started <- r.job.Start(context.Background(), JobOptions{}) }() + time.Sleep(40 * time.Millisecond) + done := make(chan JobStatus, 1) + go func() { done <- r.job.Status() }() + select { + case st := <-done: + if !st.Running { + t.Errorf("status = %+v, want it to say the job is starting", st) + } + case <-time.After(time.Second): + t.Fatal("Status waited for the count to end") + } + if err := r.job.Start(context.Background(), JobOptions{}); !errors.Is(err, ErrJobRunning) { + t.Errorf("a second start while the first counts: err = %v", err) + } + close(r.lister.countGate) + if err := <-started; err != nil { + t.Fatal(err) + } + waitDone(t, r.job) +} + +func TestAStopThatReturnsAPlainErrorIsNotAFailure(t *testing.T) { + r := newJobRig(t) + ctx, cancel := context.WithCancel(context.Background()) + r.svc.SetAlbumRemote(remoteFunc(func(context.Context, int64) (Found, error) { + cancel() // stopped while the provider is asked, and what comes back is not a context error + return Found{}, errors.New("interrupted") + })) + _ = r.job.Start(ctx, JobOptions{}) + st := waitDone(t, r.job) + if st.Last == nil || st.Last.Failed != 0 || !st.Last.Stopped { + t.Errorf("summary = %+v, want stopped and no failure", st.Last) + } +} + +func TestAJobThatEndedByItselfIsNotStopped(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{}, ErrNoImage) + _ = r.job.Start(context.Background(), JobOptions{}) + st := waitDone(t, r.job) + r.job.Stop() // too late: nothing runs + if st.Last.Stopped || r.job.Status().Last.Stopped { + t.Errorf("a finished job is marked stopped: %+v", r.job.Status().Last) + } +} + +// ---- what a refresh must never do ---- + +func TestARefreshWithTheArchiveOffKeepsTheCoversThatCameFromIt(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.RGBA{0, 90, 0, 255})}, nil) + _ = r.job.Start(context.Background(), JobOptions{}) + waitDone(t, r.job) + before := r.store.recs[key("album", 1)] + if before.Source != SourceCoverArt { + t.Fatalf("record = %+v", before) + } + // The admin switches the provider off (not purged) and refreshes. + r.remote.set(Found{}, ErrProviderDisabled) + r.now = r.now.Add(time.Hour) + _ = r.job.Start(context.Background(), JobOptions{Refresh: true}) + st := waitDone(t, r.job) + after := r.store.recs[key("album", 1)] + if after.Status != StatusOK || after.Source != SourceCoverArt || after.ETag != before.ETag { + t.Errorf("record = %+v, want the archive cover kept", after) + } + if _, err := os.Stat(filepath.Join(r.data, "images", "album-remote", "1-large.jpg")); err != nil { + t.Errorf("the file of an archive cover went when the provider was switched off: %v", err) + } + if st.Last.Failed != 0 { + t.Errorf("summary = %+v", *st.Last) + } + // An album that had none stays with none, of course. + if got := r.store.recs[key("album", 2)]; got.Status != StatusOK { + t.Errorf("album 2 = %+v", got) + } +} + +func TestACoverFromTheArchiveWhoseProviderIsOffIsNotSearchedAgainAtEveryRequest(t *testing.T) { + r := newRemoteRig(t) + r.rig.read(t, 1, Large) // from the archive + r.remote.set(Found{}, ErrProviderDisabled) + r.now = r.now.Add(CoverTTL + time.Hour) + r.rig.read(t, 1, Large) // kept, and its date renewed + reads := atomic.LoadInt32(&r.reads) + for i := 0; i < 3; i++ { + r.rig.read(t, 1, Large) + } + if got := atomic.LoadInt32(&r.reads); got != reads { + t.Errorf("the files were searched again %d times within the new period", got-reads) + } +} + +func TestAnUnreachableLibraryIsAFailureNotNoPicture(t *testing.T) { + r := newRemoteRig(t) + r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.RGBA{200, 0, 0, 255})) + first := r.rig.read(t, 1, Large) + // The mount goes away: nothing under the root can be opened. + if err := os.RemoveAll(r.root); err != nil { + t.Fatal(err) + } + r.store.clear("album", 1) // a rescan, or a refresh, wants to look again + _, err := r.svc.AlbumCover(context.Background(), 1, Large) + if !errors.Is(err, ErrFilesUnreadable) { + t.Fatalf("err = %v, want ErrFilesUnreadable", err) + } + if rec := r.store.recs[key("album", 1)]; rec.Status == StatusNone { + t.Errorf("record = %+v: an unreadable library must not become 'no cover'", rec) + } + if n := atomic.LoadInt32(&r.remote.calls); n != 0 { + t.Errorf("the archive was asked %d times about files that could not even be read", n) + } + if entries, _ := os.ReadDir(filepath.Join(r.data, "images", "album")); len(entries) != 2 { + t.Errorf("%d files left, want the two sizes still there", len(entries)) + } + _ = first +} + +func TestARefreshOfAnUnreachableLibraryGivesUpInsteadOfWipingEverything(t *testing.T) { + r := newJobRig(t) + r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.White)) + _ = r.job.Start(context.Background(), JobOptions{}) + waitDone(t, r.job) + if err := os.RemoveAll(r.root); err != nil { + t.Fatal(err) + } + _ = r.job.Start(context.Background(), JobOptions{Refresh: true}) + st := waitDone(t, r.job) + if st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.None != 0 { + t.Errorf("status = %+v, want it to give up with failures and not count 'none'", st) + } + for id := int64(1); id <= 6; id++ { + if rec := r.store.recs[key("album", id)]; rec.Status != StatusOK { + t.Errorf("album %d: %+v, want its cover kept", id, rec) + } + } +} + +func TestADeadProviderIsSeenEvenWhenOldCoversCouldBeServed(t *testing.T) { + r := newJobRig(t) + r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil) + _ = r.job.Start(context.Background(), JobOptions{}) + waitDone(t, r.job) + // Every archive cover is old now, and the archive is down. + r.now = r.now.Add(CoverTTL + time.Hour) + r.remote.set(Found{}, ErrSourceUnavailable) + atomic.StoreInt32(&r.remote.calls, 0) + _ = r.job.Start(context.Background(), JobOptions{}) + st := waitDone(t, r.job) + if st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.FromArchive != 0 { + t.Errorf("status = %+v, want it to notice the outage and give up", st) + } + if n := atomic.LoadInt32(&r.remote.calls); n != int32(MaxConsecutiveFailures) { + t.Errorf("the archive was asked %d times", n) } } diff --git a/internal/images/service.go b/internal/images/service.go index a6780e5..7bb64fa 100644 --- a/internal/images/service.go +++ b/internal/images/service.go @@ -73,6 +73,11 @@ var ( ErrNoCover = errors.New("this album has no cover") ErrInvalidSize = errors.New("invalid size") + // ErrFilesUnreadable means none of the files and folders of the album could be looked + // at (an unmounted disk, a permission): that says nothing about whether there is a picture, + // so it is not remembered and nothing already built is touched. + ErrFilesUnreadable = errors.New("the files of the album could not be read") + // Answers of an ArtistSource. ErrNoImage = errors.New("there is no free picture of this artist") ErrProviderDisabled = errors.New("the provider is switched off") @@ -269,28 +274,29 @@ func (s *Service) buildShared(ctx context.Context, albumID int64) (Record, error } } -// EnsureAlbumCover makes sure the cover of an album is built, as opening it would, and says -// where it came from (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album -// has none. With refresh the files are searched again even if a good cover is stored. It is -// what the job that builds every cover calls. +// EnsureAlbumCover makes sure the cover of an album is built, and says where it came from +// (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album has none. A stored +// cover that is still good is left alone; otherwise, or with refresh, the album is searched +// again, and a failure is returned as one: unlike an HTTP request, which is better served an +// older cover, the job that builds every cover must see that the provider is down. It is +// what that job calls. func (s *Service) EnsureAlbumCover(ctx context.Context, albumID int64, refresh bool) (string, error) { if albumID <= 0 { return "", ErrAlbumNotFound } if !refresh { - c, err := s.AlbumCover(ctx, albumID, Small) - switch { - case errors.Is(err, ErrNoCover): - return "", nil - case err != nil: - return "", err - } - _ = c.File.Close() rec, ok, err := s.store.Get(ctx, kindAlbum, albumID) - if err != nil || !ok { - return "", err + if err != nil { + return "", fmt.Errorf("read picture record: %w", err) + } + if ok && rec.Status == StatusNone && s.now().Sub(rec.CheckedAt) < NoCoverTTL { + return "", nil + } + if ok && rec.Status == StatusOK && s.now().Sub(rec.CheckedAt) < CoverTTL { + if _, err := os.Stat(s.filePath(folderOf(kindAlbum, rec.Source), albumID, Small)); err == nil { + return rec.Source, nil + } } - return rec.Source, nil } rec, err := s.buildShared(ctx, albumID) if err != nil { @@ -351,13 +357,24 @@ func (s *Service) build(albumID int64) (Record, error) { return Record{}, err } epoch := s.purgeCount() - var pic *Processed - rec, pic, err = s.search(ctx, albumID, files) + var ( + pic *Processed + off bool + ) + rec, pic, off, err = s.search(ctx, albumID, files) if err != nil { return Record{}, err } rec.Gen = cur.Gen - saved, err := s.commit(ctx, kindAlbum, albumID, rec, pic, epoch) + // The provider was switched off (its data was not taken back): a cover it gave stays, + // instead of being replaced by "no cover" and deleted. Its date is renewed so that + // the files are not searched again at every request. + keep := off && pic == nil && cur.Status == StatusOK && cur.Source == SourceCoverArt + if keep { + rec = cur + rec.CheckedAt = s.now() + } + saved, err := s.commit(ctx, kindAlbum, albumID, rec, pic, epoch, keep) if errors.Is(err, errPurged) { purged = true continue @@ -378,26 +395,26 @@ func (s *Service) build(albumID int64) (Record, error) { // search finds a picture: in the user's own files first, at most MaxParallelBuilds at a // time, and, only when they have none, from the provider of covers if there is one. The // picture is not stored yet (see commit); nil means there is none. -func (s *Service) search(ctx context.Context, albumID int64, files []File) (Record, *Processed, error) { +func (s *Service) search(ctx context.Context, albumID int64, files []File) (rec Record, pic *Processed, providerOff bool, err error) { none := Record{Status: StatusNone, CheckedAt: s.now()} p, source, ok, err := s.searchFiles(ctx, files) if err != nil { - return Record{}, nil, err + return Record{}, nil, false, err } if ok { - return s.record(p, source, Credit{}, none.CheckedAt), &p, nil + return s.record(p, source, Credit{}, none.CheckedAt), &p, false, nil } if s.remote == nil { - return none, nil, nil + return none, nil, false, nil } - rp, credit, ok, err := s.searchRemote(ctx, albumID) + rp, credit, ok, off, err := s.searchRemote(ctx, albumID) if err != nil { - return Record{}, nil, err + return Record{}, nil, false, err } if !ok { - return none, nil, nil + return none, nil, off, nil } - return s.record(rp, SourceCoverArt, credit, none.CheckedAt), &rp, nil + return s.record(rp, SourceCoverArt, credit, none.CheckedAt), &rp, false, nil } func (s *Service) searchFiles(ctx context.Context, files []File) (Processed, string, bool, error) { @@ -407,13 +424,16 @@ func (s *Service) searchFiles(ctx context.Context, files []File) (Processed, str case <-ctx.Done(): return Processed{}, "", false, ctx.Err() } - p, source, ok := s.find(files) + p, source, ok, readable := s.find(files) + if !ok && !readable { + return Processed{}, "", false, ErrFilesUnreadable + } return p, source, ok, nil } // searchRemote asks the provider of covers. "There is none" (no picture, no id, the // provider is off, a picture that is not one) is an answer; a failure is returned. -func (s *Service) searchRemote(ctx context.Context, albumID int64) (Processed, Credit, bool, error) { +func (s *Service) searchRemote(ctx context.Context, albumID int64) (p Processed, credit Credit, ok, providerOff bool, err error) { // This context is nobody's: when it ends here, the build ran out of time waiting for the // provider, which is an outage of the provider and not an internal error. outage := func(err error) error { @@ -426,20 +446,22 @@ func (s *Service) searchRemote(ctx context.Context, albumID int64) (Processed, C case s.artistSl <- struct{}{}: // the slots of the work that waits for the network defer func() { <-s.artistSl }() case <-ctx.Done(): - return Processed{}, Credit{}, false, outage(ctx.Err()) + return Processed{}, Credit{}, false, false, outage(ctx.Err()) } found, err := s.remote.AlbumImage(ctx, albumID) switch { - case errors.Is(err, ErrNoImage), errors.Is(err, ErrNoID), errors.Is(err, ErrProviderDisabled): - return Processed{}, Credit{}, false, nil + case errors.Is(err, ErrProviderDisabled): + return Processed{}, Credit{}, false, true, nil + case errors.Is(err, ErrNoImage), errors.Is(err, ErrNoID): + return Processed{}, Credit{}, false, false, nil case err != nil: - return Processed{}, Credit{}, false, outage(err) + return Processed{}, Credit{}, false, false, outage(err) } - p, err := ProcessLimited(found.Data, MaxArtistPixels) + p, err = ProcessLimited(found.Data, MaxArtistPixels) if err != nil { - return Processed{}, Credit{}, false, nil // not a usable picture: there is no cover to show + return Processed{}, Credit{}, false, false, nil // not a usable picture: there is no cover to show } - return p, found.Credit, true, nil + return p, found.Credit, true, false, nil } // record describes a picture that was found. @@ -459,7 +481,7 @@ func (s *Service) record(p Processed, source string, credit Credit, at time.Time // provider were taken back since the build began (errPurged) or a change moved the // record's generation (saved is false). Both happen under the read side of the purge lock, // so a purge waits for it and the next ones are refused. -func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, pic *Processed, epoch uint64) (bool, error) { +func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, pic *Processed, epoch uint64, keepFiles bool) (bool, error) { s.purgeMu.RLock() defer s.purgeMu.RUnlock() if s.purges != epoch { @@ -475,7 +497,7 @@ func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, if err != nil { return false, fmt.Errorf("save picture record: %w", err) } - if saved { + if saved && !keepFiles { // What the record no longer describes must not stay on disk: the files of the other // folder when the cover moved between the user's files and the archive, and all of // them when there is no cover any more. @@ -504,15 +526,19 @@ func (s *Service) removeFiles(folder string, id int64) { } // find returns the first usable picture: embedded in one of the first files, then an -// image in the folder of one of them. -func (s *Service) find(files []File) (Processed, string, bool) { +// image in the folder of one of them. readable says whether anything of the library could +// be looked at: when none of the files and none of the folders could, "no picture" would be +// a lie (an unmounted disk, a permission), and the caller must not remember it. +func (s *Service) find(files []File) (p Processed, source string, ok, readable bool) { if len(files) > MaxFilesTried { files = files[:MaxFilesTried] } for _, f := range files { - if data, ok := s.embeddedPicture(f); ok { + data, found, seen := s.embeddedPicture(f) + readable = readable || seen + if found { if p, err := Process(data); err == nil { - return p, SourceEmbedded, true + return p, SourceEmbedded, true, true } } } @@ -532,17 +558,19 @@ func (s *Service) find(files []File) (Processed, string, bool) { } } for _, d := range dirs { - for _, name := range folderPictureNames(d.Root, d.RelPath) { - data, ok := readFolderPicture(d.Root, d.RelPath, name) - if !ok { + names, seen := folderPictureNames(d.Root, d.RelPath) + readable = readable || seen + for _, name := range names { + data, got := readFolderPicture(d.Root, d.RelPath, name) + if !got { continue } if p, err := Process(data); err == nil { - return p, SourceFolder, true + return p, SourceFolder, true, true } } } - return Processed{}, "", false + return Processed{}, "", false, readable } // discFolder matches the name of a folder that holds one disc of an album: "CD1", @@ -551,21 +579,32 @@ var discFolder = regexp.MustCompile(`(?i)^(cd|dis[ck])[ _.-]*\d+$`) // embeddedPicture reads the picture inside an audio file, after checking through // os.Root that the file is a regular file inside its library. -func (s *Service) embeddedPicture(f File) ([]byte, bool) { +func (s *Service) embeddedPicture(f File) (data []byte, found, readable bool) { root, err := os.OpenRoot(filepath.Clean(f.Root)) if err != nil { - return nil, false + return nil, false, false } defer root.Close() info, err := root.Stat(f.RelPath) - if err != nil || !info.Mode().IsRegular() { - return nil, false + if err != nil { + // A link that leaves the library is refused on purpose: the library itself was read. + return nil, false, escapesLibrary(err) } - data, err := s.embedded(filepath.Join(filepath.Clean(f.Root), filepath.FromSlash(f.RelPath))) + if !info.Mode().IsRegular() { + return nil, false, true + } + data, err = s.embedded(filepath.Join(filepath.Clean(f.Root), filepath.FromSlash(f.RelPath))) if err != nil || len(data) == 0 { - return nil, false + return nil, false, true } - return data, true + return data, true, true +} + +// escapesLibrary says whether an error is os.Root refusing a link that leads outside the +// library. That is a decision of ours, not a disk that cannot be read, and os.Root has no +// exported error for it. +func escapesLibrary(err error) bool { + return strings.Contains(err.Error(), "path escapes from parent") } // folderNames are the names of picture files that mean "the cover", best first. @@ -574,15 +613,15 @@ var folderNames = []string{"cover", "folder", "front", "album", "albumart"} // folderPictureNames returns the names of the cover-like picture files of a folder of // the library, best name first. Only regular files inside the library are listed (a // link is not followed), and none above MaxSourceBytes. -func folderPictureNames(rootDir, dir string) []string { +func folderPictureNames(rootDir, dir string) (names []string, readable bool) { root, err := os.OpenRoot(filepath.Clean(rootDir)) if err != nil { - return nil + return nil, false } defer root.Close() entries, err := fs.ReadDir(root.FS(), dir) if err != nil { - return nil + return nil, escapesLibrary(err) } type pick struct { rank int @@ -617,7 +656,7 @@ func folderPictureNames(rootDir, dir string) []string { for i, p := range picks { out[i] = p.name } - return out + return out, true } // readFolderPicture reads one of the files folderPictureNames returned, through