fix(images): a refresh never wipes covers, the job sees outages and starts cleanly

- A refresh (or any rebuild) while the Cover Art Archive is switched off, but its data
  not taken back, no longer turns the covers it gave into "no cover" and deletes their
  files: the stored cover is kept and its date renewed, so the files are not searched
  again at every request.
- When none of an album's files and folders can be looked at (an unmounted disk, a
  permission), that is ErrFilesUnreadable (503 files_unreadable), not "no picture": nothing
  is remembered, nothing built is touched, the archive is not asked, and the job counts it
  as a failure and gives up after five. A link that leaves the library is still just refused.
- The job calls the build for every album that is not good yet and does not serve an older
  cover on an outage, so a dead provider shows as failures and trips the limit instead of
  being reported as covers from the archive.
- Start marks the job as running before counting the albums and does not hold the lock
  meanwhile; a stop that comes back as a plain driver error is not a failure; a job that
  finished by itself is never marked stopped.
- Spec and README say so; the tests cover each case, including a list that fails midway.
This commit is contained in:
tgiachi 2026-11-12 09:43:51 +01:00
commit 345ad8eecb
8 changed files with 370 additions and 85 deletions

View file

@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers).
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again; it never replaces a cover by "no cover" because a disk was not mounted, and keeps the covers that came from the archive if you switched it off).
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).

View file

@ -283,3 +283,21 @@ func TestAnOutageOfTheArchiveIsABadGatewayNotAnInternalError(t *testing.T) {
t.Errorf("%d %s, want 502 provider_unavailable", rec.Code, rec.Body.String())
}
}
func TestACoverOfFilesThatCannotBeReadIsServiceUnavailableNotNoCover(t *testing.T) {
e := newCoversEnv(t)
if err := os.RemoveAll(e.root); err != nil { // the disk went away
t.Fatal(err)
}
rec := e.get("GET", "/albums/"+id(e.without)+"/cover", e.token)
var body struct {
Error struct{ Code string } `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != http.StatusServiceUnavailable || body.Error.Code != "files_unreadable" {
t.Errorf("%d %s, want 503 files_unreadable", rec.Code, rec.Body.String())
}
if n := atomic.LoadInt32(&e.archive.calls); n != 0 {
t.Errorf("the archive was asked %d times about files that could not be read", n)
}
}

View file

@ -46,6 +46,10 @@ func (h *coverHandlers) album(c *gin.Context) {
case errors.Is(err, images.ErrNoCover):
writeError(c, http.StatusNotFound, "cover_not_found", "this album has no cover")
return
case errors.Is(err, images.ErrFilesUnreadable):
_ = c.Error(err)
writeError(c, http.StatusServiceUnavailable, "files_unreadable", "the files of the album could not be read, try again later")
return
case errors.Is(err, images.ErrSourceUnavailable):
_ = c.Error(err) // keep the cause in the log
writeError(c, http.StatusBadGateway, "provider_unavailable", "the cover provider could not be reached, try again later")

View file

@ -511,7 +511,10 @@ paths:
**This is the one request that goes through the whole library.** With the archive on, it sends
the release group id of every album that has no picture in its files, one after the other and
spaced out. It gives up, and says why in `last_error`, when the provider fails for five albums
in a row. `DELETE /library/covers` stops it.
in a row, and also when the files cannot be read (an unmounted disk): it never turns that into
"no cover" and never replaces a cover that was built. A cover that came from the archive is kept
when the provider was switched off since, even with `refresh=true`; only taking its data back
(`DELETE /providers/coverartarchive/data`) removes those. `DELETE /library/covers` stops it.
security: [{bearerAuth: []}]
parameters:
- name: refresh
@ -537,6 +540,11 @@ paths:
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'500':
description: The albums could not be counted (`internal`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
delete:
tags: [Scan]
operationId: stopCoversJob
@ -1310,6 +1318,14 @@ paths:
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'503':
description: |
None of the album's files or folders could be read, for example because the disk is not
mounted (`files_unreadable`). That says nothing about whether there is a picture, so it is
not remembered as "no cover" and nothing already built is touched. Try again later.
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
head:
tags: [Catalog]
operationId: headAlbumCover
@ -1339,6 +1355,7 @@ paths:
'401': {$ref: '#/components/responses/Unauthorized'}
'404': {description: "No such album, or it has no cover."}
'502': {description: "The cover provider could not be reached."}
'503': {description: "None of the album's files could be read."}
/tracks:
get:

View file

@ -119,7 +119,7 @@ func (s *Service) buildArtist(artistID int64) (Record, error) {
return Record{}, err
}
rec.Gen = cur.Gen
saved, err := s.commit(ctx, kindArtist, artistID, rec, pic, epoch)
saved, err := s.commit(ctx, kindArtist, artistID, rec, pic, epoch, false)
if errors.Is(err, errPurged) {
purged = true
continue

View file

@ -77,20 +77,32 @@ func NewJob(svc *Service, albums AlbumLister) *Job { return &Job{svc: svc, album
// Start begins the job in the background. ctx bounds it, so pass one that lives as long as the
// application, not an HTTP request. It returns ErrJobRunning when one is already running.
func (j *Job) Start(ctx context.Context, o JobOptions) error {
now := j.svc.now()
runCtx, cancel := context.WithCancel(ctx)
// The job is marked as running before the albums are counted, which can take a while on a
// big library, so that the status can be read meanwhile and nobody starts a second one.
j.mu.Lock()
defer j.mu.Unlock()
if j.status.Running {
j.mu.Unlock()
cancel()
return ErrJobRunning
}
now := j.svc.now()
total, err := j.albums.CountAlbumsToBuild(ctx, o.Refresh, now)
j.cancel = cancel
j.cur, j.total = JobSummary{}, 0
j.status.Running, j.status.StartedAt = true, now
j.mu.Unlock()
total, err := j.albums.CountAlbumsToBuild(runCtx, o.Refresh, now)
if err != nil {
j.mu.Lock()
j.status.Running = false
j.mu.Unlock()
cancel()
return fmt.Errorf("count the albums: %w", err)
}
runCtx, cancel := context.WithCancel(ctx)
j.cancel = cancel
j.cur, j.total = JobSummary{}, total
j.status.Running, j.status.StartedAt = true, now
j.mu.Lock()
j.total = total
j.mu.Unlock()
go j.run(runCtx, cancel, o, now)
return nil
}
@ -127,11 +139,14 @@ func (j *Job) run(ctx context.Context, cancel context.CancelFunc, o JobOptions,
defer cancel() // release the context when it ends by itself
reason := ""
failures := 0
stopped := false
loop:
for after := int64(0); ; {
ids, err := j.albums.AlbumsToBuild(ctx, after, jobPage, o.Refresh, started)
if err != nil {
if ctx.Err() == nil {
if ctx.Err() != nil {
stopped = true
} else {
reason = "could not list the albums: " + err.Error()
}
break
@ -142,12 +157,14 @@ loop:
for _, id := range ids {
after = id
if ctx.Err() != nil {
stopped = true
break loop
}
source, err := j.svc.EnsureAlbumCover(ctx, id, o.Refresh)
switch {
case ctx.Err() != nil && (errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded)):
break loop // stopped while this one was being built: it does not count
case err != nil && ctx.Err() != nil:
stopped = true
break loop // stopped while this one was being built: whatever came back, it does not count
case errors.Is(err, ErrAlbumNotFound):
failures = 0
j.count(func(s *JobSummary) { s.Checked++; s.Skipped++ })
@ -178,7 +195,7 @@ loop:
j.mu.Lock()
defer j.mu.Unlock()
sum := j.cur
sum.Stopped = ctx.Err() != nil
sum.Stopped = stopped
j.status.Running = false
j.status.LastFinishedAt = j.svc.now()
j.status.Last = &sum

View file

@ -4,6 +4,8 @@ import (
"context"
"errors"
"image/color"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
@ -11,11 +13,14 @@ import (
)
type fakeLister struct {
mu sync.Mutex
ids []int64
err error
refresh []bool
pages int
mu sync.Mutex
ids []int64
err error
failOnPage int // fail from this page on (1 = the first); 0 = never, only err
countGate chan struct{} // when set, CountAlbumsToBuild waits for it to be closed
refresh []bool
pages int
pageSize int // 0 = as asked
}
func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, refresh bool, _ time.Time) ([]int64, error) {
@ -23,8 +28,11 @@ func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, re
defer f.mu.Unlock()
f.refresh = append(f.refresh, refresh)
f.pages++
if f.err != nil {
return nil, f.err
if f.err != nil || (f.failOnPage > 0 && f.pages >= f.failOnPage) {
return nil, errors.New("database is locked")
}
if f.pageSize > 0 && f.pageSize < limit {
limit = f.pageSize
}
var out []int64
for _, id := range f.ids {
@ -37,8 +45,12 @@ func (f *fakeLister) AlbumsToBuild(_ context.Context, after int64, limit int, re
func (f *fakeLister) CountAlbumsToBuild(_ context.Context, _ bool, _ time.Time) (int, error) {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.ids), f.err
gate, n, err := f.countGate, len(f.ids), f.err
f.mu.Unlock()
if gate != nil {
<-gate
}
return n, err
}
type jobRig struct {
@ -247,16 +259,194 @@ func TestAnAlbumThatWentIsSkippedNotCountedAsAFailure(t *testing.T) {
}
}
func TestAProblemListingTheAlbumsEndsTheJobWithItsReason(t *testing.T) {
func TestAProblemListingTheAlbumsAtTheStartIsReportedAtOnce(t *testing.T) {
r := newJobRig(t)
r.lister.err = errors.New("database is locked")
if err := r.job.Start(context.Background(), JobOptions{}); err == nil {
// The count is read before starting, so the error comes at once.
waitDone(t, r.job)
t.Fatal("the job started although the albums could not be counted")
}
st := r.job.Status()
if st.Running {
t.Error("the job is still running after the list failed")
if st := r.job.Status(); st.Running {
t.Error("the job says it runs after it failed to start")
}
r.lister.err = nil
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Errorf("it cannot start again after the failure: %v", err)
}
waitDone(t, r.job)
}
func TestAProblemListingTheAlbumsDuringTheJobEndsItWithItsReason(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{Data: jpegBytes(t, 400, 400, color.White)}, nil)
r.lister.pageSize, r.lister.failOnPage = 2, 2 // the first page of two albums works, the second fails
if err := r.job.Start(context.Background(), JobOptions{}); err != nil {
t.Fatal(err)
}
st := waitDone(t, r.job)
if st.Last == nil || st.Last.Checked != 2 || st.LastError == "" || st.Last.Stopped {
t.Errorf("status = %+v, want two albums done and the reason the list failed", st)
}
}
func TestStartingDoesNotHoldTheStatusWhileTheAlbumsAreCounted(t *testing.T) {
r := newJobRig(t)
r.lister.countGate = make(chan struct{})
started := make(chan error, 1)
go func() { started <- r.job.Start(context.Background(), JobOptions{}) }()
time.Sleep(40 * time.Millisecond)
done := make(chan JobStatus, 1)
go func() { done <- r.job.Status() }()
select {
case st := <-done:
if !st.Running {
t.Errorf("status = %+v, want it to say the job is starting", st)
}
case <-time.After(time.Second):
t.Fatal("Status waited for the count to end")
}
if err := r.job.Start(context.Background(), JobOptions{}); !errors.Is(err, ErrJobRunning) {
t.Errorf("a second start while the first counts: err = %v", err)
}
close(r.lister.countGate)
if err := <-started; err != nil {
t.Fatal(err)
}
waitDone(t, r.job)
}
func TestAStopThatReturnsAPlainErrorIsNotAFailure(t *testing.T) {
r := newJobRig(t)
ctx, cancel := context.WithCancel(context.Background())
r.svc.SetAlbumRemote(remoteFunc(func(context.Context, int64) (Found, error) {
cancel() // stopped while the provider is asked, and what comes back is not a context error
return Found{}, errors.New("interrupted")
}))
_ = r.job.Start(ctx, JobOptions{})
st := waitDone(t, r.job)
if st.Last == nil || st.Last.Failed != 0 || !st.Last.Stopped {
t.Errorf("summary = %+v, want stopped and no failure", st.Last)
}
}
func TestAJobThatEndedByItselfIsNotStopped(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{}, ErrNoImage)
_ = r.job.Start(context.Background(), JobOptions{})
st := waitDone(t, r.job)
r.job.Stop() // too late: nothing runs
if st.Last.Stopped || r.job.Status().Last.Stopped {
t.Errorf("a finished job is marked stopped: %+v", r.job.Status().Last)
}
}
// ---- what a refresh must never do ----
func TestARefreshWithTheArchiveOffKeepsTheCoversThatCameFromIt(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.RGBA{0, 90, 0, 255})}, nil)
_ = r.job.Start(context.Background(), JobOptions{})
waitDone(t, r.job)
before := r.store.recs[key("album", 1)]
if before.Source != SourceCoverArt {
t.Fatalf("record = %+v", before)
}
// The admin switches the provider off (not purged) and refreshes.
r.remote.set(Found{}, ErrProviderDisabled)
r.now = r.now.Add(time.Hour)
_ = r.job.Start(context.Background(), JobOptions{Refresh: true})
st := waitDone(t, r.job)
after := r.store.recs[key("album", 1)]
if after.Status != StatusOK || after.Source != SourceCoverArt || after.ETag != before.ETag {
t.Errorf("record = %+v, want the archive cover kept", after)
}
if _, err := os.Stat(filepath.Join(r.data, "images", "album-remote", "1-large.jpg")); err != nil {
t.Errorf("the file of an archive cover went when the provider was switched off: %v", err)
}
if st.Last.Failed != 0 {
t.Errorf("summary = %+v", *st.Last)
}
// An album that had none stays with none, of course.
if got := r.store.recs[key("album", 2)]; got.Status != StatusOK {
t.Errorf("album 2 = %+v", got)
}
}
func TestACoverFromTheArchiveWhoseProviderIsOffIsNotSearchedAgainAtEveryRequest(t *testing.T) {
r := newRemoteRig(t)
r.rig.read(t, 1, Large) // from the archive
r.remote.set(Found{}, ErrProviderDisabled)
r.now = r.now.Add(CoverTTL + time.Hour)
r.rig.read(t, 1, Large) // kept, and its date renewed
reads := atomic.LoadInt32(&r.reads)
for i := 0; i < 3; i++ {
r.rig.read(t, 1, Large)
}
if got := atomic.LoadInt32(&r.reads); got != reads {
t.Errorf("the files were searched again %d times within the new period", got-reads)
}
}
func TestAnUnreachableLibraryIsAFailureNotNoPicture(t *testing.T) {
r := newRemoteRig(t)
r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.RGBA{200, 0, 0, 255}))
first := r.rig.read(t, 1, Large)
// The mount goes away: nothing under the root can be opened.
if err := os.RemoveAll(r.root); err != nil {
t.Fatal(err)
}
r.store.clear("album", 1) // a rescan, or a refresh, wants to look again
_, err := r.svc.AlbumCover(context.Background(), 1, Large)
if !errors.Is(err, ErrFilesUnreadable) {
t.Fatalf("err = %v, want ErrFilesUnreadable", err)
}
if rec := r.store.recs[key("album", 1)]; rec.Status == StatusNone {
t.Errorf("record = %+v: an unreadable library must not become 'no cover'", rec)
}
if n := atomic.LoadInt32(&r.remote.calls); n != 0 {
t.Errorf("the archive was asked %d times about files that could not even be read", n)
}
if entries, _ := os.ReadDir(filepath.Join(r.data, "images", "album")); len(entries) != 2 {
t.Errorf("%d files left, want the two sizes still there", len(entries))
}
_ = first
}
func TestARefreshOfAnUnreachableLibraryGivesUpInsteadOfWipingEverything(t *testing.T) {
r := newJobRig(t)
r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.White))
_ = r.job.Start(context.Background(), JobOptions{})
waitDone(t, r.job)
if err := os.RemoveAll(r.root); err != nil {
t.Fatal(err)
}
_ = r.job.Start(context.Background(), JobOptions{Refresh: true})
st := waitDone(t, r.job)
if st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.None != 0 {
t.Errorf("status = %+v, want it to give up with failures and not count 'none'", st)
}
for id := int64(1); id <= 6; id++ {
if rec := r.store.recs[key("album", id)]; rec.Status != StatusOK {
t.Errorf("album %d: %+v, want its cover kept", id, rec)
}
}
}
func TestADeadProviderIsSeenEvenWhenOldCoversCouldBeServed(t *testing.T) {
r := newJobRig(t)
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil)
_ = r.job.Start(context.Background(), JobOptions{})
waitDone(t, r.job)
// Every archive cover is old now, and the archive is down.
r.now = r.now.Add(CoverTTL + time.Hour)
r.remote.set(Found{}, ErrSourceUnavailable)
atomic.StoreInt32(&r.remote.calls, 0)
_ = r.job.Start(context.Background(), JobOptions{})
st := waitDone(t, r.job)
if st.Last.Failed != MaxConsecutiveFailures || st.LastError == "" || st.Last.FromArchive != 0 {
t.Errorf("status = %+v, want it to notice the outage and give up", st)
}
if n := atomic.LoadInt32(&r.remote.calls); n != int32(MaxConsecutiveFailures) {
t.Errorf("the archive was asked %d times", n)
}
}

View file

@ -73,6 +73,11 @@ var (
ErrNoCover = errors.New("this album has no cover")
ErrInvalidSize = errors.New("invalid size")
// ErrFilesUnreadable means none of the files and folders of the album could be looked
// at (an unmounted disk, a permission): that says nothing about whether there is a picture,
// so it is not remembered and nothing already built is touched.
ErrFilesUnreadable = errors.New("the files of the album could not be read")
// Answers of an ArtistSource.
ErrNoImage = errors.New("there is no free picture of this artist")
ErrProviderDisabled = errors.New("the provider is switched off")
@ -269,28 +274,29 @@ func (s *Service) buildShared(ctx context.Context, albumID int64) (Record, error
}
}
// EnsureAlbumCover makes sure the cover of an album is built, as opening it would, and says
// where it came from (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album
// has none. With refresh the files are searched again even if a good cover is stored. It is
// what the job that builds every cover calls.
// EnsureAlbumCover makes sure the cover of an album is built, and says where it came from
// (SourceEmbedded, SourceFolder or SourceCoverArt), or "" when the album has none. A stored
// cover that is still good is left alone; otherwise, or with refresh, the album is searched
// again, and a failure is returned as one: unlike an HTTP request, which is better served an
// older cover, the job that builds every cover must see that the provider is down. It is
// what that job calls.
func (s *Service) EnsureAlbumCover(ctx context.Context, albumID int64, refresh bool) (string, error) {
if albumID <= 0 {
return "", ErrAlbumNotFound
}
if !refresh {
c, err := s.AlbumCover(ctx, albumID, Small)
switch {
case errors.Is(err, ErrNoCover):
return "", nil
case err != nil:
return "", err
}
_ = c.File.Close()
rec, ok, err := s.store.Get(ctx, kindAlbum, albumID)
if err != nil || !ok {
return "", err
if err != nil {
return "", fmt.Errorf("read picture record: %w", err)
}
if ok && rec.Status == StatusNone && s.now().Sub(rec.CheckedAt) < NoCoverTTL {
return "", nil
}
if ok && rec.Status == StatusOK && s.now().Sub(rec.CheckedAt) < CoverTTL {
if _, err := os.Stat(s.filePath(folderOf(kindAlbum, rec.Source), albumID, Small)); err == nil {
return rec.Source, nil
}
}
return rec.Source, nil
}
rec, err := s.buildShared(ctx, albumID)
if err != nil {
@ -351,13 +357,24 @@ func (s *Service) build(albumID int64) (Record, error) {
return Record{}, err
}
epoch := s.purgeCount()
var pic *Processed
rec, pic, err = s.search(ctx, albumID, files)
var (
pic *Processed
off bool
)
rec, pic, off, err = s.search(ctx, albumID, files)
if err != nil {
return Record{}, err
}
rec.Gen = cur.Gen
saved, err := s.commit(ctx, kindAlbum, albumID, rec, pic, epoch)
// The provider was switched off (its data was not taken back): a cover it gave stays,
// instead of being replaced by "no cover" and deleted. Its date is renewed so that
// the files are not searched again at every request.
keep := off && pic == nil && cur.Status == StatusOK && cur.Source == SourceCoverArt
if keep {
rec = cur
rec.CheckedAt = s.now()
}
saved, err := s.commit(ctx, kindAlbum, albumID, rec, pic, epoch, keep)
if errors.Is(err, errPurged) {
purged = true
continue
@ -378,26 +395,26 @@ func (s *Service) build(albumID int64) (Record, error) {
// search finds a picture: in the user's own files first, at most MaxParallelBuilds at a
// time, and, only when they have none, from the provider of covers if there is one. The
// picture is not stored yet (see commit); nil means there is none.
func (s *Service) search(ctx context.Context, albumID int64, files []File) (Record, *Processed, error) {
func (s *Service) search(ctx context.Context, albumID int64, files []File) (rec Record, pic *Processed, providerOff bool, err error) {
none := Record{Status: StatusNone, CheckedAt: s.now()}
p, source, ok, err := s.searchFiles(ctx, files)
if err != nil {
return Record{}, nil, err
return Record{}, nil, false, err
}
if ok {
return s.record(p, source, Credit{}, none.CheckedAt), &p, nil
return s.record(p, source, Credit{}, none.CheckedAt), &p, false, nil
}
if s.remote == nil {
return none, nil, nil
return none, nil, false, nil
}
rp, credit, ok, err := s.searchRemote(ctx, albumID)
rp, credit, ok, off, err := s.searchRemote(ctx, albumID)
if err != nil {
return Record{}, nil, err
return Record{}, nil, false, err
}
if !ok {
return none, nil, nil
return none, nil, off, nil
}
return s.record(rp, SourceCoverArt, credit, none.CheckedAt), &rp, nil
return s.record(rp, SourceCoverArt, credit, none.CheckedAt), &rp, false, nil
}
func (s *Service) searchFiles(ctx context.Context, files []File) (Processed, string, bool, error) {
@ -407,13 +424,16 @@ func (s *Service) searchFiles(ctx context.Context, files []File) (Processed, str
case <-ctx.Done():
return Processed{}, "", false, ctx.Err()
}
p, source, ok := s.find(files)
p, source, ok, readable := s.find(files)
if !ok && !readable {
return Processed{}, "", false, ErrFilesUnreadable
}
return p, source, ok, nil
}
// searchRemote asks the provider of covers. "There is none" (no picture, no id, the
// provider is off, a picture that is not one) is an answer; a failure is returned.
func (s *Service) searchRemote(ctx context.Context, albumID int64) (Processed, Credit, bool, error) {
func (s *Service) searchRemote(ctx context.Context, albumID int64) (p Processed, credit Credit, ok, providerOff bool, err error) {
// This context is nobody's: when it ends here, the build ran out of time waiting for the
// provider, which is an outage of the provider and not an internal error.
outage := func(err error) error {
@ -426,20 +446,22 @@ func (s *Service) searchRemote(ctx context.Context, albumID int64) (Processed, C
case s.artistSl <- struct{}{}: // the slots of the work that waits for the network
defer func() { <-s.artistSl }()
case <-ctx.Done():
return Processed{}, Credit{}, false, outage(ctx.Err())
return Processed{}, Credit{}, false, false, outage(ctx.Err())
}
found, err := s.remote.AlbumImage(ctx, albumID)
switch {
case errors.Is(err, ErrNoImage), errors.Is(err, ErrNoID), errors.Is(err, ErrProviderDisabled):
return Processed{}, Credit{}, false, nil
case errors.Is(err, ErrProviderDisabled):
return Processed{}, Credit{}, false, true, nil
case errors.Is(err, ErrNoImage), errors.Is(err, ErrNoID):
return Processed{}, Credit{}, false, false, nil
case err != nil:
return Processed{}, Credit{}, false, outage(err)
return Processed{}, Credit{}, false, false, outage(err)
}
p, err := ProcessLimited(found.Data, MaxArtistPixels)
p, err = ProcessLimited(found.Data, MaxArtistPixels)
if err != nil {
return Processed{}, Credit{}, false, nil // not a usable picture: there is no cover to show
return Processed{}, Credit{}, false, false, nil // not a usable picture: there is no cover to show
}
return p, found.Credit, true, nil
return p, found.Credit, true, false, nil
}
// record describes a picture that was found.
@ -459,7 +481,7 @@ func (s *Service) record(p Processed, source string, credit Credit, at time.Time
// provider were taken back since the build began (errPurged) or a change moved the
// record's generation (saved is false). Both happen under the read side of the purge lock,
// so a purge waits for it and the next ones are refused.
func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, pic *Processed, epoch uint64) (bool, error) {
func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, pic *Processed, epoch uint64, keepFiles bool) (bool, error) {
s.purgeMu.RLock()
defer s.purgeMu.RUnlock()
if s.purges != epoch {
@ -475,7 +497,7 @@ func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record,
if err != nil {
return false, fmt.Errorf("save picture record: %w", err)
}
if saved {
if saved && !keepFiles {
// What the record no longer describes must not stay on disk: the files of the other
// folder when the cover moved between the user's files and the archive, and all of
// them when there is no cover any more.
@ -504,15 +526,19 @@ func (s *Service) removeFiles(folder string, id int64) {
}
// find returns the first usable picture: embedded in one of the first files, then an
// image in the folder of one of them.
func (s *Service) find(files []File) (Processed, string, bool) {
// image in the folder of one of them. readable says whether anything of the library could
// be looked at: when none of the files and none of the folders could, "no picture" would be
// a lie (an unmounted disk, a permission), and the caller must not remember it.
func (s *Service) find(files []File) (p Processed, source string, ok, readable bool) {
if len(files) > MaxFilesTried {
files = files[:MaxFilesTried]
}
for _, f := range files {
if data, ok := s.embeddedPicture(f); ok {
data, found, seen := s.embeddedPicture(f)
readable = readable || seen
if found {
if p, err := Process(data); err == nil {
return p, SourceEmbedded, true
return p, SourceEmbedded, true, true
}
}
}
@ -532,17 +558,19 @@ func (s *Service) find(files []File) (Processed, string, bool) {
}
}
for _, d := range dirs {
for _, name := range folderPictureNames(d.Root, d.RelPath) {
data, ok := readFolderPicture(d.Root, d.RelPath, name)
if !ok {
names, seen := folderPictureNames(d.Root, d.RelPath)
readable = readable || seen
for _, name := range names {
data, got := readFolderPicture(d.Root, d.RelPath, name)
if !got {
continue
}
if p, err := Process(data); err == nil {
return p, SourceFolder, true
return p, SourceFolder, true, true
}
}
}
return Processed{}, "", false
return Processed{}, "", false, readable
}
// discFolder matches the name of a folder that holds one disc of an album: "CD1",
@ -551,21 +579,32 @@ var discFolder = regexp.MustCompile(`(?i)^(cd|dis[ck])[ _.-]*\d+$`)
// embeddedPicture reads the picture inside an audio file, after checking through
// os.Root that the file is a regular file inside its library.
func (s *Service) embeddedPicture(f File) ([]byte, bool) {
func (s *Service) embeddedPicture(f File) (data []byte, found, readable bool) {
root, err := os.OpenRoot(filepath.Clean(f.Root))
if err != nil {
return nil, false
return nil, false, false
}
defer root.Close()
info, err := root.Stat(f.RelPath)
if err != nil || !info.Mode().IsRegular() {
return nil, false
if err != nil {
// A link that leaves the library is refused on purpose: the library itself was read.
return nil, false, escapesLibrary(err)
}
data, err := s.embedded(filepath.Join(filepath.Clean(f.Root), filepath.FromSlash(f.RelPath)))
if !info.Mode().IsRegular() {
return nil, false, true
}
data, err = s.embedded(filepath.Join(filepath.Clean(f.Root), filepath.FromSlash(f.RelPath)))
if err != nil || len(data) == 0 {
return nil, false
return nil, false, true
}
return data, true
return data, true, true
}
// escapesLibrary says whether an error is os.Root refusing a link that leads outside the
// library. That is a decision of ours, not a disk that cannot be read, and os.Root has no
// exported error for it.
func escapesLibrary(err error) bool {
return strings.Contains(err.Error(), "path escapes from parent")
}
// folderNames are the names of picture files that mean "the cover", best first.
@ -574,15 +613,15 @@ var folderNames = []string{"cover", "folder", "front", "album", "albumart"}
// folderPictureNames returns the names of the cover-like picture files of a folder of
// the library, best name first. Only regular files inside the library are listed (a
// link is not followed), and none above MaxSourceBytes.
func folderPictureNames(rootDir, dir string) []string {
func folderPictureNames(rootDir, dir string) (names []string, readable bool) {
root, err := os.OpenRoot(filepath.Clean(rootDir))
if err != nil {
return nil
return nil, false
}
defer root.Close()
entries, err := fs.ReadDir(root.FS(), dir)
if err != nil {
return nil
return nil, escapesLibrary(err)
}
type pick struct {
rank int
@ -617,7 +656,7 @@ func folderPictureNames(rootDir, dir string) []string {
for i, p := range picks {
out[i] = p.name
}
return out
return out, true
}
// readFolderPicture reads one of the files folderPictureNames returned, through