feat(images): album covers from the Cover Art Archive

- The scan reads the MusicBrainz release group id of an album from the tags of its
  files (MUSICBRAINZ_RELEASEGROUPID, a vorbis comment or an ID3 frame) and stores it in
  albums.mbid: it is taken only when the file has an album and exactly one valid id, an
  album keeps the one it has, and an id that another album owns is not given twice. A full
  rescan reads it for music that was already indexed.
- New provider "coverartarchive" (off by default, no key, contact in the User-Agent). When
  an album has no picture in the user's own files, which always come first, its cover is
  asked from the archive by that id and never by a title: only the id leaves the server.
  The same GET /albums/{id}/cover; the record keeps the source. An outage is a 502
  provider_unavailable and is not remembered, and an older cover is served in its place.
- The archive answers with three redirects (coverartarchive.org, archive.org, a data node
  whose name changes). The outbound client can now follow redirects, but only the ones the
  caller allows, up to a limit; here that is https on archive.org and its subdomains.
- A purge counter now guards the covers too: a read that started before the provider's data
  was taken back is not kept. Removing the provider's data forgets the covers that came from
  it and deletes their files, and leaves the ones from the user's files.
- Spec, contract and HTTP tests, README updated; the facts of the real archive are written down.

Closes #43
This commit is contained in:
tgiachi 2026-11-12 09:09:31 +01:00
commit d5ed0f2e9c
27 changed files with 1349 additions and 92 deletions

View file

@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers).
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere. It also takes `?access_token=`, so an `<img>` element can use it.
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere. It also takes `?access_token=`, so an `<img>` element can use it.
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
@ -188,7 +188,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
## External providers
Sonda can ask Last.fm, MusicBrainz, Wikipedia and Wikimedia Commons about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, Wikipedia (with Wikidata) for descriptions of artists, and Wikimedia Commons (with Wikidata) for free photos of artists.
Sonda can ask Last.fm, MusicBrainz, Wikipedia, Wikimedia Commons and the Cover Art Archive about artists and albums. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, Wikipedia (with Wikidata) for descriptions of artists, Wikimedia Commons (with Wikidata) for free photos of artists, and the Cover Art Archive for the cover of an album that has none in your files.
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
@ -205,7 +205,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.

View file

@ -27,6 +27,7 @@ import (
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/commons"
"codeberg.org/squidd/sonda/internal/providers/coverart"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
"codeberg.org/squidd/sonda/internal/providers/outbound"
@ -58,9 +59,10 @@ const (
// commonsMaxBody leaves room for a 960 pixel copy of a picture, which is some hundred kilobytes.
const (
commonsMinInterval = 300 * time.Millisecond
commonsAPIMaxBody = 1 << 20
commonsMaxBody = 8 << 20
coverartMinInterval = 500 * time.Millisecond
commonsMinInterval = 300 * time.Millisecond
commonsAPIMaxBody = 1 << 20
commonsMaxBody = 8 << 20
)
func main() {
@ -183,19 +185,48 @@ func run() error {
pictures.SetArtistSource(&images.CommonsSource{
Artists: discoveryRepo, Settings: providerSettings, MusicBrainz: musicbrainzClient, Commons: commonsClient,
})
// Covers from the Cover Art Archive when the user's own files have none. The archive
// answers with redirects (to archive.org and its data nodes): they are followed, to nobody else.
coverartClient := coverart.New(outbound.New(outbound.Options{
UserAgentFunc: providerSettings.UserAgentFor(providers.CoverArtArchive, version),
MinInterval: coverartMinInterval,
MaxBody: commonsMaxBody,
FollowRedirect: coverart.RedirectAllowed,
}), coverart.BaseURL)
providerSettings.SetChecker(providers.CoverArtArchive, func(ctx context.Context, _ string) error {
return coverartClient.Probe(ctx)
})
pictures.SetAlbumRemote(&images.CoverArtSource{Albums: imageRepo, Settings: providerSettings, Client: coverartClient})
providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
if name == providers.Commons {
pictures.BeginArtistPurge() // a picture being read right now must not be kept
switch name {
case providers.Commons:
pictures.BeginPurge() // a picture being read right now must not be kept
out, err := discoveryRepo.Purge(ctx, name)
if err != nil {
return out, err
}
// The records are gone; the pictures themselves are files in the data folder.
if _, err := pictures.RemoveArtistFiles(); err != nil {
return out, fmt.Errorf("remove the pictures: %w", err)
}
return out, nil
case providers.CoverArtArchive:
pictures.BeginPurge()
// The records say which albums have a cover from the archive: read them before they go.
ids, err := imageRepo.OwnersBySource(ctx, "album", images.SourceCoverArt)
if err != nil {
return providers.Purged{}, err
}
out, err := discoveryRepo.Purge(ctx, name)
if err != nil {
return out, err
}
if _, err := pictures.RemoveAlbumFiles(ids); err != nil {
return out, fmt.Errorf("remove the covers: %w", err)
}
return out, nil
}
out, err := discoveryRepo.Purge(ctx, name)
if err != nil || name != providers.Commons {
return out, err
}
// The records are gone; the pictures themselves are files in the data folder.
if _, err := pictures.RemoveArtistFiles(); err != nil {
return out, fmt.Errorf("remove the pictures: %w", err)
}
return out, nil
return discoveryRepo.Purge(ctx, name)
})
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{

View file

@ -426,8 +426,8 @@ func TestResponsesMatchTheSpec(t *testing.T) {
// ---- providers ----
listed := c.expect(c.call("GET", "/providers", token, nil), 200, "list providers")["providers"].([]any)
if len(listed) != 4 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false || listed[3].(map[string]any)["enabled"] != false {
t.Fatalf("providers = %v, want four, all off", listed)
if len(listed) != 5 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false || listed[3].(map[string]any)["enabled"] != false || listed[4].(map[string]any)["enabled"] != false {
t.Fatalf("providers = %v, want five, all off", listed)
}
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")

View file

@ -2,6 +2,7 @@ package httpapi
import (
"bytes"
"context"
"encoding/json"
"image"
"image/color"
@ -10,6 +11,7 @@ import (
"net/http/httptest"
"os"
"path/filepath"
"sync/atomic"
"testing"
"github.com/uptrace/bun"
@ -19,7 +21,19 @@ import (
"codeberg.org/squidd/sonda/internal/storage"
)
type fakeArchive struct {
found images.Found
err error
calls int32
}
func (f *fakeArchive) AlbumImage(context.Context, int64) (images.Found, error) {
atomic.AddInt32(&f.calls, 1)
return f.found, f.err
}
type coversEnv struct {
archive *fakeArchive
h http.Handler
db *bun.DB
token string
@ -46,7 +60,8 @@ func testJPEG(t *testing.T, w, h int, c color.Color) []byte {
func newCoversEnv(t *testing.T) *coversEnv {
t.Helper()
e := &coversEnv{root: t.TempDir(), data: filepath.Join(t.TempDir(), "data")}
e := &coversEnv{root: t.TempDir(), data: filepath.Join(t.TempDir(), "data"),
archive: &fakeArchive{err: images.ErrNoImage}}
for _, dir := range []string{"With Art", "Without"} {
if err := os.MkdirAll(filepath.Join(e.root, dir), 0o755); err != nil {
t.Fatal(err)
@ -60,7 +75,9 @@ func newCoversEnv(t *testing.T) *coversEnv {
}
e.h, e.db = newTestEnvDB(t, func(d *Deps, db *bun.DB) {
repo := storage.NewImageRepo(db)
d.Images = images.NewService(e.data, repo, repo)
svc := images.NewService(e.data, repo, repo)
svc.SetAlbumRemote(e.archive)
d.Images = svc
})
e.token = registerAndLogin(t, e.h)
@ -227,3 +244,40 @@ func TestCoverIsBuiltAgainAfterTheFilesChange(t *testing.T) {
t.Errorf("after the file changed: %d %s", rec.Code, rec.Body.String())
}
}
func TestAnAlbumWithNoPictureGetsTheCoverOfTheArchive(t *testing.T) {
e := newCoversEnv(t)
e.archive.found, e.archive.err = images.Found{Data: testJPEG(t, 1200, 1200, color.RGBA{10, 100, 10, 255})}, nil
path := "/albums/" + id(e.without) + "/cover"
rec := e.get("GET", path, e.token)
if rec.Code != http.StatusOK || rec.Header().Get("Content-Type") != "image/jpeg" {
t.Fatalf("%d %s", rec.Code, rec.Body.String())
}
cfg, _, err := image.DecodeConfig(bytes.NewReader(rec.Body.Bytes()))
if err != nil || cfg.Width != 1000 {
t.Errorf("cover = %v %v, want 1000 wide", cfg, err)
}
// Stored: asked again, the archive is not.
e.get("GET", path+"?size=small", e.token)
if n := atomic.LoadInt32(&e.archive.calls); n != 1 {
t.Errorf("the archive was asked %d times, want 1", n)
}
// The files of the user come first: the album that has a picture never asks the archive.
e.get("GET", "/albums/"+id(e.withArt)+"/cover", e.token)
if n := atomic.LoadInt32(&e.archive.calls); n != 1 {
t.Errorf("the archive was asked about an album that has a picture in its files (%d calls)", n)
}
}
func TestAnOutageOfTheArchiveIsABadGatewayNotAnInternalError(t *testing.T) {
e := newCoversEnv(t)
e.archive.err = images.ErrSourceUnavailable
rec := e.get("GET", "/albums/"+id(e.without)+"/cover", e.token)
var body struct {
Error struct{ Code string } `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != http.StatusBadGateway || body.Error.Code != "provider_unavailable" {
t.Errorf("%d %s, want 502 provider_unavailable", rec.Code, rec.Body.String())
}
}

View file

@ -46,6 +46,10 @@ func (h *coverHandlers) album(c *gin.Context) {
case errors.Is(err, images.ErrNoCover):
writeError(c, http.StatusNotFound, "cover_not_found", "this album has no cover")
return
case errors.Is(err, images.ErrSourceUnavailable):
_ = c.Error(err) // keep the cause in the log
writeError(c, http.StatusBadGateway, "provider_unavailable", "the cover provider could not be reached, try again later")
return
case errors.Is(err, context.Canceled):
c.AbortWithStatus(statusClientClosed) // the client left; nobody is waiting
return

View file

@ -46,7 +46,7 @@ func TestProvidersStartOffAndSayWhatTheySend(t *testing.T) {
t.Fatalf("list: %d %v", r.Code, r.Body)
}
list := r.Body["providers"].([]any)
if len(list) != 4 {
if len(list) != 5 {
t.Fatalf("providers = %v", list)
}
for _, it := range list {

View file

@ -1173,7 +1173,12 @@ paths:
pictures are ignored, and only files inside the library are read.
For an album split in folders named like `CD1` or `Disc 2`, the picture in the folder
above them is used too. At most a couple of covers are built at the same time, so a
above them is used too. When the files have no picture at all, and an admin switched on
the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the
MusicBrainz release group id that the tags of the album's files carry (never by title; a
full rescan reads that tag again for music indexed before); only that id leaves the server.
The files of the user always come first. That request can take several seconds and may
answer `provider_unavailable`. At most a couple of covers are built at the same time, so a
client that asks for many at once may wait.
An album with no picture answers `cover_not_found`, and the server does not look again
@ -1228,6 +1233,11 @@ paths:
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'502':
description: The cover provider could not be reached (`provider_unavailable`). Try again later.
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
head:
tags: [Catalog]
operationId: headAlbumCover
@ -1256,6 +1266,7 @@ paths:
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404': {description: "No such album, or it has no cover."}
'502': {description: "The cover provider could not be reached."}
/tracks:
get:
@ -1447,7 +1458,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons, coverartarchive]}
put:
tags: [Providers]
operationId: updateProvider
@ -1485,7 +1496,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons, coverartarchive]}
post:
tags: [Providers]
operationId: testProvider
@ -1520,7 +1531,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons, coverartarchive]}
delete:
tags: [Providers]
operationId: purgeProviderData
@ -1981,7 +1992,7 @@ components:
additionalProperties: false
required: [name, enabled, needs_api_key, has_api_key, contact, sends, notes, updated_at]
properties:
name: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
name: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons, coverartarchive]}
enabled: {type: boolean, description: Whether Sonda may contact this provider.}
needs_api_key: {type: boolean}
has_api_key: {type: boolean, description: Whether a key is stored. The key itself is never returned.}
@ -2064,7 +2075,7 @@ components:
properties:
similar_artists: {type: integer, description: Similar-artist links removed.}
descriptions: {type: integer, description: Artist descriptions removed (only the data of Wikipedia has any).}
images: {type: integer, description: Pictures of artists removed (only the data of Commons has any).}
images: {type: integer, description: "Pictures removed: the photos of artists for Commons, the covers that came from the archive for the Cover Art Archive."}
tags: {type: integer, description: Tags removed.}
artists: {type: integer, description: "Artists removed because only the removed data named them and nobody has files for them."}

View file

@ -0,0 +1,190 @@
package images
import (
"bytes"
"context"
"errors"
"image/color"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
)
type fakeRemote struct {
mu sync.Mutex
found Found
err error
calls int32
gate chan struct{}
}
func (f *fakeRemote) AlbumImage(_ context.Context, id int64) (Found, error) {
atomic.AddInt32(&f.calls, 1)
f.mu.Lock()
gate, found, err := f.gate, f.found, f.err
f.mu.Unlock()
if gate != nil {
<-gate
}
return found, err
}
func (f *fakeRemote) set(found Found, err error) {
f.mu.Lock()
defer f.mu.Unlock()
f.found, f.err = found, err
}
type remoteRig struct {
*rig
remote *fakeRemote
}
func newRemoteRig(t *testing.T) *remoteRig {
t.Helper()
r := &remoteRig{rig: newRig(t), remote: &fakeRemote{}}
r.remote.found = Found{Data: jpegBytes(t, 1200, 1200, color.RGBA{10, 120, 10, 255})}
r.svc.SetAlbumRemote(r.remote)
return r
}
func TestTheFilesOfTheUserComeBeforeTheArchive(t *testing.T) {
r := newRemoteRig(t)
r.write(t, "Album One/cover.jpg", jpegBytes(t, 400, 400, color.RGBA{200, 0, 0, 255}))
r.rig.read(t, 1, Large)
if n := atomic.LoadInt32(&r.remote.calls); n != 0 {
t.Errorf("the archive was asked %d times although the files had a cover", n)
}
if got := r.store.recs["album/1"].Source; got != SourceFolder {
t.Errorf("source = %q", got)
}
}
func TestAnAlbumWithNoPictureInItsFilesGetsTheCoverOfTheArchiveAndKeepsIt(t *testing.T) {
r := newRemoteRig(t)
big := r.rig.read(t, 1, Large)
if len(big) == 0 {
t.Fatal("no cover")
}
rec := r.store.recs["album/1"]
if rec.Status != StatusOK || rec.Source != SourceCoverArt || rec.ETag == "" {
t.Errorf("record = %+v", rec)
}
for i := 0; i < 3; i++ {
r.rig.read(t, 1, Small)
}
if n := atomic.LoadInt32(&r.remote.calls); n != 1 {
t.Errorf("the archive was asked %d times, want 1: the cover is stored", n)
}
info, err := os.Stat(filepath.Join(r.data, "images", "album", "1-large.jpg"))
if err != nil || info.Mode().Perm()&0o077 != 0 {
t.Errorf("file: %v %v", info, err)
}
}
func TestWhatTheArchiveSaysAboutNoCoverIsRememberedLikeAnyNone(t *testing.T) {
for _, why := range []error{ErrNoImage, ErrNoID, ErrProviderDisabled} {
r := newRemoteRig(t)
r.remote.set(Found{}, why)
if _, err := r.svc.AlbumCover(context.Background(), 1, Large); !errors.Is(err, ErrNoCover) {
t.Errorf("%v: err = %v, want ErrNoCover", why, err)
}
if rec := r.store.recs["album/1"]; rec.Status != StatusNone {
t.Errorf("%v: record = %+v", why, rec)
}
for i := 0; i < 3; i++ {
_, _ = r.svc.AlbumCover(context.Background(), 1, Large)
}
if n := atomic.LoadInt32(&r.remote.calls); n != 1 {
t.Errorf("%v: asked %d times within the day, want 1", why, n)
}
}
}
func TestAnOutageOfTheArchiveIsNotRememberedAndIsReportedAsSuch(t *testing.T) {
r := newRemoteRig(t)
r.remote.set(Found{}, ErrSourceUnavailable)
if _, err := r.svc.AlbumCover(context.Background(), 1, Large); !errors.Is(err, ErrSourceUnavailable) {
t.Errorf("err = %v, want ErrSourceUnavailable", err)
}
if len(r.store.recs) != 0 {
t.Errorf("records = %v: the next request must try again", r.store.recs)
}
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil)
if got := r.rig.read(t, 1, Large); len(got) == 0 {
t.Error("no cover once the archive is back")
}
}
func TestAnOldCoverOfTheArchiveIsServedWhenItCannotBeCheckedAgain(t *testing.T) {
r := newRemoteRig(t)
first := r.rig.read(t, 1, Large)
r.now = r.now.Add(CoverTTL + time.Hour)
r.remote.set(Found{}, ErrSourceUnavailable)
if again := r.rig.read(t, 1, Large); !bytes.Equal(first, again) {
t.Error("the stored cover was not served when the archive could not be reached")
}
// And a cover that the files now have wins over the archive's.
r.write(t, "Album One/cover.jpg", jpegBytes(t, 300, 300, color.RGBA{0, 0, 200, 255}))
r.remote.set(Found{Data: jpegBytes(t, 600, 600, color.White)}, nil)
if now := r.rig.read(t, 1, Large); bytes.Equal(first, now) {
t.Error("the cover from the files did not replace the archive's")
}
if got := r.store.recs["album/1"].Source; got != SourceFolder {
t.Errorf("source = %q", got)
}
}
func TestAPurgeDuringTheReadOfACoverIsNotUndoneByIt(t *testing.T) {
r := newRemoteRig(t)
r.remote.gate = make(chan struct{})
done := make(chan error, 1)
go func() { _, err := r.svc.AlbumCover(context.Background(), 1, Large); done <- err }()
time.Sleep(60 * time.Millisecond)
r.svc.BeginPurge()
r.remote.set(Found{}, ErrProviderDisabled)
close(r.remote.gate)
if err := <-done; !errors.Is(err, ErrNoCover) {
t.Fatalf("err = %v, want no cover: the provider is off after the purge", err)
}
if rec := r.store.recs["album/1"]; rec.Status == StatusOK {
t.Errorf("record = %+v: what the read found before the purge came back", rec)
}
if entries, _ := os.ReadDir(filepath.Join(r.data, "images", "album")); len(entries) != 0 {
t.Errorf("%d files left by a read that started before the purge", len(entries))
}
}
func TestPurgingTheArchiveCoversRemovesThoseFilesOnly(t *testing.T) {
r := newRemoteRig(t)
r.rig.read(t, 1, Large) // from the archive
r.src.files[2] = []File{{Root: r.root, RelPath: "Album One/01.mp3"}}
r.write(t, "Album One/folder.jpg", jpegBytes(t, 300, 300, color.White))
r.remote.set(Found{}, ErrNoImage)
r.rig.read(t, 2, Large) // from the folder
n, err := r.svc.RemoveAlbumFiles([]int64{1, 99})
if err != nil || n != 2 {
t.Fatalf("removed %d files err = %v, want the two sizes of album 1", n, err)
}
if _, err := os.Stat(filepath.Join(r.data, "images", "album", "1-large.jpg")); !os.IsNotExist(err) {
t.Error("the archive cover is still there")
}
if _, err := os.Stat(filepath.Join(r.data, "images", "album", "2-large.jpg")); err != nil {
t.Errorf("the cover from the folder went too: %v", err)
}
// Ids that are not numbers of an album can never reach outside the folder.
if n, err := r.svc.RemoveAlbumFiles([]int64{-1, 0}); err != nil || n != 0 {
t.Errorf("bad ids: %d %v", n, err)
}
}
func TestSomethingThatIsNotAPictureFromTheArchiveIsNoCover(t *testing.T) {
r := newRemoteRig(t)
r.remote.set(Found{Data: []byte("<html>blocked</html>")}, nil)
if _, err := r.svc.AlbumCover(context.Background(), 1, Large); !errors.Is(err, ErrNoCover) {
t.Errorf("err = %v", err)
}
}

View file

@ -2,7 +2,6 @@ package images
import (
"context"
"crypto/sha256"
"errors"
"fmt"
"os"
@ -74,10 +73,10 @@ func (s *Service) ArtistImage(ctx context.Context, artistID int64, size Size) (*
// asked from Commons is 960 pixels wide, so this leaves room for a very tall one.
const MaxArtistPixels = 4_000_000
// BeginArtistPurge must be called before the pictures of artists are taken back (their
// BeginPurge must be called before the pictures of a provider are taken back (their
// records forgotten and their files removed). A read that started before it is not kept:
// it starts again, and is refused if the provider is by then off.
func (s *Service) BeginArtistPurge() {
func (s *Service) BeginPurge() {
s.purgeMu.Lock()
s.purges++
s.purgeMu.Unlock()
@ -119,7 +118,7 @@ func (s *Service) buildArtist(artistID int64) (Record, error) {
return Record{}, err
}
rec.Gen = cur.Gen
saved, err := s.saveArtist(ctx, artistID, rec, pic, epoch)
saved, err := s.commit(ctx, kindArtist, artistID, rec, pic, epoch)
if errors.Is(err, errPurged) {
continue
}
@ -158,35 +157,7 @@ func (s *Service) searchArtist(ctx context.Context, artistID int64) (Record, *Pr
if err != nil {
return none, nil, nil // what the provider sent is not a usable picture: there is none to show
}
// The credit is part of the identity: the same picture under another author or licence
// must not be taken for the same one by a client that kept the old credit.
h := sha256.New()
h.Write(p.Large)
h.Write([]byte("\x00" + found.Credit.Author + "\x00" + found.Credit.License + "\x00" + found.Credit.LicenseURL + "\x00" + found.Credit.PageURL))
return Record{
Status: StatusOK, Source: SourceCommons, Credit: found.Credit, ETag: fmt.Sprintf("%x", h.Sum(nil)[:8]),
Width: p.Width, Height: p.Height, CheckedAt: none.CheckedAt,
}, &p, nil
}
// saveArtist keeps what was found: the files and the record, unless the pictures were
// taken back since the build began (errPurged) or a change moved the record's generation.
func (s *Service) saveArtist(ctx context.Context, artistID int64, rec Record, pic *Processed, epoch uint64) (bool, error) {
s.purgeMu.RLock()
defer s.purgeMu.RUnlock()
if s.purges != epoch {
return false, errPurged
}
if pic != nil {
if err := s.write(kindArtist, artistID, *pic); err != nil {
return false, err
}
}
saved, err := s.store.Save(ctx, kindArtist, artistID, rec)
if err != nil {
return false, fmt.Errorf("save picture record: %w", err)
}
return saved, nil
return s.record(p, SourceCommons, found.Credit, none.CheckedAt), &p, nil
}
// RemoveArtistFiles deletes the stored pictures of artists, and only those, and says how
@ -216,3 +187,27 @@ func (s *Service) RemoveArtistFiles() (int, error) {
s.tidied.Delete(dir)
return n, nil
}
// RemoveAlbumFiles deletes the stored covers of the given albums (both sizes), and says
// how many files went. It is what taking the cover provider's data back needs, for the
// albums whose cover came from it; ids that are not an album's are ignored.
func (s *Service) RemoveAlbumFiles(albumIDs []int64) (int, error) {
dir := filepath.Join(s.dir, "images", kindAlbum)
n := 0
for _, id := range albumIDs {
if id <= 0 {
continue
}
for _, size := range []Size{Large, Small} {
err := os.Remove(s.filePath(kindAlbum, id, size))
switch {
case err == nil:
n++
case errors.Is(err, os.ErrNotExist):
default:
return n, fmt.Errorf("remove a cover in %s: %w", dir, err)
}
}
}
return n, nil
}

View file

@ -220,7 +220,7 @@ func TestAPurgeDuringTheReadIsNotUndoneByIt(t *testing.T) {
done := make(chan error, 1)
go func() { _, err := r.svc.ArtistImage(context.Background(), 7, Large); done <- err }()
time.Sleep(50 * time.Millisecond) // the read has the picture and is about to keep it
r.svc.BeginArtistPurge() // the admin takes the provider's data back...
r.svc.BeginPurge() // the admin takes the provider's data back...
r.src.set(Found{}, ErrProviderDisabled) // ...and switches it off
close(r.src.gate)
if err := <-done; !errors.Is(err, ErrProviderDisabled) {
@ -240,7 +240,7 @@ func TestAReadThatStartedBeforeAPurgeStartsAgainIfTheProviderIsStillOn(t *testin
done := make(chan error, 1)
go func() { _, err := r.svc.ArtistImage(context.Background(), 7, Large); done <- err }()
time.Sleep(50 * time.Millisecond)
r.svc.BeginArtistPurge()
r.svc.BeginPurge()
close(r.src.gate)
if err := <-done; err != nil {
t.Fatalf("err = %v", err)

View file

@ -0,0 +1,64 @@
package images
import (
"context"
"errors"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/coverart"
)
// AlbumLookup gives the MusicBrainz release group id stored for an album: "" when its
// files had none, ErrAlbumNotFound for an album that does not exist.
type AlbumLookup interface {
AlbumMBID(ctx context.Context, albumID int64) (string, error)
}
// CoverClient is what the source needs from the Cover Art Archive: *coverart.Client.
type CoverClient interface {
Front(ctx context.Context, releaseGroupID string) ([]byte, error)
}
// CoverArtSource finds the cover of an album in the Cover Art Archive by the release group
// id that the tags of the album's own files carried, never by a title. It needs the
// coverartarchive provider to be on.
type CoverArtSource struct {
Albums AlbumLookup
Settings Switches
Client CoverClient
}
func (c *CoverArtSource) active(ctx context.Context) (bool, error) {
_, on, err := c.Settings.Active(ctx, providers.CoverArtArchive)
return on, err
}
// AlbumImage implements AlbumRemote.
func (c *CoverArtSource) AlbumImage(ctx context.Context, albumID int64) (Found, error) {
id, err := c.Albums.AlbumMBID(ctx, albumID)
if err != nil {
return Found{}, err
}
if on, err := c.active(ctx); err != nil {
return Found{}, err
} else if !on {
return Found{}, ErrProviderDisabled
}
if id == "" { // Sonda never looks an id up by title
return Found{}, ErrNoID
}
data, err := c.Client.Front(ctx, id)
switch {
case errors.Is(err, coverart.ErrNotFound), errors.Is(err, coverart.ErrBadRequest), errors.Is(err, coverart.ErrUnusable):
return Found{}, ErrNoImage
case err != nil:
return Found{}, failed(ctx, err)
}
// The admin may have switched the provider off while the cover was coming.
if on, err := c.active(ctx); err != nil {
return Found{}, err
} else if !on {
return Found{}, ErrProviderDisabled
}
return Found{Data: data}, nil
}

View file

@ -0,0 +1,131 @@
package images
import (
"context"
"errors"
"fmt"
"testing"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/coverart"
)
const releaseGroup = "b1392450-e666-3926-a536-22c65f834433"
type albumsFake struct{ ids map[int64]string }
func (f albumsFake) AlbumMBID(_ context.Context, id int64) (string, error) {
m, ok := f.ids[id]
if !ok {
return "", ErrAlbumNotFound
}
return m, nil
}
type coverFake struct {
data []byte
err error
calls int
got string
hook func()
}
func (f *coverFake) Front(_ context.Context, id string) ([]byte, error) {
f.calls++
f.got = id
if f.hook != nil {
f.hook()
}
return f.data, f.err
}
type archive struct {
src *CoverArtSource
sw *switchesFake
cv *coverFake
}
func newArchive() *archive {
a := &archive{
sw: &switchesFake{on: map[string]bool{providers.CoverArtArchive: true}},
cv: &coverFake{data: []byte("cover bytes")},
}
a.src = &CoverArtSource{Albums: albumsFake{map[int64]string{1: releaseGroup, 2: ""}}, Settings: a.sw, Client: a.cv}
return a
}
func TestTheArchiveGivesTheCoverByReleaseGroup(t *testing.T) {
a := newArchive()
got, err := a.src.AlbumImage(context.Background(), 1)
if err != nil || string(got.Data) != "cover bytes" || got.Credit != (Credit{}) {
t.Fatalf("got %+v err %v", got, err)
}
if a.cv.got != releaseGroup {
t.Errorf("asked for %q, want only the release group id", a.cv.got)
}
}
func TestNothingIsAskedWhenTheProviderIsOffOrTheAlbumHasNoId(t *testing.T) {
for name, tt := range map[string]struct {
prepare func(*archive)
id int64
want error
}{
"provider off": {func(a *archive) { a.sw.set(providers.CoverArtArchive, false) }, 1, ErrProviderDisabled},
"no id": {func(*archive) {}, 2, ErrNoID},
"unknown album": {func(*archive) {}, 99, ErrAlbumNotFound},
} {
a := newArchive()
tt.prepare(a)
if _, err := a.src.AlbumImage(context.Background(), tt.id); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
if a.cv.calls != 0 {
t.Errorf("%s: the archive was asked", name)
}
}
}
func TestWhereTheArchiveHasNoCoverTheAnswerIsNoImage(t *testing.T) {
for name, why := range map[string]error{
"no cover": coverart.ErrNotFound, "bad id": coverart.ErrBadRequest, "unusable": coverart.ErrUnusable,
} {
a := newArchive()
a.cv.err = why
if _, err := a.src.AlbumImage(context.Background(), 1); !errors.Is(err, ErrNoImage) {
t.Errorf("%s: err = %v, want ErrNoImage", name, err)
}
}
}
func TestAnArchiveFailureIsAnOutageEvenWhenItEndsInADeadline(t *testing.T) {
for name, why := range map[string]error{
"unavailable": coverart.ErrUnavailable,
"rate limited": coverart.ErrRateLimited,
"a deadline": fmt.Errorf("%w: %w", coverart.ErrUnavailable, context.DeadlineExceeded),
} {
a := newArchive()
a.cv.err = why
if _, err := a.src.AlbumImage(context.Background(), 1); !errors.Is(err, ErrSourceUnavailable) {
t.Errorf("%s: err = %v, want ErrSourceUnavailable", name, err)
}
}
}
func TestSwitchedOffWhileTheCoverWasComingNothingIsReturned(t *testing.T) {
a := newArchive()
a.cv.hook = func() { a.sw.set(providers.CoverArtArchive, false) }
if _, err := a.src.AlbumImage(context.Background(), 1); !errors.Is(err, ErrProviderDisabled) {
t.Errorf("err = %v, want ErrProviderDisabled", err)
}
}
func TestACallerWhoLeavesGetsTheCancellation(t *testing.T) {
a := newArchive()
ctx, cancel := context.WithCancel(context.Background())
a.cv.hook = cancel
a.cv.err = context.Canceled
if _, err := a.src.AlbumImage(ctx, 1); !errors.Is(err, context.Canceled) || errors.Is(err, ErrSourceUnavailable) {
t.Errorf("err = %v", err)
}
}

View file

@ -34,9 +34,10 @@ const (
StatusNone = "none" // the files were searched and have none
StatusStale = "stale" // a file of the album changed: look again
SourceEmbedded = "embedded" // the picture inside an audio file
SourceFolder = "folder" // a cover.jpg, folder.png ... next to the audio
SourceCommons = "commons" // a free picture of an artist from Wikimedia Commons
SourceEmbedded = "embedded" // the picture inside an audio file
SourceFolder = "folder" // a cover.jpg, folder.png ... next to the audio
SourceCommons = "commons" // a free picture of an artist from Wikimedia Commons
SourceCoverArt = "coverartarchive" // the cover of an album from the Cover Art Archive
)
// Tuning.
@ -123,6 +124,14 @@ type ArtistSource interface {
ArtistImage(ctx context.Context, artistID int64) (Found, error)
}
// AlbumRemote finds the cover of an album outside the user's files, when they have none. It
// answers like an ArtistSource: ErrNoImage, ErrNoID (the album has no MusicBrainz id),
// ErrProviderDisabled, ErrAlbumNotFound, and ErrSourceUnavailable (or any other error) for a
// failure.
type AlbumRemote interface {
AlbumImage(ctx context.Context, albumID int64) (Found, error)
}
// Record is what is known about the picture of one owner.
type Record struct {
Status string // StatusOK or StatusNone
@ -166,9 +175,10 @@ type Service struct {
slots chan struct{} // one per cover build allowed at the same time
artistSl chan struct{} // the same for artist pictures, which wait for the network
artists ArtistSource
remote AlbumRemote
buildTimeout time.Duration // how long one build may take
// purges counts the times the provider's pictures were taken back (BeginArtistPurge). A
// purges counts the times the provider's pictures were taken back (BeginPurge). A
// build that started before one must not keep what it found: its files and its record
// would bring back what the admin just removed. Keeping a result takes the read side, so
// a purge waits for the results being kept and the next ones are refused.
@ -184,6 +194,10 @@ func NewService(dir string, store Store, src Source) *Service {
buildTimeout: buildTimeout}
}
// SetAlbumRemote tells where the cover of an album comes from when the user's own files
// have none. Without one, such an album has no cover.
func (s *Service) SetAlbumRemote(r AlbumRemote) { s.remote = r }
// SetArtistSource tells where pictures of artists come from. Without one, they are
// reported as ErrProviderDisabled.
func (s *Service) SetArtistSource(src ArtistSource) { s.artists = src }
@ -230,6 +244,12 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co
return nil, ctx.Err()
case r := <-done:
if r.Err != nil {
// An older cover (from the provider) is better than none when it cannot be asked now.
if errors.Is(r.Err, ErrSourceUnavailable) && ok && rec.Status == StatusOK {
if c, err := s.open(kindAlbum, albumID, size, rec); err == nil {
return c, nil
}
}
return nil, r.Err
}
built = r.Val.(Record)
@ -262,7 +282,8 @@ func (s *Service) open(kind string, id int64, size Size, rec Record) (*Cover, er
}
// build looks for a picture, stores it and records the outcome. If a file of the
// album changed while it was working, what it found may be out of date: it starts again.
// album changed, or the pictures of a provider were taken back, while it was working,
// what it found may be out of date: it starts again.
func (s *Service) build(albumID int64) (Record, error) {
ctx, cancel := context.WithTimeout(context.Background(), s.buildTimeout)
defer cancel()
@ -276,14 +297,24 @@ func (s *Service) build(albumID int64) (Record, error) {
if err != nil {
return Record{}, err
}
rec, err = s.search(ctx, albumID, files)
epoch := s.purgeCount()
var pic *Processed
rec, pic, err = s.search(ctx, albumID, files)
if err != nil {
// Nobody calls this context: when it ends it is the build that ran out of time,
// which is an outage of the provider, not an internal error.
if ctx.Err() != nil && (errors.Is(err, context.DeadlineExceeded) || errors.Is(err, context.Canceled)) {
err = fmt.Errorf("%w: %w", ErrSourceUnavailable, err)
}
return Record{}, err
}
rec.Gen = cur.Gen
saved, err := s.store.Save(ctx, kindAlbum, albumID, rec)
saved, err := s.commit(ctx, kindAlbum, albumID, rec, pic, epoch)
if errors.Is(err, errPurged) {
continue
}
if err != nil {
return Record{}, fmt.Errorf("save picture record: %w", err)
return Record{}, err
}
if saved {
return rec, nil
@ -292,23 +323,98 @@ func (s *Service) build(albumID int64) (Record, error) {
return rec, nil // it kept changing; this one is served and the record stays stale
}
// search finds a picture and stores it, at most MaxParallelBuilds at a time.
func (s *Service) search(ctx context.Context, albumID int64, files []File) (Record, error) {
// search finds a picture: in the user's own files first, at most MaxParallelBuilds at a
// time, and, only when they have none, from the provider of covers if there is one. The
// picture is not stored yet (see commit); nil means there is none.
func (s *Service) search(ctx context.Context, albumID int64, files []File) (Record, *Processed, error) {
none := Record{Status: StatusNone, CheckedAt: s.now()}
p, source, ok, err := s.searchFiles(ctx, files)
if err != nil {
return Record{}, nil, err
}
if ok {
return s.record(p, source, Credit{}, none.CheckedAt), &p, nil
}
if s.remote == nil {
return none, nil, nil
}
rp, credit, ok, err := s.searchRemote(ctx, albumID)
if err != nil {
return Record{}, nil, err
}
if !ok {
return none, nil, nil
}
return s.record(rp, SourceCoverArt, credit, none.CheckedAt), &rp, nil
}
func (s *Service) searchFiles(ctx context.Context, files []File) (Processed, string, bool, error) {
select {
case s.slots <- struct{}{}:
defer func() { <-s.slots }()
case <-ctx.Done():
return Record{}, ctx.Err()
return Processed{}, "", false, ctx.Err()
}
rec := Record{Status: StatusNone, CheckedAt: s.now()}
if p, source, ok := s.find(files); ok {
if err := s.write(kindAlbum, albumID, p); err != nil {
return Record{}, err
p, source, ok := s.find(files)
return p, source, ok, nil
}
// searchRemote asks the provider of covers. "There is none" (no picture, no id, the
// provider is off, a picture that is not one) is an answer; a failure is returned.
func (s *Service) searchRemote(ctx context.Context, albumID int64) (Processed, Credit, bool, error) {
select {
case s.artistSl <- struct{}{}: // the slots of the work that waits for the network
defer func() { <-s.artistSl }()
case <-ctx.Done():
return Processed{}, Credit{}, false, ctx.Err()
}
found, err := s.remote.AlbumImage(ctx, albumID)
switch {
case errors.Is(err, ErrNoImage), errors.Is(err, ErrNoID), errors.Is(err, ErrProviderDisabled):
return Processed{}, Credit{}, false, nil
case err != nil:
return Processed{}, Credit{}, false, err
}
p, err := ProcessLimited(found.Data, MaxArtistPixels)
if err != nil {
return Processed{}, Credit{}, false, nil // not a usable picture: there is no cover to show
}
return p, found.Credit, true, nil
}
// record describes a picture that was found.
func (s *Service) record(p Processed, source string, credit Credit, at time.Time) Record {
h := sha256.New()
h.Write(p.Large)
// The credit is part of the identity: the same picture under another author or licence
// must not be taken for the same one by a client that kept the old credit.
h.Write([]byte("\x00" + credit.Author + "\x00" + credit.License + "\x00" + credit.LicenseURL + "\x00" + credit.PageURL))
return Record{
Status: StatusOK, Source: source, Credit: credit, ETag: hex.EncodeToString(h.Sum(nil)[:8]),
Width: p.Width, Height: p.Height, CheckedAt: at,
}
}
// commit keeps what was found: the files and the record, unless the pictures of a
// provider were taken back since the build began (errPurged) or a change moved the
// record's generation (saved is false). Both happen under the read side of the purge lock,
// so a purge waits for it and the next ones are refused.
func (s *Service) commit(ctx context.Context, kind string, id int64, rec Record, pic *Processed, epoch uint64) (bool, error) {
s.purgeMu.RLock()
defer s.purgeMu.RUnlock()
if s.purges != epoch {
return false, errPurged
}
if pic != nil {
if err := s.write(kind, id, *pic); err != nil {
return false, err
}
sum := sha256.Sum256(p.Large)
rec = Record{Status: StatusOK, Source: source, ETag: hex.EncodeToString(sum[:8]), Width: p.Width, Height: p.Height, CheckedAt: rec.CheckedAt}
}
return rec, nil
saved, err := s.store.Save(ctx, kind, id, rec)
if err != nil {
return false, fmt.Errorf("save picture record: %w", err)
}
return saved, nil
}
// find returns the first usable picture: embedded in one of the first files, then an

View file

@ -41,6 +41,11 @@ func (TaglibReader) Read(path string) (Metadata, error) {
// below is only a fallback, and an id would land on the wrong artist.
albumArtistID = singleID(tags["MUSICBRAINZ_ALBUMARTISTID"])
}
albumID := ""
if first(tags, "ALBUM") != "" {
// Like the album artist's id, it belongs to a tag: a file with no album has none to put it on.
albumID = singleID(tags["MUSICBRAINZ_RELEASEGROUPID"])
}
m := Metadata{
Tags: Tags{
Genres: CleanGenres(tags["GENRE"]),
@ -50,6 +55,7 @@ func (TaglibReader) Read(path string) (Metadata, error) {
Artists: artists,
ArtistMBIDs: pairArtistIDs(artists, tags["MUSICBRAINZ_ARTISTID"]),
AlbumArtistMBID: albumArtistID,
AlbumMBID: albumID,
TrackNo: leadingInt(first(tags, "TRACKNUMBER")),
DiscNo: leadingInt(first(tags, "DISCNUMBER")),
},

View file

@ -237,6 +237,35 @@ func TestTaglibReaderReadsMusicBrainzIDsFromMP3(t *testing.T) {
}
}
// The release group id of the album: with it the cover can be asked from the Cover Art Archive
// by id. It belongs to the album tag: a file with no album has no album to put it on.
func TestTaglibReaderReadsTheReleaseGroupID(t *testing.T) {
const rg = "b1392450-e666-3926-a536-22c65f834433"
for _, tt := range []struct {
name string
file string
meta map[string]string
want string
}{
{"flac", "song.flac", map[string]string{"title": "T", "album": "A", "MUSICBRAINZ_RELEASEGROUPID": rg}, rg},
{"mp3 frame", "song.mp3", map[string]string{"title": "T", "album": "A", "MusicBrainz Release Group Id": rg}, rg},
{"upper case", "up.flac", map[string]string{"title": "T", "album": "A", "MUSICBRAINZ_RELEASEGROUPID": strings.ToUpper(rg)}, rg},
{"not an id", "bad.flac", map[string]string{"title": "T", "album": "A", "MUSICBRAINZ_RELEASEGROUPID": "not-an-id"}, ""},
{"several ids", "two.flac", map[string]string{"title": "T", "album": "A", "MUSICBRAINZ_RELEASEGROUPID": rg + ";" + rg}, ""},
{"no album", "noalbum.flac", map[string]string{"title": "T", "MUSICBRAINZ_RELEASEGROUPID": rg}, ""},
{"none", "none.flac", map[string]string{"title": "T", "album": "A"}, ""},
} {
p := makeAudio(t, t.TempDir(), tt.file, tt.meta)
m, err := TaglibReader{}.Read(p)
if err != nil {
t.Fatalf("%s: %v", tt.name, err)
}
if m.AlbumMBID != tt.want {
t.Errorf("%s: album id = %q, want %q", tt.name, m.AlbumMBID, tt.want)
}
}
}
// ---- genres ----
func TestGenresAreReadCleanedAndBounded(t *testing.T) {

View file

@ -28,6 +28,7 @@ type Tags struct {
Artists []string
ArtistMBIDs []string
AlbumArtistMBID string
AlbumMBID string // the MusicBrainz release group id of the album, when the file has exactly one
TrackNo int
DiscNo int
}

View file

@ -0,0 +1,128 @@
// Package coverart is a small read-only client for the Cover Art Archive.
//
// Sonda asks it for the front cover of an album by the MusicBrainz release group id that
// is in the tags of the album's own files, and never by a title: a search returns
// look-alikes and picking one would be a guess. Only that id leaves the server.
//
// The archive answers with redirects (coverartarchive.org, then archive.org, then a data
// node whose name changes), so the client that talks to it must follow redirects, but only
// to archive.org over https: RedirectAllowed is what to give it. Errors never repeat the
// provider's own text.
package coverart
import (
"context"
"errors"
"fmt"
"net/url"
"regexp"
"strings"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
// BaseURL is the Cover Art Archive.
const BaseURL = "https://coverartarchive.org/"
// probeID is a public release group (Radiohead, OK Computer) used to check that the service answers.
const probeID = "b1392450-e666-3926-a536-22c65f834433"
// Errors a call can return, to be matched with errors.Is.
var (
ErrBadRequest = errors.New("coverart: not a MusicBrainz id")
ErrNotFound = errors.New("coverart: no cover art for this release group")
ErrUnusable = errors.New("coverart: the answer cannot be used")
ErrRateLimited = errors.New("coverart: too many requests")
ErrUnavailable = errors.New("coverart: service unavailable")
ErrBadResponse = errors.New("coverart: unreadable answer")
)
var idRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
// RedirectAllowed says whether a redirect may be followed: https, no credentials, no
// port, and coverartarchive.org, archive.org or a subdomain of archive.org.
func RedirectAllowed(u *url.URL) bool {
if u.Scheme != "https" || u.User != nil || u.Port() != "" {
return false
}
h := u.Hostname()
return h == "coverartarchive.org" || h == "archive.org" || strings.HasSuffix(h, ".archive.org")
}
// Getter performs a GET. *outbound.Client is the one used in production.
type Getter interface {
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
}
// Client talks to the Cover Art Archive. It is safe for concurrent use.
type Client struct {
http Getter
base string
}
// New builds a Client that sends its requests through g to base (BaseURL in production).
func New(g Getter, base string) *Client { return &Client{http: g, base: base} }
// Front returns the front cover of a release group: the 1200 pixel copy, or the 500 pixel one
// when there is no larger copy. ErrNotFound means the archive has no cover for it.
func (c *Client) Front(ctx context.Context, releaseGroupID string) ([]byte, error) {
if !idRe.MatchString(releaseGroupID) {
return nil, ErrBadRequest
}
id := strings.ToLower(releaseGroupID)
body, err := c.get(ctx, fmt.Sprintf("%srelease-group/%s/front-1200", c.base, id))
if errors.Is(err, ErrNotFound) {
body, err = c.get(ctx, fmt.Sprintf("%srelease-group/%s/front-500", c.base, id))
}
return body, err
}
// Probe asks for one small cover, to find out whether the service answers. It is a
// providers.Checker, and answers with the errors the settings understand: a release group
// with no cover still proves that the service is there.
func (c *Client) Probe(ctx context.Context) error {
_, err := c.get(ctx, fmt.Sprintf("%srelease-group/%s/front-250", c.base, probeID))
switch {
case errors.Is(err, ErrRateLimited):
return providers.ErrRateLimited
case errors.Is(err, ErrUnavailable):
return providers.ErrUnreachable
case errors.Is(err, ErrNotFound):
return nil
}
return err
}
// get sends one request and returns the body of a successful answer.
func (c *Client) get(ctx context.Context, rawURL string) ([]byte, error) {
resp, err := c.http.Get(ctx, rawURL)
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
var limited *outbound.RateLimitedError
switch {
case errors.As(err, &limited):
return nil, ErrRateLimited
// A redirect that is not allowed, or a picture over the limit, is a fault of that
// answer and not an outage: there is no usable cover.
case errors.Is(err, outbound.ErrRedirect), errors.Is(err, outbound.ErrResponseTooLarge):
return nil, fmt.Errorf("%w: the answer cannot be followed or is too big", ErrUnusable)
}
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err)
}
switch {
case resp.Status == 200:
return resp.Body, nil
case resp.Status == 404:
return nil, ErrNotFound
case resp.Status == 400:
return nil, ErrBadRequest
case resp.Status == 429:
return nil, ErrRateLimited
case resp.Status >= 500:
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
}
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
}

View file

@ -0,0 +1,199 @@
package coverart
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
const rg = "b1392450-e666-3926-a536-22c65f834433"
type server struct {
*httptest.Server
mu sync.Mutex
paths []string
h func(w http.ResponseWriter, r *http.Request)
}
func newServer(t *testing.T, h func(w http.ResponseWriter, r *http.Request)) *server {
t.Helper()
s := &server{h: h}
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
s.paths = append(s.paths, r.URL.RequestURI())
s.mu.Unlock()
s.h(w, r)
}))
t.Cleanup(s.Close)
return s
}
func (s *server) client() *Client {
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries,
FollowRedirect: func(*url.URL) bool { return true }}), s.URL+"/")
}
func (s *server) requests() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.paths...)
}
func status(code int, body string) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(code)
_, _ = w.Write([]byte(body))
}
}
func TestFrontAsksForTheLargeCopyOfTheReleaseGroupAndFollowsTheRedirects(t *testing.T) {
var s *server
s = newServer(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasPrefix(r.URL.Path, "/release-group/"):
http.Redirect(w, r, s.URL+"/archive/x", http.StatusTemporaryRedirect)
case r.URL.Path == "/archive/x":
http.Redirect(w, r, s.URL+"/node/y.jpg", http.StatusFound)
default:
w.Header().Set("Content-Type", "image/jpeg")
_, _ = w.Write([]byte("JPEG BYTES"))
}
})
got, err := s.client().Front(context.Background(), rg)
if err != nil || string(got) != "JPEG BYTES" {
t.Fatalf("got %q err %v", got, err)
}
if reqs := s.requests(); reqs[0] != "/release-group/"+rg+"/front-1200" {
t.Errorf("first request = %v: only the id leaves", reqs)
}
}
func TestFrontFallsBackToTheSmallerCopyOnlyWhenTheLargeOneIsNotThere(t *testing.T) {
s := newServer(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "front-1200") {
http.NotFound(w, r)
return
}
_, _ = w.Write([]byte("SMALL"))
})
got, err := s.client().Front(context.Background(), rg)
if err != nil || string(got) != "SMALL" {
t.Fatalf("got %q err %v", got, err)
}
if reqs := s.requests(); len(reqs) != 2 || !strings.HasSuffix(reqs[1], "front-500") {
t.Errorf("requests = %v", reqs)
}
// Neither there: no cover. Not a failure.
none := newServer(t, status(404, "<html>No cover art found</html>"))
if _, err := none.client().Front(context.Background(), rg); !errors.Is(err, ErrNotFound) {
t.Errorf("err = %v, want ErrNotFound", err)
}
if n := len(none.requests()); n != 2 {
t.Errorf("%d requests for a release group with no cover, want 2", n)
}
}
func TestFrontRefusesWhatIsNotAnIDWithoutAsking(t *testing.T) {
s := newServer(t, status(200, "x"))
for _, id := range []string{"", "not-an-id", rg + "/../x", rg + "?x=y", strings.ToUpper(rg)[:35]} {
if _, err := s.client().Front(context.Background(), id); !errors.Is(err, ErrBadRequest) {
t.Errorf("id %q: err = %v", id, err)
}
}
if len(s.requests()) != 0 {
t.Errorf("requests = %v", s.requests())
}
// An id in capitals is fine: it is sent in lower case.
if _, err := s.client().Front(context.Background(), strings.ToUpper(rg)); err != nil {
t.Errorf("capitals: %v", err)
}
if reqs := s.requests(); len(reqs) != 1 || !strings.Contains(reqs[0], rg) {
t.Errorf("requests = %v, want the id in lower case", reqs)
}
}
func TestFrontErrors(t *testing.T) {
for name, tt := range map[string]struct {
code int
want error
}{
"bad id": {400, ErrBadRequest},
"rate limited": {429, ErrRateLimited},
"server error": {503, ErrUnavailable},
"forbidden": {403, ErrBadResponse},
} {
if _, err := newServer(t, status(tt.code, "")).client().Front(context.Background(), rg); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
}
secret := "SECRET-FROM-THE-PROVIDER"
if _, err := newServer(t, status(500, secret)).client().Front(context.Background(), rg); err == nil || strings.Contains(err.Error(), secret) {
t.Errorf("err = %v", err)
}
}
func TestARedirectToAPlaceThatIsNotAllowedOrAHugePictureIsUnusable(t *testing.T) {
redirect := newServer(t, func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, "https://evil.example/", http.StatusFound)
})
c := New(outbound.New(outbound.Options{UserAgent: "x", Retries: outbound.NoRetries, FollowRedirect: RedirectAllowed}), redirect.URL+"/")
if _, err := c.Front(context.Background(), rg); !errors.Is(err, ErrUnusable) {
t.Errorf("redirect: err = %v, want ErrUnusable", err)
}
huge := newServer(t, func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write(make([]byte, 5000)) })
c = New(outbound.New(outbound.Options{UserAgent: "x", Retries: outbound.NoRetries, MaxBody: 1000}), huge.URL+"/")
if _, err := c.Front(context.Background(), rg); !errors.Is(err, ErrUnusable) {
t.Errorf("huge: err = %v, want ErrUnusable", err)
}
}
func TestOnlyArchiveOrgIsFollowed(t *testing.T) {
for raw, ok := range map[string]bool{
"https://coverartarchive.org/release-group/x/front-500": true,
"https://archive.org/download/mbid-x/y.jpg": true,
"https://dn720706.ca.archive.org/0/items/mbid-x/y.jpg": true,
"https://ia800000.us.archive.org/1/items/y.jpg": true,
"http://archive.org/download/y.jpg": false,
"https://archive.org.evil.example/y.jpg": false,
"https://evilarchive.org/y.jpg": false,
"https://notarchive.org/y.jpg": false,
"https://user@archive.org/y.jpg": false,
"https://archive.org:8443/y.jpg": false,
"https://evil.example/archive.org": false,
"https://coverartarchive.org.evil.example/x": false,
"file:///etc/passwd": false,
"https://musicbrainz.org/": false,
} {
u, err := url.Parse(raw)
if err != nil {
t.Fatal(err)
}
if got := RedirectAllowed(u); got != ok {
t.Errorf("RedirectAllowed(%q) = %v, want %v", raw, got, ok)
}
}
}
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
if err := newServer(t, status(200, "x")).client().Probe(context.Background()); err != nil {
t.Errorf("ok: %v", err)
}
// A release group that has no cover still proves the service answers.
if err := newServer(t, status(404, "")).client().Probe(context.Background()); err != nil {
t.Errorf("404: %v", err)
}
if err := newServer(t, status(429, "")).client().Probe(context.Background()); !errors.Is(err, providers.ErrRateLimited) {
t.Errorf("429: %v", err)
}
if err := newServer(t, status(503, "")).client().Probe(context.Background()); !errors.Is(err, providers.ErrUnreachable) {
t.Errorf("503: %v", err)
}
}

View file

@ -77,6 +77,12 @@ type Options struct {
MinInterval time.Duration // minimum time between requests; 0 = no limit
MaxBody int64 // largest accepted answer; default 4 MiB
// FollowRedirect, when set, is asked about the address of every redirect: only the ones it
// accepts are followed, up to MaxRedirects (4 by default). Without it no redirect is ever
// followed, since a redirect could lead anywhere. It sees the whole address, scheme included.
FollowRedirect func(*url.URL) bool
MaxRedirects int
Retries int // retries after the first attempt; 0 = default (2), NoRetries = none
BackoffBase time.Duration // first backoff; doubles each retry; default 1s
MaxRetryAfter time.Duration // longest Retry-After we will wait for; default 30s
@ -175,10 +181,19 @@ func New(o Options) *Client {
if transport == nil {
transport = http.DefaultTransport
}
maxRedirects := o.MaxRedirects
if maxRedirects <= 0 {
maxRedirects = 4
}
follow := o.FollowRedirect
c.hc = &http.Client{
Transport: transport, // no cookie jar: nothing a provider sets is ever sent back
CheckRedirect: func(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse // never follow: a redirect could lead anywhere
CheckRedirect: func(req *http.Request, via []*http.Request) error {
// The last response is then a redirect, which Get reports as ErrRedirect.
if follow == nil || len(via) > maxRedirects || !follow(req.URL) {
return http.ErrUseLastResponse
}
return nil
},
}
return c

View file

@ -5,6 +5,7 @@ import (
"errors"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"sync/atomic"
@ -719,3 +720,68 @@ func TestUserAgentFuncIsReadForEveryRequest(t *testing.T) {
t.Errorf("User-Agents = %q", seen)
}
}
// ---- redirects, only where the caller allows them ----
func TestARedirectIsFollowedOnlyToWhereTheCallerAllows(t *testing.T) {
var finalUA atomic.Value
final := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
finalUA.Store(r.Header.Get("User-Agent"))
_, _ = w.Write([]byte("the picture"))
}))
defer final.Close()
middle := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, final.URL+"/image.jpg", http.StatusFound)
}))
defer middle.Close()
first := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, middle.URL+"/next", http.StatusTemporaryRedirect)
}))
defer first.Close()
allowed := map[string]bool{strings.TrimPrefix(middle.URL, "http://"): true, strings.TrimPrefix(final.URL, "http://"): true}
c := New(Options{UserAgent: "Sonda/test ( https://example.org )", Retries: NoRetries,
FollowRedirect: func(u *url.URL) bool { return allowed[u.Host] }})
resp, err := c.Get(context.Background(), first.URL+"/start")
if err != nil || resp.Status != 200 || string(resp.Body) != "the picture" {
t.Fatalf("resp = %+v err = %v", resp, err)
}
if ua, _ := finalUA.Load().(string); ua != "Sonda/test ( https://example.org )" {
t.Errorf("the last hop saw the User-Agent %q", ua)
}
}
func TestARedirectToAHostThatIsNotAllowedIsNeverContacted(t *testing.T) {
var contacted int32
evil := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&contacted, 1)
}))
defer evil.Close()
start := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Redirect(w, r, evil.URL+"/steal", http.StatusFound)
}))
defer start.Close()
c := New(Options{UserAgent: "x", Retries: NoRetries, FollowRedirect: func(u *url.URL) bool { return u.Host == strings.TrimPrefix(start.URL, "http://") }})
if _, err := c.Get(context.Background(), start.URL); !errors.Is(err, ErrRedirect) {
t.Fatalf("err = %v, want ErrRedirect", err)
}
if atomic.LoadInt32(&contacted) != 0 {
t.Error("a host that was not allowed was contacted")
}
}
func TestRedirectsAreCapped(t *testing.T) {
var hops int32
var srv *httptest.Server
srv = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
atomic.AddInt32(&hops, 1)
http.Redirect(w, r, srv.URL+"/again", http.StatusFound) // forever
}))
defer srv.Close()
c := New(Options{UserAgent: "x", Retries: NoRetries, MaxRedirects: 3, FollowRedirect: func(*url.URL) bool { return true }})
if _, err := c.Get(context.Background(), srv.URL); !errors.Is(err, ErrRedirect) {
t.Fatalf("err = %v, want ErrRedirect", err)
}
if n := atomic.LoadInt32(&hops); n != 4 { // the first request and three hops
t.Errorf("%d requests, want 4", n)
}
}

Some files were not shown because too many files have changed in this diff Show more