Merge pull request 'feat(images): find the cover of an album that has no MusicBrainz id' (#84) from feat/album-mbid-lookup into main

This commit is contained in:
squidd 2026-11-12 16:23:36 +01:00
commit a64122efdd
12 changed files with 467 additions and 26 deletions

View file

@ -19,7 +19,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers). A listener has one Discover at a time (a second one answers `429` with `Retry-After`), and at most three run at once on the server, so the providers are never lined up with work: one that waits more than 30 seconds for its turn answers `429` too, and one that stops waiting does none.
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. To read the descriptions of the whole library in one go, an admin starts a background job like a scan: `POST /library/descriptions` (follow it with `GET /library/descriptions`, stop it with `DELETE /library/descriptions`; `?refresh=true` reads every artist again). It only goes through artists that have a MusicBrainz id, never replaces a description by a failure, and ends at once, saying so, when a provider it needs is off. See [External providers](#external-providers).
- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. To look for the pictures of the whole library in one go, an admin starts a background job like a scan: `POST /library/artist-images` (follow it with `GET /library/artist-images`, stop it with `DELETE /library/artist-images`; `?refresh=true` searches every artist again). It only goes through artists that have a MusicBrainz id, never replaces a picture by a failure, and ends at once, saying so, when a provider it needs is off. See [External providers](#external-providers).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again; it never replaces a cover by "no cover" because a disk was not mounted, and keeps the covers that came from the archive if you switched it off).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. When your files have no picture at all and an admin switched on the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the MusicBrainz release group id that the tags of the album's files carry (`MUSICBRAINZ_RELEASEGROUPID`; run a full rescan to read it for music you already indexed); your own files always come first, and only that id leaves the server. An album whose files have no id is looked up in MusicBrainz by its artist and title when the `musicbrainz` provider is on too (see the list of what leaves your server). For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere unless you switched on the `coverartarchive` provider (see [External providers](#external-providers)). It also takes `?access_token=`, so an `<img>` element can use it. To build the covers of the whole library in one go instead of one album at a time, an admin starts a background job like a scan: `POST /library/covers` (follow it with `GET /library/covers`, stop it with `DELETE /library/covers`; `?refresh=true` searches every album again; it never replaces a cover by "no cover" because a disk was not mounted, and keeps the covers that came from the archive if you switched it off).
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. The wishlist is a list for you and sends nothing anywhere; Sonda downloads nothing, but an admin can send an artist that Discover showed to their own Lidarr (see [External providers](#external-providers)), which works for an artist that has a MusicBrainz id and that Sonda knows of from a list or from your files. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
@ -253,7 +253,7 @@ The address can have a path if Lidarr runs under a URL base, and it cannot carry
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files; `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send only the MusicBrainz release group id of an album, read from the tags of its own files and never a title, and only when the files have no picture; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, the MusicBrainz id of the artist (and, only for the covers of albums that have no id, the artist's name and the album's title: see **Covers**), your IP address and a `User-Agent` that carries your contact. Never your users, your listens or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files (and, for the albums whose files have no id, with MusicBrainz on, the artist and the title, one request a second); `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send the MusicBrainz release group id of an album, read from the tags of its own files, and only when the files have no picture. When the files have no id either and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz, which is asked for the release group: a result is taken only when MusicBrainz scores it 95 or more and its title and artist are the ones asked for (compared without case, accents or punctuation), and anything doubtful is no cover. The id that comes back is used for that request and is not stored: it is not what your files say, and it is never used to read an artist or to send anything to Lidarr; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
@ -261,7 +261,7 @@ The key is stored in the database, in clear text, and is never returned by the A
**Genres from your own files.** The scan also reads the genre tag of each file (several genres separated by `;`, at most five per file) and `GET /artists/{id}/tags` adds a `files` entry: the genres of the playable tracks the artist is credited on, each weighted by how many tracks have it. It needs no provider, asks nobody, and is always there.
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz by name, because a search returns look-alikes and picking one would be a guess. A normal scan reads a file's tags when the file is new or changed. To pick up ids and genres for music that is already indexed, scan with `full=true` (`POST /library/scan?full=true`, or `POST /libraries/{id}/scan?full=true`): every file is read again, keeps its identity, and your listening history is untouched.
**MusicBrainz ids come from your files, never from a guess.** If your files carry MusicBrainz artist ids in their tags (taggers like Picard write them), the scan reads them, offline, and keeps them on your artists, but only when each id can be paired with its artist for certain. MusicBrainz is asked only about artists whose id is known; for the others its entry says `no_id`. Sonda never searches MusicBrainz for an artist by name, because a search returns look-alikes and picking one would be a guess; the one search there is is for the cover of an album that has no id (see **Covers** below), and it takes a result only when it is sure. A normal scan reads a file's tags when the file is new or changed. To pick up ids and genres for music that is already indexed, scan with `full=true` (`POST /library/scan?full=true`, or `POST /libraries/{id}/scan?full=true`): every file is read again, keeps its identity, and your listening history is untouched.
**Taking it back.** `DELETE /api/v1/providers/{name}/data` (admin) removes everything a provider gave: its similar artists, its tags and the records of when it was asked, and then the artists that only that data named and that you have no files for. Last.fm's terms ask for this when an agreement ends. Your own artists, music and listens stay, and the provider is not switched off. An artist that only that provider's lists named goes too, even if someone had it on their wishlist or dismissed. An artist that no list names any more is also removed when the list that named it is replaced, so they do not pile up.

View file

@ -203,7 +203,7 @@ func run() error {
providerSettings.SetChecker(providers.CoverArtArchive, func(ctx context.Context, _ string) error {
return coverartClient.Probe(ctx)
})
pictures.SetAlbumRemote(&images.CoverArtSource{Albums: imageRepo, Settings: providerSettings, Client: coverartClient})
pictures.SetAlbumRemote(&images.CoverArtSource{Albums: imageRepo, Settings: providerSettings, Client: coverartClient, Finder: musicbrainzClient})
// Lidarr is the admin's own server: it is read from the stored settings for every request, on or off,
// since setting it up (a test, what it offers) happens before it is switched on.
lidarrClient := lidarr.New(func(ctx context.Context) (lidarr.Config, error) {

View file

@ -1508,8 +1508,11 @@ paths:
For an album split in folders named like `CD1` or `Disc 2`, the picture in the folder
above them is used too. When the files have no picture at all, and an admin switched on
the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the
MusicBrainz release group id that the tags of the album's files carry (never by title; a
full rescan reads that tag again for music indexed before); only that id leaves the server.
MusicBrainz release group id that the tags of the album's files carry (a full rescan reads
that tag again for music indexed before); only that id leaves the server. An album whose
files have no id is looked up in MusicBrainz by its artist and title when the
`musicbrainz` provider is on too: the id is taken only when MusicBrainz is sure (an exact
match of both names), is used for that request and is not stored.
The files of the user always come first. That request can take several seconds and may
answer `provider_unavailable`. At most a couple of covers are built at the same time, so a
client that asks for many at once may wait.

View file

@ -8,10 +8,19 @@ import (
"codeberg.org/squidd/sonda/internal/providers/coverart"
)
// AlbumLookup gives the MusicBrainz release group id stored for an album: "" when its
// files had none, ErrAlbumNotFound for an album that does not exist.
// AlbumLookup gives what Sonda knows of an album. Both calls return ErrAlbumNotFound for an
// album that does not exist.
type AlbumLookup interface {
// AlbumMBID is the MusicBrainz release group id stored for the album: "" when its files had none.
AlbumMBID(ctx context.Context, albumID int64) (string, error)
// AlbumNames is the name of the album's artist ("" when it has none) and its title.
AlbumNames(ctx context.Context, albumID int64) (artist, title string, err error)
}
// ReleaseGroupFinder looks up the release group of an album by its artist and title, and gives
// "" when no result is sure: *musicbrainz.Client.
type ReleaseGroupFinder interface {
FindReleaseGroup(ctx context.Context, artist, title string) (string, error)
}
// CoverClient is what the source needs from the Cover Art Archive: *coverart.Client.
@ -20,12 +29,18 @@ type CoverClient interface {
}
// CoverArtSource finds the cover of an album in the Cover Art Archive by the release group
// id that the tags of the album's own files carried, never by a title. It needs the
// coverartarchive provider to be on.
// id that the tags of the album's own files carried. It needs the coverartarchive provider to
// be on.
//
// An album whose files carried no id is looked up in MusicBrainz by its artist and title, when
// the musicbrainz provider is on too and there is a Finder: the artist and the title go to
// MusicBrainz, which answers only when one result is sure (see musicbrainz.FindReleaseGroup).
// The id that comes back is used for this request and is not stored: it is not what the files say.
type CoverArtSource struct {
Albums AlbumLookup
Settings Switches
Client CoverClient
Finder ReleaseGroupFinder // nil: an album with no id has no cover from the archive
}
func (c *CoverArtSource) active(ctx context.Context) (bool, error) {
@ -44,8 +59,10 @@ func (c *CoverArtSource) AlbumImage(ctx context.Context, albumID int64) (Found,
} else if !on {
return Found{}, ErrProviderDisabled
}
if id == "" { // Sonda never looks an id up by title
return Found{}, ErrNoID
if id == "" {
if id, err = c.find(ctx, albumID); err != nil {
return Found{}, err
}
}
data, err := c.Client.Front(ctx, id)
switch {
@ -62,3 +79,31 @@ func (c *CoverArtSource) AlbumImage(ctx context.Context, albumID int64) (Found,
}
return Found{Data: data}, nil
}
// find looks up the release group of an album that has no id: ErrNoID when it may not be looked
// up (no finder, MusicBrainz is off, the album has no artist), ErrNoImage when no result is sure.
func (c *CoverArtSource) find(ctx context.Context, albumID int64) (string, error) {
if c.Finder == nil {
return "", ErrNoID
}
if _, on, err := c.Settings.Active(ctx, providers.MusicBrainz); err != nil {
return "", err
} else if !on {
return "", ErrNoID
}
artist, title, err := c.Albums.AlbumNames(ctx, albumID)
if err != nil {
return "", err
}
if artist == "" || title == "" {
return "", ErrNoID
}
id, err := c.Finder.FindReleaseGroup(ctx, artist, title)
switch {
case err != nil:
return "", failed(ctx, err)
case id == "":
return "", ErrNoImage
}
return id, nil
}

View file

@ -8,11 +8,23 @@ import (
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/coverart"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
)
const releaseGroup = "b1392450-e666-3926-a536-22c65f834433"
type albumsFake struct{ ids map[int64]string }
type albumsFake struct {
ids map[int64]string
names map[int64][2]string // artist, title
}
func (f albumsFake) AlbumNames(_ context.Context, id int64) (string, string, error) {
if _, ok := f.ids[id]; !ok {
return "", "", ErrAlbumNotFound
}
n := f.names[id]
return n[0], n[1], nil
}
func (f albumsFake) AlbumMBID(_ context.Context, id int64) (string, error) {
m, ok := f.ids[id]
@ -50,7 +62,7 @@ func newArchive() *archive {
sw: &switchesFake{on: map[string]bool{providers.CoverArtArchive: true}},
cv: &coverFake{data: []byte("cover bytes")},
}
a.src = &CoverArtSource{Albums: albumsFake{map[int64]string{1: releaseGroup, 2: ""}}, Settings: a.sw, Client: a.cv}
a.src = &CoverArtSource{Albums: albumsFake{ids: map[int64]string{1: releaseGroup, 2: ""}}, Settings: a.sw, Client: a.cv}
return a
}
@ -129,3 +141,122 @@ func TestACallerWhoLeavesGetsTheCancellation(t *testing.T) {
t.Errorf("err = %v", err)
}
}
// ---- albums without an id: looked up in MusicBrainz by the artist and the title ----
type finderFake struct {
id string
err error
calls int
artist string
title string
}
func (f *finderFake) FindReleaseGroup(_ context.Context, artist, title string) (string, error) {
f.calls++
f.artist, f.title = artist, title
return f.id, f.err
}
// searching is an archive whose source may look an album up: album 2 has no id and a name,
// album 3 has no id and no artist.
func searching() (*archive, *finderFake) {
a := newArchive()
a.sw.set(providers.MusicBrainz, true)
f := &finderFake{id: releaseGroup}
a.src.Albums = albumsFake{
ids: map[int64]string{1: releaseGroup, 2: "", 3: ""},
names: map[int64][2]string{1: {"Radiohead", "The Bends"}, 2: {"Radiohead", "OK Computer"}, 3: {"", "Untitled"}},
}
a.src.Finder = f
return a, f
}
func TestAnAlbumWithNoIdIsLookedUpByArtistAndTitle(t *testing.T) {
a, f := searching()
got, err := a.src.AlbumImage(context.Background(), 2)
if err != nil || string(got.Data) != "cover bytes" {
t.Fatalf("got %+v err %v", got, err)
}
if f.artist != "Radiohead" || f.title != "OK Computer" {
t.Errorf("looked up %q / %q", f.artist, f.title)
}
if a.cv.got != releaseGroup {
t.Errorf("the archive was asked for %q, want the id that was found", a.cv.got)
}
}
func TestNothingIsLookedUpWhenTheIdIsKnownOrAProviderIsOffOrTheArtistIsMissing(t *testing.T) {
for name, tt := range map[string]struct {
prepare func(*archive)
id int64
want error
}{
"the files have the id": {func(*archive) {}, 1, nil},
"musicbrainz is off": {func(a *archive) { a.sw.set(providers.MusicBrainz, false) }, 2, ErrNoID},
"the archive is off": {func(a *archive) { a.sw.set(providers.CoverArtArchive, false) }, 2, ErrProviderDisabled},
"the album has no artist": {func(*archive) {}, 3, ErrNoID},
"the album does not exist": {func(*archive) {}, 99, ErrAlbumNotFound},
} {
a, f := searching()
tt.prepare(a)
if _, err := a.src.AlbumImage(context.Background(), tt.id); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
if f.calls != 0 {
t.Errorf("%s: MusicBrainz was asked", name)
}
}
}
func TestWithoutAFinderTheAlbumIsNeverLookedUp(t *testing.T) {
a, _ := searching()
a.src.Finder = nil
if _, err := a.src.AlbumImage(context.Background(), 2); !errors.Is(err, ErrNoID) {
t.Errorf("err = %v, want ErrNoID", err)
}
}
func TestWhenNoResultIsSureThereIsNoCover(t *testing.T) {
a, f := searching()
f.id = ""
if _, err := a.src.AlbumImage(context.Background(), 2); !errors.Is(err, ErrNoImage) {
t.Errorf("err = %v, want ErrNoImage", err)
}
if a.cv.calls != 0 {
t.Error("the archive was asked without an id")
}
}
func TestAMusicBrainzFailureIsAnOutage(t *testing.T) {
for name, why := range map[string]error{
"unavailable": musicbrainz.ErrUnavailable,
"rate limited": musicbrainz.ErrRateLimited,
"unreadable": musicbrainz.ErrBadResponse,
"a deadline": fmt.Errorf("%w: %w", musicbrainz.ErrUnavailable, context.DeadlineExceeded),
} {
a, f := searching()
f.err = why
if _, err := a.src.AlbumImage(context.Background(), 2); !errors.Is(err, ErrSourceUnavailable) {
t.Errorf("%s: err = %v, want ErrSourceUnavailable", name, err)
}
}
}
func TestACallerWhoLeavesDuringTheLookupGetsTheCancellation(t *testing.T) {
a, f := searching()
f.err = context.Canceled
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := a.src.AlbumImage(ctx, 2); !errors.Is(err, context.Canceled) || errors.Is(err, ErrSourceUnavailable) {
t.Errorf("err = %v", err)
}
}
func TestSwitchedOffDuringTheLookupNothingIsReturned(t *testing.T) {
a, _ := searching()
a.cv.hook = func() { a.sw.set(providers.CoverArtArchive, false) }
if _, err := a.src.AlbumImage(context.Background(), 2); !errors.Is(err, ErrProviderDisabled) {
t.Errorf("err = %v, want ErrProviderDisabled", err)
}
}

View file

@ -1,9 +1,13 @@
// Package musicbrainz is a small read-only client for the MusicBrainz web
// service.
//
// Sonda only reads an artist whose MusicBrainz id it already knows. It never
// searches by name: a search returns look-alikes (a conductor named like a rock
// band) and picking one would be a guess. Only the id leaves the server.
// Sonda reads an artist whose MusicBrainz id it already knows, and only the id leaves the
// server. It never searches for an artist by name: a search returns look-alikes (a conductor
// named like a rock band) and picking one would be a guess.
//
// The one search there is looks for the release group of an album that has no id (see
// FindReleaseGroup): it sends the artist and the title, and takes a result only when both
// match exactly. The id it finds is never stored as the identity of the album.
//
// MusicBrainz asks every client to identify itself with a contact in the
// User-Agent and to stay at one request per second: both are the business of the
@ -16,11 +20,15 @@ import (
"encoding/json"
"errors"
"fmt"
"net/url"
"regexp"
"sort"
"strings"
"unicode"
"unicode/utf8"
"golang.org/x/text/unicode/norm"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
@ -168,6 +176,77 @@ func (c *Client) WikidataID(ctx context.Context, id string) (string, error) {
return "", nil
}
// minGroupScore is the score (0 to 100) MusicBrainz must give a release group for it to be taken.
const minGroupScore = 95
// maxNameLen bounds the artist and the title of a search: longer ones are not names.
const maxNameLen = 200
// FindReleaseGroup looks up the release group of the album that this artist made with this
// title, and returns its id, or "" when no result is sure. "No result is sure" is not an error.
//
// A result is taken only if MusicBrainz scores it at least 95, its title is the title asked for
// and one of the names of its artist credit is the artist asked for, all compared after folding
// case, accents and punctuation. The first such result in MusicBrainz's order wins. Nothing
// else is guessed: an album called "OK Computer (Remastered)" finds nothing for "OK Computer".
// Only the artist and the title are sent.
func (c *Client) FindReleaseGroup(ctx context.Context, artist, title string) (string, error) {
artist, title = strings.TrimSpace(artist), strings.TrimSpace(title)
wantArtist, wantTitle := fold(artist), fold(title)
if wantArtist == "" || wantTitle == "" || utf8.RuneCountInString(artist) > maxNameLen || utf8.RuneCountInString(title) > maxNameLen {
return "", nil
}
query := `releasegroup:"` + quoted(title) + `" AND artist:"` + quoted(artist) + `"`
body, err := c.get(ctx, c.base+"release-group?query="+url.QueryEscape(query)+"&limit=5&fmt=json")
if err != nil {
return "", err
}
var doc struct {
Groups []struct {
ID string `json:"id"`
Score int `json:"score"`
Title string `json:"title"`
Credit []struct {
Name string `json:"name"`
Artist struct {
Name string `json:"name"`
} `json:"artist"`
} `json:"artist-credit"`
} `json:"release-groups"`
}
if err := json.Unmarshal(body, &doc); err != nil || doc.Groups == nil {
return "", ErrBadResponse
}
for _, g := range doc.Groups {
if g.Score < minGroupScore || !idRe.MatchString(g.ID) || fold(g.Title) != wantTitle {
continue
}
for _, cr := range g.Credit {
if fold(cr.Name) == wantArtist || fold(cr.Artist.Name) == wantArtist {
return strings.ToLower(g.ID), nil
}
}
}
return "", nil
}
// quoted escapes what has a meaning inside the quotes of a search phrase.
func quoted(s string) string {
return strings.NewReplacer(`\`, `\\`, `"`, `\"`).Replace(s)
}
// fold is a name as it is compared: lower case, without accents and without anything that is
// not a letter or a digit. It is "" for a name made only of symbols, which never matches.
func fold(s string) string {
var b strings.Builder
for _, r := range norm.NFKD.String(s) {
if unicode.IsLetter(r) || unicode.IsDigit(r) {
b.WriteRune(unicode.ToLower(r))
}
}
return b.String()
}
// Probe asks for one public artist, to find out whether the service answers. It
// is a providers.Checker, and answers with the errors the settings understand.
func (c *Client) Probe(ctx context.Context) error {

View file

@ -3,8 +3,10 @@ package musicbrainz
import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
@ -218,3 +220,115 @@ func TestWikidataIDErrors(t *testing.T) {
t.Errorf("not json: err = %v", err)
}
}
// ---- FindReleaseGroup ----
func groups(items ...string) string { return `{"release-groups":[` + strings.Join(items, ",") + `]}` }
func group(id string, score int, title, artist string) string {
return fmt.Sprintf(`{"id":%q,"score":%d,"title":%q,"artist-credit":[{"name":%q,"artist":{"id":"x","name":%q}}]}`, id, score, title, artist, artist)
}
const rg1 = "aaaaaaaa-1111-4222-8333-444444444444"
const rg2 = "bbbbbbbb-1111-4222-8333-444444444444"
func TestFindReleaseGroupSendsOnlyTheArtistAndTheTitle(t *testing.T) {
s := newServer(t, 200, groups(group(rg1, 100, "OK Computer", "Radiohead")))
got, err := s.client().FindReleaseGroup(context.Background(), "Radiohead", "OK Computer")
if err != nil || got != rg1 {
t.Fatalf("got %q err %v", got, err)
}
reqs := s.requests()
if len(reqs) != 1 || !strings.HasPrefix(reqs[0], "/ws/2/release-group?") {
t.Fatalf("requests = %v", reqs)
}
u, err := url.Parse(reqs[0])
if err != nil {
t.Fatal(err)
}
q := u.Query()
if q.Get("query") != `releasegroup:"OK Computer" AND artist:"Radiohead"` || q.Get("fmt") != "json" || q.Get("limit") == "" {
t.Errorf("query = %v", q)
}
}
func TestFindReleaseGroupEscapesWhatIsInsideTheQuotes(t *testing.T) {
s := newServer(t, 200, groups())
if _, err := s.client().FindReleaseGroup(context.Background(), `AC/DC "live"`, `Back \ In "Black"`); err != nil {
t.Fatal(err)
}
u, _ := url.Parse(s.requests()[0])
if got, want := u.Query().Get("query"), `releasegroup:"Back \\ In \"Black\"" AND artist:"AC/DC \"live\""`; got != want {
t.Errorf("query = %s, want %s", got, want)
}
}
func TestFindReleaseGroupAcceptsOnlyAnExactMatchThatScoresHigh(t *testing.T) {
tests := []struct {
name, body, artist, title, want string
}{
{"exact", groups(group(rg1, 100, "OK Computer", "Radiohead")), "Radiohead", "OK Computer", rg1},
{"score 95 is enough", groups(group(rg1, 95, "OK Computer", "Radiohead")), "Radiohead", "OK Computer", rg1},
{"score 94 is not", groups(group(rg1, 94, "OK Computer", "Radiohead")), "Radiohead", "OK Computer", ""},
{"case, accents and punctuation do not matter", groups(group(rg1, 100, "Ágætis byrjun", "Sigur Rós")), "sigur ros", "AGÆTIS byrjun!", rg1},
{"another title", groups(group(rg1, 100, "OK Computer OKNOTOK 1997 2017", "Radiohead")), "Radiohead", "OK Computer", ""},
{"another artist", groups(group(rg1, 100, "OK Computer", "Radiohead Tribute")), "Radiohead", "OK Computer", ""},
{"no result", groups(), "Radiohead", "OK Computer", ""},
{"the first that matches, not the first one", groups(group(rg1, 100, "Pablo Honey", "Radiohead"), group(rg2, 98, "OK Computer", "Radiohead")), "Radiohead", "OK Computer", rg2},
{"the first of two that match", groups(group(rg1, 100, "OK Computer", "Radiohead"), group(rg2, 99, "OK Computer", "Radiohead")), "Radiohead", "OK Computer", rg1},
{"a title of only symbols is not compared", groups(group(rg1, 100, "???", "Radiohead")), "Radiohead", "!!!", ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
s := newServer(t, 200, tt.body)
got, err := s.client().FindReleaseGroup(context.Background(), tt.artist, tt.title)
if err != nil || got != tt.want {
t.Errorf("got %q err %v, want %q", got, err, tt.want)
}
})
}
}
func TestFindReleaseGroupAcceptsAnyNameOfTheCredit(t *testing.T) {
body := `{"release-groups":[{"id":"` + rg1 + `","score":100,"title":"Watch the Throne",
"artist-credit":[{"name":"Jay-Z","joinphrase":" & ","artist":{"id":"1","name":"Jay-Z"}},{"name":"Kanye West","artist":{"id":"2","name":"Kanye West"}}]}]}`
s := newServer(t, 200, body)
got, err := s.client().FindReleaseGroup(context.Background(), "Kanye West", "Watch the Throne")
if err != nil || got != rg1 {
t.Errorf("got %q err %v", got, err)
}
}
func TestFindReleaseGroupAsksNothingWithoutBothNames(t *testing.T) {
s := newServer(t, 200, groups(group(rg1, 100, "x", "y")))
long := strings.Repeat("a", 300)
for _, c := range [][2]string{{"", "OK Computer"}, {"Radiohead", ""}, {" ", "OK Computer"}, {long, "OK Computer"}, {"Radiohead", long}} {
got, err := s.client().FindReleaseGroup(context.Background(), c[0], c[1])
if err != nil || got != "" {
t.Errorf("%q / %q: got %q err %v", c[0], c[1], got, err)
}
}
if n := len(s.requests()); n != 0 {
t.Errorf("%d requests, want none", n)
}
}
func TestFindReleaseGroupErrors(t *testing.T) {
tests := []struct {
status int
body string
want error
}{
{503, "", ErrUnavailable},
{429, "", ErrRateLimited},
{400, "", ErrBadRequest},
{200, "not json", ErrBadResponse},
{200, `{"count":0}`, ErrBadResponse},
}
for _, tt := range tests {
s := newServer(t, tt.status, tt.body)
if _, err := s.client().FindReleaseGroup(context.Background(), "Radiohead", "OK Computer"); !errors.Is(err, tt.want) {
t.Errorf("status %d body %q: err = %v, want %v", tt.status, tt.body, err, tt.want)
}
}
}

View file

@ -148,9 +148,11 @@ var known = []meta{
},
{
name: MusicBrainz,
sends: "The MusicBrainz id of the artist you look up (never a name: Sonda does not search), your " +
"IP address and the User-Agent, which carries the contact below. Never your users, your listens, " +
"track titles or file paths.",
sends: "The MusicBrainz id of the artist you look up (never an artist name: Sonda does not search for artists). " +
"Only when the Cover Art Archive provider is on too and an album has no picture and no id in its files: the name " +
"of the album's artist and its title, to find the album (a result is taken only when both match exactly). " +
"Your IP address and the User-Agent, which carries the contact below. Never your users, your listens " +
"or file paths.",
note: "Core data (artist names and ids) is public domain. Tags and genres are CC BY-NC-SA 3.0: " +
"non-commercial, with attribution to MusicBrainz.",
},
@ -178,9 +180,10 @@ var known = []meta{
name: CoverArtArchive,
sends: "Only when an album has no picture in your own files, when someone opens its cover or an admin starts the " +
"covers job (which goes through every album): the MusicBrainz release group id that the tags of " +
"its files carry (never a title: Sonda does not search), your IP address and the User-Agent, which carries " +
"the contact below. Redirects are followed only to archive.org, over https. Never your users, your listens, " +
"track titles or file paths.",
"its files carry, your IP address and the User-Agent, which carries the contact below. When the files have no id " +
"and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz to find " +
"the id, and the id found goes to the archive. Redirects are followed only to archive.org, over https. " +
"Never your users, your listens or file paths.",
note: "Covers are uploaded by MusicBrainz users and have no licence of their own: Sonda keeps a copy only for " +
"the albums of your own library, for your use, and never shows it as free.",
},

View file

@ -477,6 +477,23 @@ func TestCommonsIsAProviderThatNeedsNoKeyAndSaysWhatItSendsAndKeeps(t *testing.T
}
}
func TestMusicBrainzSaysItIsToldTheArtistAndTheTitleOfAnAlbumWithNoId(t *testing.T) {
svc, _ := newSvc()
infos, _ := svc.List(context.Background())
for _, in := range infos {
if in.Name != MusicBrainz {
continue
}
for _, want := range []string{"name of the album's artist and its title", "never an artist name"} {
if !strings.Contains(in.Sends, want) {
t.Errorf("the text does not say %q: %q", want, in.Sends)
}
}
return
}
t.Fatal("no musicbrainz provider")
}
func TestCoverArtArchiveIsAProviderThatNeedsNoKeyAndSaysWhatItSends(t *testing.T) {
svc, _ := newSvc()
infos, _ := svc.List(context.Background())
@ -492,7 +509,7 @@ func TestCoverArtArchiveIsAProviderThatNeedsNoKeyAndSaysWhatItSends(t *testing.T
if c.Enabled || c.NeedsAPIKey {
t.Errorf("coverartarchive = %+v: off by default, no key", c)
}
for _, want := range []string{"release group", "never a title", "album"} {
for _, want := range []string{"release group", "album", "its title go to MusicBrainz"} {
if !strings.Contains(c.Sends+" "+c.Notes, want) {
t.Errorf("the text does not say %q: %q %q", want, c.Sends, c.Notes)
}

View file

@ -128,6 +128,21 @@ func (r *ImageRepo) AlbumMBID(ctx context.Context, albumID int64) (string, error
return m.String, nil
}
// AlbumNames returns the name of the artist of an album ("" when it has none) and its title, or
// images.ErrAlbumNotFound.
func (r *ImageRepo) AlbumNames(ctx context.Context, albumID int64) (artist, title string, err error) {
var a sql.NullString
err = r.db.QueryRowContext(ctx,
"SELECT al.title, ar.name FROM albums al LEFT JOIN artists ar ON ar.id = al.artist_id WHERE al.id = ?", albumID).Scan(&title, &a)
if errors.Is(err, sql.ErrNoRows) {
return "", "", images.ErrAlbumNotFound
}
if err != nil {
return "", "", fmt.Errorf("read album names: %w", err)
}
return a.String, title, nil
}
// albumsToBuildWhere is the condition for "has something to play and no cover that is still
// good". A cover is good while it is younger than its time to live: a found one CoverTTL, "no
// cover" NoCoverTTL. A stale record (a file changed) and an album with no record are not good.

View file

@ -335,6 +335,37 @@ func TestImageRepoAlbumMBID(t *testing.T) {
})
}
func TestImageRepoAlbumNames(t *testing.T) {
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
ctx := context.Background()
lib := NewLibraryRepo(db)
l, _ := lib.CreateLibrary(ctx, "m", "/lib/m")
if _, err := lib.SaveFile(ctx, l.ID, pf("a/1.flac", "T", "OK Computer", "Radiohead", 1, 1, "flac", "Radiohead"), 0); err != nil {
t.Fatal(err)
}
if _, err := db.ExecContext(ctx, "INSERT INTO albums (title, title_key, search_key) VALUES ('Orphan', 'orphan', 'orphan')"); err != nil {
t.Fatal(err)
}
id := func(title string) int64 {
var id int64
if err := db.QueryRow("SELECT id FROM albums WHERE title = ?", title).Scan(&id); err != nil {
t.Fatal(err)
}
return id
}
repo := NewImageRepo(db)
if artist, title, err := repo.AlbumNames(ctx, id("OK Computer")); err != nil || artist != "Radiohead" || title != "OK Computer" {
t.Errorf("names = %q / %q err = %v", artist, title, err)
}
if artist, title, err := repo.AlbumNames(ctx, id("Orphan")); err != nil || artist != "" || title != "Orphan" {
t.Errorf("an album with no artist: %q / %q err = %v", artist, title, err)
}
if _, _, err := repo.AlbumNames(ctx, 99999); !errors.Is(err, images.ErrAlbumNotFound) {
t.Errorf("unknown album: err = %v", err)
}
})
}
func TestPurgingTheArchiveForgetsOnlyTheCoversThatCameFromIt(t *testing.T) {
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
ctx := context.Background()

View file

@ -1034,8 +1034,11 @@ export interface paths {
* For an album split in folders named like `CD1` or `Disc 2`, the picture in the folder
* above them is used too. When the files have no picture at all, and an admin switched on
* the `coverartarchive` provider, the cover is asked from the Cover Art Archive by the
* MusicBrainz release group id that the tags of the album's files carry (never by title; a
* full rescan reads that tag again for music indexed before); only that id leaves the server.
* MusicBrainz release group id that the tags of the album's files carry (a full rescan reads
* that tag again for music indexed before); only that id leaves the server. An album whose
* files have no id is looked up in MusicBrainz by its artist and title when the
* `musicbrainz` provider is on too: the id is taken only when MusicBrainz is sure (an exact
* match of both names), is used for that request and is not stored.
* The files of the user always come first. That request can take several seconds and may
* answer `provider_unavailable`. At most a couple of covers are built at the same time, so a
* client that asks for many at once may wait.