feat(images): ask again without the disc or edition note of a title
Many titles end with a note taken from the tags: (disc 2), (bonus disc), (Deluxe Edition), sometimes cut off by a tag that is too short. A release group is named after the album, not after a disc, so those albums found nothing. The lookup now asks for the title as it is first and, only when MusicBrainz has no sure result, once more without the notes at its end (baseTitle). A group in brackets with none of the words of a note stays part of the title, and a title that would be left empty is kept. The rule for accepting a result does not change. On 25 albums of a real library with such notes, 19 found a cover (none before). Closes #87
This commit is contained in:
parent
a64122efdd
commit
2669f6c6dc
6 changed files with 149 additions and 19 deletions
|
|
@ -253,7 +253,7 @@ The address can have a path if Lidarr runs under a URL base, and it cannot carry
|
|||
|
||||
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
|
||||
|
||||
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, the MusicBrainz id of the artist (and, only for the covers of albums that have no id, the artist's name and the album's title: see **Covers**), your IP address and a `User-Agent` that carries your contact. Never your users, your listens or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files (and, for the albums whose files have no id, with MusicBrainz on, the artist and the title, one request a second); `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send the MusicBrainz release group id of an album, read from the tags of its own files, and only when the files have no picture. When the files have no id either and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz, which is asked for the release group: a result is taken only when MusicBrainz scores it 95 or more and its title and artist are the ones asked for (compared without case, accents or punctuation), and anything doubtful is no cover. The id that comes back is used for that request and is not stored: it is not what your files say, and it is never used to read an artist or to send anything to Lidarr; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key.
|
||||
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, the MusicBrainz id of the artist (and, only for the covers of albums that have no id, the artist's name and the album's title: see **Covers**), your IP address and a `User-Agent` that carries your contact. Never your users, your listens or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files (and, for the albums whose files have no id, with MusicBrainz on, the artist and the title, one request a second); `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send the MusicBrainz release group id of an album, read from the tags of its own files, and only when the files have no picture. When the files have no id either and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz, which is asked for the release group: a result is taken only when MusicBrainz scores it 95 or more and its title and artist are the ones asked for (compared without case, accents or punctuation), and anything doubtful is no cover. A title that ends with a note about the disc or the edition (`(disc 2)`, `(bonus disc)`, `(Deluxe Edition)`) is asked as it is first and, if MusicBrainz is not sure, once more without the note, because a release group is named after the album and not after a disc. The id that comes back is used for that request and is not stored: it is not what your files say, and it is never used to read an artist or to send anything to Lidarr; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key.
|
||||
|
||||
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
|
||||
|
||||
|
|
|
|||
|
|
@ -1512,7 +1512,8 @@ paths:
|
|||
that tag again for music indexed before); only that id leaves the server. An album whose
|
||||
files have no id is looked up in MusicBrainz by its artist and title when the
|
||||
`musicbrainz` provider is on too: the id is taken only when MusicBrainz is sure (an exact
|
||||
match of both names), is used for that request and is not stored.
|
||||
match of both names; a title that ends with a note like `(disc 2)` or `(Deluxe Edition)` is
|
||||
tried again without it), is used for that request and is not stored.
|
||||
The files of the user always come first. That request can take several seconds and may
|
||||
answer `provider_unavailable`. At most a couple of covers are built at the same time, so a
|
||||
client that asks for many at once may wait.
|
||||
|
|
|
|||
|
|
@ -3,6 +3,8 @@ package images
|
|||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/coverart"
|
||||
|
|
@ -98,12 +100,43 @@ func (c *CoverArtSource) find(ctx context.Context, albumID int64) (string, error
|
|||
if artist == "" || title == "" {
|
||||
return "", ErrNoID
|
||||
}
|
||||
id, err := c.Finder.FindReleaseGroup(ctx, artist, title)
|
||||
switch {
|
||||
case err != nil:
|
||||
return "", failed(ctx, err)
|
||||
case id == "":
|
||||
return "", ErrNoImage
|
||||
// The title as the files have it comes first: a release group is named after the album
|
||||
// and not after a disc, so a title with a disc or an edition note is tried again without it.
|
||||
tries := []string{title}
|
||||
if base := baseTitle(title); base != title {
|
||||
tries = append(tries, base)
|
||||
}
|
||||
return id, nil
|
||||
for _, t := range tries {
|
||||
id, err := c.Finder.FindReleaseGroup(ctx, artist, t)
|
||||
if err != nil {
|
||||
return "", failed(ctx, err)
|
||||
}
|
||||
if id != "" {
|
||||
return id, nil
|
||||
}
|
||||
}
|
||||
return "", ErrNoImage
|
||||
}
|
||||
|
||||
// notesRe is a note at the end of a title that says which disc or which edition it is: a group
|
||||
// in brackets (closed, or cut off by a tag that is too short) that has one of the words of a
|
||||
// note, like "(disc 2)", "(bonus disc: Rarities)" or "[2011 Deluxe Remaster]". A group without
|
||||
// them ("(The Mastermind Redux)") is part of the title.
|
||||
var notesRe = regexp.MustCompile(`(?i)\s*[(\[][^()\[\]]*\b(?:disc|disk|cd|bonus|deluxe|remaster(?:ed)?|edition|expanded|anniversary|reissue|special|limited)\b[^()\[\]]*[)\]]?\s*$`)
|
||||
|
||||
// baseTitle is a title without the notes at its end, or the title itself when taking them off
|
||||
// would leave nothing.
|
||||
func baseTitle(title string) string {
|
||||
base := strings.TrimSpace(title)
|
||||
for {
|
||||
next := strings.TrimSpace(notesRe.ReplaceAllString(base, ""))
|
||||
if next == base || next == "" {
|
||||
break
|
||||
}
|
||||
base = next
|
||||
}
|
||||
if base == "" {
|
||||
return title
|
||||
}
|
||||
return base
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import (
|
|||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"slices"
|
||||
"testing"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
|
|
@ -145,17 +146,26 @@ func TestACallerWhoLeavesGetsTheCancellation(t *testing.T) {
|
|||
// ---- albums without an id: looked up in MusicBrainz by the artist and the title ----
|
||||
|
||||
type finderFake struct {
|
||||
id string
|
||||
err error
|
||||
calls int
|
||||
artist string
|
||||
title string
|
||||
id string
|
||||
answers map[string]string // by title: when set, a title it does not list has no answer
|
||||
err error
|
||||
calls int
|
||||
artist string
|
||||
title string
|
||||
titles []string
|
||||
}
|
||||
|
||||
func (f *finderFake) FindReleaseGroup(_ context.Context, artist, title string) (string, error) {
|
||||
f.calls++
|
||||
f.artist, f.title = artist, title
|
||||
return f.id, f.err
|
||||
f.titles = append(f.titles, title)
|
||||
if f.err != nil {
|
||||
return "", f.err
|
||||
}
|
||||
if f.answers != nil {
|
||||
return f.answers[title], nil
|
||||
}
|
||||
return f.id, nil
|
||||
}
|
||||
|
||||
// searching is an archive whose source may look an album up: album 2 has no id and a name,
|
||||
|
|
@ -165,8 +175,11 @@ func searching() (*archive, *finderFake) {
|
|||
a.sw.set(providers.MusicBrainz, true)
|
||||
f := &finderFake{id: releaseGroup}
|
||||
a.src.Albums = albumsFake{
|
||||
ids: map[int64]string{1: releaseGroup, 2: "", 3: ""},
|
||||
names: map[int64][2]string{1: {"Radiohead", "The Bends"}, 2: {"Radiohead", "OK Computer"}, 3: {"", "Untitled"}},
|
||||
ids: map[int64]string{1: releaseGroup, 2: "", 3: "", 4: "", 5: ""},
|
||||
names: map[int64][2]string{
|
||||
1: {"Radiohead", "The Bends"}, 2: {"Radiohead", "OK Computer"}, 3: {"", "Untitled"},
|
||||
4: {"Aerosmith", "Alive II (disc 2)"}, 5: {"Pearl Jam", "Ten"},
|
||||
},
|
||||
}
|
||||
a.src.Finder = f
|
||||
return a, f
|
||||
|
|
@ -260,3 +273,84 @@ func TestSwitchedOffDuringTheLookupNothingIsReturned(t *testing.T) {
|
|||
t.Errorf("err = %v, want ErrProviderDisabled", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestADiscOrEditionSuffixIsTriedWithoutItWhenTheFullTitleHasNoSureResult(t *testing.T) {
|
||||
a, f := searching()
|
||||
f.answers = map[string]string{"Alive II": releaseGroup}
|
||||
got, err := a.src.AlbumImage(context.Background(), 4)
|
||||
if err != nil || string(got.Data) != "cover bytes" {
|
||||
t.Fatalf("got %+v err %v", got, err)
|
||||
}
|
||||
if want := []string{"Alive II (disc 2)", "Alive II"}; !slices.Equal(f.titles, want) {
|
||||
t.Errorf("titles asked = %q, want %q", f.titles, want)
|
||||
}
|
||||
if a.cv.got != releaseGroup {
|
||||
t.Errorf("the archive was asked for %q", a.cv.got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheFullTitleComesFirstAndIsEnoughWhenItIsSure(t *testing.T) {
|
||||
a, f := searching()
|
||||
f.answers = map[string]string{"Alive II (disc 2)": releaseGroup, "Alive II": "cccccccc-1111-4222-8333-444444444444"}
|
||||
if _, err := a.src.AlbumImage(context.Background(), 4); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if want := []string{"Alive II (disc 2)"}; !slices.Equal(f.titles, want) {
|
||||
t.Errorf("titles asked = %q, want only the full one", f.titles)
|
||||
}
|
||||
if a.cv.got != releaseGroup {
|
||||
t.Errorf("the archive was asked for %q", a.cv.got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestATitleWithoutASuffixIsAskedOnce(t *testing.T) {
|
||||
a, f := searching()
|
||||
f.id = ""
|
||||
if _, err := a.src.AlbumImage(context.Background(), 5); !errors.Is(err, ErrNoImage) {
|
||||
t.Errorf("err = %v, want ErrNoImage", err)
|
||||
}
|
||||
if f.calls != 1 {
|
||||
t.Errorf("MusicBrainz was asked %d times, want once", f.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWhenNeitherTitleHasASureResultThereIsNoCoverAndAFailureStops(t *testing.T) {
|
||||
a, f := searching()
|
||||
f.id = ""
|
||||
if _, err := a.src.AlbumImage(context.Background(), 4); !errors.Is(err, ErrNoImage) || f.calls != 2 {
|
||||
t.Errorf("err = %v after %d calls, want ErrNoImage after 2", err, f.calls)
|
||||
}
|
||||
a, f = searching()
|
||||
f.err = musicbrainz.ErrUnavailable
|
||||
if _, err := a.src.AlbumImage(context.Background(), 4); !errors.Is(err, ErrSourceUnavailable) || f.calls != 1 {
|
||||
t.Errorf("err = %v after %d calls, want an outage after 1", err, f.calls)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBaseTitleTakesOffTheDiscAndEditionNotes(t *testing.T) {
|
||||
for in, want := range map[string]string{
|
||||
"Alive II (disc 2)": "Alive II",
|
||||
"Crossroads (Disc 2)": "Crossroads",
|
||||
"The Boogie House Tapes (Disc 1": "The Boogie House Tapes",
|
||||
"Ummagumma (disc 1: Live Album)": "Ummagumma",
|
||||
"Seventeen Seconds (bonus disc: Rarities 1979-1980)": "Seventeen Seconds",
|
||||
"Pornography (bonus disc: Rarit": "Pornography",
|
||||
"Exciter (bonus disc)": "Exciter",
|
||||
"A-Ha Shake Heartbreak (Bonus CD)": "A-Ha Shake Heartbreak",
|
||||
"Live At Wembley Stadium (2011 Deluxe Remaster)": "Live At Wembley Stadium",
|
||||
"Extreme Behavior (Special Edit": "Extreme Behavior",
|
||||
"Album [Remastered]": "Album",
|
||||
"Album (Deluxe Edition) (disc 2)": "Album",
|
||||
"Soundtrack to the Apocalypse (disc 5: Live at The Grove in Anaheim, CA 5/2/02)": "Soundtrack to the Apocalypse",
|
||||
// What is part of the title stays.
|
||||
"Cobras And Fire (The Mastermind Redux)": "Cobras And Fire (The Mastermind Redux)",
|
||||
"This Type Of Thinking (Could Do Us In)": "This Type Of Thinking (Could Do Us In)",
|
||||
"Ten": "Ten",
|
||||
"(disc 1)": "(disc 1)", // nothing would be left
|
||||
"Disc One": "Disc One", // not in brackets
|
||||
} {
|
||||
if got := baseTitle(in); got != want {
|
||||
t.Errorf("baseTitle(%q) = %q, want %q", in, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -188,7 +188,8 @@ const maxNameLen = 200
|
|||
// A result is taken only if MusicBrainz scores it at least 95, its title is the title asked for
|
||||
// and one of the names of its artist credit is the artist asked for, all compared after folding
|
||||
// case, accents and punctuation. The first such result in MusicBrainz's order wins. Nothing
|
||||
// else is guessed: an album called "OK Computer (Remastered)" finds nothing for "OK Computer".
|
||||
// else is guessed: an album called "OK Computer (Remastered)" finds nothing for "OK Computer"
|
||||
// (the caller may ask again with the title without the note).
|
||||
// Only the artist and the title are sent.
|
||||
func (c *Client) FindReleaseGroup(ctx context.Context, artist, title string) (string, error) {
|
||||
artist, title = strings.TrimSpace(artist), strings.TrimSpace(title)
|
||||
|
|
|
|||
3
web/src/api/schema.d.ts
vendored
3
web/src/api/schema.d.ts
vendored
|
|
@ -1038,7 +1038,8 @@ export interface paths {
|
|||
* that tag again for music indexed before); only that id leaves the server. An album whose
|
||||
* files have no id is looked up in MusicBrainz by its artist and title when the
|
||||
* `musicbrainz` provider is on too: the id is taken only when MusicBrainz is sure (an exact
|
||||
* match of both names), is used for that request and is not stored.
|
||||
* match of both names; a title that ends with a note like `(disc 2)` or `(Deluxe Edition)` is
|
||||
* tried again without it), is used for that request and is not stored.
|
||||
* The files of the user always come first. That request can take several seconds and may
|
||||
* answer `provider_unavailable`. At most a couple of covers are built at the same time, so a
|
||||
* client that asks for many at once may wait.
|
||||
|
|
|
|||
Loading…
Reference in a new issue