diff --git a/README.md b/README.md index e25b491..98c252c 100644 --- a/README.md +++ b/README.md @@ -257,7 +257,7 @@ The address can have a path if Lidarr runs under a URL base, and it cannot carry The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends. -**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, the MusicBrainz id of the artist (and, only for the covers of albums that have no id, the artist's name and the album's title: see **Covers**), your IP address and a `User-Agent` that carries your contact. Never your users, your listens or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files (and, for the albums whose files have no id, with MusicBrainz on, the artist and the title, one request a second); `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send the MusicBrainz release group id of an album, read from the tags of its own files, and only when the files have no picture. When the files have no id either and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz, which is asked for the release group: a result is taken only when MusicBrainz scores it 95 or more and its title and artist are the ones asked for (compared without case, accents or punctuation), and anything doubtful is no cover. A title that ends with a note about the disc or the edition (`(disc 2)`, `(bonus disc)`, `(Deluxe Edition)`) is asked as it is first and, if MusicBrainz is not sure, once more without the note, because a release group is named after the album and not after a disc. The id that comes back is used for that request and is not stored: it is not what your files say, and it is never used to read an artist or to send anything to Lidarr; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key. +**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, the MusicBrainz id of the artist (and, only for the covers of albums that have no id, the artist's name and the album's title: see **Covers**), your IP address and a `User-Agent` that carries your contact. Never your users, your listens or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library (the exceptions are the three jobs that an admin starts on purpose, each of them one request that goes through the whole library, one item after the other, spaced out, and each stops when its providers keep failing: `POST /library/covers`, with the Cover Art Archive on, sends the release group id of every album that has no picture in its files (and, for the albums whose files have no id, with MusicBrainz on, the artist and the title, one request a second); `POST /library/artist-images`, with Commons and MusicBrainz on, sends the MusicBrainz id of every artist that has one, to MusicBrainz, Wikidata and Commons; `POST /library/descriptions`, with Wikipedia and MusicBrainz on, sends it to MusicBrainz, Wikidata and Wikipedia), and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. When MusicBrainz answers that an id was merged into another, Sonda follows that one redirect, and only to an artist or a release group of musicbrainz.org over https; the new id is used for that request and is not stored, and any other redirect is refused. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Covers** (the Cover Art Archive provider) send the MusicBrainz release group id of an album, read from the tags of its own files, and only when the files have no picture. When the files have no id either and the MusicBrainz provider is on too, the name of the album's artist and its title go to MusicBrainz, which is asked for the release group: a result is taken only when MusicBrainz scores it 95 or more and its title and artist are the ones asked for (compared without case, accents or punctuation), and anything doubtful is no cover. A title that ends with a note about the disc or the edition (`(disc 2)`, `(bonus disc)`, `(Deluxe Edition)`) is asked as it is first and, if MusicBrainz is not sure, once more without the note, because a release group is named after the album and not after a disc. The id that comes back is used for that request and is not stored: it is not what your files say, and it is never used to read an artist or to send anything to Lidarr; the redirects of the archive are followed only to archive.org, over https. Its covers have no licence of their own: Sonda keeps a copy for your library and never treats it as free. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. **Lidarr** (your own server, at the address an admin set) gets the MusicBrainz id of an artist an admin sends, or of an artist someone opens `GET /artists/{id}/lidarr` for, and the API key, nothing else; asking it what it offers and testing the key send only the key. Never a name, your users, your listens or your files. What Lidarr then does is up to its own indexers and download clients. Besides the three jobs, the other exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key (or, for Lidarr, the address and the key); it sends nothing but the key. **Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer. diff --git a/cmd/sondad/main.go b/cmd/sondad/main.go index e3649ad..8ca19dc 100644 --- a/cmd/sondad/main.go +++ b/cmd/sondad/main.go @@ -155,8 +155,9 @@ func run() error { // MusicBrainz wants a contact in the User-Agent, which an admin can change: it // is read again for every request. musicbrainzClient := musicbrainz.New(outbound.New(outbound.Options{ - UserAgentFunc: providerSettings.UserAgentFor(providers.MusicBrainz, version), - MinInterval: musicbrainzMinInterval, + UserAgentFunc: providerSettings.UserAgentFor(providers.MusicBrainz, version), + MinInterval: musicbrainzMinInterval, + FollowRedirect: musicbrainz.RedirectAllowed, // an id that was merged leads to the new one, and to nothing else }), musicbrainz.BaseURL) providerSettings.SetChecker(providers.MusicBrainz, func(ctx context.Context, _ string) error { return musicbrainzClient.Probe(ctx) diff --git a/internal/providers/musicbrainz/musicbrainz.go b/internal/providers/musicbrainz/musicbrainz.go index 9d71ea3..6412804 100644 --- a/internal/providers/musicbrainz/musicbrainz.go +++ b/internal/providers/musicbrainz/musicbrainz.go @@ -13,6 +13,10 @@ // User-Agent and to stay at one request per second: both are the business of the // outbound client this one sends its requests through. Errors never repeat the // provider's own text. +// +// When two entries are merged, the id of the old one answers with a redirect to the new one. +// That is the only redirect the client that talks to MusicBrainz may follow (RedirectAllowed is +// what to give it); any other is refused, as everywhere else. package musicbrainz import ( @@ -53,6 +57,18 @@ var ( var idRe = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) +// redirectPathRe is where a merged id may lead: the same kind of thing, in the web service. +var redirectPathRe = regexp.MustCompile(`^/ws/2/(artist|release-group)/[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`) + +// RedirectAllowed says whether a redirect may be followed: https, no credentials, no port, +// musicbrainz.org itself (not a subdomain), and a path that is again an artist or a release +// group id of the web service. It is how a merged id reaches the entry it was merged into. +// The follow-up request goes out at once, a second in a row for this one id: merges are rare. +func RedirectAllowed(u *url.URL) bool { + return u.Scheme == "https" && u.User == nil && u.Port() == "" && u.Hostname() == "musicbrainz.org" && + redirectPathRe.MatchString(u.Path) +} + // Getter performs a GET. *outbound.Client is the one used in production. type Getter interface { Get(ctx context.Context, rawURL string) (*outbound.Response, error) diff --git a/internal/providers/musicbrainz/musicbrainz_test.go b/internal/providers/musicbrainz/musicbrainz_test.go index 60146fd..df4baa4 100644 --- a/internal/providers/musicbrainz/musicbrainz_test.go +++ b/internal/providers/musicbrainz/musicbrainz_test.go @@ -4,6 +4,7 @@ import ( "context" "errors" "fmt" + "io" "net/http" "net/http/httptest" "net/url" @@ -332,3 +333,99 @@ func TestFindReleaseGroupErrors(t *testing.T) { } } } + +// --- an id that was merged ---------------------------------------------------------------------- + +func TestRedirectAllowedOnlyToAnArtistOrReleaseGroupOnMusicBrainz(t *testing.T) { + const id = "4e0dffde-ad2d-45b7-9c75-d57ce55de061" + cases := []struct { + url string + want bool + }{ + {"https://musicbrainz.org/ws/2/artist/" + id + "?inc=url-rels&fmt=json", true}, + {"https://musicbrainz.org/ws/2/release-group/" + id, true}, + {"http://musicbrainz.org/ws/2/artist/" + id, false}, // not https + {"https://musicbrainz.org:8443/ws/2/artist/" + id, false}, // a port + {"https://user:pw@musicbrainz.org/ws/2/artist/" + id, false}, // credentials + {"https://evil.example/ws/2/artist/" + id, false}, // another host + {"https://musicbrainz.org.evil.example/ws/2/artist/" + id, false}, // look-alike host + {"https://beta.musicbrainz.org/ws/2/artist/" + id, false}, // a subdomain is not asked + {"https://musicbrainz.org/ws/2/artist/not-an-id", false}, // not an id + {"https://musicbrainz.org/ws/2/artist/" + id + "/../../login", false}, // a longer path + {"https://musicbrainz.org/login", false}, // not the web service + {"https://musicbrainz.org/ws/2/recording/" + id, false}, // not a thing Sonda asks for + } + for _, c := range cases { + u, err := url.Parse(c.url) + if err != nil { + t.Fatal(err) + } + if got := RedirectAllowed(u); got != c.want { + t.Errorf("RedirectAllowed(%s) = %v, want %v", c.url, got, c.want) + } + } +} + +// roundTrip lets a test answer for https://musicbrainz.org without the network. +type roundTrip func(*http.Request) *http.Response + +func (f roundTrip) RoundTrip(r *http.Request) (*http.Response, error) { return f(r), nil } + +func reply(status int, location, body string) *http.Response { + h := http.Header{"Content-Type": {"application/json"}} + if location != "" { + h.Set("Location", location) + } + return &http.Response{StatusCode: status, Header: h, Body: io.NopCloser(strings.NewReader(body))} +} + +func realClient(rt roundTrip) *Client { + return New(outbound.New(outbound.Options{ + UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries, + Transport: rt, FollowRedirect: RedirectAllowed, + }), BaseURL) +} + +const ( + oldID = "1cf2a9a4-ce75-4bff-b545-a788535ca3a5" + newID = "7fa7fc04-1011-4876-8095-ecd232edea87" +) + +func TestAnArtistWhoseIdWasMergedIsReadFromTheNewId(t *testing.T) { + var asked []string + c := realClient(func(r *http.Request) *http.Response { + asked = append(asked, r.URL.Host+r.URL.Path) + if strings.HasSuffix(r.URL.Path, oldID) { + return reply(301, "https://musicbrainz.org/ws/2/artist/"+newID+"?inc=url-rels&fmt=json", "") + } + return reply(200, "", `{"id":"`+newID+`","relations":[{"type":"wikidata","ended":false,"url":{"resource":"https://www.wikidata.org/wiki/Q42"}}]}`) + }) + got, err := c.WikidataID(context.Background(), oldID) + if err != nil || got != "Q42" { + t.Fatalf("WikidataID = %q, %v; want Q42", got, err) + } + want := []string{"musicbrainz.org/ws/2/artist/" + oldID, "musicbrainz.org/ws/2/artist/" + newID} + if strings.Join(asked, " ") != strings.Join(want, " ") { + t.Errorf("asked %v, want %v", asked, want) + } +} + +func TestARedirectToAnywhereElseIsStillRefused(t *testing.T) { + for _, to := range []string{ + "https://evil.example/ws/2/artist/" + newID, + "https://musicbrainz.org/login", + "http://musicbrainz.org/ws/2/artist/" + newID, + } { + var asked []string + c := realClient(func(r *http.Request) *http.Response { + asked = append(asked, r.URL.String()) + return reply(301, to, "") + }) + if _, err := c.WikidataID(context.Background(), oldID); !errors.Is(err, ErrUnavailable) { + t.Errorf("redirect to %s: err = %v, want ErrUnavailable", to, err) + } + if len(asked) != 1 { + t.Errorf("redirect to %s: %d requests, want only the first: %v", to, len(asked), asked) + } + } +}