diff --git a/README.md b/README.md index 1e34ab6..a70029b 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr - **Your taste.** `GET /me/taste` shows what Sonda has learned from your own listening: the artists you like and the kinds of music (tags) they have in common. A complete listen counts for more than a short one, a skip counts against the artist, and older listens weigh less (they halve every 90 days, `taste.half_life_days`; only the last ten half-lives are read). It is worked out on every request from data already in your database; nothing is sent to any provider. - **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers). - **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers). +- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers). - **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere. It also takes `?access_token=`, so an `` element can use it. - **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it. - **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who. @@ -187,7 +188,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another ## External providers -Sonda can ask Last.fm, MusicBrainz and Wikipedia about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, and Wikipedia (with Wikidata) for descriptions of artists. +Sonda can ask Last.fm, MusicBrainz, Wikipedia and Wikimedia Commons about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, Wikipedia (with Wikidata) for descriptions of artists, and Wikimedia Commons (with Wikidata) for free photos of artists. - **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it. - **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to. @@ -204,7 +205,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends. -**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key. +**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key. **Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer. @@ -232,6 +233,7 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er | signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). | | signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. | | signed in | `GET /artists/{id}/tags` | What Last.fm and MusicBrainz say about the artist's kind of music, from each provider that is on. | +| signed in | `GET`, `HEAD /artists/{id}/image`, `GET /artists/{id}/image/credit` | A free picture of the artist from Wikimedia Commons (query `size=small\|large`, also `?access_token=`) and its credit. | | signed in | `GET /artists/{id}/description` | What Wikipedia says about the artist, with its page and licence, from the Wikipedia provider when an admin switched it on. | | signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 50). | | signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. | diff --git a/cmd/sondad/main.go b/cmd/sondad/main.go index 24d6cfd..9298c17 100644 --- a/cmd/sondad/main.go +++ b/cmd/sondad/main.go @@ -26,6 +26,7 @@ import ( "codeberg.org/squidd/sonda/internal/library" "codeberg.org/squidd/sonda/internal/listening" "codeberg.org/squidd/sonda/internal/providers" + "codeberg.org/squidd/sonda/internal/providers/commons" "codeberg.org/squidd/sonda/internal/providers/lastfm" "codeberg.org/squidd/sonda/internal/providers/musicbrainz" "codeberg.org/squidd/sonda/internal/providers/outbound" @@ -55,6 +56,12 @@ const ( wikipediaMaxBody = 1 << 20 ) +// commonsMaxBody leaves room for a 960 pixel copy of a picture, which is some hundred kilobytes. +const ( + commonsMinInterval = 300 * time.Millisecond + commonsMaxBody = 8 << 20 +) + func main() { if err := run(); err != nil { fmt.Fprintln(os.Stderr, "sondad:", err) @@ -151,9 +158,34 @@ func run() error { providerSettings.SetChecker(providers.Wikipedia, func(ctx context.Context, _ string) error { return wikipediaClient.Probe(ctx) }) + // Pictures of artists from Wikimedia Commons, through Wikidata, under free licences only. + commonsClient := commons.New(outbound.New(outbound.Options{ + UserAgentFunc: providerSettings.UserAgentFor(providers.Commons, version), + MinInterval: commonsMinInterval, + MaxBody: commonsMaxBody, + }), commons.WikidataURL, commons.CommonsURL) + providerSettings.SetChecker(providers.Commons, func(ctx context.Context, _ string) error { + return commonsClient.Probe(ctx) + }) + if err := os.MkdirAll(cfg.Storage.DataDir, 0o700); err != nil { + return fmt.Errorf("create the data folder: %w", err) + } + imageRepo := storage.NewImageRepo(db) + pictures := images.NewService(cfg.Storage.DataDir, imageRepo, imageRepo) discoveryRepo := storage.NewDiscoveryRepo(db) + pictures.SetArtistSource(&images.CommonsSource{ + Artists: discoveryRepo, Settings: providerSettings, MusicBrainz: musicbrainzClient, Commons: commonsClient, + }) providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) { - return discoveryRepo.Purge(ctx, name) + out, err := discoveryRepo.Purge(ctx, name) + if err != nil || name != providers.Commons { + return out, err + } + // The records are gone; the pictures themselves are files in the data folder. + if _, err := pictures.RemoveArtistFiles(); err != nil { + return out, fmt.Errorf("remove the pictures: %w", err) + } + return out, nil }) discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger) tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{ @@ -165,30 +197,27 @@ func run() error { tasteService := taste.NewService(tasteRepo, taste.Options{MinListen: cfg.Listening.MinListen(), HalfLife: cfg.Taste.HalfLife()}) userSettings := usersettings.NewService(storage.NewUserSettingsRepo(db)) feedbacks := feedback.NewService(storage.NewFeedbackRepo(db)) - if err := os.MkdirAll(cfg.Storage.DataDir, 0o700); err != nil { - return fmt.Errorf("create the data folder: %w", err) - } - imageRepo := storage.NewImageRepo(db) weights := discover.Weights{ Affinity: cfg.Scoring.WeightAffinity, Genre: cfg.Scoring.WeightGenre, Preference: cfg.Scoring.WeightPreference, Discovery: cfg.Scoring.WeightDiscovery, } router := httpapi.NewRouter(httpapi.Deps{ - Logger: logger, - Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()), - Tokens: tokens, - Listening: listening.NewService(storage.NewListeningRepo(db), cfg.Listening.MinListen()), - Scan: scanControl{ctx: ctx, m: scans}, - Libraries: libraries, - Catalog: catalog.NewService(storage.NewCatalogRepo(db)), - Providers: providerSettings, - Discovery: discoveries, - Tags: tags, - Taste: tasteService, - Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights), - Feedback: feedbacks, - Images: images.NewService(cfg.Storage.DataDir, imageRepo, imageRepo), + Logger: logger, + Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()), + Tokens: tokens, + Listening: listening.NewService(storage.NewListeningRepo(db), cfg.Listening.MinListen()), + Scan: scanControl{ctx: ctx, m: scans}, + Libraries: libraries, + Catalog: catalog.NewService(storage.NewCatalogRepo(db)), + Providers: providerSettings, + Discovery: discoveries, + Tags: tags, + Taste: tasteService, + Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights), + Feedback: feedbacks, + Images: pictures, + ArtistImages: pictures, About: about.NewService(discoveryRepo, about.WikipediaSource{MusicBrainz: musicbrainzClient, Wikipedia: wikipediaClient}, providerSettings, cfg.Wikipedia.Languages, logger), Settings: userSettings, diff --git a/internal/api/http/api_test.go b/internal/api/http/api_test.go index 6f73078..5cbf2ec 100644 --- a/internal/api/http/api_test.go +++ b/internal/api/http/api_test.go @@ -99,7 +99,8 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun feedbackSvc := feedback.NewService(storage.NewFeedbackRepo(db)) deps.Feedback = feedbackSvc imageRepo := storage.NewImageRepo(db) - deps.Images = images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo) + imageSvc := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo) + deps.Images, deps.ArtistImages = imageSvc, imageSvc mutate(&deps, db) if deps.About == nil { deps.About = about.NewService(storage.NewDiscoveryRepo(db), noWikipedia{}, prov, []string{"en"}, zap.NewNop()) diff --git a/internal/api/http/artist_image_test.go b/internal/api/http/artist_image_test.go new file mode 100644 index 0000000..a613da3 --- /dev/null +++ b/internal/api/http/artist_image_test.go @@ -0,0 +1,189 @@ +package httpapi + +import ( + "bytes" + "context" + "encoding/json" + "image" + "image/color" + "net/http" + "net/http/httptest" + "path/filepath" + "strconv" + "sync/atomic" + "testing" + + "github.com/uptrace/bun" + + "codeberg.org/squidd/sonda/internal/images" + "codeberg.org/squidd/sonda/internal/providers" + "codeberg.org/squidd/sonda/internal/storage" +) + +type fakeArtistPictures struct { + found images.Found + err error + calls int32 +} + +func (f *fakeArtistPictures) ArtistImage(context.Context, int64) (images.Found, error) { + atomic.AddInt32(&f.calls, 1) + return f.found, f.err +} + +type artistImageEnv struct { + *similarEnv + pics *fakeArtistPictures +} + +func newArtistImageEnv(t *testing.T) *artistImageEnv { + t.Helper() + e := &artistImageEnv{pics: &fakeArtistPictures{found: images.Found{ + Data: testJPEG(t, 1500, 1200, color.RGBA{30, 60, 200, 255}), + Credit: images.Credit{Author: "Raph_PH", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0", + PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg"}, + }}} + e.similarEnv = newSimilarEnvWith(t, func(d *Deps, db *bun.DB, _ *providers.Service) { + repo := storage.NewImageRepo(db) + svc := images.NewService(filepath.Join(t.TempDir(), "data"), repo, repo) + svc.SetArtistSource(e.pics) + d.ArtistImages = svc + }) + return e +} + +func (e *artistImageEnv) get(method, path, token string, header ...string) *httptest.ResponseRecorder { + req := httptest.NewRequest(method, "/api/v1"+path, nil) + if token != "" { + req.Header.Set("Authorization", "Bearer "+token) + } + for i := 0; i+1 < len(header); i += 2 { + req.Header.Set(header[i], header[i+1]) + } + rec := httptest.NewRecorder() + e.h.ServeHTTP(rec, req) + return rec +} + +func (e *artistImageEnv) path(id int64, rest string) string { + return "/artists/" + strconv.FormatInt(id, 10) + "/image" + rest +} + +func TestArtistImageNeedsASignedInUserAndTakesTheTokenInTheQuery(t *testing.T) { + e := newArtistImageEnv(t) + if rec := e.get("GET", e.path(e.alpha, ""), ""); rec.Code != http.StatusUnauthorized { + t.Errorf("no token: %d", rec.Code) + } + if rec := e.get("GET", e.path(e.alpha, "/credit"), ""); rec.Code != http.StatusUnauthorized { + t.Errorf("credit without a token: %d", rec.Code) + } + if rec := e.get("GET", e.path(e.alpha, "?access_token="+e.admin), ""); rec.Code != http.StatusOK { + t.Errorf("token in the query (an cannot send a header): %d", rec.Code) + } + // The credit is JSON for a program: a token in the query is not accepted there. + if rec := e.get("GET", e.path(e.alpha, "/credit?access_token="+e.admin), ""); rec.Code != http.StatusUnauthorized { + t.Errorf("credit with the token in the query: %d, want 401", rec.Code) + } +} + +func TestArtistImageIsAJPEGInTheSizeAskedAndCached(t *testing.T) { + e := newArtistImageEnv(t) + for query, width := range map[string]int{"": 1000, "?size=large": 1000, "?size=small": 300} { + rec := e.get("GET", e.path(e.alpha, query), e.admin) + if rec.Code != http.StatusOK || rec.Header().Get("Content-Type") != "image/jpeg" { + t.Fatalf("%q: %d %s", query, rec.Code, rec.Header().Get("Content-Type")) + } + cfg, format, err := image.DecodeConfig(bytes.NewReader(rec.Body.Bytes())) + if err != nil || format != "jpeg" || cfg.Width != width { + t.Errorf("%q: %v %s %v, want %d wide", query, cfg, format, err, width) + } + if rec.Header().Get("X-Content-Type-Options") != "nosniff" || rec.Header().Get("Cache-Control") != "private, max-age=3600" { + t.Errorf("%q: headers %v", query, rec.Header()) + } + } + if n := atomic.LoadInt32(&e.pics.calls); n != 1 { + t.Errorf("the provider was asked %d times, want 1", n) + } + first := e.get("GET", e.path(e.alpha, ""), e.admin) + etag := first.Header().Get("ETag") + if etag == "" || etag[0] != '"' { + t.Fatalf("etag = %q", etag) + } + if rec := e.get("GET", e.path(e.alpha, ""), e.admin, "If-None-Match", etag); rec.Code != http.StatusNotModified || rec.Body.Len() != 0 { + t.Errorf("If-None-Match: %d", rec.Code) + } + if rec := e.get("HEAD", e.path(e.alpha, ""), e.admin); rec.Code != http.StatusOK || rec.Body.Len() != 0 { + t.Errorf("HEAD: %d, %d bytes", rec.Code, rec.Body.Len()) + } +} + +func TestArtistImageCreditSaysWhoseItIsAndUnderWhichLicence(t *testing.T) { + e := newArtistImageEnv(t) + rec := e.get("GET", e.path(e.alpha, "/credit"), e.admin) + if rec.Code != http.StatusOK { + t.Fatalf("%d %s", rec.Code, rec.Body.String()) + } + var got map[string]any + if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil { + t.Fatal(err) + } + sameKeys(t, "credit", got, "source", "author", "license", "license_url", "page_url") + if got["source"] != "Wikimedia Commons" || got["author"] != "Raph_PH" || got["license"] != "CC BY 4.0" || + got["license_url"] != "https://creativecommons.org/licenses/by/4.0" || got["page_url"] != "https://commons.wikimedia.org/wiki/File:Band.jpg" { + t.Errorf("credit = %v", got) + } + // The credit comes from the record, with no new request. + if n := atomic.LoadInt32(&e.pics.calls); n != 1 { + t.Errorf("the provider was asked %d times, want 1", n) + } +} + +func TestArtistImageErrors(t *testing.T) { + for name, tt := range map[string]struct { + err error + code int + want string + }{ + "no free picture": {images.ErrNoImage, 404, "image_not_found"}, + "provider off": {images.ErrProviderDisabled, 404, "provider_disabled"}, + "no id": {images.ErrNoID, 404, "no_musicbrainz_id"}, + "unknown artist": {images.ErrArtistNotFound, 404, "artist_not_found"}, + "provider down": {images.ErrSourceUnavailable, 502, "provider_unavailable"}, + } { + e := newArtistImageEnv(t) + e.pics.err = tt.err + for _, path := range []string{e.path(e.alpha, ""), e.path(e.alpha, "/credit")} { + rec := e.get("GET", path, e.admin) + var body struct { + Error struct{ Code string } `json:"error"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &body) + if rec.Code != tt.code || body.Error.Code != tt.want { + t.Errorf("%s %s: %d %s, want %d %s", name, path, rec.Code, rec.Body.String(), tt.code, tt.want) + } + } + } +} + +func TestArtistImageRefusesWhatItCannotUnderstand(t *testing.T) { + e := newArtistImageEnv(t) + for name, path := range map[string]string{ + "a bad id": "/artists/abc/image", + "id zero": "/artists/0/image", + "a bad size": e.path(e.alpha, "?size=huge"), + "size twice": e.path(e.alpha, "?size=small&size=large"), + "credit bad id": "/artists/abc/image/credit", + } { + rec := e.get("GET", path, e.admin) + var body struct { + Error struct{ Code string } `json:"error"` + } + _ = json.Unmarshal(rec.Body.Bytes(), &body) + if rec.Code != http.StatusBadRequest || body.Error.Code != "invalid_input" { + t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String()) + } + } + if n := atomic.LoadInt32(&e.pics.calls); n != 0 { + t.Errorf("the provider was asked %d times for requests that were refused", n) + } +} diff --git a/internal/api/http/contract_test.go b/internal/api/http/contract_test.go index 8a62b21..9b81a1a 100644 --- a/internal/api/http/contract_test.go +++ b/internal/api/http/contract_test.go @@ -5,6 +5,7 @@ import ( "context" "encoding/json" "image" + "image/color" "image/png" "io" "mime" @@ -25,6 +26,7 @@ import ( "codeberg.org/squidd/sonda/internal/catalog" "codeberg.org/squidd/sonda/internal/discovery" + "codeberg.org/squidd/sonda/internal/images" "codeberg.org/squidd/sonda/internal/library" "codeberg.org/squidd/sonda/internal/providers" "codeberg.org/squidd/sonda/internal/providers/lastfm" @@ -132,6 +134,11 @@ func newContract(t *testing.T) *contract { d.Libraries = libs d.Catalog = catalog.NewService(storage.NewCatalogRepo(db)) d.Streams = streaming.NewService(repo) + imageRepo := storage.NewImageRepo(db) + pictures := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo) + pictures.SetArtistSource(&fakeArtistPictures{found: images.Found{Data: testJPEG(t, 800, 600, color.White), + Credit: images.Credit{Author: "Someone", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0", PageURL: "https://commons.wikimedia.org/wiki/File:X.jpg"}}}) + d.Images, d.ArtistImages = pictures, pictures }) router, err := legacy.NewRouter(loadSpec(t)) @@ -419,8 +426,8 @@ func TestResponsesMatchTheSpec(t *testing.T) { // ---- providers ---- listed := c.expect(c.call("GET", "/providers", token, nil), 200, "list providers")["providers"].([]any) - if len(listed) != 3 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false { - t.Fatalf("providers = %v, want three, all off", listed) + if len(listed) != 4 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false || listed[3].(map[string]any)["enabled"] != false { + t.Fatalf("providers = %v, want four, all off", listed) } c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input") c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input") @@ -574,6 +581,33 @@ func TestResponsesMatchTheSpec(t *testing.T) { t.Fatalf("cover HEAD: %d", rec.Code) } + // ---- the picture of an artist ---- + photo := "/artists/" + artistID + "/image" + if rec := c.call("GET", photo, token, nil); rec.Code != 200 || rec.Header().Get("Content-Type") != "image/jpeg" { + t.Fatalf("artist image: %d %s", rec.Code, rec.Header().Get("Content-Type")) + } + if rec := c.call("GET", photo+"?size=small", token, nil); rec.Code != 200 { + t.Fatalf("small artist image: %d", rec.Code) + } + if rec := c.call("GET", photo, token, nil, "Range", "bytes=0-9"); rec.Code != 206 { + t.Fatalf("artist image range: %d", rec.Code) + } + if rec := c.call("HEAD", photo, token, nil); rec.Code != 200 { + t.Fatalf("artist image HEAD: %d", rec.Code) + } + if rec := c.call("GET", photo+"?access_token="+token, "", nil); rec.Code != 200 { + t.Fatalf("artist image with the token in the query: %d", rec.Code) + } + if rec := c.call("GET", photo, token, nil, "If-None-Match", c.call("GET", photo, token, nil).Header().Get("ETag")); rec.Code != 304 { + t.Fatalf("artist image If-None-Match: %d", rec.Code) + } + if cr := c.expect(c.call("GET", photo+"/credit", token, nil), 200, "the credit of the picture"); cr["license"] != "CC BY 4.0" || cr["source"] != "Wikimedia Commons" { + t.Fatalf("credit = %v", cr) + } + c.expect(c.call("GET", photo+"?size=huge", token, nil), 400, "an artist image of a size that does not exist", "invalid_input") + c.expect(c.call("GET", "/artists/abc/image/credit", token, nil), 400, "the credit of a bad id", "invalid_input") + c.expect(c.call("GET", photo+"/credit", "", nil), 401, "the credit without a token", "unauthorized") + // ---- discover ---- c.expect(c.call("GET", "/discover", token, nil), 400, "discover with no seed and no permission", "seed_required") found := c.expect(c.call("GET", "/discover?seed="+artistID+"&limit=5", token, nil), 200, "discover from a named seed") diff --git a/internal/api/http/description_test.go b/internal/api/http/description_test.go index aa4e690..c5ede88 100644 --- a/internal/api/http/description_test.go +++ b/internal/api/http/description_test.go @@ -190,7 +190,7 @@ func TestPurgingWikipediaTakesTheDescriptionsBack(t *testing.T) { if p.Code != http.StatusOK { t.Fatalf("purge: %d %v", p.Code, p.Body) } - sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "artists") + sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "images", "artists") if p.Body["descriptions"] != float64(1) { t.Errorf("purge = %v, want one description", p.Body) } diff --git a/internal/api/http/handlers_artist_image.go b/internal/api/http/handlers_artist_image.go new file mode 100644 index 0000000..7ff9c5a --- /dev/null +++ b/internal/api/http/handlers_artist_image.go @@ -0,0 +1,87 @@ +package httpapi + +import ( + "context" + "errors" + "net/http" + + "github.com/gin-gonic/gin" + + "codeberg.org/squidd/sonda/internal/images" +) + +type artistImageHandlers struct{ svc artistImageService } + +type creditJSON struct { + Source string `json:"source"` + Author string `json:"author"` + License string `json:"license"` + LicenseURL string `json:"license_url"` + PageURL string `json:"page_url"` +} + +// writeArtistImageError answers for what can go wrong; it reports whether it did. +func writeArtistImageError(c *gin.Context, err error) bool { + switch { + case err == nil: + return false + case errors.Is(err, images.ErrArtistNotFound): + writeError(c, http.StatusNotFound, "artist_not_found", "no such artist") + case errors.Is(err, images.ErrNoImage): + writeError(c, http.StatusNotFound, "image_not_found", "there is no free picture of this artist") + case errors.Is(err, images.ErrProviderDisabled): + writeError(c, http.StatusNotFound, "provider_disabled", "the picture provider is switched off") + case errors.Is(err, images.ErrNoID): + writeError(c, http.StatusNotFound, "no_musicbrainz_id", "the artist has no MusicBrainz id, and Sonda does not look one up by name") + case errors.Is(err, images.ErrSourceUnavailable): + _ = c.Error(err) // keep the cause in the log + writeError(c, http.StatusBadGateway, "provider_unavailable", "the picture provider could not be reached, try again later") + case errors.Is(err, context.Canceled): + c.AbortWithStatus(statusClientClosed) // the client left; nobody is waiting + default: + _ = c.Error(err) + writeError(c, http.StatusInternalServerError, "internal", "internal error") + } + return true +} + +// image serves the free picture of an artist: a JPEG made from the copy found on Wikimedia +// Commons, built on the first request and kept in the data folder. +func (h *artistImageHandlers) image(c *gin.Context) { + id, ok := pathID(c) + if !ok { + return + } + size, ok := sizeParam(c) + if !ok { + return + } + pic, err := h.svc.ArtistImage(c.Request.Context(), id, size) + if writeArtistImageError(c, err) { + return + } + defer pic.File.Close() + hdr := c.Writer.Header() + hdr.Set("Content-Type", "image/jpeg") + hdr.Set("ETag", `"`+pic.ETag+`"`) + hdr.Set("Cache-Control", "private, max-age=3600") + hdr.Set("X-Content-Type-Options", "nosniff") + http.ServeContent(c.Writer, c.Request, "image.jpg", pic.ModTime, pic.File) +} + +// credit says whose the picture is and under which licence, to be shown next to it. +func (h *artistImageHandlers) credit(c *gin.Context) { + id, ok := pathID(c) + if !ok { + return + } + pic, err := h.svc.ArtistImage(c.Request.Context(), id, images.Small) + if writeArtistImageError(c, err) { + return + } + _ = pic.File.Close() + c.JSON(http.StatusOK, creditJSON{ + Source: "Wikimedia Commons", Author: pic.Credit.Author, License: pic.Credit.License, + LicenseURL: pic.Credit.LicenseURL, PageURL: pic.Credit.PageURL, + }) +} diff --git a/internal/api/http/handlers_providers.go b/internal/api/http/handlers_providers.go index 50fc14a..8de50ab 100644 --- a/internal/api/http/handlers_providers.go +++ b/internal/api/http/handlers_providers.go @@ -97,5 +97,5 @@ func (h *providersHandlers) purge(c *gin.Context) { writeProviderError(c, err) return } - c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "descriptions": res.Descriptions, "artists": res.Artists}) + c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "descriptions": res.Descriptions, "images": res.Images, "artists": res.Artists}) } diff --git a/internal/api/http/providers_test.go b/internal/api/http/providers_test.go index 5e3b958..3bf006a 100644 --- a/internal/api/http/providers_test.go +++ b/internal/api/http/providers_test.go @@ -46,7 +46,7 @@ func TestProvidersStartOffAndSayWhatTheySend(t *testing.T) { t.Fatalf("list: %d %v", r.Code, r.Body) } list := r.Body["providers"].([]any) - if len(list) != 3 { + if len(list) != 4 { t.Fatalf("providers = %v", list) } for _, it := range list { @@ -263,7 +263,7 @@ func TestPurgeRemovesWhatTheProviderGaveAndNothingOfMine(t *testing.T) { if p.Code != http.StatusOK { t.Fatalf("%d %v", p.Code, p.Body) } - sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "artists") + sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "images", "artists") if p.Body["similar_artists"] != float64(2) || p.Body["tags"] != float64(0) || p.Body["artists"] != float64(1) { t.Errorf("purge = %v, want 2 similar artists, no tags and the one artist Gamma", p.Body) } diff --git a/internal/api/http/router.go b/internal/api/http/router.go index 94084f2..a2ff1d1 100644 --- a/internal/api/http/router.go +++ b/internal/api/http/router.go @@ -86,6 +86,10 @@ type feedbackService interface { List(ctx context.Context, userID int64, kind feedback.Kind, p feedback.Page) (feedback.Listing, error) } +type artistImageService interface { + ArtistImage(ctx context.Context, artistID int64, size images.Size) (*images.Cover, error) +} + type aboutService interface { Description(ctx context.Context, artistID int64) (about.Result, error) } @@ -121,23 +125,24 @@ type libraryService interface { // Deps are the services the API needs. type Deps struct { - Logger *zap.Logger - Auth authService - Tokens tokenParser - Listening listenService - Scan scanService - Libraries libraryService - Catalog catalogService - Providers providerService - Discovery discoveryService - Tags tagService - Taste tasteService - Discover discoverService - Feedback feedbackService - Images coverService - About aboutService - Settings userSettingsService - Streams streamService + Logger *zap.Logger + Auth authService + Tokens tokenParser + Listening listenService + Scan scanService + Libraries libraryService + Catalog catalogService + Providers providerService + Discovery discoveryService + Tags tagService + Taste tasteService + Discover discoverService + Feedback feedbackService + Images coverService + About aboutService + ArtistImages artistImageService + Settings userSettingsService + Streams streamService } // NewRouter builds the gin engine with all routes. @@ -192,6 +197,7 @@ func NewRouter(d Deps) *gin.Engine { protected.GET("/artists/:id/tags", tg.tags) ab := &aboutHandlers{svc: d.About} protected.GET("/artists/:id/description", ab.description) + protected.GET("/artists/:id/image/credit", (&artistImageHandlers{svc: d.ArtistImages}).credit) protected.GET("/albums", cat.listAlbums) protected.GET("/albums/:id", cat.getAlbum) protected.GET("/tracks", cat.listTracks) @@ -207,6 +213,9 @@ func NewRouter(d Deps) *gin.Engine { cv := &coverHandlers{svc: d.Images} audio.GET("/albums/:id/cover", cv.album) audio.HEAD("/albums/:id/cover", cv.album) + ai := &artistImageHandlers{svc: d.ArtistImages} + audio.GET("/artists/:id/image", ai.image) + audio.HEAD("/artists/:id/image", ai.image) // Admins only. admin := protected.Group("", requireAdmin(d.Auth)) diff --git a/internal/api/openapi/openapi.yaml b/internal/api/openapi/openapi.yaml index 30259fb..e08ca70 100644 --- a/internal/api/openapi/openapi.yaml +++ b/internal/api/openapi/openapi.yaml @@ -971,6 +971,144 @@ paths: application/json: schema: {$ref: '#/components/schemas/Error'} + /artists/{id}/image: + parameters: + - {$ref: '#/components/parameters/Id'} + get: + tags: [Discovery] + operationId: getArtistImage + summary: A free picture of an artist + description: | + A JPEG made from a picture of the artist on Wikimedia Commons, only when its licence is + free: public domain, CC0, CC BY or CC BY-SA. Non-commercial, no-derivatives, fair use and + unknown licences are refused, because Sonda keeps a copy. **Where you show it, show its + credit** (`GET /artists/{id}/image/credit`): the author, the licence and the page of the file. + + The way to it is a chain of identifiers, never a search by name: the artist's MusicBrainz + id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons + for its licence and a copy. It needs the `commons` provider and the `musicbrainz` one, which + an admin switches on. It is built the first time it is asked for (that request can take a few + seconds), scaled to fit, saved in the data folder, and served from there afterwards: for 30 + days, or a week when there is no free picture; an older copy is served when Commons cannot + be reached. What is sent: the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, + the name of the file to Commons, and the address of a copy to Wikimedia, with your server's + IP address and a `User-Agent` with a contact. + + An `` element cannot send headers, so, like the stream and the covers, this accepts + the access token as the `access_token` query parameter. A header, when present, always wins. + security: + - bearerAuth: [] + - streamQueryToken: [] + parameters: + - name: size + in: query + description: "`small` (300 pixels at most on the longest side) or `large` (1000). Large when absent." + schema: {type: string, enum: [small, large], default: large} + - name: If-None-Match + in: header + required: false + schema: {type: string} + - name: Range + in: header + required: false + schema: {type: string} + responses: + '200': + description: The picture. + headers: + ETag: {$ref: '#/components/headers/ETag'} + Cache-Control: {$ref: '#/components/headers/CacheControl'} + Content-Length: {$ref: '#/components/headers/ContentLength'} + Last-Modified: {$ref: '#/components/headers/LastModified'} + content: + image/jpeg: + schema: {type: string, format: binary} + '206': + description: The requested byte range of the picture. + headers: + Content-Range: + schema: {type: string} + content: + image/jpeg: + schema: {type: string, format: binary} + '304': {description: Not modified since the `ETag` sent.} + '400': {$ref: '#/components/responses/InvalidInput'} + '401': {$ref: '#/components/responses/Unauthorized'} + '404': + description: | + No such artist (`artist_not_found`), no free picture of it (`image_not_found`), the + provider is off (`provider_disabled`), or the artist has no MusicBrainz id + (`no_musicbrainz_id`). + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} + '502': + description: The picture provider could not be reached (`provider_unavailable`). Try again later. + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} + head: + tags: [Discovery] + operationId: headArtistImage + summary: Headers of a picture + description: Same as `GET`, without the body. + security: + - bearerAuth: [] + - streamQueryToken: [] + parameters: + - name: size + in: query + schema: {type: string, enum: [small, large], default: large} + - name: If-None-Match + in: header + required: false + schema: {type: string} + responses: + '200': + description: Headers of the picture. + headers: + ETag: {$ref: '#/components/headers/ETag'} + Cache-Control: {$ref: '#/components/headers/CacheControl'} + Content-Length: {$ref: '#/components/headers/ContentLength'} + Last-Modified: {$ref: '#/components/headers/LastModified'} + '304': {description: Not modified since the `ETag` sent.} + '400': {$ref: '#/components/responses/InvalidInput'} + '401': {$ref: '#/components/responses/Unauthorized'} + '404': {description: "No such artist, no free picture, the provider is off or the artist has no MusicBrainz id."} + '502': {description: "The picture provider could not be reached."} + + /artists/{id}/image/credit: + parameters: + - {$ref: '#/components/parameters/Id'} + get: + tags: [Discovery] + operationId: getArtistImageCredit + summary: The credit of a picture of an artist + description: | + Whose the picture is and under which licence, to be shown next to it (the licences ask for + it). The author is plain text and may be empty; `license_url` may be empty when the licence + has no address; `page_url` is the page of the file on Commons, where everything is. + It answers like `GET /artists/{id}/image` (same errors), because the credit exists only with a picture. + security: [{bearerAuth: []}] + responses: + '200': + description: The credit. + content: + application/json: + schema: {$ref: '#/components/schemas/PictureCredit'} + '400': {$ref: '#/components/responses/InvalidInput'} + '401': {$ref: '#/components/responses/Unauthorized'} + '404': + description: No such artist (`artist_not_found`), no free picture (`image_not_found`), the provider is off (`provider_disabled`) or the artist has no MusicBrainz id (`no_musicbrainz_id`). + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} + '502': + description: The picture provider could not be reached (`provider_unavailable`). + content: + application/json: + schema: {$ref: '#/components/schemas/Error'} + /albums: get: tags: [Catalog] @@ -1308,7 +1446,7 @@ paths: - name: name in: path required: true - schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]} + schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]} put: tags: [Providers] operationId: updateProvider @@ -1346,7 +1484,7 @@ paths: - name: name in: path required: true - schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]} + schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]} post: tags: [Providers] operationId: testProvider @@ -1381,7 +1519,7 @@ paths: - name: name in: path required: true - schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]} + schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]} delete: tags: [Providers] operationId: purgeProviderData @@ -1492,7 +1630,7 @@ components: type: apiKey in: query name: access_token - description: The same access token, for stream URLs used by an `