feat(images): free photos of artists from Wikimedia Commons

- GET and HEAD /artists/{id}/image serve a free picture of an artist as a JPEG
  (size=small 300 px or large 1000 px, token in the query allowed for an <img>), and
  GET /artists/{id}/image/credit gives the author, the licence and the page of the file,
  to be shown next to it.
- New provider "commons" (off by default, no key, contact in the User-Agent), which also
  needs MusicBrainz on. The way to the picture uses ids only, never a name: the artist's
  MusicBrainz id, the Wikidata item MusicBrainz links to it, the file Wikidata names for
  it (P18), Commons for its licence, and a 960 px copy from a Wikimedia picture host.
- Only free licences: public domain, CC0, CC BY and CC BY-SA. Non-commercial,
  no-derivatives, GFDL alone, fair use and unknown licences are refused, since a copy is
  kept. The author is plain text, the licence address is kept only on creativecommons.org,
  and the download only from https thumb.wikimedia.org or upload.wikimedia.org.
- Built on the first request, kept in the data folder: 30 days, or a week when there is no
  free picture; an older copy is served when Commons cannot be reached; a failure is not
  remembered. Credit changes change the ETag.
- Removing the provider's data forgets the records and deletes the files; a read that started
  before it is not saved over it.
- images table gets the credit (migration 0015); the purge reports images.
- Spec, contract test, README updated; the facts of the real APIs are written down.

Closes #41
This commit is contained in:
tgiachi 2026-11-11 22:03:32 +01:00
commit e752d15536
26 changed files with 2176 additions and 81 deletions

View file

@ -18,6 +18,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Your taste.** `GET /me/taste` shows what Sonda has learned from your own listening: the artists you like and the kinds of music (tags) they have in common. A complete listen counts for more than a short one, a skip counts against the artist, and older listens weigh less (they halve every 90 days, `taste.half_life_days`; only the last ten half-lives are read). It is worked out on every request from data already in your database; nothing is sent to any provider.
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers).
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
- **Artist photos.** `GET /artists/{id}/image` serves a free picture of an artist from Wikimedia Commons, and `GET /artists/{id}/image/credit` says whose it is and under which licence, which you must show next to it. Only free licences are kept (public domain, CC0, CC BY, CC BY-SA): non-commercial, no-derivatives, fair-use and unknown ones are refused, because Sonda stores a copy (`?size=small` is 300 pixels, `large` 1000) in the data folder. With the `commons` provider and the `musicbrainz` one switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons for its licence. Many artists will have none. Kept for 30 days (a week when there is none) and removed, files included, with the provider's data. See [External providers](#external-providers).
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere. It also takes `?access_token=`, so an `<img>` element can use it.
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
@ -187,7 +188,7 @@ curl -s -X POST $BASE/users -H "$AUTH" -d '{"username":"bob","password":"another
## External providers
Sonda can ask Last.fm, MusicBrainz and Wikipedia about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, and Wikipedia (with Wikidata) for descriptions of artists.
Sonda can ask Last.fm, MusicBrainz, Wikipedia and Wikimedia Commons about artists. This is **off by default**: until an admin turns a provider on, nothing leaves your server. Today Last.fm is used for similar artists and tags, MusicBrainz for tags and for the link to Wikidata, Wikipedia (with Wikidata) for descriptions of artists, and Wikimedia Commons (with Wikidata) for free photos of artists.
- **Last.fm** needs an API key. Create one for your own use at `last.fm/api` (it needs a Last.fm account). No user name or password is involved and the "shared secret" is not needed. Last.fm data is for non-commercial use only, and you should credit Last.fm when you show it.
- **MusicBrainz** needs no key, but it asks every client to identify itself with a contact (a URL or an e-mail address) in the `User-Agent`. By default Sonda uses the address of the project, so no personal address is sent unless you choose to.
@ -204,7 +205,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Photos** (the Commons provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata, the name of the picture to Commons and the address of a copy to Wikimedia, with your IP address and a `User-Agent` with a contact, and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
@ -232,6 +233,7 @@ Everything lives under `/api/v1`. Bodies are JSON. Errors always look like `{"er
| signed in | `GET /auth/me`, `POST /auth/password` | Your account; change your password (signs you out everywhere). |
| signed in | `GET /artists`, `/albums`, `/tracks` and `/…/{id}` | Browse and search. Query: `q`, `limit` (default 50, max 200), `offset`, `library_id`, `artist_id`, `album_id`. |
| signed in | `GET /artists/{id}/tags` | What Last.fm and MusicBrainz say about the artist's kind of music, from each provider that is on. |
| signed in | `GET`, `HEAD /artists/{id}/image`, `GET /artists/{id}/image/credit` | A free picture of the artist from Wikimedia Commons (query `size=small\|large`, also `?access_token=`) and its credit. |
| signed in | `GET /artists/{id}/description` | What Wikipedia says about the artist, with its page and licence, from the Wikipedia provider when an admin switched it on. |
| signed in | `GET /artists/{id}/similar` | Artists similar to one, from Last.fm when an admin switched it on. Query: `limit` (default 20, max 50). |
| signed in | `GET`, `HEAD /tracks/{id}/stream` | Stream the original file. Also accepts `?access_token=`. |

View file

@ -26,6 +26,7 @@ import (
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/commons"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
"codeberg.org/squidd/sonda/internal/providers/outbound"
@ -55,6 +56,12 @@ const (
wikipediaMaxBody = 1 << 20
)
// commonsMaxBody leaves room for a 960 pixel copy of a picture, which is some hundred kilobytes.
const (
commonsMinInterval = 300 * time.Millisecond
commonsMaxBody = 8 << 20
)
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, "sondad:", err)
@ -151,9 +158,34 @@ func run() error {
providerSettings.SetChecker(providers.Wikipedia, func(ctx context.Context, _ string) error {
return wikipediaClient.Probe(ctx)
})
// Pictures of artists from Wikimedia Commons, through Wikidata, under free licences only.
commonsClient := commons.New(outbound.New(outbound.Options{
UserAgentFunc: providerSettings.UserAgentFor(providers.Commons, version),
MinInterval: commonsMinInterval,
MaxBody: commonsMaxBody,
}), commons.WikidataURL, commons.CommonsURL)
providerSettings.SetChecker(providers.Commons, func(ctx context.Context, _ string) error {
return commonsClient.Probe(ctx)
})
if err := os.MkdirAll(cfg.Storage.DataDir, 0o700); err != nil {
return fmt.Errorf("create the data folder: %w", err)
}
imageRepo := storage.NewImageRepo(db)
pictures := images.NewService(cfg.Storage.DataDir, imageRepo, imageRepo)
discoveryRepo := storage.NewDiscoveryRepo(db)
pictures.SetArtistSource(&images.CommonsSource{
Artists: discoveryRepo, Settings: providerSettings, MusicBrainz: musicbrainzClient, Commons: commonsClient,
})
providerSettings.SetPurger(func(ctx context.Context, name string) (providers.Purged, error) {
return discoveryRepo.Purge(ctx, name)
out, err := discoveryRepo.Purge(ctx, name)
if err != nil || name != providers.Commons {
return out, err
}
// The records are gone; the pictures themselves are files in the data folder.
if _, err := pictures.RemoveArtistFiles(); err != nil {
return out, fmt.Errorf("remove the pictures: %w", err)
}
return out, nil
})
discoveries := discovery.NewService(discoveryRepo, lastfmClient, providerSettings, logger)
tags := discovery.NewTagService(discoveryRepo, map[string]discovery.TagSource{
@ -165,30 +197,27 @@ func run() error {
tasteService := taste.NewService(tasteRepo, taste.Options{MinListen: cfg.Listening.MinListen(), HalfLife: cfg.Taste.HalfLife()})
userSettings := usersettings.NewService(storage.NewUserSettingsRepo(db))
feedbacks := feedback.NewService(storage.NewFeedbackRepo(db))
if err := os.MkdirAll(cfg.Storage.DataDir, 0o700); err != nil {
return fmt.Errorf("create the data folder: %w", err)
}
imageRepo := storage.NewImageRepo(db)
weights := discover.Weights{
Affinity: cfg.Scoring.WeightAffinity, Genre: cfg.Scoring.WeightGenre,
Preference: cfg.Scoring.WeightPreference, Discovery: cfg.Scoring.WeightDiscovery,
}
router := httpapi.NewRouter(httpapi.Deps{
Logger: logger,
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()),
Tokens: tokens,
Listening: listening.NewService(storage.NewListeningRepo(db), cfg.Listening.MinListen()),
Scan: scanControl{ctx: ctx, m: scans},
Libraries: libraries,
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
Providers: providerSettings,
Discovery: discoveries,
Tags: tags,
Taste: tasteService,
Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights),
Feedback: feedbacks,
Images: images.NewService(cfg.Storage.DataDir, imageRepo, imageRepo),
Logger: logger,
Auth: auth.NewService(storage.NewUserRepo(db), storage.NewRefreshRepo(db), tokens, cfg.Auth.RefreshTTL()),
Tokens: tokens,
Listening: listening.NewService(storage.NewListeningRepo(db), cfg.Listening.MinListen()),
Scan: scanControl{ctx: ctx, m: scans},
Libraries: libraries,
Catalog: catalog.NewService(storage.NewCatalogRepo(db)),
Providers: providerSettings,
Discovery: discoveries,
Tags: tags,
Taste: tasteService,
Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights),
Feedback: feedbacks,
Images: pictures,
ArtistImages: pictures,
About: about.NewService(discoveryRepo, about.WikipediaSource{MusicBrainz: musicbrainzClient, Wikipedia: wikipediaClient},
providerSettings, cfg.Wikipedia.Languages, logger),
Settings: userSettings,

View file

@ -99,7 +99,8 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
feedbackSvc := feedback.NewService(storage.NewFeedbackRepo(db))
deps.Feedback = feedbackSvc
imageRepo := storage.NewImageRepo(db)
deps.Images = images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo)
imageSvc := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo)
deps.Images, deps.ArtistImages = imageSvc, imageSvc
mutate(&deps, db)
if deps.About == nil {
deps.About = about.NewService(storage.NewDiscoveryRepo(db), noWikipedia{}, prov, []string{"en"}, zap.NewNop())

View file

@ -0,0 +1,189 @@
package httpapi
import (
"bytes"
"context"
"encoding/json"
"image"
"image/color"
"net/http"
"net/http/httptest"
"path/filepath"
"strconv"
"sync/atomic"
"testing"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/images"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/storage"
)
type fakeArtistPictures struct {
found images.Found
err error
calls int32
}
func (f *fakeArtistPictures) ArtistImage(context.Context, int64) (images.Found, error) {
atomic.AddInt32(&f.calls, 1)
return f.found, f.err
}
type artistImageEnv struct {
*similarEnv
pics *fakeArtistPictures
}
func newArtistImageEnv(t *testing.T) *artistImageEnv {
t.Helper()
e := &artistImageEnv{pics: &fakeArtistPictures{found: images.Found{
Data: testJPEG(t, 1500, 1200, color.RGBA{30, 60, 200, 255}),
Credit: images.Credit{Author: "Raph_PH", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0",
PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg"},
}}}
e.similarEnv = newSimilarEnvWith(t, func(d *Deps, db *bun.DB, _ *providers.Service) {
repo := storage.NewImageRepo(db)
svc := images.NewService(filepath.Join(t.TempDir(), "data"), repo, repo)
svc.SetArtistSource(e.pics)
d.ArtistImages = svc
})
return e
}
func (e *artistImageEnv) get(method, path, token string, header ...string) *httptest.ResponseRecorder {
req := httptest.NewRequest(method, "/api/v1"+path, nil)
if token != "" {
req.Header.Set("Authorization", "Bearer "+token)
}
for i := 0; i+1 < len(header); i += 2 {
req.Header.Set(header[i], header[i+1])
}
rec := httptest.NewRecorder()
e.h.ServeHTTP(rec, req)
return rec
}
func (e *artistImageEnv) path(id int64, rest string) string {
return "/artists/" + strconv.FormatInt(id, 10) + "/image" + rest
}
func TestArtistImageNeedsASignedInUserAndTakesTheTokenInTheQuery(t *testing.T) {
e := newArtistImageEnv(t)
if rec := e.get("GET", e.path(e.alpha, ""), ""); rec.Code != http.StatusUnauthorized {
t.Errorf("no token: %d", rec.Code)
}
if rec := e.get("GET", e.path(e.alpha, "/credit"), ""); rec.Code != http.StatusUnauthorized {
t.Errorf("credit without a token: %d", rec.Code)
}
if rec := e.get("GET", e.path(e.alpha, "?access_token="+e.admin), ""); rec.Code != http.StatusOK {
t.Errorf("token in the query (an <img> cannot send a header): %d", rec.Code)
}
// The credit is JSON for a program: a token in the query is not accepted there.
if rec := e.get("GET", e.path(e.alpha, "/credit?access_token="+e.admin), ""); rec.Code != http.StatusUnauthorized {
t.Errorf("credit with the token in the query: %d, want 401", rec.Code)
}
}
func TestArtistImageIsAJPEGInTheSizeAskedAndCached(t *testing.T) {
e := newArtistImageEnv(t)
for query, width := range map[string]int{"": 1000, "?size=large": 1000, "?size=small": 300} {
rec := e.get("GET", e.path(e.alpha, query), e.admin)
if rec.Code != http.StatusOK || rec.Header().Get("Content-Type") != "image/jpeg" {
t.Fatalf("%q: %d %s", query, rec.Code, rec.Header().Get("Content-Type"))
}
cfg, format, err := image.DecodeConfig(bytes.NewReader(rec.Body.Bytes()))
if err != nil || format != "jpeg" || cfg.Width != width {
t.Errorf("%q: %v %s %v, want %d wide", query, cfg, format, err, width)
}
if rec.Header().Get("X-Content-Type-Options") != "nosniff" || rec.Header().Get("Cache-Control") != "private, max-age=3600" {
t.Errorf("%q: headers %v", query, rec.Header())
}
}
if n := atomic.LoadInt32(&e.pics.calls); n != 1 {
t.Errorf("the provider was asked %d times, want 1", n)
}
first := e.get("GET", e.path(e.alpha, ""), e.admin)
etag := first.Header().Get("ETag")
if etag == "" || etag[0] != '"' {
t.Fatalf("etag = %q", etag)
}
if rec := e.get("GET", e.path(e.alpha, ""), e.admin, "If-None-Match", etag); rec.Code != http.StatusNotModified || rec.Body.Len() != 0 {
t.Errorf("If-None-Match: %d", rec.Code)
}
if rec := e.get("HEAD", e.path(e.alpha, ""), e.admin); rec.Code != http.StatusOK || rec.Body.Len() != 0 {
t.Errorf("HEAD: %d, %d bytes", rec.Code, rec.Body.Len())
}
}
func TestArtistImageCreditSaysWhoseItIsAndUnderWhichLicence(t *testing.T) {
e := newArtistImageEnv(t)
rec := e.get("GET", e.path(e.alpha, "/credit"), e.admin)
if rec.Code != http.StatusOK {
t.Fatalf("%d %s", rec.Code, rec.Body.String())
}
var got map[string]any
if err := json.Unmarshal(rec.Body.Bytes(), &got); err != nil {
t.Fatal(err)
}
sameKeys(t, "credit", got, "source", "author", "license", "license_url", "page_url")
if got["source"] != "Wikimedia Commons" || got["author"] != "Raph_PH" || got["license"] != "CC BY 4.0" ||
got["license_url"] != "https://creativecommons.org/licenses/by/4.0" || got["page_url"] != "https://commons.wikimedia.org/wiki/File:Band.jpg" {
t.Errorf("credit = %v", got)
}
// The credit comes from the record, with no new request.
if n := atomic.LoadInt32(&e.pics.calls); n != 1 {
t.Errorf("the provider was asked %d times, want 1", n)
}
}
func TestArtistImageErrors(t *testing.T) {
for name, tt := range map[string]struct {
err error
code int
want string
}{
"no free picture": {images.ErrNoImage, 404, "image_not_found"},
"provider off": {images.ErrProviderDisabled, 404, "provider_disabled"},
"no id": {images.ErrNoID, 404, "no_musicbrainz_id"},
"unknown artist": {images.ErrArtistNotFound, 404, "artist_not_found"},
"provider down": {images.ErrSourceUnavailable, 502, "provider_unavailable"},
} {
e := newArtistImageEnv(t)
e.pics.err = tt.err
for _, path := range []string{e.path(e.alpha, ""), e.path(e.alpha, "/credit")} {
rec := e.get("GET", path, e.admin)
var body struct {
Error struct{ Code string } `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != tt.code || body.Error.Code != tt.want {
t.Errorf("%s %s: %d %s, want %d %s", name, path, rec.Code, rec.Body.String(), tt.code, tt.want)
}
}
}
}
func TestArtistImageRefusesWhatItCannotUnderstand(t *testing.T) {
e := newArtistImageEnv(t)
for name, path := range map[string]string{
"a bad id": "/artists/abc/image",
"id zero": "/artists/0/image",
"a bad size": e.path(e.alpha, "?size=huge"),
"size twice": e.path(e.alpha, "?size=small&size=large"),
"credit bad id": "/artists/abc/image/credit",
} {
rec := e.get("GET", path, e.admin)
var body struct {
Error struct{ Code string } `json:"error"`
}
_ = json.Unmarshal(rec.Body.Bytes(), &body)
if rec.Code != http.StatusBadRequest || body.Error.Code != "invalid_input" {
t.Errorf("%s: %d %s", name, rec.Code, rec.Body.String())
}
}
if n := atomic.LoadInt32(&e.pics.calls); n != 0 {
t.Errorf("the provider was asked %d times for requests that were refused", n)
}
}

View file

@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"image"
"image/color"
"image/png"
"io"
"mime"
@ -25,6 +26,7 @@ import (
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/images"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/lastfm"
@ -132,6 +134,11 @@ func newContract(t *testing.T) *contract {
d.Libraries = libs
d.Catalog = catalog.NewService(storage.NewCatalogRepo(db))
d.Streams = streaming.NewService(repo)
imageRepo := storage.NewImageRepo(db)
pictures := images.NewService(filepath.Join(t.TempDir(), "data"), imageRepo, imageRepo)
pictures.SetArtistSource(&fakeArtistPictures{found: images.Found{Data: testJPEG(t, 800, 600, color.White),
Credit: images.Credit{Author: "Someone", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0", PageURL: "https://commons.wikimedia.org/wiki/File:X.jpg"}}})
d.Images, d.ArtistImages = pictures, pictures
})
router, err := legacy.NewRouter(loadSpec(t))
@ -419,8 +426,8 @@ func TestResponsesMatchTheSpec(t *testing.T) {
// ---- providers ----
listed := c.expect(c.call("GET", "/providers", token, nil), 200, "list providers")["providers"].([]any)
if len(listed) != 3 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false {
t.Fatalf("providers = %v, want three, all off", listed)
if len(listed) != 4 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false || listed[3].(map[string]any)["enabled"] != false {
t.Fatalf("providers = %v, want four, all off", listed)
}
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
@ -574,6 +581,33 @@ func TestResponsesMatchTheSpec(t *testing.T) {
t.Fatalf("cover HEAD: %d", rec.Code)
}
// ---- the picture of an artist ----
photo := "/artists/" + artistID + "/image"
if rec := c.call("GET", photo, token, nil); rec.Code != 200 || rec.Header().Get("Content-Type") != "image/jpeg" {
t.Fatalf("artist image: %d %s", rec.Code, rec.Header().Get("Content-Type"))
}
if rec := c.call("GET", photo+"?size=small", token, nil); rec.Code != 200 {
t.Fatalf("small artist image: %d", rec.Code)
}
if rec := c.call("GET", photo, token, nil, "Range", "bytes=0-9"); rec.Code != 206 {
t.Fatalf("artist image range: %d", rec.Code)
}
if rec := c.call("HEAD", photo, token, nil); rec.Code != 200 {
t.Fatalf("artist image HEAD: %d", rec.Code)
}
if rec := c.call("GET", photo+"?access_token="+token, "", nil); rec.Code != 200 {
t.Fatalf("artist image with the token in the query: %d", rec.Code)
}
if rec := c.call("GET", photo, token, nil, "If-None-Match", c.call("GET", photo, token, nil).Header().Get("ETag")); rec.Code != 304 {
t.Fatalf("artist image If-None-Match: %d", rec.Code)
}
if cr := c.expect(c.call("GET", photo+"/credit", token, nil), 200, "the credit of the picture"); cr["license"] != "CC BY 4.0" || cr["source"] != "Wikimedia Commons" {
t.Fatalf("credit = %v", cr)
}
c.expect(c.call("GET", photo+"?size=huge", token, nil), 400, "an artist image of a size that does not exist", "invalid_input")
c.expect(c.call("GET", "/artists/abc/image/credit", token, nil), 400, "the credit of a bad id", "invalid_input")
c.expect(c.call("GET", photo+"/credit", "", nil), 401, "the credit without a token", "unauthorized")
// ---- discover ----
c.expect(c.call("GET", "/discover", token, nil), 400, "discover with no seed and no permission", "seed_required")
found := c.expect(c.call("GET", "/discover?seed="+artistID+"&limit=5", token, nil), 200, "discover from a named seed")

View file

@ -190,7 +190,7 @@ func TestPurgingWikipediaTakesTheDescriptionsBack(t *testing.T) {
if p.Code != http.StatusOK {
t.Fatalf("purge: %d %v", p.Code, p.Body)
}
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "artists")
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "images", "artists")
if p.Body["descriptions"] != float64(1) {
t.Errorf("purge = %v, want one description", p.Body)
}

View file

@ -0,0 +1,87 @@
package httpapi
import (
"context"
"errors"
"net/http"
"github.com/gin-gonic/gin"
"codeberg.org/squidd/sonda/internal/images"
)
type artistImageHandlers struct{ svc artistImageService }
type creditJSON struct {
Source string `json:"source"`
Author string `json:"author"`
License string `json:"license"`
LicenseURL string `json:"license_url"`
PageURL string `json:"page_url"`
}
// writeArtistImageError answers for what can go wrong; it reports whether it did.
func writeArtistImageError(c *gin.Context, err error) bool {
switch {
case err == nil:
return false
case errors.Is(err, images.ErrArtistNotFound):
writeError(c, http.StatusNotFound, "artist_not_found", "no such artist")
case errors.Is(err, images.ErrNoImage):
writeError(c, http.StatusNotFound, "image_not_found", "there is no free picture of this artist")
case errors.Is(err, images.ErrProviderDisabled):
writeError(c, http.StatusNotFound, "provider_disabled", "the picture provider is switched off")
case errors.Is(err, images.ErrNoID):
writeError(c, http.StatusNotFound, "no_musicbrainz_id", "the artist has no MusicBrainz id, and Sonda does not look one up by name")
case errors.Is(err, images.ErrSourceUnavailable):
_ = c.Error(err) // keep the cause in the log
writeError(c, http.StatusBadGateway, "provider_unavailable", "the picture provider could not be reached, try again later")
case errors.Is(err, context.Canceled):
c.AbortWithStatus(statusClientClosed) // the client left; nobody is waiting
default:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
}
return true
}
// image serves the free picture of an artist: a JPEG made from the copy found on Wikimedia
// Commons, built on the first request and kept in the data folder.
func (h *artistImageHandlers) image(c *gin.Context) {
id, ok := pathID(c)
if !ok {
return
}
size, ok := sizeParam(c)
if !ok {
return
}
pic, err := h.svc.ArtistImage(c.Request.Context(), id, size)
if writeArtistImageError(c, err) {
return
}
defer pic.File.Close()
hdr := c.Writer.Header()
hdr.Set("Content-Type", "image/jpeg")
hdr.Set("ETag", `"`+pic.ETag+`"`)
hdr.Set("Cache-Control", "private, max-age=3600")
hdr.Set("X-Content-Type-Options", "nosniff")
http.ServeContent(c.Writer, c.Request, "image.jpg", pic.ModTime, pic.File)
}
// credit says whose the picture is and under which licence, to be shown next to it.
func (h *artistImageHandlers) credit(c *gin.Context) {
id, ok := pathID(c)
if !ok {
return
}
pic, err := h.svc.ArtistImage(c.Request.Context(), id, images.Small)
if writeArtistImageError(c, err) {
return
}
_ = pic.File.Close()
c.JSON(http.StatusOK, creditJSON{
Source: "Wikimedia Commons", Author: pic.Credit.Author, License: pic.Credit.License,
LicenseURL: pic.Credit.LicenseURL, PageURL: pic.Credit.PageURL,
})
}

View file

@ -97,5 +97,5 @@ func (h *providersHandlers) purge(c *gin.Context) {
writeProviderError(c, err)
return
}
c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "descriptions": res.Descriptions, "artists": res.Artists})
c.JSON(http.StatusOK, gin.H{"similar_artists": res.SimilarArtists, "tags": res.Tags, "descriptions": res.Descriptions, "images": res.Images, "artists": res.Artists})
}

View file

@ -46,7 +46,7 @@ func TestProvidersStartOffAndSayWhatTheySend(t *testing.T) {
t.Fatalf("list: %d %v", r.Code, r.Body)
}
list := r.Body["providers"].([]any)
if len(list) != 3 {
if len(list) != 4 {
t.Fatalf("providers = %v", list)
}
for _, it := range list {
@ -263,7 +263,7 @@ func TestPurgeRemovesWhatTheProviderGaveAndNothingOfMine(t *testing.T) {
if p.Code != http.StatusOK {
t.Fatalf("%d %v", p.Code, p.Body)
}
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "artists")
sameKeys(t, "purge", p.Body, "similar_artists", "tags", "descriptions", "images", "artists")
if p.Body["similar_artists"] != float64(2) || p.Body["tags"] != float64(0) || p.Body["artists"] != float64(1) {
t.Errorf("purge = %v, want 2 similar artists, no tags and the one artist Gamma", p.Body)
}

View file

@ -86,6 +86,10 @@ type feedbackService interface {
List(ctx context.Context, userID int64, kind feedback.Kind, p feedback.Page) (feedback.Listing, error)
}
type artistImageService interface {
ArtistImage(ctx context.Context, artistID int64, size images.Size) (*images.Cover, error)
}
type aboutService interface {
Description(ctx context.Context, artistID int64) (about.Result, error)
}
@ -121,23 +125,24 @@ type libraryService interface {
// Deps are the services the API needs.
type Deps struct {
Logger *zap.Logger
Auth authService
Tokens tokenParser
Listening listenService
Scan scanService
Libraries libraryService
Catalog catalogService
Providers providerService
Discovery discoveryService
Tags tagService
Taste tasteService
Discover discoverService
Feedback feedbackService
Images coverService
About aboutService
Settings userSettingsService
Streams streamService
Logger *zap.Logger
Auth authService
Tokens tokenParser
Listening listenService
Scan scanService
Libraries libraryService
Catalog catalogService
Providers providerService
Discovery discoveryService
Tags tagService
Taste tasteService
Discover discoverService
Feedback feedbackService
Images coverService
About aboutService
ArtistImages artistImageService
Settings userSettingsService
Streams streamService
}
// NewRouter builds the gin engine with all routes.
@ -192,6 +197,7 @@ func NewRouter(d Deps) *gin.Engine {
protected.GET("/artists/:id/tags", tg.tags)
ab := &aboutHandlers{svc: d.About}
protected.GET("/artists/:id/description", ab.description)
protected.GET("/artists/:id/image/credit", (&artistImageHandlers{svc: d.ArtistImages}).credit)
protected.GET("/albums", cat.listAlbums)
protected.GET("/albums/:id", cat.getAlbum)
protected.GET("/tracks", cat.listTracks)
@ -207,6 +213,9 @@ func NewRouter(d Deps) *gin.Engine {
cv := &coverHandlers{svc: d.Images}
audio.GET("/albums/:id/cover", cv.album)
audio.HEAD("/albums/:id/cover", cv.album)
ai := &artistImageHandlers{svc: d.ArtistImages}
audio.GET("/artists/:id/image", ai.image)
audio.HEAD("/artists/:id/image", ai.image)
// Admins only.
admin := protected.Group("", requireAdmin(d.Auth))

View file

@ -971,6 +971,144 @@ paths:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/artists/{id}/image:
parameters:
- {$ref: '#/components/parameters/Id'}
get:
tags: [Discovery]
operationId: getArtistImage
summary: A free picture of an artist
description: |
A JPEG made from a picture of the artist on Wikimedia Commons, only when its licence is
free: public domain, CC0, CC BY or CC BY-SA. Non-commercial, no-derivatives, fair use and
unknown licences are refused, because Sonda keeps a copy. **Where you show it, show its
credit** (`GET /artists/{id}/image/credit`): the author, the licence and the page of the file.
The way to it is a chain of identifiers, never a search by name: the artist's MusicBrainz
id, the Wikidata item MusicBrainz links to it, the picture Wikidata names for it, and Commons
for its licence and a copy. It needs the `commons` provider and the `musicbrainz` one, which
an admin switches on. It is built the first time it is asked for (that request can take a few
seconds), scaled to fit, saved in the data folder, and served from there afterwards: for 30
days, or a week when there is no free picture; an older copy is served when Commons cannot
be reached. What is sent: the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata,
the name of the file to Commons, and the address of a copy to Wikimedia, with your server's
IP address and a `User-Agent` with a contact.
An `<img>` element cannot send headers, so, like the stream and the covers, this accepts
the access token as the `access_token` query parameter. A header, when present, always wins.
security:
- bearerAuth: []
- streamQueryToken: []
parameters:
- name: size
in: query
description: "`small` (300 pixels at most on the longest side) or `large` (1000). Large when absent."
schema: {type: string, enum: [small, large], default: large}
- name: If-None-Match
in: header
required: false
schema: {type: string}
- name: Range
in: header
required: false
schema: {type: string}
responses:
'200':
description: The picture.
headers:
ETag: {$ref: '#/components/headers/ETag'}
Cache-Control: {$ref: '#/components/headers/CacheControl'}
Content-Length: {$ref: '#/components/headers/ContentLength'}
Last-Modified: {$ref: '#/components/headers/LastModified'}
content:
image/jpeg:
schema: {type: string, format: binary}
'206':
description: The requested byte range of the picture.
headers:
Content-Range:
schema: {type: string}
content:
image/jpeg:
schema: {type: string, format: binary}
'304': {description: Not modified since the `ETag` sent.}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: |
No such artist (`artist_not_found`), no free picture of it (`image_not_found`), the
provider is off (`provider_disabled`), or the artist has no MusicBrainz id
(`no_musicbrainz_id`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'502':
description: The picture provider could not be reached (`provider_unavailable`). Try again later.
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
head:
tags: [Discovery]
operationId: headArtistImage
summary: Headers of a picture
description: Same as `GET`, without the body.
security:
- bearerAuth: []
- streamQueryToken: []
parameters:
- name: size
in: query
schema: {type: string, enum: [small, large], default: large}
- name: If-None-Match
in: header
required: false
schema: {type: string}
responses:
'200':
description: Headers of the picture.
headers:
ETag: {$ref: '#/components/headers/ETag'}
Cache-Control: {$ref: '#/components/headers/CacheControl'}
Content-Length: {$ref: '#/components/headers/ContentLength'}
Last-Modified: {$ref: '#/components/headers/LastModified'}
'304': {description: Not modified since the `ETag` sent.}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404': {description: "No such artist, no free picture, the provider is off or the artist has no MusicBrainz id."}
'502': {description: "The picture provider could not be reached."}
/artists/{id}/image/credit:
parameters:
- {$ref: '#/components/parameters/Id'}
get:
tags: [Discovery]
operationId: getArtistImageCredit
summary: The credit of a picture of an artist
description: |
Whose the picture is and under which licence, to be shown next to it (the licences ask for
it). The author is plain text and may be empty; `license_url` may be empty when the licence
has no address; `page_url` is the page of the file on Commons, where everything is.
It answers like `GET /artists/{id}/image` (same errors), because the credit exists only with a picture.
security: [{bearerAuth: []}]
responses:
'200':
description: The credit.
content:
application/json:
schema: {$ref: '#/components/schemas/PictureCredit'}
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: No such artist (`artist_not_found`), no free picture (`image_not_found`), the provider is off (`provider_disabled`) or the artist has no MusicBrainz id (`no_musicbrainz_id`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'502':
description: The picture provider could not be reached (`provider_unavailable`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/albums:
get:
tags: [Catalog]
@ -1308,7 +1446,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
put:
tags: [Providers]
operationId: updateProvider
@ -1346,7 +1484,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
post:
tags: [Providers]
operationId: testProvider
@ -1381,7 +1519,7 @@ paths:
- name: name
in: path
required: true
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia]}
schema: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
delete:
tags: [Providers]
operationId: purgeProviderData
@ -1492,7 +1630,7 @@ components:
type: apiKey
in: query
name: access_token
description: The same access token, for stream URLs used by an `<audio>` element. Accepted only on `/tracks/{id}/stream` and `/albums/{id}/cover`.
description: The same access token, for stream URLs used by an `<audio>` element. Accepted only on `/tracks/{id}/stream`, `/albums/{id}/cover` and `/artists/{id}/image`.
parameters:
Id:
@ -1842,7 +1980,7 @@ components:
additionalProperties: false
required: [name, enabled, needs_api_key, has_api_key, contact, sends, notes, updated_at]
properties:
name: {type: string, enum: [lastfm, musicbrainz, wikipedia]}
name: {type: string, enum: [lastfm, musicbrainz, wikipedia, commons]}
enabled: {type: boolean, description: Whether Sonda may contact this provider.}
needs_api_key: {type: boolean}
has_api_key: {type: boolean, description: Whether a key is stored. The key itself is never returned.}
@ -1907,13 +2045,25 @@ components:
license: {type: string, description: "The licence of the text."}
license_url: {type: string, format: uri}
PictureCredit:
type: object
additionalProperties: false
required: [source, author, license, license_url, page_url]
properties:
source: {type: string, description: "Where the picture comes from: `Wikimedia Commons`."}
author: {type: string, description: "Plain text. May be empty."}
license: {type: string, description: "Like `CC BY-SA 4.0`."}
license_url: {type: string, description: "The licence, on creativecommons.org. May be empty."}
page_url: {type: string, format: uri, description: The page of the file on Commons.}
ProviderPurge:
type: object
additionalProperties: false
required: [similar_artists, tags, descriptions, artists]
required: [similar_artists, tags, descriptions, images, artists]
properties:
similar_artists: {type: integer, description: Similar-artist links removed.}
descriptions: {type: integer, description: Artist descriptions removed (only the data of Wikipedia has any).}
images: {type: integer, description: Pictures of artists removed (only the data of Commons has any).}
tags: {type: integer, description: Tags removed.}
artists: {type: integer, description: "Artists removed because only the removed data named them and nobody has files for them."}

162
internal/images/artist.go Normal file
View file

@ -0,0 +1,162 @@
package images
import (
"context"
"crypto/sha256"
"errors"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
)
// ArtistImage returns the picture of an artist in the given size. It is built from
// the free picture the source finds, the first time it is asked for, and kept in the
// data folder: 30 days, or a week when there is none. When the provider cannot be asked
// and an older picture is stored, the old one is served. Anything that is not an answer
// about the picture (the provider is off, the artist has no id, the network failed) is
// returned and not remembered, so the next request asks again.
func (s *Service) ArtistImage(ctx context.Context, artistID int64, size Size) (*Cover, error) {
if size != Small && size != Large {
return nil, fmt.Errorf("%w: %q", ErrInvalidSize, string(size))
}
if artistID <= 0 {
return nil, ErrArtistNotFound
}
if err := ctx.Err(); err != nil {
return nil, err
}
rec, ok, err := s.store.Get(ctx, kindArtist, artistID)
if err != nil {
return nil, fmt.Errorf("read picture record: %w", err)
}
if ok && rec.Status == StatusNone && s.now().Sub(rec.CheckedAt) < NoImageTTL {
return nil, ErrNoImage
}
if ok && rec.Status == StatusOK && s.now().Sub(rec.CheckedAt) < ArtistTTL {
if c, err := s.open(kindArtist, artistID, size, rec); err == nil {
return c, nil
}
}
done := s.flight.DoChan("artist/"+strconv.FormatInt(artistID, 10), func() (v any, err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("building a picture panicked: %v", r)
}
}()
return s.buildArtist(artistID)
})
var built Record
select {
case <-ctx.Done():
return nil, ctx.Err()
case r := <-done:
if r.Err != nil {
// An older picture is better than none when the provider cannot be asked now.
if (errors.Is(r.Err, ErrProviderDisabled) || errors.Is(r.Err, ErrSourceUnavailable)) && ok && rec.Status == StatusOK {
if c, err := s.open(kindArtist, artistID, size, rec); err == nil {
return c, nil
}
}
return nil, r.Err
}
built = r.Val.(Record)
}
if built.Status == StatusNone {
return nil, ErrNoImage
}
return s.open(kindArtist, artistID, size, built)
}
// buildArtist asks the source, stores the picture and records the outcome. If the
// record was cleared while it worked, what it found may be out of date: it starts again.
func (s *Service) buildArtist(artistID int64) (Record, error) {
if s.artists == nil {
return Record{}, ErrProviderDisabled
}
ctx, cancel := context.WithTimeout(context.Background(), buildTimeout)
defer cancel()
var rec Record
for attempt := 0; attempt < 3; attempt++ {
cur, _, err := s.store.Get(ctx, kindArtist, artistID)
if err != nil {
return Record{}, fmt.Errorf("read picture record: %w", err)
}
rec, err = s.searchArtist(ctx, artistID)
if err != nil {
return Record{}, err
}
rec.Gen = cur.Gen
saved, err := s.store.Save(ctx, kindArtist, artistID, rec)
if err != nil {
return Record{}, fmt.Errorf("save picture record: %w", err)
}
if saved {
return rec, nil
}
}
return rec, nil
}
func (s *Service) searchArtist(ctx context.Context, artistID int64) (Record, error) {
select {
case s.artistSl <- struct{}{}:
defer func() { <-s.artistSl }()
case <-ctx.Done():
return Record{}, ctx.Err()
}
none := Record{Status: StatusNone, CheckedAt: s.now()}
found, err := s.artists.ArtistImage(ctx, artistID)
switch {
case errors.Is(err, ErrNoImage):
return none, nil
case err != nil:
return Record{}, err
}
p, err := Process(found.Data)
if err != nil {
return none, nil // what the provider sent is not a picture: there is none to show
}
if err := s.write(kindArtist, artistID, p); err != nil {
return Record{}, err
}
// The credit is part of the identity: the same picture under another author or licence
// must not be taken for the same one by a client that kept the old credit.
h := sha256.New()
h.Write(p.Large)
h.Write([]byte("\x00" + found.Credit.Author + "\x00" + found.Credit.License + "\x00" + found.Credit.LicenseURL + "\x00" + found.Credit.PageURL))
return Record{
Status: StatusOK, Source: SourceCommons, Credit: found.Credit, ETag: fmt.Sprintf("%x", h.Sum(nil)[:8]),
Width: p.Width, Height: p.Height, CheckedAt: none.CheckedAt,
}, nil
}
// RemoveArtistFiles deletes the stored pictures of artists, and only those, and says how
// many files went. It is what taking the picture provider's data back needs, together with
// forgetting their records.
func (s *Service) RemoveArtistFiles() (int, error) {
dir := filepath.Join(s.dir, "images", kindArtist)
entries, err := os.ReadDir(dir)
if errors.Is(err, os.ErrNotExist) {
return 0, nil
}
if err != nil {
return 0, fmt.Errorf("list the artist pictures: %w", err)
}
n := 0
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".jpg") {
continue
}
if err := os.Remove(filepath.Join(dir, e.Name())); err != nil {
return n, fmt.Errorf("remove a picture: %w", err)
}
n++
}
// The folder goes too when nothing else is in it (a temporary file may be).
_ = os.Remove(dir)
s.tidied.Delete(dir)
return n, nil
}

View file

@ -0,0 +1,254 @@
package images
import (
"bytes"
"context"
"errors"
"image"
"image/color"
"io"
"os"
"path/filepath"
"sync"
"sync/atomic"
"testing"
"time"
)
type fakeArtists struct {
mu sync.Mutex
found Found
err error
calls int32
gate chan struct{}
}
func (f *fakeArtists) ArtistImage(_ context.Context, id int64) (Found, error) {
atomic.AddInt32(&f.calls, 1)
f.mu.Lock()
gate, found, err := f.gate, f.found, f.err
f.mu.Unlock()
if gate != nil {
<-gate
}
return found, err
}
func (f *fakeArtists) set(found Found, err error) {
f.mu.Lock()
defer f.mu.Unlock()
f.found, f.err = found, err
}
var credit = Credit{Author: "Raph_PH", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0", PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg"}
type artistRig struct {
*rig
src *fakeArtists
}
func newArtistRig(t *testing.T) *artistRig {
t.Helper()
r := &artistRig{rig: newRig(t), src: &fakeArtists{}}
r.src.found = Found{Data: jpegBytes(t, 1500, 1000, color.RGBA{30, 30, 200, 255}), Credit: credit}
r.svc.SetArtistSource(r.src)
return r
}
func (r *artistRig) get(t *testing.T, id int64, size Size) (*Cover, []byte) {
t.Helper()
c, err := r.svc.ArtistImage(context.Background(), id, size)
if err != nil {
t.Fatalf("ArtistImage: %v", err)
}
defer c.File.Close()
b, _ := io.ReadAll(c.File)
return c, b
}
func TestAnArtistPictureIsBuiltOnceStoredAndCredited(t *testing.T) {
r := newArtistRig(t)
c, big := r.get(t, 7, Large)
cfg, format, err := image.DecodeConfig(bytes.NewReader(big))
if err != nil || format != "jpeg" || cfg.Width != 1000 || cfg.Height != 666 {
t.Errorf("large = %v %s %v, want a jpeg 1000 wide", cfg, format, err)
}
if c.Credit != credit {
t.Errorf("credit = %+v, want %+v", c.Credit, credit)
}
if _, small := r.get(t, 7, Small); len(small) == 0 {
t.Error("no small picture")
}
rec := r.store.recs["artist/7"]
if rec.Status != StatusOK || rec.Source != SourceCommons || rec.Credit != credit || rec.ETag == "" {
t.Errorf("record = %+v", rec)
}
if n := atomic.LoadInt32(&r.src.calls); n != 1 {
t.Errorf("the provider was asked %d times, want 1: the picture is stored", n)
}
for _, size := range []string{"7-large.jpg", "7-small.jpg"} {
info, err := os.Stat(filepath.Join(r.data, "images", "artist", size))
if err != nil || info.Mode().Perm()&0o077 != 0 {
t.Errorf("%s: %v %v, want a private file under images/artist", size, info, err)
}
}
}
func TestAnArtistWithNoFreePictureIsRememberedForAWeek(t *testing.T) {
r := newArtistRig(t)
r.src.set(Found{}, ErrNoImage)
if _, err := r.svc.ArtistImage(context.Background(), 7, Large); !errors.Is(err, ErrNoImage) {
t.Fatalf("err = %v", err)
}
if rec := r.store.recs["artist/7"]; rec.Status != StatusNone {
t.Fatalf("record = %+v", rec)
}
for i := 0; i < 5; i++ {
if _, err := r.svc.ArtistImage(context.Background(), 7, Large); !errors.Is(err, ErrNoImage) {
t.Fatalf("err = %v", err)
}
}
if n := atomic.LoadInt32(&r.src.calls); n != 1 {
t.Errorf("asked %d times within the week, want 1", n)
}
r.now = r.now.Add(NoImageTTL + time.Hour)
r.src.set(Found{Data: jpegBytes(t, 300, 300, color.White), Credit: credit}, nil)
if _, b := r.get(t, 7, Large); len(b) == 0 {
t.Error("a picture that appeared was not found after the week")
}
}
func TestSomethingThatIsNotAPictureCountsAsNoPicture(t *testing.T) {
r := newArtistRig(t)
r.src.set(Found{Data: []byte("<html>not a picture</html>"), Credit: credit}, nil)
if _, err := r.svc.ArtistImage(context.Background(), 7, Large); !errors.Is(err, ErrNoImage) {
t.Errorf("err = %v, want ErrNoImage", err)
}
if rec := r.store.recs["artist/7"]; rec.Status != StatusNone {
t.Errorf("record = %+v", rec)
}
}
func TestWhatIsNotAnAnswerAboutThePictureIsNotRemembered(t *testing.T) {
for _, want := range []error{ErrProviderDisabled, ErrNoID, ErrSourceUnavailable, ErrArtistNotFound} {
r := newArtistRig(t)
r.src.set(Found{}, want)
if _, err := r.svc.ArtistImage(context.Background(), 7, Large); !errors.Is(err, want) {
t.Errorf("err = %v, want %v", err, want)
}
if len(r.store.recs) != 0 {
t.Errorf("%v: records = %v, want none: the next request must ask again", want, r.store.recs)
}
}
// No source at all is a provider that is off.
r := newRig(t)
if _, err := r.svc.ArtistImage(context.Background(), 7, Large); !errors.Is(err, ErrProviderDisabled) {
t.Errorf("no source: err = %v", err)
}
}
func TestAnOldPictureIsKeptWhenTheProviderCannotBeAsked(t *testing.T) {
for _, why := range []error{ErrProviderDisabled, ErrSourceUnavailable} {
r := newArtistRig(t)
_, first := r.get(t, 7, Large)
r.now = r.now.Add(ArtistTTL + time.Hour)
r.src.set(Found{}, why)
c, again := r.get(t, 7, Large)
if !bytes.Equal(first, again) || c.Credit != credit {
t.Errorf("%v: the stored picture was not served, or lost its credit", why)
}
}
// A picture that was changed is fetched again after a month.
r := newArtistRig(t)
_, first := r.get(t, 7, Large)
r.now = r.now.Add(ArtistTTL + time.Hour)
r.src.set(Found{Data: jpegBytes(t, 800, 800, color.RGBA{200, 0, 0, 255}), Credit: Credit{Author: "Someone", License: "CC0", PageURL: "https://commons.wikimedia.org/wiki/File:New.jpg"}}, nil)
c, second := r.get(t, 7, Large)
if bytes.Equal(first, second) || c.Credit.Author != "Someone" {
t.Error("the new picture and its credit were not taken after a month")
}
}
func TestACoverAndAnArtistPictureWithTheSameIdDoNotMix(t *testing.T) {
r := newArtistRig(t)
r.write(t, "Album One/cover.jpg", jpegBytes(t, 400, 400, color.RGBA{200, 0, 0, 255}))
_, artist := r.get(t, 1, Large)
album := r.rig.read(t, 1, Large)
if bytes.Equal(artist, album) {
t.Error("the picture of artist 1 and the cover of album 1 are the same file")
}
if len(r.store.recs) != 2 {
t.Errorf("records = %v, want one for the artist and one for the album", r.store.recs)
}
}
func TestManyAsksForAnArtistPictureAskTheProviderOnce(t *testing.T) {
r := newArtistRig(t)
r.src.gate = make(chan struct{})
var wg sync.WaitGroup
for i := 0; i < 15; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if c, err := r.svc.ArtistImage(context.Background(), 7, Large); err != nil {
t.Errorf("err = %v", err)
} else {
c.File.Close()
}
}()
}
time.Sleep(80 * time.Millisecond)
close(r.src.gate)
wg.Wait()
if n := atomic.LoadInt32(&r.src.calls); n != 1 {
t.Errorf("asked %d times, want 1", n)
}
}
func TestAChangeDuringTheReadOfAnArtistPictureIsNotUndoneByIt(t *testing.T) {
r := newArtistRig(t)
r.src.gate = make(chan struct{})
done := make(chan error, 1)
go func() { _, err := r.svc.ArtistImage(context.Background(), 7, Large); done <- err }()
time.Sleep(50 * time.Millisecond)
r.store.clear("artist", 7) // for example the purge of the provider
close(r.src.gate)
<-done
// What the read that started before the change found was not kept as the answer.
if rec := r.store.recs["artist/7"]; rec.Status == StatusOK && rec.Gen == 0 {
t.Errorf("record = %+v: an out of date build was saved over the change", rec)
}
}
func TestPurgingTheArtistPicturesRemovesTheFilesAndOnlyThose(t *testing.T) {
r := newArtistRig(t)
r.get(t, 7, Large)
r.get(t, 8, Large)
r.write(t, "Album One/cover.jpg", jpegBytes(t, 400, 400, color.White))
r.rig.read(t, 1, Large)
n, err := r.svc.RemoveArtistFiles()
if err != nil || n != 4 {
t.Fatalf("removed %d files err = %v, want 4 (two sizes of two pictures)", n, err)
}
if _, err := os.Stat(filepath.Join(r.data, "images", "artist")); !os.IsNotExist(err) {
t.Error("the artist folder is still there")
}
if _, err := os.Stat(filepath.Join(r.data, "images", "album", "1-large.jpg")); err != nil {
t.Errorf("a cover was removed with them: %v", err)
}
// Nothing to remove is not an error.
if n, err := r.svc.RemoveArtistFiles(); err != nil || n != 0 {
t.Errorf("again: %d %v", n, err)
}
}
func TestACreditIsPartOfTheETag(t *testing.T) {
r := newArtistRig(t)
c1, _ := r.get(t, 7, Large)
r.store.clear("artist", 7)
r.src.set(Found{Data: r.src.found.Data, Credit: Credit{Author: "Other", License: "CC0", PageURL: "https://commons.wikimedia.org/wiki/File:Other.jpg"}}, nil)
c2, _ := r.get(t, 7, Large)
if c1.ETag == c2.ETag {
t.Error("the picture is the same but its credit changed: the etag must change too, or a client keeps the old credit")
}
}

View file

@ -0,0 +1,136 @@
package images
import (
"context"
"errors"
"fmt"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/commons"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
)
// ArtistLookup finds an artist by id.
type ArtistLookup interface {
Artist(ctx context.Context, id int64) (discovery.Artist, error)
}
// Switches says whether a provider may be used right now.
type Switches interface {
Active(ctx context.Context, name string) (providers.Settings, bool, error)
}
// WikidataLink gives the Wikidata item that MusicBrainz links to an artist.
type WikidataLink interface {
WikidataID(ctx context.Context, musicbrainzID string) (string, error)
}
// CommonsAPI is what the chain needs from Wikidata and Commons: *commons.Client.
type CommonsAPI interface {
FileName(ctx context.Context, item string) (string, error)
Info(ctx context.Context, name string) (commons.File, error)
Download(ctx context.Context, pictureURL string) ([]byte, error)
}
// CommonsSource finds the picture of an artist by following identifiers, never a name:
// the artist's MusicBrainz id, the Wikidata item MusicBrainz links to it, the Commons file
// Wikidata names for it, and, if its licence is free, a copy of it. It needs the commons
// provider and the musicbrainz one to be on, since the artist's id goes to both.
type CommonsSource struct {
Artists ArtistLookup
Settings Switches
MusicBrainz WikidataLink
Commons CommonsAPI
}
// active says whether both providers are on.
func (c *CommonsSource) active(ctx context.Context) (bool, error) {
for _, name := range []string{providers.Commons, providers.MusicBrainz} {
_, on, err := c.Settings.Active(ctx, name)
if err != nil {
return false, err
}
if !on {
return false, nil
}
}
return true, nil
}
// failed turns what went wrong into ErrSourceUnavailable, keeping the cause. A request
// that was cancelled is not a failure of the provider and is returned as it is.
func failed(err error) error {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return err
}
return fmt.Errorf("%w: %w", ErrSourceUnavailable, err)
}
// ArtistImage implements ArtistSource.
func (c *CommonsSource) ArtistImage(ctx context.Context, artistID int64) (Found, error) {
artist, err := c.Artists.Artist(ctx, artistID)
if errors.Is(err, discovery.ErrNotFound) {
return Found{}, ErrArtistNotFound
}
if err != nil {
return Found{}, err
}
if on, err := c.active(ctx); err != nil {
return Found{}, err
} else if !on {
return Found{}, ErrProviderDisabled
}
if artist.MBID == "" { // Sonda never looks an id up by name
return Found{}, ErrNoID
}
item, err := c.MusicBrainz.WikidataID(ctx, artist.MBID)
switch {
case errors.Is(err, musicbrainz.ErrNotFound), errors.Is(err, musicbrainz.ErrBadRequest):
return Found{}, ErrNoImage
case err != nil:
return Found{}, failed(err)
case item == "":
return Found{}, ErrNoImage
}
name, err := c.Commons.FileName(ctx, item)
switch {
case errors.Is(err, commons.ErrNotFound), errors.Is(err, commons.ErrBadRequest):
return Found{}, ErrNoImage
case err != nil:
return Found{}, failed(err)
case name == "":
return Found{}, ErrNoImage
}
file, err := c.Commons.Info(ctx, name)
switch {
case errors.Is(err, commons.ErrNotFound), errors.Is(err, commons.ErrNotFree),
errors.Is(err, commons.ErrUnusable), errors.Is(err, commons.ErrBadRequest):
return Found{}, ErrNoImage
case err != nil:
return Found{}, failed(err)
}
// The admin may have switched the provider off while the chain was being followed:
// do not download, and do not hand over, what they no longer allow.
if on, err := c.active(ctx); err != nil {
return Found{}, err
} else if !on {
return Found{}, ErrProviderDisabled
}
data, err := c.Commons.Download(ctx, file.ThumbURL)
switch {
case errors.Is(err, commons.ErrBadRequest), errors.Is(err, commons.ErrNotFound):
return Found{}, ErrNoImage
case err != nil:
return Found{}, failed(err)
}
if on, err := c.active(ctx); err != nil {
return Found{}, err
} else if !on {
return Found{}, ErrProviderDisabled
}
return Found{Data: data, Credit: Credit{
Author: file.Author, License: file.License, LicenseURL: file.LicenseURL, PageURL: file.PageURL,
}}, nil
}

View file

@ -0,0 +1,203 @@
package images
import (
"context"
"errors"
"sync"
"testing"
"codeberg.org/squidd/sonda/internal/discovery"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/commons"
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
)
const mbid = "11111111-2222-4333-8444-555555555555"
type artistsFake struct{ artists map[int64]discovery.Artist }
func (f artistsFake) Artist(_ context.Context, id int64) (discovery.Artist, error) {
a, ok := f.artists[id]
if !ok {
return discovery.Artist{}, discovery.ErrNotFound
}
return a, nil
}
type switchesFake struct {
mu sync.Mutex
on map[string]bool
}
func (f *switchesFake) Active(_ context.Context, name string) (providers.Settings, bool, error) {
f.mu.Lock()
defer f.mu.Unlock()
return providers.Settings{Provider: name}, f.on[name], nil
}
func (f *switchesFake) set(name string, on bool) {
f.mu.Lock()
defer f.mu.Unlock()
f.on[name] = on
}
type mbFake struct {
item string
err error
calls int
got string
hook func()
}
func (f *mbFake) WikidataID(_ context.Context, id string) (string, error) {
f.calls++
f.got = id
if f.hook != nil {
f.hook()
}
return f.item, f.err
}
type commonsFake struct {
name string
nameErr, infoErr, loadErr error
file commons.File
data []byte
nameCalls, infoCalls, load int
loaded string
}
func (f *commonsFake) FileName(context.Context, string) (string, error) {
f.nameCalls++
return f.name, f.nameErr
}
func (f *commonsFake) Info(context.Context, string) (commons.File, error) {
f.infoCalls++
return f.file, f.infoErr
}
func (f *commonsFake) Download(_ context.Context, url string) ([]byte, error) {
f.load++
f.loaded = url
return f.data, f.loadErr
}
type chain struct {
src *CommonsSource
sw *switchesFake
mb *mbFake
cm *commonsFake
}
func newChain() *chain {
c := &chain{
sw: &switchesFake{on: map[string]bool{providers.Commons: true, providers.MusicBrainz: true}},
mb: &mbFake{item: "Q44190"},
cm: &commonsFake{
name: "Band.jpg",
file: commons.File{Name: "Band.jpg", PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg", ThumbURL: "https://thumb.wikimedia.org/x/960px-Band.jpg",
Author: "Raph", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0"},
data: []byte("picture bytes"),
},
}
c.src = &CommonsSource{
Artists: artistsFake{map[int64]discovery.Artist{1: {ID: 1, Name: "With Id", MBID: mbid}, 2: {ID: 2, Name: "No Id"}}},
Settings: c.sw,
MusicBrainz: c.mb,
Commons: c.cm,
}
return c
}
func TestTheChainGivesTheFreePictureWithItsCredit(t *testing.T) {
c := newChain()
got, err := c.src.ArtistImage(context.Background(), 1)
if err != nil {
t.Fatal(err)
}
if string(got.Data) != "picture bytes" || got.Credit != (Credit{Author: "Raph", License: "CC BY 4.0",
LicenseURL: "https://creativecommons.org/licenses/by/4.0", PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg"}) {
t.Errorf("found = %+v", got)
}
if c.mb.got != mbid || c.cm.loaded != "https://thumb.wikimedia.org/x/960px-Band.jpg" {
t.Errorf("asked MusicBrainz about %q and downloaded %q", c.mb.got, c.cm.loaded)
}
}
func TestNothingIsAskedWhenAnythingNeededIsOffOrThereIsNoId(t *testing.T) {
for name, tt := range map[string]struct {
prepare func(*chain)
id int64
want error
}{
"commons off": {func(c *chain) { c.sw.set(providers.Commons, false) }, 1, ErrProviderDisabled},
"musicbrainz off": {func(c *chain) { c.sw.set(providers.MusicBrainz, false) }, 1, ErrProviderDisabled},
"no musicbrainz id": {func(*chain) {}, 2, ErrNoID},
"unknown artist": {func(*chain) {}, 99, ErrArtistNotFound},
} {
c := newChain()
tt.prepare(c)
if _, err := c.src.ArtistImage(context.Background(), tt.id); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
if c.mb.calls+c.cm.nameCalls+c.cm.infoCalls+c.cm.load != 0 {
t.Errorf("%s: a provider was asked", name)
}
}
}
func TestWhereThereIsNoPictureTheAnswerIsNoImage(t *testing.T) {
for name, tt := range map[string]func(*chain){
"no wikidata link": func(c *chain) { c.mb.item = "" },
"musicbrainz does not know": func(c *chain) { c.mb.err = musicbrainz.ErrNotFound },
"no image on wikidata": func(c *chain) { c.cm.name = "" },
"wikidata does not know": func(c *chain) { c.cm.nameErr = commons.ErrNotFound },
"the file is gone": func(c *chain) { c.cm.infoErr = commons.ErrNotFound },
"not a free licence": func(c *chain) { c.cm.infoErr = commons.ErrNotFree },
"not a usable picture": func(c *chain) { c.cm.infoErr = commons.ErrUnusable },
"an address that is refused": func(c *chain) { c.cm.loadErr = commons.ErrBadRequest },
} {
c := newChain()
tt(c)
if _, err := c.src.ArtistImage(context.Background(), 1); !errors.Is(err, ErrNoImage) {
t.Errorf("%s: err = %v, want ErrNoImage", name, err)
}
}
}
func TestAFailureIsNotTakenForNoPicture(t *testing.T) {
boom := errors.New("boom")
for name, tt := range map[string]func(*chain){
"musicbrainz fails": func(c *chain) { c.mb.err = boom },
"wikidata fails": func(c *chain) { c.cm.nameErr = commons.ErrUnavailable },
"commons fails": func(c *chain) { c.cm.infoErr = commons.ErrRateLimited },
"the download fails": func(c *chain) { c.cm.loadErr = commons.ErrUnavailable },
} {
c := newChain()
tt(c)
_, err := c.src.ArtistImage(context.Background(), 1)
if !errors.Is(err, ErrSourceUnavailable) || errors.Is(err, ErrNoImage) {
t.Errorf("%s: err = %v, want ErrSourceUnavailable", name, err)
}
}
}
func TestSwitchedOffWhileItWasWorkingNothingIsReturned(t *testing.T) {
c := newChain()
c.mb.hook = func() { c.sw.set(providers.Commons, false) } // the admin took the data back
if _, err := c.src.ArtistImage(context.Background(), 1); !errors.Is(err, ErrProviderDisabled) {
t.Errorf("err = %v, want ErrProviderDisabled: a picture read before the switch must not be kept", err)
}
if c.cm.load != 0 {
t.Error("a picture was downloaded after the provider was switched off")
}
}
func TestACancelledRequestIsNotAFailureOfTheProvider(t *testing.T) {
c := newChain()
c.mb.err = context.Canceled
if _, err := c.src.ArtistImage(context.Background(), 1); !errors.Is(err, context.Canceled) {
t.Errorf("err = %v, want the cancellation itself", err)
}
}

View file

@ -23,7 +23,10 @@ import (
)
// Kinds of owner a picture belongs to.
const kindAlbum = "album"
const (
kindAlbum = "album"
kindArtist = "artist"
)
// What is stored about a picture, and where it came from.
const (
@ -33,6 +36,7 @@ const (
SourceEmbedded = "embedded" // the picture inside an audio file
SourceFolder = "folder" // a cover.jpg, folder.png ... next to the audio
SourceCommons = "commons" // a free picture of an artist from Wikimedia Commons
)
// Tuning.
@ -49,6 +53,10 @@ const (
CoverTTL = 7 * 24 * time.Hour
// tempMaxAge is how old a leftover temporary file must be to be removed.
tempMaxAge = time.Hour
// ArtistTTL is how long a picture of an artist is believed; NoImageTTL how long "there is
// no free picture". Pictures of artists change rarely and every check costs three requests.
ArtistTTL = 30 * 24 * time.Hour
NoImageTTL = 7 * 24 * time.Hour
// NoCoverTTL is how long "no cover" is believed. Adding a picture to a folder
// changes no audio file, so the answer has to be checked again now and then.
NoCoverTTL = 24 * time.Hour
@ -61,6 +69,13 @@ var (
ErrAlbumNotFound = errors.New("album not found")
ErrNoCover = errors.New("this album has no cover")
ErrInvalidSize = errors.New("invalid size")
// Answers of an ArtistSource.
ErrNoImage = errors.New("there is no free picture of this artist")
ErrProviderDisabled = errors.New("the provider is switched off")
ErrNoID = errors.New("the artist has no MusicBrainz id")
ErrArtistNotFound = errors.New("artist not found")
ErrSourceUnavailable = errors.New("the provider could not be reached")
)
// Size is one of the two stored sizes.
@ -85,13 +100,37 @@ type Source interface {
AlbumFiles(ctx context.Context, albumID int64) ([]File, error)
}
// Credit says whose picture it is and under which licence, as it has to be shown next
// to it. Pictures taken from the user's own files have none.
type Credit struct {
Author string // plain text; may be empty
License string // like "CC BY-SA 4.0"
LicenseURL string // may be empty
PageURL string // the page of the file, where the details are
}
// Found is a free picture of an artist, as the source got it, with its credit.
type Found struct {
Data []byte
Credit Credit
}
// ArtistSource finds a free picture of an artist. It returns ErrNoImage when there is
// none, ErrProviderDisabled when the provider is off, ErrNoID when the artist has no
// MusicBrainz id, ErrArtistNotFound for an unknown artist and ErrSourceUnavailable (or
// any other error) when it could not get an answer.
type ArtistSource interface {
ArtistImage(ctx context.Context, artistID int64) (Found, error)
}
// Record is what is known about the picture of one owner.
type Record struct {
Status string // StatusOK or StatusNone
Source string // SourceEmbedded or SourceFolder; empty for StatusNone
Source string // SourceEmbedded, SourceFolder or SourceCommons; empty for StatusNone
Credit Credit // empty for what comes from the user's own files
ETag string // identifies the stored picture
// Gen counts the changes of the album's files. A build saves its result only if
// the count is still the one it started from, so that it cannot bring back an
// Gen counts the changes of the owner's files or data. A build saves its result only
// if the count is still the one it started from, so that it cannot bring back an
// answer that a change made out of date.
Gen int64
Width int // of the original
@ -112,6 +151,7 @@ type Cover struct {
File *os.File
ModTime time.Time
ETag string
Credit Credit // empty for a cover from the user's own files
}
// Service builds covers from the user's own files, on the first request, and keeps
@ -123,16 +163,22 @@ type Service struct {
embedded func(path string) ([]byte, error)
now func() time.Time
flight singleflight.Group
slots chan struct{} // one per build allowed at the same time
tidy sync.Once
slots chan struct{} // one per cover build allowed at the same time
artistSl chan struct{} // the same for artist pictures, which wait for the network
artists ArtistSource
tidied sync.Map // the folders whose old temporary files were already removed
}
// NewService builds a Service that keeps its files under dir.
func NewService(dir string, store Store, src Source) *Service {
return &Service{dir: dir, store: store, src: src, embedded: taglib.ReadImage, now: time.Now,
slots: make(chan struct{}, MaxParallelBuilds)}
slots: make(chan struct{}, MaxParallelBuilds), artistSl: make(chan struct{}, MaxParallelBuilds)}
}
// SetArtistSource tells where pictures of artists come from. Without one, they are
// reported as ErrProviderDisabled.
func (s *Service) SetArtistSource(src ArtistSource) { s.artists = src }
// AlbumCover returns the cover of an album in the given size, building it first when
// it was never built or its record was cleared (a changed file does that), and
// searching again for an album that had none after NoCoverTTL.
@ -154,7 +200,7 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co
return nil, ErrNoCover
}
if ok && rec.Status == StatusOK && s.now().Sub(rec.CheckedAt) < CoverTTL {
if c, err := s.open(albumID, size, rec); err == nil {
if c, err := s.open(kindAlbum, albumID, size, rec); err == nil {
return c, nil
} // the stored file is gone: build it again
}
@ -182,15 +228,15 @@ func (s *Service) AlbumCover(ctx context.Context, albumID int64, size Size) (*Co
if built.Status == StatusNone {
return nil, ErrNoCover
}
return s.open(albumID, size, built)
return s.open(kindAlbum, albumID, size, built)
}
func (s *Service) filePath(albumID int64, size Size) string {
return filepath.Join(s.dir, "images", kindAlbum, strconv.FormatInt(albumID, 10)+"-"+string(size)+".jpg")
func (s *Service) filePath(kind string, id int64, size Size) string {
return filepath.Join(s.dir, "images", kind, strconv.FormatInt(id, 10)+"-"+string(size)+".jpg")
}
func (s *Service) open(albumID int64, size Size, rec Record) (*Cover, error) {
f, err := os.Open(s.filePath(albumID, size))
func (s *Service) open(kind string, id int64, size Size, rec Record) (*Cover, error) {
f, err := os.Open(s.filePath(kind, id, size))
if err != nil {
return nil, err
}
@ -203,7 +249,7 @@ func (s *Service) open(albumID int64, size Size, rec Record) (*Cover, error) {
if size == Small {
etag += "s"
}
return &Cover{File: f, ModTime: info.ModTime(), ETag: etag}, nil
return &Cover{File: f, ModTime: info.ModTime(), ETag: etag, Credit: rec.Credit}, nil
}
// build looks for a picture, stores it and records the outcome. If a file of the
@ -247,7 +293,7 @@ func (s *Service) search(ctx context.Context, albumID int64, files []File) (Reco
}
rec := Record{Status: StatusNone, CheckedAt: s.now()}
if p, source, ok := s.find(files); ok {
if err := s.write(albumID, p); err != nil {
if err := s.write(kindAlbum, albumID, p); err != nil {
return Record{}, err
}
sum := sha256.Sum256(p.Large)
@ -398,12 +444,14 @@ func readFolderPicture(rootDir, dir, name string) ([]byte, bool) {
// write stores both sizes. Each goes to a temporary file first and is renamed, so a
// reader never sees half a picture.
func (s *Service) write(albumID int64, p Processed) error {
dir := filepath.Dir(s.filePath(albumID, Large))
func (s *Service) write(kind string, id int64, p Processed) error {
dir := filepath.Dir(s.filePath(kind, id, Large))
if err := os.MkdirAll(dir, 0o700); err != nil {
return fmt.Errorf("create the data folder: %w", err)
}
s.tidy.Do(func() { removeOldTemps(dir, time.Now()) }) // file times are the disk's, not the clock of the service
if _, done := s.tidied.LoadOrStore(dir, true); !done {
removeOldTemps(dir, time.Now()) // file times are the disk's, not the clock of the service
}
for size, data := range map[Size][]byte{Large: p.Large, Small: p.Small} {
tmp, err := os.CreateTemp(dir, "tmp-*")
if err != nil {
@ -416,7 +464,7 @@ func (s *Service) write(albumID int64, p Processed) error {
_ = os.Remove(tmp.Name())
return fmt.Errorf("write a picture file: %w", errors.Join(werr, serr, cerr))
}
if err := os.Rename(tmp.Name(), s.filePath(albumID, size)); err != nil {
if err := os.Rename(tmp.Name(), s.filePath(kind, id, size)); err != nil {
_ = os.Remove(tmp.Name())
return fmt.Errorf("store a picture file: %w", err)
}

View file

@ -0,0 +1,354 @@
// Package commons is a small read-only client for the picture of an artist on
// Wikimedia Commons, reached through Wikidata.
//
// Like the Wikipedia client, it follows an identifier and never searches by name: the
// Wikidata item that MusicBrainz links to an artist names a Commons file (property
// P18), and Commons says under which licence the file is. Only a free licence is
// accepted (public domain, CC0, CC BY, CC BY-SA), because Sonda keeps a copy of the
// picture on the user's server: non-commercial, no-derivatives, fair use and unknown
// licences are refused. What leaves the server is the item, the name of the file and the
// contact in the User-Agent, which Wikimedia requires (it answers 403 without one).
//
// Errors never repeat the provider's own text.
package commons
import (
"context"
"encoding/json"
"errors"
"fmt"
"html"
"net/url"
"regexp"
"strings"
"unicode"
"unicode/utf8"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
// Addresses in production.
const (
WikidataURL = "https://www.wikidata.org/w/api.php"
CommonsURL = "https://commons.wikimedia.org/w/api.php"
)
// ThumbWidth is the width of the copy asked from Commons. It is one of the sizes
// Wikimedia serves: another is rounded up to the next one.
const ThumbWidth = 960
const (
maxNameLen = 300
maxAuthorLen = 300
probeItem = "Q44190" // a public item (Radiohead), to check that the service answers
)
// Errors a call can return, to be matched with errors.Is.
var (
ErrBadRequest = errors.New("commons: not a usable item, file name or address")
ErrNotFound = errors.New("commons: no such item or file")
ErrNotFree = errors.New("commons: the picture is not under a free licence")
ErrUnusable = errors.New("commons: the picture cannot be used")
ErrRateLimited = errors.New("commons: too many requests")
ErrUnavailable = errors.New("commons: service unavailable")
ErrBadResponse = errors.New("commons: unreadable answer")
)
var itemRe = regexp.MustCompile(`^Q[1-9][0-9]{0,15}$`)
// Getter performs a GET. *outbound.Client is the one used in production.
type Getter interface {
Get(ctx context.Context, rawURL string) (*outbound.Response, error)
}
// Client talks to Wikidata and Commons. It is safe for concurrent use.
type Client struct {
http Getter
wikidata string
commons string
}
// New builds a Client that sends its requests through g, to wikidata and commons
// (WikidataURL and CommonsURL in production).
func New(g Getter, wikidata, commons string) *Client {
return &Client{http: g, wikidata: wikidata, commons: commons}
}
// File is a Commons picture and what has to be shown with it.
type File struct {
Name string // the file name, without the "File:"
PageURL string // the page of the file on Commons, where author and licence are
ThumbURL string // a copy 960 pixels wide, on a Wikimedia picture host
Mime string
Width int // of the original
Height int
Author string // plain text; may be empty
License string // like "CC BY-SA 4.0"
LicenseURL string // on creativecommons.org; may be empty
}
// FileName returns the name of the Commons file that Wikidata gives as the image of an
// item, or "" when there is none. A preferred statement wins, then the first normal
// one; deprecated ones and values that are not a file name are skipped.
func (c *Client) FileName(ctx context.Context, item string) (string, error) {
if !itemRe.MatchString(item) {
return "", ErrBadRequest
}
q := url.Values{}
q.Set("action", "wbgetclaims")
q.Set("entity", item)
q.Set("property", "P18")
q.Set("maxlag", "5") // Wikimedia asks automated clients to back off when its databases lag
q.Set("format", "json")
body, err := c.get(ctx, c.wikidata+"?"+q.Encode())
if err != nil {
return "", err
}
var doc struct {
Error *struct {
Code string `json:"code"`
} `json:"error"`
Claims map[string][]struct {
Rank string `json:"rank"`
Mainsnak struct {
Snaktype string `json:"snaktype"`
Datavalue struct {
Value json.RawMessage `json:"value"`
} `json:"datavalue"`
} `json:"mainsnak"`
} `json:"claims"`
}
if err := json.Unmarshal(body, &doc); err != nil {
return "", ErrBadResponse
}
if doc.Error != nil { // Wikidata answers 200 to a failed request
switch doc.Error.Code {
case "no-such-entity":
return "", ErrNotFound
case "maxlag":
return "", fmt.Errorf("%w: lagged", ErrUnavailable)
}
return "", ErrBadResponse
}
var first string
for _, cl := range doc.Claims["P18"] {
if cl.Rank == "deprecated" || cl.Mainsnak.Snaktype != "value" {
continue
}
var name string
if json.Unmarshal(cl.Mainsnak.Datavalue.Value, &name) != nil || !validFileName(name) {
continue
}
if cl.Rank == "preferred" {
return name, nil
}
if first == "" {
first = name
}
}
return first, nil
}
// Info reads the licence, the author and a copy of a Commons file. ErrNotFound means
// the file does not exist, ErrNotFree that its licence is not a free one and
// ErrUnusable that it is not a picture Sonda can use.
func (c *Client) Info(ctx context.Context, name string) (File, error) {
if !validFileName(name) {
return File{}, ErrBadRequest
}
q := url.Values{}
q.Set("action", "query")
q.Set("titles", "File:"+name)
q.Set("prop", "imageinfo")
q.Set("iiprop", "url|size|mime|extmetadata")
q.Set("iiurlwidth", fmt.Sprint(ThumbWidth))
q.Set("iiextmetadatafilter", "LicenseShortName|LicenseUrl|Artist|AttributionRequired|Copyrighted|UsageTerms|NonFree")
q.Set("format", "json")
q.Set("formatversion", "2")
body, err := c.get(ctx, c.commons+"?"+q.Encode())
if err != nil {
return File{}, err
}
var doc struct {
Query struct {
Pages []struct {
Missing bool `json:"missing"`
Imageinfo []struct {
ThumbURL string `json:"thumburl"`
Width int `json:"width"`
Height int `json:"height"`
Mime string `json:"mime"`
DescriptionURL string `json:"descriptionurl"`
Extmetadata map[string]struct {
Value json.RawMessage `json:"value"`
} `json:"extmetadata"`
} `json:"imageinfo"`
} `json:"pages"`
} `json:"query"`
}
if err := json.Unmarshal(body, &doc); err != nil || len(doc.Query.Pages) != 1 {
return File{}, ErrBadResponse
}
page := doc.Query.Pages[0]
if page.Missing {
return File{}, ErrNotFound
}
if len(page.Imageinfo) != 1 {
return File{}, ErrBadResponse
}
ii := page.Imageinfo[0]
meta := func(key string) string {
var s string
if v, ok := ii.Extmetadata[key]; ok {
_ = json.Unmarshal(v.Value, &s)
}
return s
}
if strings.EqualFold(strings.TrimSpace(meta("NonFree")), "true") || !Free(meta("LicenseShortName")) {
return File{}, ErrNotFree
}
switch ii.Mime {
case "image/jpeg", "image/png", "image/gif":
default:
return File{}, ErrUnusable
}
if !allowedPictureURL(ii.ThumbURL) {
return File{}, ErrUnusable
}
licence := strings.TrimSpace(meta("LicenseShortName"))
return File{
Name: name, PageURL: pageURL(ii.DescriptionURL, name), ThumbURL: ii.ThumbURL, Mime: ii.Mime,
Width: ii.Width, Height: ii.Height, Author: plainText(meta("Artist")),
License: licence, LicenseURL: licenseURL(licence, meta("LicenseUrl")),
}, nil
}
// Download reads a picture from a Wikimedia picture host. Any other address is refused
// without a request.
func (c *Client) Download(ctx context.Context, pictureURL string) ([]byte, error) {
if !allowedPictureURL(pictureURL) {
return nil, ErrBadRequest
}
return c.get(ctx, pictureURL)
}
// Probe asks about one public item, to find out whether the service answers. It is a
// providers.Checker, and answers with the errors the settings understand.
func (c *Client) Probe(ctx context.Context) error {
_, err := c.FileName(ctx, probeItem)
switch {
case errors.Is(err, ErrRateLimited):
return providers.ErrRateLimited
case errors.Is(err, ErrUnavailable):
return providers.ErrUnreachable
}
return err
}
var (
pdRe = regexp.MustCompile(`(?i)^(public domain|pd(-[a-z0-9]+)+)$`)
cc0Re = regexp.MustCompile(`(?i)^cc0( 1\.0)?$`)
ccRe = regexp.MustCompile(`(?i)^cc by(-sa)? [0-9](\.[0-9])?( [a-z]{2,3})?$`)
)
// Free says whether a licence short name, as Commons writes it, is one Sonda accepts:
// public domain, CC0, CC BY and CC BY-SA. A copy is kept and scaled, so no-derivatives
// and non-commercial licences, and anything unknown, are refused.
func Free(shortName string) bool {
n := strings.TrimSpace(shortName)
return pdRe.MatchString(n) || cc0Re.MatchString(n) || ccRe.MatchString(n)
}
// licenseURL returns the address of the licence: the one Commons gives if it is on
// creativecommons.org, else the standard one of the public domain and CC0, else "".
func licenseURL(short, given string) string {
if strings.HasPrefix(given, "https://creativecommons.org/") {
return given
}
switch n := strings.TrimSpace(short); {
case cc0Re.MatchString(n):
return "https://creativecommons.org/publicdomain/zero/1.0/"
case pdRe.MatchString(n):
return "https://creativecommons.org/publicdomain/mark/1.0/"
}
return ""
}
// pageURL is the page of the file, shown as the source: it must be on Commons.
func pageURL(given, name string) string {
if strings.HasPrefix(given, "https://commons.wikimedia.org/wiki/") {
return given
}
return "https://commons.wikimedia.org/wiki/File:" + url.PathEscape(strings.ReplaceAll(name, " ", "_"))
}
var tagRe = regexp.MustCompile(`<[^>]*>`)
// plainText turns the HTML Commons gives for an author into text: tags removed,
// entities decoded, control characters dropped, spaces collapsed, at most maxAuthorLen.
func plainText(s string) string {
s = html.UnescapeString(tagRe.ReplaceAllString(s, ""))
s = strings.Map(func(r rune) rune {
if unicode.IsControl(r) && !unicode.IsSpace(r) {
return -1
}
return r
}, s)
s = strings.Join(strings.Fields(s), " ")
if utf8.RuneCountInString(s) > maxAuthorLen {
s = string([]rune(s)[:maxAuthorLen])
}
return s
}
// validFileName accepts what can be a Commons file name: not empty, not long, with no
// control character and none of the characters a title cannot hold.
func validFileName(n string) bool {
if strings.TrimSpace(n) == "" || utf8.RuneCountInString(n) > maxNameLen || !utf8.ValidString(n) {
return false
}
for _, r := range n {
if unicode.IsControl(r) || strings.ContainsRune("|[]{}<>#", r) {
return false
}
}
return true
}
// allowedPictureURL says whether an address is on a Wikimedia picture host, over https,
// with no credentials and no port.
func allowedPictureURL(raw string) bool {
u, err := url.Parse(raw)
if err != nil || u.Scheme != "https" || u.User != nil || u.Opaque != "" {
return false
}
return u.Host == "thumb.wikimedia.org" || u.Host == "upload.wikimedia.org"
}
// get sends one request and returns the body of a successful answer.
func (c *Client) get(ctx context.Context, rawURL string) ([]byte, error) {
resp, err := c.http.Get(ctx, rawURL)
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
var limited *outbound.RateLimitedError
if errors.As(err, &limited) {
return nil, ErrRateLimited
}
return nil, fmt.Errorf("%w: %w", ErrUnavailable, err)
}
switch {
case resp.Status == 200:
return resp.Body, nil
case resp.Status == 404:
return nil, ErrNotFound
case resp.Status == 429:
return nil, ErrRateLimited
case resp.Status >= 500:
return nil, fmt.Errorf("%w: status %d", ErrUnavailable, resp.Status)
}
return nil, fmt.Errorf("%w: status %d", ErrBadResponse, resp.Status)
}

View file

@ -0,0 +1,309 @@
package commons
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"codeberg.org/squidd/sonda/internal/providers"
"codeberg.org/squidd/sonda/internal/providers/outbound"
)
type server struct {
*httptest.Server
mu sync.Mutex
paths []string
status int
body string
}
func newServer(t *testing.T, status int, body string) *server {
t.Helper()
s := &server{status: status, body: body}
s.Server = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
s.mu.Lock()
s.paths = append(s.paths, r.URL.RequestURI())
status, body := s.status, s.body
s.mu.Unlock()
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.WriteHeader(status)
_, _ = w.Write([]byte(body))
}))
t.Cleanup(s.Close)
return s
}
func (s *server) client() *Client {
return New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries}),
s.URL+"/wikidata/w/api.php", s.URL+"/commons/w/api.php")
}
func (s *server) requests() []string {
s.mu.Lock()
defer s.mu.Unlock()
return append([]string(nil), s.paths...)
}
func TestFileNameReadsTheImageOfAnItem(t *testing.T) {
body := `{"claims":{"P18":[{"rank":"normal","mainsnak":{"snaktype":"value","datatype":"commonsMedia","datavalue":{"value":"RadioheadO2211125 composite.jpg"}}}]}}`
s := newServer(t, 200, body)
got, err := s.client().FileName(context.Background(), "Q44190")
if err != nil || got != "RadioheadO2211125 composite.jpg" {
t.Fatalf("name = %q err = %v", got, err)
}
reqs := s.requests()
if len(reqs) != 1 || !strings.Contains(reqs[0], "action=wbgetclaims") || !strings.Contains(reqs[0], "entity=Q44190") ||
!strings.Contains(reqs[0], "property=P18") || !strings.Contains(reqs[0], "maxlag=5") {
t.Errorf("requests = %v", reqs)
}
}
func TestFileNameChoosesThePreferredThenTheFirstAndSkipsWhatIsNotAFile(t *testing.T) {
claim := func(rank, snak, value string) string {
return `{"rank":"` + rank + `","mainsnak":{"snaktype":"` + snak + `","datavalue":{"value":"` + value + `"}}}`
}
for name, tt := range map[string]struct{ claims, want string }{
"preferred wins": {claim("normal", "value", "A.jpg") + "," + claim("preferred", "value", "B.jpg"), "B.jpg"},
"first of the normals": {claim("normal", "value", "A.jpg") + "," + claim("normal", "value", "B.jpg"), "A.jpg"},
"deprecated is skipped": {claim("deprecated", "value", "A.jpg") + "," + claim("normal", "value", "B.jpg"), "B.jpg"},
"only deprecated": {claim("deprecated", "value", "A.jpg"), ""},
"unknown value": {claim("normal", "somevalue", "A.jpg"), ""},
"no value": {claim("normal", "novalue", "A.jpg"), ""},
"a name with a pipe": {claim("normal", "value", "A|B.jpg") + "," + claim("normal", "value", "C.jpg"), "C.jpg"},
"a name with a bracket": {claim("normal", "value", "A[x].jpg"), ""},
"a long name": {claim("normal", "value", strings.Repeat("a", 400)), ""},
} {
body := `{"claims":{"P18":[` + tt.claims + `]}}`
got, err := newServer(t, 200, body).client().FileName(context.Background(), "Q1")
if err != nil || got != tt.want {
t.Errorf("%s: name = %q err = %v, want %q", name, got, err, tt.want)
}
}
}
func TestFileNameIsEmptyWhenThereIsNoImage(t *testing.T) {
for name, body := range map[string]string{"no claims": `{"claims":{}}`, "no P18": `{"claims":{"P31":[]}}`} {
if got, err := newServer(t, 200, body).client().FileName(context.Background(), "Q1"); err != nil || got != "" {
t.Errorf("%s: %q %v", name, got, err)
}
}
}
func TestFileNameRefusesABadItemAndReadsWikidataErrors(t *testing.T) {
s := newServer(t, 200, `{"claims":{}}`)
for _, item := range []string{"", "q1", "P18", "Q1&x=y", "Q0", "Q99999999999999999999"} {
if _, err := s.client().FileName(context.Background(), item); !errors.Is(err, ErrBadRequest) {
t.Errorf("item %q: err = %v", item, err)
}
}
if len(s.requests()) != 0 {
t.Errorf("requests for a bad item: %v", s.requests())
}
for name, tt := range map[string]struct {
status int
body string
want error
}{
"no such item (HTTP 200)": {200, `{"error":{"code":"no-such-entity"}}`, ErrNotFound},
"lag": {200, `{"error":{"code":"maxlag"}}`, ErrUnavailable},
"another error": {200, `{"error":{"code":"x"}}`, ErrBadResponse},
"rate limited": {429, ``, ErrRateLimited},
"server error": {503, ``, ErrUnavailable},
"not json": {200, `<html>`, ErrBadResponse},
"forbidden": {403, `no`, ErrBadResponse},
} {
if _, err := newServer(t, tt.status, tt.body).client().FileName(context.Background(), "Q1"); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
}
}
const infoBody = `{"continue":{},"query":{"pages":[{"pageid":1,"ns":6,"title":"File:Band.jpg","imagerepository":"local","imageinfo":[{
"thumburl":"https://thumb.wikimedia.org/wikipedia/commons/thumb/a/a1/Band.jpg/960px-Band.jpg?utm_source=commons.wikimedia.org",
"thumbwidth":960,"thumbheight":349,"width":5940,"height":2160,"mime":"image/jpeg","size":9175547,
"descriptionurl":"https://commons.wikimedia.org/wiki/File:Band.jpg",
"extmetadata":{"LicenseShortName":{"value":"CC BY 4.0"},"LicenseUrl":{"value":"https://creativecommons.org/licenses/by/4.0"},
"Artist":{"value":"<a rel=\"nofollow\" class=\"external text\" href=\"https://www.flickr.com/photos/raph_ph/\">Raph_PH</a>"},
"AttributionRequired":{"value":"true"},"Copyrighted":{"value":"True"},"UsageTerms":{"value":"Creative Commons Attribution 4.0"}}}]}]}}`
func TestInfoReadsTheLicenceTheAuthorAndTheMiniature(t *testing.T) {
s := newServer(t, 200, infoBody)
got, err := s.client().Info(context.Background(), "Band.jpg")
if err != nil {
t.Fatal(err)
}
want := File{
Name: "Band.jpg", PageURL: "https://commons.wikimedia.org/wiki/File:Band.jpg", Mime: "image/jpeg", Width: 5940, Height: 2160,
ThumbURL: "https://thumb.wikimedia.org/wikipedia/commons/thumb/a/a1/Band.jpg/960px-Band.jpg?utm_source=commons.wikimedia.org",
Author: "Raph_PH", License: "CC BY 4.0", LicenseURL: "https://creativecommons.org/licenses/by/4.0",
}
if got != want {
t.Errorf("file = %+v\nwant %+v", got, want)
}
reqs := s.requests()
if len(reqs) != 1 || !strings.Contains(reqs[0], "iiurlwidth=960") || !strings.Contains(reqs[0], "formatversion=2") ||
!strings.Contains(reqs[0], "titles=File%3ABand.jpg") || !strings.Contains(reqs[0], "iiextmetadatafilter=") {
t.Errorf("requests = %v", reqs)
}
}
func withLicense(short, url, extra string) string {
body := strings.Replace(infoBody, `"LicenseShortName":{"value":"CC BY 4.0"}`, `"LicenseShortName":{"value":"`+short+`"}`, 1)
body = strings.Replace(body, `"LicenseUrl":{"value":"https://creativecommons.org/licenses/by/4.0"}`, `"LicenseUrl":{"value":"`+url+`"}`, 1)
if extra != "" {
body = strings.Replace(body, `"AttributionRequired"`, extra+`,"AttributionRequired"`, 1)
}
return body
}
func TestOnlyFreeLicencesAreAccepted(t *testing.T) {
for short, free := range map[string]bool{
"CC BY 4.0": true, "CC BY 2.0": true, "CC BY-SA 4.0": true, "CC BY-SA 3.0": true, "CC BY-SA 2.5": true, "CC BY-SA 3.0 de": true,
"CC0": true, "CC0 1.0": true, "Public domain": true, "PD-old-100": true, "PD-US": true, "PD-self": true,
"CC BY-NC 4.0": false, "CC BY-NC-SA 3.0": false, "CC BY-ND 4.0": false, "CC BY-NC-ND 2.0": false,
"Fair use": false, "GFDL": false, "GFDL 1.2": false, "": false, "All rights reserved": false, "Attribution": false,
"CC BY": false, "Copyrighted free use": false, "Public domain but not really": false, "cc by-nd": false,
} {
_, err := newServer(t, 200, withLicense(short, "", "")).client().Info(context.Background(), "Band.jpg")
if free && err != nil {
t.Errorf("%q refused: %v", short, err)
}
if !free && !errors.Is(err, ErrNotFree) {
t.Errorf("%q: err = %v, want ErrNotFree", short, err)
}
if Free(short) != free {
t.Errorf("Free(%q) = %v, want %v", short, Free(short), free)
}
}
}
func TestANonFreeFlagWinsOverTheName(t *testing.T) {
body := withLicense("CC BY 4.0", "https://creativecommons.org/licenses/by/4.0", `"NonFree":{"value":"true"}`)
if _, err := newServer(t, 200, body).client().Info(context.Background(), "Band.jpg"); !errors.Is(err, ErrNotFree) {
t.Errorf("err = %v, want ErrNotFree", err)
}
}
func TestTheLicenceAddressIsKeptOnlyOnCreativeCommons(t *testing.T) {
for url, want := range map[string]string{
"https://creativecommons.org/licenses/by/4.0": "https://creativecommons.org/licenses/by/4.0",
"https://evil.example/licenses/by/4.0": "",
"javascript:alert(1)": "",
"http://creativecommons.org/licenses/by/4.0": "",
"": "",
} {
got, err := newServer(t, 200, withLicense("CC BY 4.0", url, "")).client().Info(context.Background(), "Band.jpg")
if err != nil || got.LicenseURL != want {
t.Errorf("%q: LicenseURL = %q err = %v, want %q", url, got.LicenseURL, err, want)
}
}
// The public domain has no address in Commons: the standard mark is given.
got, _ := newServer(t, 200, withLicense("Public domain", "", "")).client().Info(context.Background(), "Band.jpg")
if got.LicenseURL != "https://creativecommons.org/publicdomain/mark/1.0/" {
t.Errorf("public domain address = %q", got.LicenseURL)
}
got, _ = newServer(t, 200, withLicense("CC0 1.0", "", "")).client().Info(context.Background(), "Band.jpg")
if got.LicenseURL != "https://creativecommons.org/publicdomain/zero/1.0/" {
t.Errorf("CC0 address = %q", got.LicenseURL)
}
}
func TestTheAuthorIsPlainText(t *testing.T) {
for artist, want := range map[string]string{
`<a href=\"https://x\">Raph_PH</a>`: "Raph_PH",
`Jane &amp; John <b>Doe</b>`: "Jane & John Doe",
`<script>alert(1)</script>Eve`: "alert(1)Eve",
"Line one\\nLine two": "Line one Line two",
` <span> spaced </span> `: "spaced",
`&lt;b&gt;not a tag&lt;/b&gt;`: "<b>not a tag</b>",
`<a href=\"https://x\"></a>`: "",
strings.Repeat("a", 500): strings.Repeat("a", maxAuthorLen),
} {
body := strings.Replace(infoBody, `<a rel=\"nofollow\" class=\"external text\" href=\"https://www.flickr.com/photos/raph_ph/\">Raph_PH</a>`, artist, 1)
got, err := newServer(t, 200, body).client().Info(context.Background(), "Band.jpg")
if err != nil || got.Author != want {
t.Errorf("%q: author = %q err = %v, want %q", artist, got.Author, err, want)
}
}
}
func TestInfoRefusesWhatCannotBeUsed(t *testing.T) {
for name, tt := range map[string]struct {
body string
want error
}{
"no such file": {`{"query":{"pages":[{"title":"File:X.jpg","missing":true}]}}`, ErrNotFound},
"not a picture": {strings.Replace(infoBody, `"mime":"image/jpeg"`, `"mime":"audio/ogg"`, 1), ErrUnusable},
"a drawing": {strings.Replace(infoBody, `"mime":"image/jpeg"`, `"mime":"image/svg+xml"`, 1), ErrUnusable},
"no thumbnail": {strings.Replace(infoBody, `"thumburl"`, `"thumbxx"`, 1), ErrUnusable},
"a thumbnail on another host": {strings.Replace(infoBody, "thumb.wikimedia.org", "evil.example", 1), ErrUnusable},
"a thumbnail with a password": {strings.Replace(infoBody, "https://thumb.wikimedia.org", "https://user:pw@thumb.wikimedia.org", 1), ErrUnusable},
"a thumbnail over http": {strings.Replace(infoBody, "https://thumb.wikimedia.org", "http://thumb.wikimedia.org", 1), ErrUnusable},
"no pages": {`{"query":{"pages":[]}}`, ErrBadResponse},
"not json": {`<html>`, ErrBadResponse},
} {
if _, err := newServer(t, 200, tt.body).client().Info(context.Background(), "Band.jpg"); !errors.Is(err, tt.want) {
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
}
}
for _, bad := range []string{"", strings.Repeat("a", 400), "A\nB", "A|B", "A[B]", "A{B}"} {
if _, err := newServer(t, 200, infoBody).client().Info(context.Background(), bad); !errors.Is(err, ErrBadRequest) {
t.Errorf("name %q: err = %v", bad, err)
}
}
}
func TestDownloadOnlyFromWikimediaPictureHosts(t *testing.T) {
for url, ok := range map[string]bool{
"https://thumb.wikimedia.org/wikipedia/commons/thumb/a/a1/B.jpg/960px-B.jpg": true,
"https://upload.wikimedia.org/wikipedia/commons/a/a1/B.jpg": true,
"http://upload.wikimedia.org/wikipedia/commons/a/a1/B.jpg": false,
"https://upload.wikimedia.org.evil.example/a.jpg": false,
"https://evil.example/upload.wikimedia.org/a.jpg": false,
"https://user@upload.wikimedia.org/a.jpg": false,
"https://upload.wikimedia.org:8443/a.jpg": false,
"https://commons.wikimedia.org/w/api.php": false,
"file:///etc/passwd": false,
"": false,
} {
if got := allowedPictureURL(url); got != ok {
t.Errorf("allowedPictureURL(%q) = %v, want %v", url, got, ok)
}
}
// A refused address is never requested.
s := newServer(t, 200, "x")
if _, err := s.client().Download(context.Background(), s.URL+"/a.jpg"); !errors.Is(err, ErrBadRequest) {
t.Errorf("err = %v", err)
}
if len(s.requests()) != 0 {
t.Errorf("requests = %v", s.requests())
}
}
func TestErrorsDoNotCopyTheProvidersText(t *testing.T) {
secret := "SECRET-FROM-THE-PROVIDER"
_, err := newServer(t, 200, `{"error":{"code":"x","info":"`+secret+`"}}`).client().FileName(context.Background(), "Q1")
if err == nil || strings.Contains(err.Error(), secret) {
t.Errorf("err = %v", err)
}
_, err = newServer(t, 500, secret).client().Info(context.Background(), "A.jpg")
if err == nil || strings.Contains(err.Error(), secret) {
t.Errorf("err = %v", err)
}
}
func TestProbeSpeaksTheLanguageOfTheSettings(t *testing.T) {
if err := newServer(t, 200, `{"claims":{}}`).client().Probe(context.Background()); err != nil {
t.Errorf("ok: %v", err)
}
if err := newServer(t, 429, ``).client().Probe(context.Background()); !errors.Is(err, providers.ErrRateLimited) {
t.Errorf("429: %v", err)
}
if err := newServer(t, 503, ``).client().Probe(context.Background()); !errors.Is(err, providers.ErrUnreachable) {
t.Errorf("503: %v", err)
}
}

View file

@ -22,6 +22,7 @@ const (
LastFM = "lastfm"
MusicBrainz = "musicbrainz"
Wikipedia = "wikipedia"
Commons = "commons"
)
// Errors returned by the service.
@ -116,6 +117,16 @@ var known = []meta{
note: "Wikipedia text is CC BY-SA 4.0: show the name of the page, a link to it and the licence next to " +
"the text, and say if you changed it.",
},
{
name: Commons,
sends: "It needs the MusicBrainz provider on too. The MusicBrainz id of the artist you look up goes to MusicBrainz " +
"(never a name: Sonda does not search), with the contact set for MusicBrainz; the Wikidata item that " +
"MusicBrainz links to it goes to Wikidata, the name of the picture Wikidata gives it goes to Wikimedia " +
"Commons, and a copy of the picture is downloaded from Wikimedia, with the contact below in the User-Agent. " +
"Your IP address goes to all of them. Never your users, your listens, track titles or file paths.",
note: "Only pictures under free licences are kept (public domain, CC0, CC BY, CC BY-SA); the author, the licence " +
"and the page of the file are stored with the picture and must be shown next to it.",
},
}
func lookup(name string) (meta, bool) {
@ -372,6 +383,7 @@ type Purged struct {
SimilarArtists int // lists of similar artists
Tags int // tags
Descriptions int // descriptions
Images int // pictures of artists
Artists int // artists that only the removed data named
}

View file

@ -53,8 +53,8 @@ func TestEveryProviderIsOffByDefault(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if len(infos) != 3 || infos[0].Name != LastFM || infos[1].Name != MusicBrainz || infos[2].Name != Wikipedia {
t.Fatalf("providers = %+v, want lastfm, musicbrainz, then wikipedia", infos)
if len(infos) != 4 || infos[0].Name != LastFM || infos[1].Name != MusicBrainz || infos[2].Name != Wikipedia || infos[3].Name != Commons {
t.Fatalf("providers = %+v, want lastfm, musicbrainz, wikipedia, then commons", infos)
}
for _, in := range infos {
if in.Enabled || in.HasAPIKey {
@ -222,7 +222,7 @@ func TestListMergesStoredRowsInAFixedOrder(t *testing.T) {
st.rows[MusicBrainz] = Settings{Provider: MusicBrainz, Enabled: true, Contact: "c", UpdatedAt: time.Unix(1, 0)}
st.rows["stale-unknown"] = Settings{Provider: "stale-unknown", Enabled: true} // a row of a provider that no longer exists
infos, err := s.List(context.Background())
if err != nil || len(infos) != 3 {
if err != nil || len(infos) != 4 {
t.Fatalf("infos = %+v err=%v, want exactly the known providers", infos, err)
}
if infos[0].Name != LastFM || infos[1].Name != MusicBrainz || !infos[1].Enabled || infos[1].Contact != "c" {
@ -447,3 +447,32 @@ func TestWikipediaIsAProviderThatNeedsNoKey(t *testing.T) {
t.Errorf("contact: %+v err=%v", info, err)
}
}
func TestCommonsIsAProviderThatNeedsNoKeyAndSaysWhatItSendsAndKeeps(t *testing.T) {
svc, _ := newSvc()
infos, _ := svc.List(context.Background())
var c *Info
for i := range infos {
if infos[i].Name == Commons {
c = &infos[i]
}
}
if c == nil {
t.Fatalf("providers = %+v, want commons among them", infos)
}
if c.Enabled || c.NeedsAPIKey || c.Sends == "" || c.Notes == "" {
t.Errorf("commons = %+v: off by default, no key, and it says what is sent and under which licences", c)
}
for _, want := range []string{"MusicBrainz", "free licences"} {
if !strings.Contains(c.Sends+c.Notes, want) {
t.Errorf("the text does not say %q: %q %q", want, c.Sends, c.Notes)
}
}
on := true
if _, err := svc.Update(context.Background(), Commons, Patch{Enabled: &on}); err != nil {
t.Fatalf("enable: %v", err)
}
if _, active, _ := svc.Active(context.Background(), Commons); !active {
t.Error("not active after being switched on")
}
}

Some files were not shown because too many files have changed in this diff Show more