fix(about): unusable pages fall through, MusicBrainz switch, merged items
- A page that cannot be used (no text, an address off Wikipedia, a redirect that is never followed) is now an answer: the chain goes on to the next language and the outcome is stored. Before, it was a failure asked about again at every view, and redirects counted against the provider's breaker. - The chain also needs the MusicBrainz provider to be on, since the id goes to it with its own contact: nothing is sent to MusicBrainz that the admin did not allow. The disclosure in the provider settings, the spec and the README say so. - A Wikidata item that was merged into another is understood through the target: the entity's own id is returned, compared with the page's item and stored. Verified on the real API. - The title shown is the one Wikidata names (with spaces), control characters are dropped from the text (PostgreSQL refuses NUL), the text is cut to at most 4000 characters with the mark included, and a stored description is reported disabled when the provider is off. - A read that started before the provider was switched off does not save its result. - Tests: the real clients with a redirect and a page without text, merged items, MusicBrainz off, switching off during a read.
This commit is contained in:
parent
48cb903115
commit
762d424816
12 changed files with 367 additions and 98 deletions
|
|
@ -17,7 +17,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
|
|||
- **Listening history.** Clients report explicit `play`, `complete` and `skip` events. An event counts as a listen when it is `complete`, or when it was played for at least 20 seconds (`listening.min_listen_seconds`); the rule is applied when the history is read, so changing it re-reads the past. You can read your own history, with a `counted` flag on every event. It stays in your database and is never sent anywhere.
|
||||
- **Your taste.** `GET /me/taste` shows what Sonda has learned from your own listening: the artists you like and the kinds of music (tags) they have in common. A complete listen counts for more than a short one, a skip counts against the artist, and older listens weigh less (they halve every 90 days, `taste.half_life_days`; only the last ten half-lives are read). It is worked out on every request from data already in your database; nothing is sent to any provider.
|
||||
- **Discover.** `GET /discover` suggests artists you may like, in your library or not, each with a score from 0 to 1 and the parts it is made of: how close it is to the artists you like (affinity), how close its tags are to yours (genre), what your listening says about it (preference) and a bonus for what you have not played (discovery). The weights are in `[scoring]` in the config. A part with no data is left out and the rest rescaled, and every answer says which seeds led to each suggestion. It starts from the artists you name with `?seed=`, or, only if you turned on `discover_from_listening`, from the artists you play most. It needs Last.fm to be switched on to ask for anything new: when Last.fm is off and a seed's similar artists are not stored yet, the answer says `disabled` (what was stored earlier is still used). See [External providers](#external-providers).
|
||||
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider switched on, Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
|
||||
- **Artist descriptions.** `GET /artists/{id}/description` shows what Wikipedia says about an artist: a short text, with the page it comes from, a link to it and the licence (CC BY-SA 4.0), which you must show next to it. With the `wikipedia` provider and the `musicbrainz` one switched on (the link to Wikidata comes from MusicBrainz), Sonda follows ids and never a name: the artist's MusicBrainz id, then the Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in the first of your languages (`[wikipedia] languages`, the code of that Wikipedia, for example `["it", "en"]`) that has one. An artist without a MusicBrainz id has no description. It is kept for 30 days and removed with the provider's data. See [External providers](#external-providers).
|
||||
- **Album covers.** `GET /albums/{id}/cover` serves the cover of an album from your own files: the picture embedded in them, or else a `cover`, `folder`, `front`, `album` or `albumart` image in the album's folder. It is built the first time it is asked for (a few at a time), scaled (`?size=small` is 300 pixels, `large` 1000) and kept in the data folder (`storage.data_dir`, you can delete it: covers are built again). A picture you replace in a folder is noticed within a week, or at once after a full rescan. For an album split in `CD1`/`Disc 2` folders, the picture in the folder above is used. Nothing is sent anywhere. It also takes `?access_token=`, so an `<img>` element can use it.
|
||||
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. Nothing is downloaded or sent anywhere: the wishlist is a list for you, and for a download client you may set up later. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
|
||||
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
|
||||
|
|
@ -204,7 +204,7 @@ curl -s -X PUT $BASE/providers/musicbrainz -H "$AUTH" -d '{"enabled":true,"conta
|
|||
|
||||
The key is stored in the database, in clear text, and is never returned by the API or written to the log: the list only says `has_api_key`. Because the database holds it, treat database backups like secrets. Removing the key turns the provider off, and each provider describes in the list exactly what it sends.
|
||||
|
||||
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries your contact (Wikimedia refuses requests without one), and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
|
||||
**What leaves your server, when a provider is on:** to Last.fm, the artist name (or MusicBrainz id) being looked up, your API key, your IP address and the `User-Agent`; to MusicBrainz, only the MusicBrainz id of the artist, your IP address and a `User-Agent` that carries your contact. Never your users, your listens, track titles or file paths. Requests only happen when someone asks for them, never as a bulk pass over your library, and they are spaced out, retried with backoff, and stopped by a circuit breaker if a provider is down. **Descriptions** (the Wikipedia provider, which also needs MusicBrainz on) send the MusicBrainz id to MusicBrainz, the Wikidata item to Wikidata and the title of the page to Wikipedia, with your IP address and a `User-Agent` that carries a contact (the one set for each provider; Wikimedia refuses requests without one), and never a name. **Discover** asks Last.fm for the similar artists of each seed (the seed's name or MusicBrainz id) and, for the best suggestions that have no tags yet, for the tags of that suggestion (its name); never MusicBrainz. Seeds you name are sent when you ask. Seeds taken from your listening, which are the names of the artists you play most, are sent only if you turned on `discover_from_listening` yourself; otherwise Discover refuses to run without a `seed`. The one exception to "only when someone asks" is `POST /providers/{name}/test`, which an admin sends to check a key; it sends nothing but the key.
|
||||
|
||||
**Similar artists.** `GET /api/v1/artists/{id}/similar` asks Last.fm for the artists similar to an artist (one of yours, or one Last.fm named earlier) and keeps the answer in your database: 7 days, or 1 day when Last.fm does not know the artist. Until then a second look costs nothing and sends nothing, and if Last.fm is down you still get the last answer, marked `stale`. If the provider is off the answer says `disabled` instead of failing. Each similar artist says whether you have it (`in_library`); the ones you do not have are stored as artists without files. They can be opened with `GET /artists/{id}` but are never part of your artist lists, and if you later add their music, the scan finds them there. Last.fm's `score` is relative to each answer (the best one is usually 1), so compare artists only inside one answer.
|
||||
|
||||
|
|
|
|||
|
|
@ -114,8 +114,9 @@ type Summary struct {
|
|||
type Source interface {
|
||||
// WikidataID returns the Wikidata item MusicBrainz links to the artist, or "".
|
||||
WikidataID(ctx context.Context, musicbrainzID string) (string, error)
|
||||
// Pages returns the title of the page of the item in each of the languages that has one.
|
||||
Pages(ctx context.Context, item string, languages []string) (map[string]string, error)
|
||||
// Pages returns the title of the page of the item in each of the languages that has
|
||||
// one, and the item they belong to: the same, or the one it was merged into.
|
||||
Pages(ctx context.Context, item string, languages []string) (string, map[string]string, error)
|
||||
Summary(ctx context.Context, language, title string) (Summary, error)
|
||||
}
|
||||
|
||||
|
|
@ -152,13 +153,16 @@ func (s *Service) Description(ctx context.Context, artistID int64) (Result, erro
|
|||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if snap.Fetched && s.fresh(snap) {
|
||||
return result(snap, discovery.StatusOK, false), nil
|
||||
}
|
||||
if _, active, err := s.settings.Active(ctx, Provider); err != nil {
|
||||
fresh := snap.Fetched && s.fresh(snap)
|
||||
active, err := s.active(ctx)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
} else if !active {
|
||||
return result(snap, discovery.StatusDisabled, snap.Fetched), nil
|
||||
}
|
||||
switch {
|
||||
case !active:
|
||||
return result(snap, discovery.StatusDisabled, snap.Fetched && !fresh), nil
|
||||
case fresh:
|
||||
return result(snap, discovery.StatusOK, false), nil
|
||||
}
|
||||
if artist.MBID == "" { // Sonda never looks an id up by name
|
||||
return result(snap, discovery.StatusNoID, snap.Fetched), nil
|
||||
|
|
@ -180,6 +184,9 @@ func (s *Service) Description(ctx context.Context, artistID int64) (Result, erro
|
|||
if errors.Is(r.Err, discovery.ErrNotFound) {
|
||||
return Result{}, discovery.ErrNotFound // removed while it was being read
|
||||
}
|
||||
if errors.Is(r.Err, errSwitchedOff) {
|
||||
return result(snap, discovery.StatusDisabled, snap.Fetched), nil
|
||||
}
|
||||
if r.Err != nil {
|
||||
s.log.Warn("the description could not be read", zap.String("provider", Provider), zap.Error(r.Err))
|
||||
return result(snap, discovery.StatusError, snap.Fetched), nil
|
||||
|
|
@ -192,6 +199,25 @@ func (s *Service) Description(ctx context.Context, artistID int64) (Result, erro
|
|||
return result(snap, discovery.StatusOK, false), nil
|
||||
}
|
||||
|
||||
// errSwitchedOff is what a read returns when the providers were switched off while it ran.
|
||||
var errSwitchedOff = errors.New("the provider was switched off")
|
||||
|
||||
// active says whether the chain may be followed: Wikipedia is on, and so is MusicBrainz,
|
||||
// which gives the link to Wikidata. Without that second switch nothing would stop the
|
||||
// artist's id going to a provider the admin never allowed.
|
||||
func (s *Service) active(ctx context.Context) (bool, error) {
|
||||
for _, name := range []string{Provider, providers.MusicBrainz} {
|
||||
_, on, err := s.settings.Active(ctx, name)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if !on {
|
||||
return false, nil
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func (s *Service) fresh(snap Snapshot) bool {
|
||||
ttl := PositiveTTL
|
||||
if snap.None {
|
||||
|
|
@ -224,11 +250,11 @@ func (s *Service) refresh(a discovery.Artist) error {
|
|||
return err
|
||||
}
|
||||
if item == "" || err != nil {
|
||||
return s.store.SaveDescription(ctx, a.ID, s.now(), "", nil)
|
||||
return s.save(ctx, a.ID, "", nil)
|
||||
}
|
||||
pages, err := s.source.Pages(ctx, item, s.languages)
|
||||
resolved, pages, err := s.source.Pages(ctx, item, s.languages)
|
||||
if errors.Is(err, ErrSourceNotFound) {
|
||||
return s.store.SaveDescription(ctx, a.ID, s.now(), item, nil)
|
||||
return s.save(ctx, a.ID, item, nil)
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
|
|
@ -246,20 +272,29 @@ func (s *Service) refresh(a discovery.Artist) error {
|
|||
return err
|
||||
}
|
||||
text := strings.TrimSpace(sum.Extract)
|
||||
// An article about this very item, with something in it. Anything else (a page
|
||||
// of disambiguation, a redirect to another subject) is not a description.
|
||||
if sum.Type != "standard" || sum.WikibaseItem != item || text == "" {
|
||||
// An article about this very item (the one the pages belong to, which is the
|
||||
// target when the item was merged), with some text, and an address to credit.
|
||||
// Anything else (a page of disambiguation, a redirect to another subject, a page
|
||||
// with no text) is not a description.
|
||||
if sum.Type != "standard" || sum.WikibaseItem != resolved || text == "" || sum.URL == "" {
|
||||
continue
|
||||
}
|
||||
title = sum.Title
|
||||
if title == "" {
|
||||
title = pages[lang]
|
||||
}
|
||||
return s.store.SaveDescription(ctx, a.ID, s.now(), item, &Description{
|
||||
return s.save(ctx, a.ID, resolved, &Description{
|
||||
Text: cut(text), Short: sum.Description, Language: lang, Title: title, URL: sum.URL,
|
||||
})
|
||||
}
|
||||
return s.store.SaveDescription(ctx, a.ID, s.now(), item, nil)
|
||||
return s.save(ctx, a.ID, resolved, nil)
|
||||
}
|
||||
|
||||
// save stores the outcome, unless the providers were switched off since the read began:
|
||||
// an admin who took the data back and switched it off must not see it come back.
|
||||
func (s *Service) save(ctx context.Context, id int64, item string, d *Description) error {
|
||||
if on, err := s.active(ctx); err != nil {
|
||||
return err
|
||||
} else if !on {
|
||||
return errSwitchedOff
|
||||
}
|
||||
return s.store.SaveDescription(ctx, id, s.now(), item, d)
|
||||
}
|
||||
|
||||
// cut keeps at most MaxTextLen characters and marks a text that was shortened.
|
||||
|
|
@ -267,5 +302,5 @@ func cut(s string) string {
|
|||
if utf8.RuneCountInString(s) <= MaxTextLen {
|
||||
return s
|
||||
}
|
||||
return string([]rune(s)[:MaxTextLen]) + "…"
|
||||
return string([]rune(s)[:MaxTextLen-1]) + "…" // the mark counts: never more than MaxTextLen
|
||||
}
|
||||
|
|
|
|||
|
|
@ -80,6 +80,7 @@ func (f *fakeStore) SaveDescription(_ context.Context, id int64, at time.Time, q
|
|||
type fakeSource struct {
|
||||
mu sync.Mutex
|
||||
qid string
|
||||
resolved string // the item the pages belong to; empty means the same as qid
|
||||
qidErr error
|
||||
pages map[string]string
|
||||
pagesErr error
|
||||
|
|
@ -108,12 +109,16 @@ func (s *fakeSource) WikidataID(_ context.Context, id string) (string, error) {
|
|||
return s.qid, s.qidErr
|
||||
}
|
||||
|
||||
func (s *fakeSource) Pages(_ context.Context, qid string, langs []string) (map[string]string, error) {
|
||||
func (s *fakeSource) Pages(_ context.Context, qid string, langs []string) (string, map[string]string, error) {
|
||||
s.mu.Lock()
|
||||
defer s.mu.Unlock()
|
||||
s.calls.pages++
|
||||
s.langs = append([]string(nil), langs...)
|
||||
return s.pages, s.pagesErr
|
||||
resolved := s.resolved
|
||||
if resolved == "" {
|
||||
resolved = qid
|
||||
}
|
||||
return resolved, s.pages, s.pagesErr
|
||||
}
|
||||
|
||||
func (s *fakeSource) Summary(_ context.Context, lang, title string) (Summary, error) {
|
||||
|
|
@ -133,13 +138,26 @@ func (s *fakeSource) total() int {
|
|||
return s.calls.qid + s.calls.pages + s.calls.summary
|
||||
}
|
||||
|
||||
type on map[string]bool
|
||||
// switches says which providers are on, and can be changed while a read is running.
|
||||
type switches struct {
|
||||
mu sync.Mutex
|
||||
m map[string]bool
|
||||
}
|
||||
|
||||
func (p on) Active(_ context.Context, name string) (providers.Settings, bool, error) {
|
||||
return providers.Settings{Provider: name, Enabled: p[name]}, p[name], nil
|
||||
func (p *switches) Active(_ context.Context, name string) (providers.Settings, bool, error) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
return providers.Settings{Provider: name, Enabled: p.m[name]}, p.m[name], nil
|
||||
}
|
||||
|
||||
func (p *switches) set(name string, on bool) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.m[name] = on
|
||||
}
|
||||
|
||||
type rig struct {
|
||||
sw *switches
|
||||
svc *Service
|
||||
store *fakeStore
|
||||
src *fakeSource
|
||||
|
|
@ -162,7 +180,8 @@ func newRig(enabled bool) *rig {
|
|||
},
|
||||
sumErrs: map[string]error{},
|
||||
}
|
||||
r.svc = NewService(r.store, r.src, on{providers.Wikipedia: enabled}, []string{"it", "en"}, zap.NewNop())
|
||||
r.sw = &switches{m: map[string]bool{providers.Wikipedia: enabled, providers.MusicBrainz: enabled}}
|
||||
r.svc = NewService(r.store, r.src, r.sw, []string{"it", "en"}, zap.NewNop())
|
||||
r.svc.now = func() time.Time { return r.now }
|
||||
return r
|
||||
}
|
||||
|
|
@ -402,3 +421,93 @@ func TestAnArtistRemovedWhileItWasReadIsNotFound(t *testing.T) {
|
|||
t.Errorf("err = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestItNeedsMusicBrainzToBeOnToo(t *testing.T) {
|
||||
// The link to Wikidata comes from MusicBrainz: with MusicBrainz switched off nothing is sent to it.
|
||||
r := newRig(true)
|
||||
r.sw.set(providers.MusicBrainz, false)
|
||||
res := r.get(t, 1)
|
||||
if res.Status != discovery.StatusDisabled || res.Description != nil || r.src.total() != 0 {
|
||||
t.Errorf("result = %+v, calls %d: nobody may be asked while MusicBrainz is off", res, r.src.total())
|
||||
}
|
||||
r.sw.set(providers.MusicBrainz, true)
|
||||
r.sw.set(providers.Wikipedia, false)
|
||||
if res = r.get(t, 1); res.Status != discovery.StatusDisabled || r.src.total() != 0 {
|
||||
t.Errorf("with Wikipedia off: %+v calls %d", res, r.src.total())
|
||||
}
|
||||
}
|
||||
|
||||
func TestAMergedItemIsUnderstoodThroughTheItemItWasMergedInto(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.qid, r.src.resolved = "Q1", "Q2" // MusicBrainz knows the old id; the pages say Q2
|
||||
for lang, sum := range r.src.summaries {
|
||||
sum.WikibaseItem = "Q2"
|
||||
r.src.summaries[lang] = sum
|
||||
}
|
||||
res := r.get(t, 1)
|
||||
if res.Description == nil || res.Description.Language != "it" {
|
||||
t.Fatalf("result = %+v, want the description of the item it was merged into", res)
|
||||
}
|
||||
if len(r.store.saves) != 1 || r.store.saves[0].qid != "Q2" {
|
||||
t.Errorf("saves = %+v, want the resolved item stored", r.store.saves)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheTitleShownIsTheOneWikidataNames(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.pages = map[string]string{"it": "Guns N' Roses"}
|
||||
s := r.src.summaries["it"]
|
||||
s.Title = "Guns_N'_Roses" // the REST title is the database key, with underscores
|
||||
r.src.summaries["it"] = s
|
||||
if d := r.get(t, 1).Description; d == nil || d.Title != "Guns N' Roses" {
|
||||
t.Errorf("description = %+v", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAPageWithNoAddressOnWikipediaIsNotUsed(t *testing.T) {
|
||||
r := newRig(true)
|
||||
s := r.src.summaries["it"]
|
||||
s.URL = "" // the client drops an address that is not on that Wikipedia
|
||||
r.src.summaries["it"] = s
|
||||
if d := r.get(t, 1).Description; d == nil || d.Language != "en" {
|
||||
t.Errorf("description = %+v, want the next language: a text without its source cannot be shown", d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTheTextNeverExceedsTheLimit(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.summaries["it"] = standard("it", "Q44190", strings.Repeat("è", MaxTextLen+500))
|
||||
if n := len([]rune(r.get(t, 1).Description.Text)); n != MaxTextLen {
|
||||
t.Errorf("text has %d characters, want exactly %d with the mark that it was cut", n, MaxTextLen)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAStoredDescriptionIsStillReportedAsDisabledWhenTheProviderIsOff(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.get(t, 1)
|
||||
r.sw.set(providers.Wikipedia, false)
|
||||
res := r.get(t, 1) // fresh, but the switch is off
|
||||
if res.Status != discovery.StatusDisabled || res.Stale || res.Description == nil {
|
||||
t.Errorf("result = %+v, want the description, not stale, with status disabled", res)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSwitchedOffWhileItWasBeingReadNothingIsSaved(t *testing.T) {
|
||||
r := newRig(true)
|
||||
r.src.release = make(chan struct{})
|
||||
done := make(chan Result, 1)
|
||||
go func() {
|
||||
res, _ := r.svc.Description(context.Background(), 1)
|
||||
done <- res
|
||||
}()
|
||||
time.Sleep(50 * time.Millisecond)
|
||||
r.sw.set(providers.Wikipedia, false) // the admin took the data back and switched it off
|
||||
close(r.src.release)
|
||||
res := <-done
|
||||
if len(r.store.saves) != 0 {
|
||||
t.Errorf("saves = %+v: what a read that started before the switch found must not be kept", r.store.saves)
|
||||
}
|
||||
if res.Status != discovery.StatusDisabled || res.Description != nil {
|
||||
t.Errorf("result = %+v", res)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,9 +33,9 @@ func (w WikipediaSource) WikidataID(ctx context.Context, musicbrainzID string) (
|
|||
}
|
||||
|
||||
// Pages implements Source.
|
||||
func (w WikipediaSource) Pages(ctx context.Context, item string, languages []string) (map[string]string, error) {
|
||||
pages, err := w.Wikipedia.Pages(ctx, item, languages)
|
||||
return pages, notFound(err)
|
||||
func (w WikipediaSource) Pages(ctx context.Context, item string, languages []string) (string, map[string]string, error) {
|
||||
resolved, pages, err := w.Wikipedia.Pages(ctx, item, languages)
|
||||
return resolved, pages, notFound(err)
|
||||
}
|
||||
|
||||
// Summary implements Source.
|
||||
|
|
|
|||
|
|
@ -8,6 +8,10 @@ import (
|
|||
"strings"
|
||||
"testing"
|
||||
|
||||
"go.uber.org/zap"
|
||||
|
||||
"codeberg.org/squidd/sonda/internal/discovery"
|
||||
"codeberg.org/squidd/sonda/internal/providers"
|
||||
"codeberg.org/squidd/sonda/internal/providers/musicbrainz"
|
||||
"codeberg.org/squidd/sonda/internal/providers/outbound"
|
||||
"codeberg.org/squidd/sonda/internal/providers/wikipedia"
|
||||
|
|
@ -15,6 +19,12 @@ import (
|
|||
|
||||
// fakeWeb answers for MusicBrainz, Wikidata and Wikipedia on one server.
|
||||
func fakeWeb(t *testing.T, wikidataLink string) WikipediaSource {
|
||||
t.Helper()
|
||||
return fakeWebWith(t, wikidataLink, "ok")
|
||||
}
|
||||
|
||||
// fakeWebWith can make the Italian page a redirect ("redirect") or a page with no text ("noextract").
|
||||
func fakeWebWith(t *testing.T, wikidataLink, italian string) WikipediaSource {
|
||||
t.Helper()
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
switch {
|
||||
|
|
@ -27,7 +37,13 @@ func fakeWeb(t *testing.T, wikidataLink string) WikipediaSource {
|
|||
case strings.HasPrefix(r.URL.Path, "/ws/2/artist/"):
|
||||
http.NotFound(w, r)
|
||||
case r.URL.Path == "/w/api.php":
|
||||
_, _ = w.Write([]byte(`{"entities":{"Q44190":{"sitelinks":{"itwiki":{"title":"Radiohead"},"enwiki":{"title":"Radiohead"}}}}}`))
|
||||
_, _ = w.Write([]byte(`{"entities":{"Q44190":{"id":"Q44190","sitelinks":{"itwiki":{"title":"Radiohead"},"enwiki":{"title":"Radiohead"}}}}}`))
|
||||
case r.URL.Path == "/it/summary/Radiohead" && italian == "redirect":
|
||||
http.Redirect(w, r, "https://evil.example/", http.StatusFound)
|
||||
case r.URL.Path == "/it/summary/Radiohead" && italian == "noextract":
|
||||
_, _ = w.Write([]byte(`{"type":"no-extract","title":"Radiohead","lang":"it","extract":"","wikibase_item":"Q44190","content_urls":{"desktop":{"page":"https://it.wikipedia.org/wiki/Radiohead"}}}`))
|
||||
case r.URL.Path == "/en/summary/Radiohead":
|
||||
_, _ = w.Write([]byte(`{"type":"standard","title":"Radiohead","lang":"en","extract":"Text.","description":"band","wikibase_item":"Q44190","content_urls":{"desktop":{"page":"https://en.wikipedia.org/wiki/Radiohead"}}}`))
|
||||
case r.URL.Path == "/it/summary/Radiohead":
|
||||
_, _ = w.Write([]byte(`{"type":"standard","title":"Radiohead","lang":"it","extract":"Testo.","description":"gruppo","wikibase_item":"Q44190","content_urls":{"desktop":{"page":"https://it.wikipedia.org/wiki/Radiohead"}}}`))
|
||||
default:
|
||||
|
|
@ -49,9 +65,9 @@ func TestTheRealClientsFollowTheChain(t *testing.T) {
|
|||
if err != nil || item != "Q44190" {
|
||||
t.Fatalf("item = %q err = %v", item, err)
|
||||
}
|
||||
pages, err := src.Pages(ctx, item, []string{"it", "en"})
|
||||
if err != nil || pages["it"] != "Radiohead" || pages["en"] != "Radiohead" {
|
||||
t.Fatalf("pages = %v err = %v", pages, err)
|
||||
resolved, pages, err := src.Pages(ctx, item, []string{"it", "en"})
|
||||
if err != nil || resolved != "Q44190" || pages["it"] != "Radiohead" || pages["en"] != "Radiohead" {
|
||||
t.Fatalf("item %q pages = %v err = %v", resolved, pages, err)
|
||||
}
|
||||
sum, err := src.Summary(ctx, "it", "Radiohead")
|
||||
if err != nil || sum.WikibaseItem != "Q44190" || sum.Extract != "Testo." || sum.Type != "standard" || sum.Description != "gruppo" {
|
||||
|
|
@ -75,3 +91,21 @@ func TestWhatThereIsNothingToFindIsNotAFailure(t *testing.T) {
|
|||
t.Errorf("no page: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// With the real clients, a page that cannot be used (a redirect that is never followed, a
|
||||
// page with no text) is an answer: the chain goes on to the next language, and the outcome
|
||||
// is stored, so the same page is not asked about again at every view.
|
||||
func TestAPageThatCannotBeUsedFallsThroughToTheNextLanguage(t *testing.T) {
|
||||
for _, italian := range []string{"redirect", "noextract"} {
|
||||
store := newStore()
|
||||
svc := NewService(store, fakeWebWith(t, "https://www.wikidata.org/wiki/Q44190", italian),
|
||||
&switches{m: map[string]bool{providers.Wikipedia: true, providers.MusicBrainz: true}}, []string{"it", "en"}, zap.NewNop())
|
||||
res, err := svc.Description(context.Background(), 1)
|
||||
if err != nil || res.Status != discovery.StatusOK || res.Description == nil || res.Description.Language != "en" {
|
||||
t.Errorf("Italian page %s: result = %+v err = %v, want the English description", italian, res, err)
|
||||
}
|
||||
if len(store.saves) != 1 {
|
||||
t.Errorf("Italian page %s: %d saves, want the outcome stored once", italian, len(store.saves))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -117,8 +117,8 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
|
|||
type noWikipedia struct{}
|
||||
|
||||
func (noWikipedia) WikidataID(context.Context, string) (string, error) { return "", nil }
|
||||
func (noWikipedia) Pages(context.Context, string, []string) (map[string]string, error) {
|
||||
return nil, about.ErrSourceNotFound
|
||||
func (noWikipedia) Pages(context.Context, string, []string) (string, map[string]string, error) {
|
||||
return "", nil, about.ErrSourceNotFound
|
||||
}
|
||||
func (noWikipedia) Summary(context.Context, string, string) (about.Summary, error) {
|
||||
return about.Summary{}, about.ErrSourceNotFound
|
||||
|
|
|
|||
|
|
@ -420,7 +420,7 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
// ---- providers ----
|
||||
listed := c.expect(c.call("GET", "/providers", token, nil), 200, "list providers")["providers"].([]any)
|
||||
if len(listed) != 3 || listed[0].(map[string]any)["enabled"] != false || listed[1].(map[string]any)["enabled"] != false || listed[2].(map[string]any)["enabled"] != false {
|
||||
t.Fatalf("providers = %v, want two, both off", listed)
|
||||
t.Fatalf("providers = %v, want three, all off", listed)
|
||||
}
|
||||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"enabled": true}), 400, "enable without a key", "invalid_input")
|
||||
c.expect(c.call("PUT", "/providers/lastfm", token, map[string]any{"api_key": "short"}), 400, "a bad key", "invalid_input")
|
||||
|
|
@ -542,7 +542,9 @@ func TestResponsesMatchTheSpec(t *testing.T) {
|
|||
t.Fatalf("description = %v", desc)
|
||||
}
|
||||
c.expect(c.call("PUT", "/providers/wikipedia", token, map[string]any{"enabled": true, "contact": "https://example.org/me"}), 200, "switch Wikipedia on")
|
||||
c.expect(c.call("GET", "/artists/"+artistID+"/description", token, nil), 200, "description of an artist without an id")
|
||||
if noID := c.expect(c.call("GET", "/artists/"+artistID+"/description", token, nil), 200, "description of an artist without an id"); noID["provider_status"] != "no_id" {
|
||||
t.Fatalf("description = %v, want no_id", noID)
|
||||
}
|
||||
c.expect(c.call("GET", "/artists/999/description", token, nil), 404, "description of an unknown artist", "artist_not_found")
|
||||
c.expect(c.call("GET", "/artists/abc/description", token, nil), 400, "description of a bad id", "invalid_input")
|
||||
c.expect(c.call("GET", "/artists/"+artistID+"/description", "", nil), 401, "description without a token", "unauthorized")
|
||||
|
|
|
|||
|
|
@ -34,11 +34,11 @@ func (f *fakeWikipedia) WikidataID(_ context.Context, id string) (string, error)
|
|||
return f.item, nil
|
||||
}
|
||||
|
||||
func (f *fakeWikipedia) Pages(context.Context, string, []string) (map[string]string, error) {
|
||||
func (f *fakeWikipedia) Pages(_ context.Context, item string, _ []string) (string, map[string]string, error) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.calls++
|
||||
return f.pages, nil
|
||||
return item, f.pages, nil
|
||||
}
|
||||
|
||||
func (f *fakeWikipedia) Summary(_ context.Context, lang, _ string) (about.Summary, error) {
|
||||
|
|
@ -77,10 +77,13 @@ func (e *descEnv) path(id int64) string {
|
|||
return "/api/v1/artists/" + strconv.FormatInt(id, 10) + "/description"
|
||||
}
|
||||
|
||||
// enable switches on Wikipedia and MusicBrainz: the link to Wikidata comes from the second.
|
||||
func (e *descEnv) enable(t *testing.T) {
|
||||
t.Helper()
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/wikipedia", e.admin, map[string]any{"enabled": true}); r.Code != http.StatusOK {
|
||||
t.Fatalf("enable wikipedia: %d %v", r.Code, r.Body)
|
||||
for _, name := range []string{"wikipedia", "musicbrainz"} {
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/"+name, e.admin, map[string]any{"enabled": true}); r.Code != http.StatusOK {
|
||||
t.Fatalf("enable %s: %d %v", name, r.Code, r.Body)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -198,3 +201,15 @@ func TestPurgingWikipediaTakesTheDescriptionsBack(t *testing.T) {
|
|||
t.Errorf("after the purge: %v", r.Body)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDescriptionNeedsMusicBrainzOnToo(t *testing.T) {
|
||||
e := newDescEnv(t)
|
||||
e.giveAlphaAnID(t)
|
||||
if r := do(t, e.h, "PUT", "/api/v1/providers/wikipedia", e.admin, map[string]any{"enabled": true}); r.Code != http.StatusOK {
|
||||
t.Fatalf("enable wikipedia: %d", r.Code)
|
||||
}
|
||||
r := do(t, e.h, "GET", e.path(e.alpha), e.admin, nil)
|
||||
if r.Code != http.StatusOK || r.Body["provider_status"] != "disabled" || r.Body["description"] != nil || e.wp.count() != 0 {
|
||||
t.Errorf("%d %v calls %d: with MusicBrainz off nothing may be sent to it", r.Code, r.Body, e.wp.count())
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -939,14 +939,17 @@ paths:
|
|||
A short text about the artist from Wikipedia, with the page it comes from. The way to it
|
||||
is a chain of identifiers, never a search by name: the artist's MusicBrainz id, then the
|
||||
Wikidata item that MusicBrainz links to it, then the Wikipedia page that Wikidata names in
|
||||
the first of the server's languages (`[wikipedia] languages`) that has one. A page is used
|
||||
the first of the server's languages (`[wikipedia] languages`, the code of the Wikipedia:
|
||||
`it`, `en`, `no` for Norwegian; a language with no Wikipedia finds nothing) that has one. A page is used
|
||||
only if it is an article about that same item (not a page of disambiguation). An artist
|
||||
with no MusicBrainz id answers `no_id`: Sonda does not look an id up by name.
|
||||
|
||||
It needs the `wikipedia` provider, which an admin switches on; nothing is sent while it is
|
||||
off (`disabled`). What is sent: the MusicBrainz id to MusicBrainz, the Wikidata item to
|
||||
Wikidata, the title of the page to Wikipedia, and your server's IP address and a
|
||||
`User-Agent` with the contact of the admin to all three. The answer is stored: 30 days, or a
|
||||
It needs the `wikipedia` provider and also the `musicbrainz` one, because the link to
|
||||
Wikidata comes from MusicBrainz; an admin switches them on, and nothing is sent while
|
||||
either is off (`disabled`, also when an answer is stored). What is sent: the MusicBrainz id
|
||||
to MusicBrainz (with the contact set for MusicBrainz), the Wikidata item to Wikidata and the
|
||||
title of the page to Wikipedia (with the contact set for Wikipedia), and your server's IP
|
||||
address with them. The answer is stored: 30 days, or a
|
||||
day when there was nothing to find, and an old one is returned as `stale` if Wikipedia
|
||||
fails. When there is nothing to show, `description` is `null` and `provider_status` is
|
||||
`ok`.
|
||||
|
|
@ -1886,6 +1889,7 @@ components:
|
|||
retrieved_at: {type: string, format: date-time, nullable: true, description: When it was read; null when nothing is known.}
|
||||
attribution: {type: string, description: What to show where the text is shown.}
|
||||
description:
|
||||
type: object
|
||||
nullable: true
|
||||
allOf:
|
||||
- {$ref: '#/components/schemas/Description'}
|
||||
|
|
|
|||
|
|
@ -108,10 +108,11 @@ var known = []meta{
|
|||
},
|
||||
{
|
||||
name: Wikipedia,
|
||||
sends: "The MusicBrainz id of the artist you look up goes to MusicBrainz (never a name: Sonda does not search); " +
|
||||
"the Wikidata item that MusicBrainz links to it goes to Wikidata, and the title of the Wikipedia page that " +
|
||||
"Wikidata names goes to Wikipedia. Your IP address and the User-Agent, which carries the contact below, go " +
|
||||
"to all three. Never your users, your listens, track titles or file paths.",
|
||||
sends: "It needs the MusicBrainz provider on too. The MusicBrainz id of the artist you look up goes to MusicBrainz " +
|
||||
"(never a name: Sonda does not search), with the contact set for MusicBrainz; the Wikidata item that " +
|
||||
"MusicBrainz links to it goes to Wikidata, and the title of the Wikipedia page that Wikidata names goes to " +
|
||||
"Wikipedia, with the contact below in the User-Agent. Your IP address goes to all three. Never your users, " +
|
||||
"your listens, track titles or file paths.",
|
||||
note: "Wikipedia text is CC BY-SA 4.0: show the name of the page, a link to it and the licence next to " +
|
||||
"the text, and say if you changed it.",
|
||||
},
|
||||
|
|
|
|||
|
|
@ -88,17 +88,19 @@ type Summary struct {
|
|||
}
|
||||
|
||||
// Pages returns, for each of the languages that has one, the title of the Wikipedia
|
||||
// page of a Wikidata item. A language with no page is left out; an item with no page
|
||||
// at all is an empty map, not an error.
|
||||
func (c *Client) Pages(ctx context.Context, item string, langs []string) (map[string]string, error) {
|
||||
// page of a Wikidata item, and the item the answer is about: the same one, or, for an
|
||||
// item that was merged into another, the other (Wikidata answers with the item asked
|
||||
// for as the key and the target as the entity's own id). A language with no page is
|
||||
// left out; an item with no page at all is an empty map, not an error.
|
||||
func (c *Client) Pages(ctx context.Context, item string, langs []string) (string, map[string]string, error) {
|
||||
if !itemRe.MatchString(item) || len(langs) == 0 || len(langs) > MaxLanguages {
|
||||
return nil, ErrBadRequest
|
||||
return "", nil, ErrBadRequest
|
||||
}
|
||||
sites := make([]string, len(langs))
|
||||
asked := map[string]bool{}
|
||||
for i, l := range langs {
|
||||
if !langRe.MatchString(l) || asked[l] {
|
||||
return nil, ErrBadRequest
|
||||
return "", nil, ErrBadRequest
|
||||
}
|
||||
asked[l] = true
|
||||
sites[i] = l + "wiki"
|
||||
|
|
@ -112,35 +114,41 @@ func (c *Client) Pages(ctx context.Context, item string, langs []string) (map[st
|
|||
q.Set("format", "json")
|
||||
body, err := c.get(ctx, c.wikidata+"?"+q.Encode())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return "", nil, err
|
||||
}
|
||||
var doc struct {
|
||||
Error *struct {
|
||||
Code string `json:"code"`
|
||||
} `json:"error"`
|
||||
Entities map[string]struct {
|
||||
ID string `json:"id"`
|
||||
Sitelinks map[string]struct {
|
||||
Title string `json:"title"`
|
||||
} `json:"sitelinks"`
|
||||
} `json:"entities"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil {
|
||||
return nil, ErrBadResponse
|
||||
return "", nil, ErrBadResponse
|
||||
}
|
||||
if doc.Error != nil { // Wikidata answers 200 to a failed request
|
||||
switch doc.Error.Code {
|
||||
case "no-such-entity":
|
||||
return nil, ErrNotFound
|
||||
return "", nil, ErrNotFound
|
||||
case "maxlag":
|
||||
return nil, fmt.Errorf("%w: lagged", ErrUnavailable)
|
||||
return "", nil, fmt.Errorf("%w: lagged", ErrUnavailable)
|
||||
}
|
||||
return nil, ErrBadResponse
|
||||
return "", nil, ErrBadResponse
|
||||
}
|
||||
if len(doc.Entities) != 1 { // a merged item comes under the new id: take the only one
|
||||
return nil, ErrBadResponse
|
||||
if len(doc.Entities) != 1 { // a merged item comes under the key asked for: take the only one
|
||||
return "", nil, ErrBadResponse
|
||||
}
|
||||
out := map[string]string{}
|
||||
resolved := ""
|
||||
for _, entity := range doc.Entities {
|
||||
if !itemRe.MatchString(entity.ID) { // it is compared with a page's item later: it must be one
|
||||
return "", nil, ErrBadResponse
|
||||
}
|
||||
resolved = entity.ID
|
||||
for _, l := range langs { // only what was asked for, whatever the answer holds
|
||||
if sl, ok := entity.Sitelinks[l+"wiki"]; ok {
|
||||
if t := strings.TrimSpace(sl.Title); t != "" && validTitle(t) {
|
||||
|
|
@ -149,7 +157,7 @@ func (c *Client) Pages(ctx context.Context, item string, langs []string) (map[st
|
|||
}
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
return resolved, out, nil
|
||||
}
|
||||
|
||||
// Summary reads the summary of a page. ErrNotFound means the page does not exist.
|
||||
|
|
@ -158,6 +166,9 @@ func (c *Client) Summary(ctx context.Context, lang, title string) (Summary, erro
|
|||
return Summary{}, ErrBadRequest
|
||||
}
|
||||
body, err := c.get(ctx, fmt.Sprintf(c.summary, lang)+url.PathEscape(strings.ReplaceAll(title, " ", "_")))
|
||||
if errors.Is(err, outbound.ErrRedirect) {
|
||||
return Summary{}, ErrNotFound // never followed, and not a failure of the service: no usable page
|
||||
}
|
||||
if err != nil {
|
||||
return Summary{}, err
|
||||
}
|
||||
|
|
@ -174,25 +185,29 @@ func (c *Client) Summary(ctx context.Context, lang, title string) (Summary, erro
|
|||
} `json:"desktop"`
|
||||
} `json:"content_urls"`
|
||||
}
|
||||
if err := json.Unmarshal(body, &doc); err != nil || strings.TrimSpace(doc.Extract) == "" {
|
||||
if err := json.Unmarshal(body, &doc); err != nil {
|
||||
return Summary{}, ErrBadResponse
|
||||
}
|
||||
// The address is shown to people as the source: it must be on that Wikipedia.
|
||||
if !strings.HasPrefix(doc.ContentURLs.Desktop.Page, "https://"+lang+".wikipedia.org/wiki/") {
|
||||
return Summary{}, ErrBadResponse
|
||||
// What the page says is returned as it is, even a page with no text or of another kind:
|
||||
// whether it can be used is for the caller to say, and an unusable page is an answer, not
|
||||
// a failure that would be asked about again at every view.
|
||||
// The address is shown to people as the source: it must be on that Wikipedia, or it is dropped.
|
||||
link := doc.ContentURLs.Desktop.Page
|
||||
if !strings.HasPrefix(link, "https://"+lang+".wikipedia.org/wiki/") {
|
||||
link = ""
|
||||
}
|
||||
return Summary{
|
||||
Type: doc.Type, Title: doc.Title, Lang: doc.Lang, WikibaseItem: doc.WikibaseItem,
|
||||
Extract: truncate(strings.TrimSpace(doc.Extract), maxExtractLen),
|
||||
Description: truncate(strings.TrimSpace(doc.Description), maxShortLen),
|
||||
URL: doc.ContentURLs.Desktop.Page,
|
||||
Type: doc.Type, Title: clean(doc.Title, false), Lang: doc.Lang, WikibaseItem: doc.WikibaseItem,
|
||||
Extract: truncate(strings.TrimSpace(clean(doc.Extract, true)), maxExtractLen),
|
||||
Description: truncate(strings.TrimSpace(clean(doc.Description, false)), maxShortLen),
|
||||
URL: link,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Probe asks about one public item, to find out whether the service answers. It is a
|
||||
// providers.Checker, and answers with the errors the settings understand.
|
||||
func (c *Client) Probe(ctx context.Context) error {
|
||||
_, err := c.Pages(ctx, probeItem, []string{"en"})
|
||||
_, _, err := c.Pages(ctx, probeItem, []string{"en"})
|
||||
switch {
|
||||
case errors.Is(err, ErrRateLimited):
|
||||
return providers.ErrRateLimited
|
||||
|
|
@ -214,6 +229,17 @@ func validTitle(t string) bool {
|
|||
return true
|
||||
}
|
||||
|
||||
// clean drops the control characters of a text (a NUL is refused by PostgreSQL), keeping
|
||||
// line breaks when asked.
|
||||
func clean(s string, keepNewlines bool) string {
|
||||
return strings.Map(func(r rune) rune {
|
||||
if unicode.IsControl(r) && !(keepNewlines && r == '\n') {
|
||||
return -1
|
||||
}
|
||||
return r
|
||||
}, s)
|
||||
}
|
||||
|
||||
func truncate(s string, max int) string {
|
||||
if utf8.RuneCountInString(s) <= max {
|
||||
return s
|
||||
|
|
|
|||
|
|
@ -55,13 +55,13 @@ const entityBody = `{"entities":{"Q44190":{"type":"item","id":"Q44190","sitelink
|
|||
|
||||
func TestPagesReturnsTheTitleForEachLanguageAsked(t *testing.T) {
|
||||
s := newServer(t, 200, entityBody)
|
||||
got, err := s.client().Pages(context.Background(), "Q44190", []string{"it", "en"})
|
||||
item, got, err := s.client().Pages(context.Background(), "Q44190", []string{"it", "en"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Only the languages asked for, even if the answer holds more.
|
||||
if len(got) != 2 || got["it"] != "Radiohead" || got["en"] != "Radiohead" {
|
||||
t.Errorf("pages = %v", got)
|
||||
if item != "Q44190" || len(got) != 2 || got["it"] != "Radiohead" || got["en"] != "Radiohead" {
|
||||
t.Errorf("item %q pages = %v", item, got)
|
||||
}
|
||||
reqs := s.requests()
|
||||
if len(reqs) != 1 || !strings.Contains(reqs[0], "ids=Q44190") || !strings.Contains(reqs[0], "sitefilter=itwiki%7Cenwiki") ||
|
||||
|
|
@ -70,17 +70,23 @@ func TestPagesReturnsTheTitleForEachLanguageAsked(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestPagesAcceptsAnEntityUnderAnotherKeyAndLeavesOutWhatIsMissing(t *testing.T) {
|
||||
// A merged item comes back under the id that was asked for or the new one: take the only entity.
|
||||
body := `{"entities":{"Q2":{"type":"item","id":"Q2","sitelinks":{"itwiki":{"site":"itwiki","title":"Terra"}}}},"success":1}`
|
||||
got, err := newServer(t, 200, body).client().Pages(context.Background(), "Q1", []string{"it", "en"})
|
||||
if err != nil || len(got) != 1 || got["it"] != "Terra" {
|
||||
t.Errorf("pages = %v err = %v", got, err)
|
||||
func TestAMergedItemAnswersWithTheItemItWasMergedInto(t *testing.T) {
|
||||
// Verified on the real API: the entity is keyed by the id that was asked for, but its
|
||||
// "id" is the target, and "redirects" says so. The pages belong to the target.
|
||||
body := `{"entities":{"Q19746657":{"redirects":{"from":"Q19746657","to":"Q1000"},"type":"item","id":"Q1000","sitelinks":{"itwiki":{"site":"itwiki","title":"Gabon"}}}},"success":1}`
|
||||
item, got, err := newServer(t, 200, body).client().Pages(context.Background(), "Q19746657", []string{"it", "en"})
|
||||
if err != nil || item != "Q1000" || len(got) != 1 || got["it"] != "Gabon" {
|
||||
t.Errorf("item %q pages = %v err = %v, want the target item Q1000", item, got, err)
|
||||
}
|
||||
// An entity whose id is not an item id is not believed.
|
||||
bad := `{"entities":{"Q1":{"id":"../x","sitelinks":{"itwiki":{"title":"A"}}}},"success":1}`
|
||||
if _, _, err := newServer(t, 200, bad).client().Pages(context.Background(), "Q1", []string{"it"}); !errors.Is(err, ErrBadResponse) {
|
||||
t.Errorf("a bad entity id: err = %v", err)
|
||||
}
|
||||
// No page in any of the languages is an empty answer, not an error.
|
||||
got, err = newServer(t, 200, `{"entities":{"Q1":{"type":"item","id":"Q1","sitelinks":{}}},"success":1}`).client().Pages(context.Background(), "Q1", []string{"it"})
|
||||
if err != nil || len(got) != 0 {
|
||||
t.Errorf("no pages: %v err = %v", got, err)
|
||||
item, got, err = newServer(t, 200, `{"entities":{"Q1":{"type":"item","id":"Q1","sitelinks":{}}},"success":1}`).client().Pages(context.Background(), "Q1", []string{"it"})
|
||||
if err != nil || item != "Q1" || len(got) != 0 {
|
||||
t.Errorf("no pages: %q %v err = %v", item, got, err)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -105,7 +111,7 @@ func TestPagesRefusesWhatCouldMakeWikidataSendEverything(t *testing.T) {
|
|||
"an injection": {"Q44190&action=x", []string{"it"}},
|
||||
"empty": {"", []string{"it"}},
|
||||
} {
|
||||
if _, err := c.Pages(context.Background(), tt.qid, tt.langs); !errors.Is(err, ErrBadRequest) {
|
||||
if _, _, err := c.Pages(context.Background(), tt.qid, tt.langs); !errors.Is(err, ErrBadRequest) {
|
||||
t.Errorf("%s: err = %v, want ErrBadRequest", name, err)
|
||||
}
|
||||
}
|
||||
|
|
@ -130,7 +136,7 @@ func TestPagesErrors(t *testing.T) {
|
|||
"not json": {200, `<html>`, ErrBadResponse},
|
||||
"no entities": {200, `{"success":1}`, ErrBadResponse},
|
||||
} {
|
||||
if _, err := newServer(t, tt.status, tt.body).client().Pages(context.Background(), "Q9", langs); !errors.Is(err, tt.want) {
|
||||
if _, _, err := newServer(t, tt.status, tt.body).client().Pages(context.Background(), "Q9", langs); !errors.Is(err, tt.want) {
|
||||
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
|
||||
}
|
||||
}
|
||||
|
|
@ -194,14 +200,52 @@ func TestSummaryRefusesBadInput(t *testing.T) {
|
|||
}
|
||||
|
||||
func TestSummaryKeepsTheLinkOnWikipediaOnly(t *testing.T) {
|
||||
// A page address must be on that language's Wikipedia: it is shown to people as the source.
|
||||
body := strings.Replace(summaryBody, "https://it.wikipedia.org/wiki/Radiohead", "https://evil.example/wiki/Radiohead", 1)
|
||||
if _, err := newServer(t, 200, body).client().Summary(context.Background(), "it", "Radiohead"); !errors.Is(err, ErrBadResponse) {
|
||||
t.Errorf("err = %v, want ErrBadResponse", err)
|
||||
// A page address must be on that language's Wikipedia: it is shown to people as the
|
||||
// source. One that is not is dropped (empty), and the caller decides what that means.
|
||||
for name, link := range map[string]string{
|
||||
"another host": "https://evil.example/wiki/Radiohead",
|
||||
"a script address": "javascript:alert(1)",
|
||||
"another language": "https://en.wikipedia.org/wiki/Radiohead",
|
||||
"a look-alike host": "https://it.wikipedia.org.evil.example/wiki/Radiohead",
|
||||
} {
|
||||
body := strings.Replace(summaryBody, "https://it.wikipedia.org/wiki/Radiohead", link, 1)
|
||||
got, err := newServer(t, 200, body).client().Summary(context.Background(), "it", "Radiohead")
|
||||
if err != nil || got.URL != "" {
|
||||
t.Errorf("%s: URL = %q err = %v, want the address dropped and no error", name, got.URL, err)
|
||||
}
|
||||
}
|
||||
body = strings.Replace(summaryBody, "https://it.wikipedia.org/wiki/Radiohead", "javascript:alert(1)", 1)
|
||||
if _, err := newServer(t, 200, body).client().Summary(context.Background(), "it", "Radiohead"); !errors.Is(err, ErrBadResponse) {
|
||||
t.Errorf("a script address: err = %v", err)
|
||||
}
|
||||
|
||||
// A page that cannot be used is an answer, not a failure: the caller goes on to the next
|
||||
// language, and the same page is not asked about again at every view.
|
||||
func TestSummaryGivesBackPagesWithoutTextAndTheCallerDecides(t *testing.T) {
|
||||
body := `{"type":"no-extract","title":"X","lang":"it","extract":"","wikibase_item":"Q1","content_urls":{"desktop":{"page":"https://it.wikipedia.org/wiki/X"}}}`
|
||||
got, err := newServer(t, 200, body).client().Summary(context.Background(), "it", "X")
|
||||
if err != nil || got.Type != "no-extract" || got.Extract != "" {
|
||||
t.Errorf("summary = %+v err = %v, want it returned as it is", got, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummaryTakesARedirectAsNoPage(t *testing.T) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Redirect(w, r, "https://evil.example/", http.StatusFound)
|
||||
}))
|
||||
defer s.Close()
|
||||
c := New(outbound.New(outbound.Options{UserAgent: "Sonda/test ( https://example.org )", Retries: outbound.NoRetries}), s.URL, s.URL+"/%s/summary/")
|
||||
if _, err := c.Summary(context.Background(), "it", "X"); !errors.Is(err, ErrNotFound) {
|
||||
t.Errorf("err = %v, want ErrNotFound: a redirect is never followed and is not a failure of the service", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSummaryCleansTheText(t *testing.T) {
|
||||
body := `{"type":"standard","title":"X","lang":"it","extract":"Uno\u0000 due\u0007\ntre","description":"a\u0000b","wikibase_item":"Q1","content_urls":{"desktop":{"page":"https://it.wikipedia.org/wiki/X"}}}`
|
||||
got, err := newServer(t, 200, body).client().Summary(context.Background(), "it", "X")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// No NUL (PostgreSQL refuses it in text), no other control character; a line break stays.
|
||||
if got.Extract != "Uno due\ntre" || got.Description != "ab" {
|
||||
t.Errorf("extract = %q description = %q", got.Extract, got.Description)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -216,7 +260,6 @@ func TestSummaryErrors(t *testing.T) {
|
|||
"forbidden": {403, `blocked`, ErrBadResponse},
|
||||
"server error": {500, ``, ErrUnavailable},
|
||||
"not json": {200, `<html>`, ErrBadResponse},
|
||||
"no text": {200, `{"type":"standard","title":"X"}`, ErrBadResponse},
|
||||
} {
|
||||
if _, err := newServer(t, tt.status, tt.body).client().Summary(context.Background(), "it", "X"); !errors.Is(err, tt.want) {
|
||||
t.Errorf("%s: err = %v, want %v", name, err, tt.want)
|
||||
|
|
@ -226,7 +269,7 @@ func TestSummaryErrors(t *testing.T) {
|
|||
|
||||
func TestErrorsDoNotCopyTheProvidersText(t *testing.T) {
|
||||
secret := "SECRET-FROM-THE-PROVIDER"
|
||||
_, err := newServer(t, 200, `{"error":{"code":"badtoken","info":"`+secret+`"}}`).client().Pages(context.Background(), "Q9", []string{"it"})
|
||||
_, _, err := newServer(t, 200, `{"error":{"code":"badtoken","info":"`+secret+`"}}`).client().Pages(context.Background(), "Q9", []string{"it"})
|
||||
if err == nil || strings.Contains(err.Error(), secret) {
|
||||
t.Errorf("err = %v", err)
|
||||
}
|
||||
|
|
|
|||
Loading…
Reference in a new issue