Commit graph sonda/internal/discovery
Author SHA1 Message Date
tgiachi
28c249c268 feat(discover): artists I may like, with a score I can explain
- GET /discover suggests artists, in my library or not, with a score from 0 to 1
  and its parts: affinity (similar to the artists I like), genre (tags close to
  mine), preference (what my listening says) and discovery (a bonus for what I
  have not played). Default weights 0.40, 0.25, 0.25, 0.10, set in [scoring].
- A part with no data (no tags) is left out and the others rescaled; the answer
  says which parts were there, and which seeds led to each suggestion.
- Seeds are the artists I name with ?seed=, or, only when I turned on
  discover_from_listening, the artists I play most. Without either the request is
  refused and nothing is sent to anyone.
- The tags of the best candidates are read from Last.fm only, only for those with
  none stored, a few at a time.
- taste.Service.Learn gives the whole picture (every preference, the liked
  artists, the tag profile); TagService.ProviderTags asks one provider.
- Spec, contract test, README and example config updated.

Closes #33
2026-11-11 19:42:09 +01:00
tgiachi
b0a7a562a6 fix(discovery): purge lock, cleanup chains and rescans that cannot read
- fix: on PostgreSQL a purge could delete a discovered artist while a scan was
  adopting it. The scan then failed on the missing artist for the whole file,
  and the two could also deadlock. Purge now takes the same artist lock as the
  scanner and discovery before it removes anything. Saving a list or tags does
  too, and an artist that is gone by then is reported as not found instead of a
  foreign key error, so a user who opened an artist that was removed meanwhile
  gets a 404
- fix: a discovered artist that was only the source of a list stayed forever and
  kept the artists of its own list alive, so chains built up. Cleanup now removes
  such an artist and then follows the chain until nothing more goes
- fix: in a full rescan a file that came back but could not be read stayed
  marked missing. It is restored first, as in a normal scan, and read as well
- fix: ?full given twice is refused, so a typo cannot start a scan of another
  kind
- fix: genre cleaning lives in one place, used by the reader and the store
- docs: the files genres count the tracks the artist is credited on, and the spec
  says every files tag is a genre
- test: a purge colliding with a scan on both databases (it fails without the
  lock), a chain of discovered artists, saving for an artist that is gone, a
  file that returns unreadable during a full scan
2026-11-11 17:36:14 +01:00
tgiachi
1857bb9c53 feat(discovery): remove provider data, genres from my files, full rescans
- feat: DELETE /api/v1/providers/{name}/data (admin) removes everything a
  provider gave: its lists of similar artists, its tags and the records of
  when it was asked, then the artists that only that data named and that
  nobody has files for. Last.fm's terms ask for this when an agreement ends.
  The user's own artists, music, listens and settings stay, the provider is not
  switched off, and a second call removes nothing
- feat: a discovered artist that no list names any more is removed when the
  list that named it is replaced, together with its tags, so they do not pile
  up. Real artists, and any artist that another list still names, are kept
- feat: the scan reads the GENRE tag of each file, offline (several genres
  separated by ";", lower case, at most five per file), and
  GET /artists/{id}/tags gets a third entry, files: the genres of the artist's
  own playable tracks, each weighted by how many tracks have it. It asks nobody
  and is always there
- feat: POST /library/scan and POST /libraries/{id}/scan take full=true to read
  every file again, not only the new or changed ones, so MusicBrainz ids and
  genres reach music that was indexed before. Files keep their identity, so
  listening history is kept. Anything but true or false is a 400
- test: the plan of a full scan, the scanner and the manager; the purge on both
  databases, checked by removing the rule that spares real artists; the
  cleanup of replaced lists; genres per file and per artist
- docs: README, OpenAPI and the todo list updated
2026-11-11 17:27:29 +01:00
tgiachi
175b87a4cc fix(discovery): ids from files, adoption and the scanner lock
- fix: a discovered artist adopted by id under another spelling kept its old
  match key, so the next file with the new spelling and no id created a second
  artist. The key now follows the name
- fix: when a real artist without an id and a discovered artist holding the id
  have the same name, the id now goes to the real artist and the discovered one
  lets go of it. Before, the scan created a second real artist and a second
  album, and could not repair it because the id was taken
- fix: when a file without an id adopts a discovered artist that had a provider
  id, the tags, the fetch records and the similar artists read under that id
  are dropped, since they are not known to be about the user's artist. Links
  from other artists to it stay. A file that confirms the very same id keeps
  everything
- fix: the album artist id is only read when the file has an album artist tag;
  with the tag missing the name is a fallback and the id would have landed on the
  wrong artist (a Various Artists id on the track's own artist)
- fix: one artist credit with one id ("X feat. Y" with the id of X, how beets
  writes it) is not paired: no id is safer than a wrong one. A name that is
  itself a single letter or contains a slash still gets its id
- fix: on PostgreSQL the scanner took the artist lock after it had already
  changed artist rows, which could deadlock with a concurrent similar-artists
  save. SaveFile now takes the lock first, once
- fix: Last.fm does not know every MusicBrainz id: when it does not know the id
  and there is a name, the name is tried, never autocorrected, as before ids
  were read from files
- fix: waiting for one provider's tags is bounded at 20 seconds, so a rate
  limited provider cannot hold a page for a minute
- docs: the tag range is 1 to 100 (a tag with no votes is dropped), the spec says
  where an artist's id can come from, and the README says when Last.fm gets an id
- test: adoption by spelling, the real artist with a discovered namesake,
  dropping and keeping what was read, album artist fallback, credits, an ID3
  frame, and the id-then-name fallback of Last.fm
2026-11-11 17:13:31 +01:00
tgiachi
07ea997cd0 feat(discovery): artist tags from Last.fm and MusicBrainz, and ids from my files
- feat: GET /api/v1/artists/{id}/tags lists, for each provider that is on, the
  tags it gives to an artist with its own weights, kept as they came: Last.fm's
  are relative to the artist, MusicBrainz's are counts of votes, and they are
  never mixed. MusicBrainz also marks the tags it lists as genres. The same
  rules as similar artists apply: asked only on demand, kept 7 days (1 for an
  unknown artist), old data served when a provider fails, and one call for
  simultaneous requests. A provider that is off, failing or lacking an id does
  not hide the other
- feat: a thin MusicBrainz client that reads an artist's tags and genres by its
  id. It never searches by name, because a search returns look-alikes and
  picking one would be a guess. The id is checked before it reaches the path,
  and errors never repeat the provider's text
- feat: the User-Agent can change while the server runs, so the contact an
  admin sets for MusicBrainz applies to the next request; if the settings
  cannot be read it falls back to the project address
- feat: Last.fm top tags, by id when known and by name otherwise, never
  autocorrected
- feat: the scan reads MusicBrainz artist ids from the file tags, offline, and
  keeps them on the artists. An id is only used when it can be paired with its
  artist for certain: the same number of ids as artists, all valid. The same id
  is the same artist whatever the spelling, an artist without an id takes the
  one its file gives, and another id under the same name is another artist. A
  file that confirms the id a provider gave to a discovered artist keeps it
- feat: POST /providers/musicbrainz/test checks that the service answers
- test: what leaves the server for MusicBrainz is checked from the other end:
  the id of the artist and the admin's contact, nothing from the library;
  removing the safeguards on ids and pairing makes tests fail
- docs: README, OpenAPI and the provider text updated
2026-11-11 17:01:43 +01:00
tgiachi
b769ad7ac5 fix(discovery): concurrent writes, adopted artists and a client that leaves
- fix: on SQLite a transaction that read and then wrote failed at once with
  "database is locked" when another writer committed in between, because the
  busy timeout does not apply there. Transactions now start with BEGIN
  IMMEDIATE. Before this change a scan running while someone opened similar
  artists made files fail, and parallel requests stored nothing
- fix: on PostgreSQL two requests naming the same new artist could fail on the
  unique MusicBrainz id or leave two rows with one name. Creating artists, in
  the discovery store and in the scanner, now takes turns through an advisory
  lock and looks again once it has the turn
- fix: music arriving for an artist that a provider only named used to keep the
  provider's spelling and its unchecked MusicBrainz id, which later lookups
  would then send instead of the name. Such artists are now marked
  (artists.discovered); the first scan that finds them renames them to the
  spelling of the file tags, drops the id and clears the mark
- fix: a panic in the shared provider call is turned into a failure; the
  singleflight group would have re-raised it where nothing can recover
- fix: a client that leaves while Last.fm is being asked no longer gets a 500
  and an error in the log
- fix: no page can ask for more than is stored (limit tops out at 50), and the
  docs no longer say the best score is always 1
- test: concurrent saves plus a scan on both databases, checked by removing each
  fix; adoption keeps neither the name nor the id; the panic; the client that
  leaves
2026-11-11 16:47:33 +01:00
tgiachi
aafb2dd906 feat(discovery): similar artists from Last.fm
- feat: GET /api/v1/artists/{id}/similar lists the artists Last.fm finds similar
  to one in the library, best first, each with the provider's own score and
  rank and a flag saying whether it is in the library. It answers 200 when the
  provider is off or failing, with provider_status, stale and retrieved_at
- feat: a thin Last.fm client on top of the outbound client. It always sends
  autocorrect=0 so a misspelled local name cannot turn into another artist,
  prefers the MusicBrainz id when known, always sets a limit, and reads the
  error field even when the status is 200. Errors never carry the URL or the
  provider's own message, because the key travels in the query string
- feat: results are stored in artist_similar with a freshness ledger in
  provider_fetches: 7 days for an answer, 1 day for "unknown artist". Old data
  is served when the provider fails, and concurrent requests for the same
  artist share one call that does not depend on the first caller staying
- feat: artists named by a provider but without files become artists without
  tracks. GET /artists/{id} can show them (track_count 0), the lists of the
  library never include them, and a later scan adopts them by name. A provider
  id is never written onto an existing artist; names match only when at least
  one side has no id, and different ids always mean different artists
- feat: POST /providers/{name}/test (admin) checks a stored key with one cheap
  request, whether the provider is on or off, and reports ok, no_api_key,
  key_refused, rate_limited, unreachable or error
- test: the whole request is checked from the receiving end, with a user, a
  listen, a title and a library path in the database: only the artist and the
  key leave, with a User-Agent that names the project
- docs: README and OpenAPI updated; the contract test calls the new operations
2026-11-11 16:42:52 +01:00