Commit graph sonda/internal/storage/library.go
Author SHA1 Message Date
tgiachi
a880b87a9e chore: move the module path to git.kmalloc.org/squid/sonda
Migrating hosting from Codeberg to git.kmalloc.org. Renames the
module path in go.mod, every internal import, and the OCI image
source label in the Dockerfile. Build and full test suite verified
green after the rename.
2026-11-15 17:30:29 +01:00
tgiachi
a92326c1b4 feat(player): play every song at a similar volume with ReplayGain
I want songs from different albums to play at about the same loudness without touching the volume.

- The scan reads REPLAYGAIN_TRACK_GAIN, REPLAYGAIN_ALBUM_GAIN and the two peaks from each file, offline, and keeps them on the file (migration 0029, SQLite and PostgreSQL). A tag that makes no sense (not a number, more than 60 dB) is left out. Files indexed before have none until a full rescan.
- A track in the API carries them as replay_gain (null when its first playable file says nothing).
- The web player levels each song with them: the sound passes through a gain node, so a quiet song can be raised too, but never so much that its peak would clip (and never more than 12 dB when there is no peak). Without Web Audio the element is turned down instead.
- A new Playback section in Settings chooses song by song (default), album by album or off; the choice stays on the device.
- README: the ReplayGain entry, and the old line that said the year was not in the index is removed.
- English and Italian.
2026-11-14 16:02:23 +01:00
tgiachi
d3695d00d6 feat(smart): smart playlists by the year a song came out
I can now ask for songs from a year to a year, for example the eighties.

- The year is read from the tags of each file (the original date first, then the date) and kept on the track: migration 0028 adds tracks.year (SQLite and PostgreSQL).
- Two new rules, min_year and max_year, both included and checked (1000 to 2999, the first not after the second). The index answers them. A song with no year never matches.
- Files indexed before have no year until a full rescan reads their tags again; the editor and the API description say so.
- The smart playlist editor gets From year and Up to year, and the rules are described in words.
- English and Italian.
2026-11-14 10:19:39 +01:00
tgiachi
89826fcd07 feat(federation): shared libraries and the signed read API for friends
A friend's server can now read what an admin chose to share.

- A library has a shared flag, off by default (migration 0024), set with
  PATCH /libraries/{id}. Changing it never waits for a scan.
- Requests between friends are signed with each server's Ed25519 key: the
  method, the path and query from /api/v1/peer on, a date, a nonce and the
  hash of the empty body. A date more than two minutes off, a nonce used
  before, an unknown or removed key and a bad signature all get the same
  401. A friend still waiting for approval gets 403, and while federation is
  off the routes are missing.
- Routes under /api/v1/peer: info, artists, albums, tracks, covers and
  streaming with Range. The catalogue, the counts and the format come from
  the shared libraries only, and the file served is the one in a shared
  library. No path, library id, listen, taste, user or provider data is part
  of an answer.
- A friend never makes this server build a cover, read an audio file or ask
  a provider: only covers already built from the audio files are served.
- Per-friend limits, set by the admin: parallel streams (default 3) and KiB
  a second (default 4096, 0 for none), shared by all the streams of a friend.
- A log of the last requests friends made (route, status, bytes; never the
  query), kept for 7 days, in GET /federation/requests.
- POST /federation/friends/{id}/check lets the side that pasted an invite
  find out that the other admin approved; a signed call from a friend this
  server is waiting for makes it active too.
- Admin: a Shared with friends switch on each library, which asks first, and
  limits, recent requests and a Check button in the Federation section, in
  English and Italian.
- OpenAPI paths and schemas, contract test, two real servers checked end to
  end with tagged audio files.
2026-11-13 15:04:07 +01:00
tgiachi
65edfc4918 feat(storage): candidates for duplicates, file fingerprints and removing a copy from the index
Candidates are the playable files whose title another file shares. A fingerprint is kept only for the file that was read (same size and date), and a scan that finds the file changed forgets it. Removing a copy moves its listens to the track that stays, because they would be deleted with the track, then removes the track with its file, credits and genres. Tested on SQLite and PostgreSQL.
2026-11-12 16:37:15 +01:00
tgiachi
77be31ea4a perf(storage): fix stale match and search keys in batches on PostgreSQL
The refresh at start fixed each stale row with its own UPDATE. On PostgreSQL that
is one round trip per row: 100,000 stale rows took 5.3 s on localhost, and more
over a network. The rows now go 500 at a time in one UPDATE ... FROM (VALUES ...):
1.9 s for the same rows. SQLite keeps one UPDATE per row, which is faster there
(no round trip to save). A run with nothing to fix is unchanged (about 0.1 s per
100,000 rows).

A test with 1,250 stale rows covers more than two batches on both databases.
2026-11-12 14:08:40 +01:00
tgiachi
271c88743c feat(search): search ignores accents
- A search key apart from the key that identifies artists, albums and tracks:
  lower case, accents taken off (Latin, Greek and Cyrillic marks only, so kana
  and Hangul stay intact), and letters like ß, æ, ø, ł written plain.
- GET /artists, /albums and /tracks look in it, so "bjork" finds "Björk" and
  "strasse" finds "Straße".
- "Beyoncé" and "Beyonce" stay two artists: the identity key keeps accents.
- Migration 0016 adds the column on SQLite and PostgreSQL; RefreshMatchKeys
  fills it for existing rows at start.
- golang.org/x/text is now a direct dependency.
2026-11-12 10:17:21 +01:00
tgiachi
5fecd394e4 fix(images): purge removes every archive cover, the id of an emptied album moves on
- The covers that come from the Cover Art Archive now live in a folder of their own
  (images/album-remote). Taking the provider's data back deletes that folder, so nothing
  from the archive is left whatever became of its records: a rescan makes a record stale
  and a later search that finds nothing replaces it, and neither of those used to remove
  the files. A cover that moves between the user's files and the archive, or goes away,
  no longer leaves the old files behind.
- A redirect to a place that is not allowed is an outage (502, not remembered, an older
  cover served), not "no cover" for a day for every album. A picture over the limit is
  still "no cover".
- A track that moves to another album (a fixed title) leaves the first with no track: it
  gives up its release group id, so the album the track went to can take it. An album
  that has an id keeps it; an id another album owns is not given twice.
- Waiting for a local slot is no longer reported as the provider being unreachable, and a
  build whose reads were purged three times is not served with files that were never written.
- README and spec no longer say covers never contact anybody.
- Tests: the new folder and the purge after a rescan, files left by a replaced cover, the
  local wait, repeated purges, a refused redirect, the update path with a taken id and the
  emptied album.
2026-11-12 09:21:11 +01:00
tgiachi
d5ed0f2e9c feat(images): album covers from the Cover Art Archive
- The scan reads the MusicBrainz release group id of an album from the tags of its
  files (MUSICBRAINZ_RELEASEGROUPID, a vorbis comment or an ID3 frame) and stores it in
  albums.mbid: it is taken only when the file has an album and exactly one valid id, an
  album keeps the one it has, and an id that another album owns is not given twice. A full
  rescan reads it for music that was already indexed.
- New provider "coverartarchive" (off by default, no key, contact in the User-Agent). When
  an album has no picture in the user's own files, which always come first, its cover is
  asked from the archive by that id and never by a title: only the id leaves the server.
  The same GET /albums/{id}/cover; the record keeps the source. An outage is a 502
  provider_unavailable and is not remembered, and an older cover is served in its place.
- The archive answers with three redirects (coverartarchive.org, archive.org, a data node
  whose name changes). The outbound client can now follow redirects, but only the ones the
  caller allows, up to a limit; here that is https on archive.org and its subdomains.
- A purge counter now guards the covers too: a read that started before the provider's data
  was taken back is not kept. Removing the provider's data forgets the covers that came from
  it and deletes their files, and leaves the ones from the user's files.
- Spec, contract and HTTP tests, README updated; the facts of the real archive are written down.

Closes #43
2026-11-12 09:09:31 +01:00
tgiachi
0e121ead8e feat(images): album covers from my own files
- GET and HEAD /albums/{id}/cover serve a JPEG made from the picture embedded in
  the album's files, or else a cover, folder, front, album or albumart image
  (jpg, jpeg, png) in the album's folder. ?size=small (300 px) or large (1000 px).
  The token can be in the query, so an <img> element can use it.
- Built the first time it is asked for, scaled, and kept in storage.data_dir
  (SONDA_DATA_DIR, default "data"); an album with no cover is remembered for a day.
  A new or changed file of the album, or a full rescan, clears the record.
- Nothing is sent anywhere. Pictures are untrusted: files are opened inside the
  library only (links are not followed), over 20 MB or 16 million pixels are
  ignored, and what is served is always a JPEG made here. Stored files are 0600.
- images table (migration 0013, SQLite and PostgreSQL).
- Spec, contract test, README and example config updated.

Closes #37
2026-11-11 20:52:36 +01:00
tgiachi
b0a7a562a6 fix(discovery): purge lock, cleanup chains and rescans that cannot read
- fix: on PostgreSQL a purge could delete a discovered artist while a scan was
  adopting it. The scan then failed on the missing artist for the whole file,
  and the two could also deadlock. Purge now takes the same artist lock as the
  scanner and discovery before it removes anything. Saving a list or tags does
  too, and an artist that is gone by then is reported as not found instead of a
  foreign key error, so a user who opened an artist that was removed meanwhile
  gets a 404
- fix: a discovered artist that was only the source of a list stayed forever and
  kept the artists of its own list alive, so chains built up. Cleanup now removes
  such an artist and then follows the chain until nothing more goes
- fix: in a full rescan a file that came back but could not be read stayed
  marked missing. It is restored first, as in a normal scan, and read as well
- fix: ?full given twice is refused, so a typo cannot start a scan of another
  kind
- fix: genre cleaning lives in one place, used by the reader and the store
- docs: the files genres count the tracks the artist is credited on, and the spec
  says every files tag is a genre
- test: a purge colliding with a scan on both databases (it fails without the
  lock), a chain of discovered artists, saving for an artist that is gone, a
  file that returns unreadable during a full scan
2026-11-11 17:36:14 +01:00
tgiachi
1857bb9c53 feat(discovery): remove provider data, genres from my files, full rescans
- feat: DELETE /api/v1/providers/{name}/data (admin) removes everything a
  provider gave: its lists of similar artists, its tags and the records of
  when it was asked, then the artists that only that data named and that
  nobody has files for. Last.fm's terms ask for this when an agreement ends.
  The user's own artists, music, listens and settings stay, the provider is not
  switched off, and a second call removes nothing
- feat: a discovered artist that no list names any more is removed when the
  list that named it is replaced, together with its tags, so they do not pile
  up. Real artists, and any artist that another list still names, are kept
- feat: the scan reads the GENRE tag of each file, offline (several genres
  separated by ";", lower case, at most five per file), and
  GET /artists/{id}/tags gets a third entry, files: the genres of the artist's
  own playable tracks, each weighted by how many tracks have it. It asks nobody
  and is always there
- feat: POST /library/scan and POST /libraries/{id}/scan take full=true to read
  every file again, not only the new or changed ones, so MusicBrainz ids and
  genres reach music that was indexed before. Files keep their identity, so
  listening history is kept. Anything but true or false is a 400
- test: the plan of a full scan, the scanner and the manager; the purge on both
  databases, checked by removing the rule that spares real artists; the
  cleanup of replaced lists; genres per file and per artist
- docs: README, OpenAPI and the todo list updated
2026-11-11 17:27:29 +01:00
tgiachi
175b87a4cc fix(discovery): ids from files, adoption and the scanner lock
- fix: a discovered artist adopted by id under another spelling kept its old
  match key, so the next file with the new spelling and no id created a second
  artist. The key now follows the name
- fix: when a real artist without an id and a discovered artist holding the id
  have the same name, the id now goes to the real artist and the discovered one
  lets go of it. Before, the scan created a second real artist and a second
  album, and could not repair it because the id was taken
- fix: when a file without an id adopts a discovered artist that had a provider
  id, the tags, the fetch records and the similar artists read under that id
  are dropped, since they are not known to be about the user's artist. Links
  from other artists to it stay. A file that confirms the very same id keeps
  everything
- fix: the album artist id is only read when the file has an album artist tag;
  with the tag missing the name is a fallback and the id would have landed on the
  wrong artist (a Various Artists id on the track's own artist)
- fix: one artist credit with one id ("X feat. Y" with the id of X, how beets
  writes it) is not paired: no id is safer than a wrong one. A name that is
  itself a single letter or contains a slash still gets its id
- fix: on PostgreSQL the scanner took the artist lock after it had already
  changed artist rows, which could deadlock with a concurrent similar-artists
  save. SaveFile now takes the lock first, once
- fix: Last.fm does not know every MusicBrainz id: when it does not know the id
  and there is a name, the name is tried, never autocorrected, as before ids
  were read from files
- fix: waiting for one provider's tags is bounded at 20 seconds, so a rate
  limited provider cannot hold a page for a minute
- docs: the tag range is 1 to 100 (a tag with no votes is dropped), the spec says
  where an artist's id can come from, and the README says when Last.fm gets an id
- test: adoption by spelling, the real artist with a discovered namesake,
  dropping and keeping what was read, album artist fallback, credits, an ID3
  frame, and the id-then-name fallback of Last.fm
2026-11-11 17:13:31 +01:00
tgiachi
07ea997cd0 feat(discovery): artist tags from Last.fm and MusicBrainz, and ids from my files
- feat: GET /api/v1/artists/{id}/tags lists, for each provider that is on, the
  tags it gives to an artist with its own weights, kept as they came: Last.fm's
  are relative to the artist, MusicBrainz's are counts of votes, and they are
  never mixed. MusicBrainz also marks the tags it lists as genres. The same
  rules as similar artists apply: asked only on demand, kept 7 days (1 for an
  unknown artist), old data served when a provider fails, and one call for
  simultaneous requests. A provider that is off, failing or lacking an id does
  not hide the other
- feat: a thin MusicBrainz client that reads an artist's tags and genres by its
  id. It never searches by name, because a search returns look-alikes and
  picking one would be a guess. The id is checked before it reaches the path,
  and errors never repeat the provider's text
- feat: the User-Agent can change while the server runs, so the contact an
  admin sets for MusicBrainz applies to the next request; if the settings
  cannot be read it falls back to the project address
- feat: Last.fm top tags, by id when known and by name otherwise, never
  autocorrected
- feat: the scan reads MusicBrainz artist ids from the file tags, offline, and
  keeps them on the artists. An id is only used when it can be paired with its
  artist for certain: the same number of ids as artists, all valid. The same id
  is the same artist whatever the spelling, an artist without an id takes the
  one its file gives, and another id under the same name is another artist. A
  file that confirms the id a provider gave to a discovered artist keeps it
- feat: POST /providers/musicbrainz/test checks that the service answers
- test: what leaves the server for MusicBrainz is checked from the other end:
  the id of the artist and the admin's contact, nothing from the library;
  removing the safeguards on ids and pairing makes tests fail
- docs: README, OpenAPI and the provider text updated
2026-11-11 17:01:43 +01:00
tgiachi
b769ad7ac5 fix(discovery): concurrent writes, adopted artists and a client that leaves
- fix: on SQLite a transaction that read and then wrote failed at once with
  "database is locked" when another writer committed in between, because the
  busy timeout does not apply there. Transactions now start with BEGIN
  IMMEDIATE. Before this change a scan running while someone opened similar
  artists made files fail, and parallel requests stored nothing
- fix: on PostgreSQL two requests naming the same new artist could fail on the
  unique MusicBrainz id or leave two rows with one name. Creating artists, in
  the discovery store and in the scanner, now takes turns through an advisory
  lock and looks again once it has the turn
- fix: music arriving for an artist that a provider only named used to keep the
  provider's spelling and its unchecked MusicBrainz id, which later lookups
  would then send instead of the name. Such artists are now marked
  (artists.discovered); the first scan that finds them renames them to the
  spelling of the file tags, drops the id and clears the mark
- fix: a panic in the shared provider call is turned into a failure; the
  singleflight group would have re-raised it where nothing can recover
- fix: a client that leaves while Last.fm is being asked no longer gets a 500
  and an error in the log
- fix: no page can ask for more than is stored (limit tops out at 50), and the
  docs no longer say the best score is always 1
- test: concurrent saves plus a scan on both databases, checked by removing each
  fix; adoption keeps neither the name nor the id; the panic; the client that
  leaves
2026-11-11 16:47:33 +01:00
tgiachi
e14c523828 feat(catalog): add the read API for artists, albums and tracks
- feat(api): GET /api/v1/artists, /albums and /tracks with q (case and accent
  insensitive substring), limit and offset (default 50, at most 200) and a
  total; filters library_id, artist_id and album_id; GET /artists/{id},
  /albums/{id} and /tracks/{id}; any signed-in user can read
- feat(catalog): only tracks with a file that is not flagged missing are
  visible, so everything shown can be played; a track with several files is
  listed once; album artists without credited tracks (compilations) are listed
- feat(storage): results are ordered the same way on SQLite and PostgreSQL
  (plain byte order), LIKE wildcards in the search text are escaped, and a
  page costs a fixed number of queries: related items and counts are loaded in
  batches
- feat(storage): tracks get a title_key computed in Go (migration 0006, with
  a startup refresh), like artists and albums
- test: ordering, paging, search with accents and wildcards, library filter,
  duplicate encodings, missing files, details, and a query-count guard on
  both databases
2026-11-11 13:43:38 +01:00
tgiachi
46463846b3 feat(library): named media libraries managed through the API
- feat(storage): libraries table (name, path); audio_files points at a library
  by id instead of repeating the root path; migration 0005 for both drivers
  turns every existing root into a library and keeps file ids, tracks and the
  missing flag
- feat(library): Libraries service validates names and paths: absolute, an
  existing directory, never the filesystem root, never inside or around
  another library (checked on real paths, so symlinks cannot hide an overlap)
- feat(api): admin-only GET/POST /libraries, PATCH/DELETE /libraries/{id} and
  POST /libraries/{id}/scan; deleting a library drops its index entries and
  keeps tracks and listening history
- feat(library): the scanner and the scan manager work on libraries; scanning
  with no libraries answers 409 no_libraries
- feat(streaming): the library directory comes from the database instead of
  a configured list of roots
- breaking(config): library.roots is removed; an old config fails with a hint
  to use POST /api/v1/libraries
- test: CRUD and duplicates on both databases, path change keeps files,
  upgrade of a database written before libraries existed
2026-11-11 13:30:25 +01:00
tgiachi
49f1051b3f feat(streaming): stream tracks with HTTP Range support
- feat(streaming): open a track's file only after checking it against the
  configured library roots: relative paths only, no "..", symlinks resolved on
  both sides so none can leave the root, regular files only
- feat(api): GET and HEAD /api/v1/tracks/{id}/stream via http.ServeContent
  (206, 416, Content-Length, conditional requests), MIME type from the
  extension, nosniff, no-referrer and private caching headers
- feat(api): the stream route also accepts the access token as access_token in
  the query string, because an audio element cannot send headers; no other
  route does, and the query string is never logged
- feat(storage): PlayableFile picks the oldest file of a track that is not
  flagged missing
- fix(api): internal errors set by handlers are now written to the request log
- test: range variants, HEAD, conditional requests, auth variants, traversal,
  symlink escape, symlinked root, missing file, directory instead of file
2026-11-11 12:56:20 +01:00
tgiachi
11aa47e79f fix: refresh match keys at startup and re-check file type after stat
- fix(storage): RefreshMatchKeys recomputes name_key and title_key for rows
  created before migration 0003, whose SQL lower() key is ASCII-only on SQLite;
  idempotent, runs at every start
- fix(library): re-check that the stat result is a regular file, closing the
  window where a path is swapped for a FIFO or symlink after the directory read
- docs(library): note that OnFinish callbacks of consecutive scans can overlap
- test: old rows with approximate keys are matched after the refresh
2026-11-11 12:30:43 +01:00
tgiachi
d6e9e09ce7 fix(library): harden the scanner against special files, stat errors and lossy logging
- fix: skip FIFOs, sockets and devices named like audio files; opening a named
  pipe blocked a scan forever and left the manager stuck in "running"
- fix: a file that cannot be stat'ed for a transient reason is unknown, not
  gone, so it is never marked missing or used as a rename source
- fix: log the scan result from Manager.OnFinish instead of the lossy event
  hub, so the line can no longer be dropped
- fix(storage): match artists and albums on a Unicode-aware key computed in Go
  (migration 0003, both drivers), so SQLite and PostgreSQL agree
- test: FIFO, stat errors, OnFinish, Unicode artist and album matching
2026-11-11 12:28:44 +01:00
tgiachi
4c7e99bd6a feat(library): add concurrent read-only scanner with progress events
- feat(library): walk each root, plan the changes, read tags with N parallel
  workers and write from a single goroutine
- feat(library): detect renames by size and mtime only when the match is
  unambiguous; mark vanished files missing instead of deleting them
- fix-safe: an unreadable root, an unreadable subdirectory or an interrupted
  scan never marks files as missing; symlinks are skipped
- feat(library): event hub that fans scan progress out to subscribers without
  ever blocking the scan (ready for a websocket client)
- feat(library): TagLib (WASM, no CGO) reader for MP3 and FLAC tags and
  audio properties, with title and artist fallbacks
- feat(storage): library repository for SQLite and PostgreSQL
- feat(api): POST/GET /api/v1/library/scan to start a scan and read its status
- feat(config): library.scan_workers
2026-11-11 12:16:45 +01:00