Every 15 minutes (library.auto_scan_minutes, SONDA_AUTO_SCAN_MINUTES, 0 turns it off)
Sonda takes the signature of each library, a hash of the names of everything in it folder
by folder, which costs only the names of the files and so suits a NAS that sleeps. A
library whose signature changed (a song or a folder came, went or was renamed) is
scanned the usual way; every library is also scanned at least once a day, to find a song
tagged again where it stands.
It does not scan while another scan runs, tries again at the next look when a scan failed,
leaves alone a folder that cannot be read (a NAS that is not mounted), and ignores the
housekeeping folders of a NAS.
I want songs from different albums to play at about the same loudness without touching the volume.
- The scan reads REPLAYGAIN_TRACK_GAIN, REPLAYGAIN_ALBUM_GAIN and the two peaks from each file, offline, and keeps them on the file (migration 0029, SQLite and PostgreSQL). A tag that makes no sense (not a number, more than 60 dB) is left out. Files indexed before have none until a full rescan.
- A track in the API carries them as replay_gain (null when its first playable file says nothing).
- The web player levels each song with them: the sound passes through a gain node, so a quiet song can be raised too, but never so much that its peak would clip (and never more than 12 dB when there is no peak). Without Web Audio the element is turned down instead.
- A new Playback section in Settings chooses song by song (default), album by album or off; the choice stays on the device.
- README: the ReplayGain entry, and the old line that said the year was not in the index is removed.
- English and Italian.
I can now ask for songs from a year to a year, for example the eighties.
- The year is read from the tags of each file (the original date first, then the date) and kept on the track: migration 0028 adds tracks.year (SQLite and PostgreSQL).
- Two new rules, min_year and max_year, both included and checked (1000 to 2999, the first not after the second). The index answers them. A song with no year never matches.
- Files indexed before have no year until a full rescan reads their tags again; the editor and the API description say so.
- The smart playlist editor gets From year and Up to year, and the rules are described in words.
- English and Italian.
A friend's server can now read what an admin chose to share.
- A library has a shared flag, off by default (migration 0024), set with
PATCH /libraries/{id}. Changing it never waits for a scan.
- Requests between friends are signed with each server's Ed25519 key: the
method, the path and query from /api/v1/peer on, a date, a nonce and the
hash of the empty body. A date more than two minutes off, a nonce used
before, an unknown or removed key and a bad signature all get the same
401. A friend still waiting for approval gets 403, and while federation is
off the routes are missing.
- Routes under /api/v1/peer: info, artists, albums, tracks, covers and
streaming with Range. The catalogue, the counts and the format come from
the shared libraries only, and the file served is the one in a shared
library. No path, library id, listen, taste, user or provider data is part
of an answer.
- A friend never makes this server build a cover, read an audio file or ask
a provider: only covers already built from the audio files are served.
- Per-friend limits, set by the admin: parallel streams (default 3) and KiB
a second (default 4096, 0 for none), shared by all the streams of a friend.
- A log of the last requests friends made (route, status, bytes; never the
query), kept for 7 days, in GET /federation/requests.
- POST /federation/friends/{id}/check lets the side that pasted an invite
find out that the other admin approved; a signed call from a friend this
server is waiting for makes it active too.
- Admin: a Shared with friends switch on each library, which asks first, and
limits, recent requests and a Check button in the Federation section, in
English and Italian.
- OpenAPI paths and schemas, contract test, two real servers checked end to
end with tagged audio files.
- The scan reads the MusicBrainz release group id of an album from the tags of its
files (MUSICBRAINZ_RELEASEGROUPID, a vorbis comment or an ID3 frame) and stores it in
albums.mbid: it is taken only when the file has an album and exactly one valid id, an
album keeps the one it has, and an id that another album owns is not given twice. A full
rescan reads it for music that was already indexed.
- New provider "coverartarchive" (off by default, no key, contact in the User-Agent). When
an album has no picture in the user's own files, which always come first, its cover is
asked from the archive by that id and never by a title: only the id leaves the server.
The same GET /albums/{id}/cover; the record keeps the source. An outage is a 502
provider_unavailable and is not remembered, and an older cover is served in its place.
- The archive answers with three redirects (coverartarchive.org, archive.org, a data node
whose name changes). The outbound client can now follow redirects, but only the ones the
caller allows, up to a limit; here that is https on archive.org and its subdomains.
- A purge counter now guards the covers too: a read that started before the provider's data
was taken back is not kept. Removing the provider's data forgets the covers that came from
it and deletes their files, and leaves the ones from the user's files.
- Spec, contract and HTTP tests, README updated; the facts of the real archive are written down.
Closes#43
- fix: on PostgreSQL a purge could delete a discovered artist while a scan was
adopting it. The scan then failed on the missing artist for the whole file,
and the two could also deadlock. Purge now takes the same artist lock as the
scanner and discovery before it removes anything. Saving a list or tags does
too, and an artist that is gone by then is reported as not found instead of a
foreign key error, so a user who opened an artist that was removed meanwhile
gets a 404
- fix: a discovered artist that was only the source of a list stayed forever and
kept the artists of its own list alive, so chains built up. Cleanup now removes
such an artist and then follows the chain until nothing more goes
- fix: in a full rescan a file that came back but could not be read stayed
marked missing. It is restored first, as in a normal scan, and read as well
- fix: ?full given twice is refused, so a typo cannot start a scan of another
kind
- fix: genre cleaning lives in one place, used by the reader and the store
- docs: the files genres count the tracks the artist is credited on, and the spec
says every files tag is a genre
- test: a purge colliding with a scan on both databases (it fails without the
lock), a chain of discovered artists, saving for an artist that is gone, a
file that returns unreadable during a full scan
- feat: DELETE /api/v1/providers/{name}/data (admin) removes everything a
provider gave: its lists of similar artists, its tags and the records of
when it was asked, then the artists that only that data named and that
nobody has files for. Last.fm's terms ask for this when an agreement ends.
The user's own artists, music, listens and settings stay, the provider is not
switched off, and a second call removes nothing
- feat: a discovered artist that no list names any more is removed when the
list that named it is replaced, together with its tags, so they do not pile
up. Real artists, and any artist that another list still names, are kept
- feat: the scan reads the GENRE tag of each file, offline (several genres
separated by ";", lower case, at most five per file), and
GET /artists/{id}/tags gets a third entry, files: the genres of the artist's
own playable tracks, each weighted by how many tracks have it. It asks nobody
and is always there
- feat: POST /library/scan and POST /libraries/{id}/scan take full=true to read
every file again, not only the new or changed ones, so MusicBrainz ids and
genres reach music that was indexed before. Files keep their identity, so
listening history is kept. Anything but true or false is a 400
- test: the plan of a full scan, the scanner and the manager; the purge on both
databases, checked by removing the rule that spares real artists; the
cleanup of replaced lists; genres per file and per artist
- docs: README, OpenAPI and the todo list updated
- fix: a discovered artist adopted by id under another spelling kept its old
match key, so the next file with the new spelling and no id created a second
artist. The key now follows the name
- fix: when a real artist without an id and a discovered artist holding the id
have the same name, the id now goes to the real artist and the discovered one
lets go of it. Before, the scan created a second real artist and a second
album, and could not repair it because the id was taken
- fix: when a file without an id adopts a discovered artist that had a provider
id, the tags, the fetch records and the similar artists read under that id
are dropped, since they are not known to be about the user's artist. Links
from other artists to it stay. A file that confirms the very same id keeps
everything
- fix: the album artist id is only read when the file has an album artist tag;
with the tag missing the name is a fallback and the id would have landed on the
wrong artist (a Various Artists id on the track's own artist)
- fix: one artist credit with one id ("X feat. Y" with the id of X, how beets
writes it) is not paired: no id is safer than a wrong one. A name that is
itself a single letter or contains a slash still gets its id
- fix: on PostgreSQL the scanner took the artist lock after it had already
changed artist rows, which could deadlock with a concurrent similar-artists
save. SaveFile now takes the lock first, once
- fix: Last.fm does not know every MusicBrainz id: when it does not know the id
and there is a name, the name is tried, never autocorrected, as before ids
were read from files
- fix: waiting for one provider's tags is bounded at 20 seconds, so a rate
limited provider cannot hold a page for a minute
- docs: the tag range is 1 to 100 (a tag with no votes is dropped), the spec says
where an artist's id can come from, and the README says when Last.fm gets an id
- test: adoption by spelling, the real artist with a discovered namesake,
dropping and keeping what was read, album artist fallback, credits, an ID3
frame, and the id-then-name fallback of Last.fm
- feat: GET /api/v1/artists/{id}/tags lists, for each provider that is on, the
tags it gives to an artist with its own weights, kept as they came: Last.fm's
are relative to the artist, MusicBrainz's are counts of votes, and they are
never mixed. MusicBrainz also marks the tags it lists as genres. The same
rules as similar artists apply: asked only on demand, kept 7 days (1 for an
unknown artist), old data served when a provider fails, and one call for
simultaneous requests. A provider that is off, failing or lacking an id does
not hide the other
- feat: a thin MusicBrainz client that reads an artist's tags and genres by its
id. It never searches by name, because a search returns look-alikes and
picking one would be a guess. The id is checked before it reaches the path,
and errors never repeat the provider's text
- feat: the User-Agent can change while the server runs, so the contact an
admin sets for MusicBrainz applies to the next request; if the settings
cannot be read it falls back to the project address
- feat: Last.fm top tags, by id when known and by name otherwise, never
autocorrected
- feat: the scan reads MusicBrainz artist ids from the file tags, offline, and
keeps them on the artists. An id is only used when it can be paired with its
artist for certain: the same number of ids as artists, all valid. The same id
is the same artist whatever the spelling, an artist without an id takes the
one its file gives, and another id under the same name is another artist. A
file that confirms the id a provider gave to a discovered artist keeps it
- feat: POST /providers/musicbrainz/test checks that the service answers
- test: what leaves the server for MusicBrainz is checked from the other end:
the id of the artist and the admin's contact, nothing from the library;
removing the safeguards on ids and pairing makes tests fail
- docs: README, OpenAPI and the provider text updated
- feat: the scan summary has an unsupported counter for audio files in formats
Sonda cannot read yet (m4a, ogg, opus, wav, wma, ...); they are counted and
never opened, instead of disappearing without a trace. Video containers such
as .mpg are not counted
- feat: folders named .@__thumb, @Recycle, #recycle, @eaDir, $RECYCLE.BIN and
System Volume Information are skipped, matched by exact name ignoring case,
so a NAS recycle bin does not end up in the library; a library whose own
folder carries one of these names is still scanned, and files indexed before
from such a folder are flagged missing like any vanished file
- docs: the README lists the new counter and the skipped folders
- test: system folders at any depth, near-miss names, a library inside a
system-named folder, entries flagged missing, unsupported counting
Closes#14
- fix(migration): roots that differ only in case (/Music and /music are two
directories on Linux) used to make migration 0005 fail on the unique name;
the later ones now get a numeric suffix
- feat(library): at startup, log a warning for every library whose directory
cannot be read or that overlaps another, because migrated rows skip
validation and may still hold roots that were removed from the old config
- fix(library): creating or updating a library is now serialised, so two
concurrent requests cannot both pass the overlap check
- fix(library): changing a library's directory or deleting it is refused with
409 scan_running while a scan runs; the scan works from the list it started
with and would write under a stale path
- fix(library): library names clash ignoring case and accents the same way on
SQLite and PostgreSQL, checked in Go
- fix(library): changing only the path no longer revalidates an untouched
name, which can be longer than the limit after a migration
- test: roots differing in case on both databases; concurrent creates with a
slow store, verified to fail without the lock; the busy guard; Problems
- feat(storage): libraries table (name, path); audio_files points at a library
by id instead of repeating the root path; migration 0005 for both drivers
turns every existing root into a library and keeps file ids, tracks and the
missing flag
- feat(library): Libraries service validates names and paths: absolute, an
existing directory, never the filesystem root, never inside or around
another library (checked on real paths, so symlinks cannot hide an overlap)
- feat(api): admin-only GET/POST /libraries, PATCH/DELETE /libraries/{id} and
POST /libraries/{id}/scan; deleting a library drops its index entries and
keeps tracks and listening history
- feat(library): the scanner and the scan manager work on libraries; scanning
with no libraries answers 409 no_libraries
- feat(streaming): the library directory comes from the database instead of
a configured list of roots
- breaking(config): library.roots is removed; an old config fails with a hint
to use POST /api/v1/libraries
- test: CRUD and duplicates on both databases, path change keeps files,
upgrade of a database written before libraries existed
- fix(storage): RefreshMatchKeys recomputes name_key and title_key for rows
created before migration 0003, whose SQL lower() key is ASCII-only on SQLite;
idempotent, runs at every start
- fix(library): re-check that the stat result is a regular file, closing the
window where a path is swapped for a FIFO or symlink after the directory read
- docs(library): note that OnFinish callbacks of consecutive scans can overlap
- test: old rows with approximate keys are matched after the refresh
- fix: skip FIFOs, sockets and devices named like audio files; opening a named
pipe blocked a scan forever and left the manager stuck in "running"
- fix: a file that cannot be stat'ed for a transient reason is unknown, not
gone, so it is never marked missing or used as a rename source
- fix: log the scan result from Manager.OnFinish instead of the lossy event
hub, so the line can no longer be dropped
- fix(storage): match artists and albums on a Unicode-aware key computed in Go
(migration 0003, both drivers), so SQLite and PostgreSQL agree
- test: FIFO, stat errors, OnFinish, Unicode artist and album matching
- fix: follow a root that is itself a symlink (e.g. /srv/music -> /mnt/nas/music).
Before, the walk saw a non-directory, found nothing and flagged every indexed
file as missing. Symlinks inside a root are still skipped
- fix: when a root is empty but files are indexed (typical for an unmounted
NAS mount point), return an error instead of flagging the whole library
- test: symlinked root, symlink to a file, empty root with and without indexed
files
- feat(library): walk each root, plan the changes, read tags with N parallel
workers and write from a single goroutine
- feat(library): detect renames by size and mtime only when the match is
unambiguous; mark vanished files missing instead of deleting them
- fix-safe: an unreadable root, an unreadable subdirectory or an interrupted
scan never marks files as missing; symlinks are skipped
- feat(library): event hub that fans scan progress out to subscribers without
ever blocking the scan (ready for a websocket client)
- feat(library): TagLib (WASM, no CGO) reader for MP3 and FLAC tags and
audio properties, with title and artist fallbacks
- feat(storage): library repository for SQLite and PostgreSQL
- feat(api): POST/GET /api/v1/library/scan to start a scan and read its status
- feat(config): library.scan_workers