Migrating hosting from Codeberg to git.kmalloc.org. Renames the
module path in go.mod, every internal import, and the OCI image
source label in the Dockerfile. Build and full test suite verified
green after the rename.
A friend's server can now read what an admin chose to share.
- A library has a shared flag, off by default (migration 0024), set with
PATCH /libraries/{id}. Changing it never waits for a scan.
- Requests between friends are signed with each server's Ed25519 key: the
method, the path and query from /api/v1/peer on, a date, a nonce and the
hash of the empty body. A date more than two minutes off, a nonce used
before, an unknown or removed key and a bad signature all get the same
401. A friend still waiting for approval gets 403, and while federation is
off the routes are missing.
- Routes under /api/v1/peer: info, artists, albums, tracks, covers and
streaming with Range. The catalogue, the counts and the format come from
the shared libraries only, and the file served is the one in a shared
library. No path, library id, listen, taste, user or provider data is part
of an answer.
- A friend never makes this server build a cover, read an audio file or ask
a provider: only covers already built from the audio files are served.
- Per-friend limits, set by the admin: parallel streams (default 3) and KiB
a second (default 4096, 0 for none), shared by all the streams of a friend.
- A log of the last requests friends made (route, status, bytes; never the
query), kept for 7 days, in GET /federation/requests.
- POST /federation/friends/{id}/check lets the side that pasted an invite
find out that the other admin approved; a signed call from a friend this
server is waiting for makes it active too.
- Admin: a Shared with friends switch on each library, which asks first, and
limits, recent requests and a Check button in the Federation section, in
English and Italian.
- OpenAPI paths and schemas, contract test, two real servers checked end to
end with tagged audio files.
Many titles end with a note taken from the tags: (disc 2), (bonus disc),
(Deluxe Edition), sometimes cut off by a tag that is too short. A release group is
named after the album, not after a disc, so those albums found nothing.
The lookup now asks for the title as it is first and, only when MusicBrainz has no
sure result, once more without the notes at its end (baseTitle). A group in brackets
with none of the words of a note stays part of the title, and a title that would be
left empty is kept. The rule for accepting a result does not change.
On 25 albums of a real library with such notes, 19 found a cover (none before).
Closes#87
The covers asked the Cover Art Archive only for albums whose files carried a
release group id. In a library tagged without MusicBrainz, 3,321 of 3,433 albums
have none, so almost no cover could come from the archive.
An album with no id is now looked up in MusicBrainz by the name of its artist and
its title (musicbrainz.Client.FindReleaseGroup), when the musicbrainz provider is
on too. A result is taken only if MusicBrainz scores it 95 or more and its title
and one name of its artist credit equal the ones asked for after folding case,
accents and punctuation; the first such result wins and anything doubtful is no
cover. The id is used for that request and is not stored on the album.
Artists are still never looked up by name. The MusicBrainz and Cover Art Archive
provider texts, the README and the API spec say what is sent.
On 47 albums sampled from a real library, 23 got a cover (all checked as the right
ones by eye or by exact title and artist); a title like 'Live (disc 2)' finds nothing.
Closes#83
- A panic while counting no longer leaves the job marked as running; a panic in the
work of one item is a failure of that item, and a panic in the list ends the job
with a reason.
- A stop while the items are counted is recorded as a stop instead of a 500, and a
count that fails leaves the status as the previous job left it.
- An artist that lost its MusicBrainz id since it was listed is skipped by the
pictures job, as the spec says, instead of counting as a failure.
- A picture that cannot be used now (refused download, bytes that are not an image)
is a failure when a good picture is stored, so a refresh never throws it away.
- The outcomes the artist jobs report share their constants in the jobs package.
- The spec no longer calls each job "the one request" that goes through the library.
- The test that waited with a sleep for the count now waits for a signal.
Add one engine, internal/jobs, that runs the jobs which go through the whole
library: start, stop, progress, one at a time, give up after five failures in
a row, end at once when a provider is off.
- The covers job now runs on the engine; its API is unchanged.
- POST/GET/DELETE /library/artist-images looks for the picture of every
artist of the library that has a MusicBrainz id.
- POST/GET/DELETE /library/descriptions does the same for the Wikipedia
description.
- ?refresh=true searches everything again and never replaces a stored result
by a failure; artists without an id are never touched, nothing is looked up
by name.
- images.Service.EnsureArtistImage and about.Service.Ensure do the work for one
artist; the lists come from ImageRepo and DiscoveryRepo.
- jobs.Runner binds a job to the application context, replacing the two
hand-written controls.
- OpenAPI, contract test and README describe the new endpoints and say that
they are bulk requests.
- A refresh (or any rebuild) while the Cover Art Archive is switched off, but its data
not taken back, no longer turns the covers it gave into "no cover" and deletes their
files: the stored cover is kept and its date renewed, so the files are not searched
again at every request.
- When none of an album's files and folders can be looked at (an unmounted disk, a
permission), that is ErrFilesUnreadable (503 files_unreadable), not "no picture": nothing
is remembered, nothing built is touched, the archive is not asked, and the job counts it
as a failure and gives up after five. A link that leaves the library is still just refused.
- The job calls the build for every album that is not good yet and does not serve an older
cover on an outage, so a dead provider shows as failures and trips the limit instead of
being reported as covers from the archive.
- Start marks the job as running before counting the albums and does not hold the lock
meanwhile; a stop that comes back as a plain driver error is not a failure; a job that
finished by itself is never marked stopped.
- Spec and README say so; the tests cover each case, including a list that fails midway.
- POST /library/covers (admin) starts, in the background and one at a time like a scan,
a job that builds the cover of every album that has something to play and no cover that
is still good; GET /library/covers shows the progress and how the last job ended;
DELETE /library/covers stops it. ?refresh=true searches every album again.
- For each album it does what opening its cover does: the user's own files first, then the
Cover Art Archive if the provider is on. It gives up, and says why, when the provider
fails for five albums in a row; an album that vanished is skipped, not a failure.
- This is the one request that goes through the whole library, and the README and the
provider text say so: with the archive on it sends the release group id of each album with
no picture in its files, one after the other, spaced out.
- NewRouter now refuses to build when a service was not wired, naming it, instead of
panicking at the first request (the wiring of this job in main had been missed, which no
test could see).
- The outbound spacing test measures where the request leaves the client, so a busy
machine cannot make it fail.
- Spec, contract and HTTP tests, README updated.
Closes#45
- The covers that come from the Cover Art Archive now live in a folder of their own
(images/album-remote). Taking the provider's data back deletes that folder, so nothing
from the archive is left whatever became of its records: a rescan makes a record stale
and a later search that finds nothing replaces it, and neither of those used to remove
the files. A cover that moves between the user's files and the archive, or goes away,
no longer leaves the old files behind.
- A redirect to a place that is not allowed is an outage (502, not remembered, an older
cover served), not "no cover" for a day for every album. A picture over the limit is
still "no cover".
- A track that moves to another album (a fixed title) leaves the first with no track: it
gives up its release group id, so the album the track went to can take it. An album
that has an id keeps it; an id another album owns is not given twice.
- Waiting for a local slot is no longer reported as the provider being unreachable, and a
build whose reads were purged three times is not served with files that were never written.
- README and spec no longer say covers never contact anybody.
- Tests: the new folder and the purge after a rescan, files left by a replaced cover, the
local wait, repeated purges, a refused redirect, the update path with a taken id and the
emptied album.
- The scan reads the MusicBrainz release group id of an album from the tags of its
files (MUSICBRAINZ_RELEASEGROUPID, a vorbis comment or an ID3 frame) and stores it in
albums.mbid: it is taken only when the file has an album and exactly one valid id, an
album keeps the one it has, and an id that another album owns is not given twice. A full
rescan reads it for music that was already indexed.
- New provider "coverartarchive" (off by default, no key, contact in the User-Agent). When
an album has no picture in the user's own files, which always come first, its cover is
asked from the archive by that id and never by a title: only the id leaves the server.
The same GET /albums/{id}/cover; the record keeps the source. An outage is a 502
provider_unavailable and is not remembered, and an older cover is served in its place.
- The archive answers with three redirects (coverartarchive.org, archive.org, a data node
whose name changes). The outbound client can now follow redirects, but only the ones the
caller allows, up to a limit; here that is https on archive.org and its subdomains.
- A purge counter now guards the covers too: a read that started before the provider's data
was taken back is not kept. Removing the provider's data forgets the covers that came from
it and deletes their files, and leaves the ones from the user's files.
- Spec, contract and HTTP tests, README updated; the facts of the real archive are written down.
Closes#43
- A timeout inside the provider (the outbound client's own deadline, which wraps
context.DeadlineExceeded) was mistaken for the caller leaving and came out as a 500, so
an older stored picture was not served. The decision now uses the caller's own context,
and a build that runs out of its two minutes is an outage (502) too.
- A purge of the Commons data could be undone by a read that was already running: the row
was deleted, so the generation guard started from zero, and the files were written after
the files were removed. Pictures are now kept only under a purge counter; BeginArtistPurge
is called before the purge and a read that started earlier starts again, or is refused if
the provider is off by then.
- A redirect or an oversized picture from the picture host is a fault of that picture (no
picture), not an outage, and the pictures are downloaded through a client of their own so
that one bad file cannot open the breaker of the API calls.
- The author drops the invisible characters that reorder or hide text. Artist pictures are
decoded only up to 4 million pixels.
- Tests: an outage that ends in a deadline, a build that times out, a purge during a read,
a redirect and a huge picture from the host, a separate download client.
- GET and HEAD /artists/{id}/image serve a free picture of an artist as a JPEG
(size=small 300 px or large 1000 px, token in the query allowed for an <img>), and
GET /artists/{id}/image/credit gives the author, the licence and the page of the file,
to be shown next to it.
- New provider "commons" (off by default, no key, contact in the User-Agent), which also
needs MusicBrainz on. The way to the picture uses ids only, never a name: the artist's
MusicBrainz id, the Wikidata item MusicBrainz links to it, the file Wikidata names for
it (P18), Commons for its licence, and a 960 px copy from a Wikimedia picture host.
- Only free licences: public domain, CC0, CC BY and CC BY-SA. Non-commercial,
no-derivatives, GFDL alone, fair use and unknown licences are refused, since a copy is
kept. The author is plain text, the licence address is kept only on creativecommons.org,
and the download only from https thumb.wikimedia.org or upload.wikimedia.org.
- Built on the first request, kept in the data folder: 30 days, or a week when there is no
free picture; an older copy is served when Commons cannot be reached; a failure is not
remembered. Credit changes change the ETag.
- Removing the provider's data forgets the records and deletes the files; a read that started
before it is not saved over it.
- images table gets the credit (migration 0015); the purge reports images.
- Spec, contract test, README updated; the facts of the real APIs are written down.
Closes#41
- At most two covers are built at the same time: a page of albums asked for at once
could decode dozens of large pictures together.
- A change to an album's files marks its record stale and moves a generation instead
of deleting the row. A build that started before the change is not saved over it
and starts again, so a "no cover" found too early cannot win for a day.
- A cover that was found is checked again after a week, so a picture replaced in a
folder is noticed.
- For CD1/Disc 2 folders the picture in the folder above is used. Folder pictures
are read one at a time, and the pixel count is computed in 64 bits.
- Stored files are synced before they get their name, an empty one counts as
missing, and old temporary files from a crash are removed.
- Files without track or disc numbers sort the same on both databases.
- storage.data_dir defaults next to the SQLite database, so a service started from
another folder does not need a writable working directory.
- Tests: a listener who leaves, a change during a build, the build limit, a named
pipe, an embedded picture of a file outside the library that exists, builds that
really overlap.
- GET and HEAD /albums/{id}/cover serve a JPEG made from the picture embedded in
the album's files, or else a cover, folder, front, album or albumart image
(jpg, jpeg, png) in the album's folder. ?size=small (300 px) or large (1000 px).
The token can be in the query, so an <img> element can use it.
- Built the first time it is asked for, scaled, and kept in storage.data_dir
(SONDA_DATA_DIR, default "data"); an album with no cover is remembered for a day.
A new or changed file of the album, or a full rescan, clears the record.
- Nothing is sent anywhere. Pictures are untrusted: files are opened inside the
library only (links are not followed), over 20 MB or 16 million pixels are
ignored, and what is served is always a JPEG made here. Stored files are 0600.
- images table (migration 0013, SQLite and PostgreSQL).
- Spec, contract test, README and example config updated.
Closes#37