Merge pull request 'Suggestions learned from my own listening' (#186) from feat/suggestions into main

This commit is contained in:
squidd 2026-11-15 11:10:26 +01:00
commit 9ce7beb42d
25 changed files with 2588 additions and 5 deletions

View file

@ -28,6 +28,7 @@ It is a single executable, one database (SQLite by default, PostgreSQL if you pr
- **Wishlist and "not interested".** `PUT /wishlist/{artist}` puts an artist on your wishlist (`GET /wishlist` lists it, `DELETE` takes it off), and `POST /discover/{artist}/dismiss` hides an artist from your Discover for good (`DELETE` brings it back, `GET /discover/dismissed` lists them). An artist is wished or dismissed, never both, and both lists are yours alone. Discover marks the wished artists it shows (`wishlisted`) and never suggests a dismissed one. The wishlist is a list for you and sends nothing anywhere; Sonda downloads nothing, but an admin can send an artist that Discover showed to their own Lidarr (see [External providers](#external-providers)), which works for an artist that has a MusicBrainz id and that Sonda knows of from a list or from your files. An artist that only Last.fm named is kept while you have it marked (your lists still show it, but other users cannot see that it exists), and goes with the rest of Last.fm's data if an admin removes Last.fm's data. Removing MusicBrainz's data does not touch it.
- **Your own settings.** `GET` and `PATCH /me/settings` hold choices that belong to you, not to the server. Everyone starts private: `discover_from_listening` is off, so this setting never sends your listening to a provider. When you turn it on, Discover may start from the artists you listen to most, and their names are sent to Last.fm (only if an admin switched it on). What Last.fm answers is kept in the shared database like any other lookup, so another local user can see, from `retrieved_at` of `GET /artists/{id}/similar`, that someone looked up an artist, but not who.
- **Importing from Spotify.** `POST /me/import/spotify` brings your listening history and your playlists from a Spotify data export into your own account. Ask Spotify for your data (Account, Privacy, "Extended streaming history" and "Account data") and send what it gives, as a multipart form with a `files` field for each file: the zip itself, or the JSON files in it (`Streaming_History_Audio_*.json`, the older `StreamingHistory_music_*.json` and `Playlist*.json`); a request may be up to 512 MB, and the rest of the export (the account, the searches, the library) is never opened. It all happens on your server and nothing is sent anywhere; the files are read from the request and not kept, and of each play only the song, the artist, the album, the time and how it ended are read, never the IP addresses, the country or the platform in the file. Of a play of a song you do not have that counts as listened, the name of the artist and the time are kept (and nothing else), so that Sonda can learn which artists you play together. A play is matched to a song of your library by artist and title (case, accents and punctuation do not matter, a `Remastered` note on either side does not, any credited artist will do, and a play of a compilation is matched by its album when Spotify says `Various Artists`). A match is recorded as a listen at the time of the play: `complete` when it ran to its end (or nine tenths were played), `skip` when Spotify says it was skipped or it lasted less than a listen, `play` otherwise; nothing is posted to Mastodon. Importing the same files again records nothing twice, and the same play in the extended and in the older history is one. A playlist becomes a playlist of yours with the songs that were found, in order; one whose name you already have is left alone, and one with none of its songs in your library is not made. The report counts what was added, what was already there and what is not in your library (and lists those artists, most played first), so you can see what is missing. One import of yours runs at a time. The report is kept: `GET /me/import/spotify` lists your past imports, newest first, with the numbers of each (never the files). The web app has an Import page for the form and for those numbers.
- **Suggestions from your own listening.** `GET /me/suggestions` learns which artists you play together: your history is split into sittings (plays less than 30 minutes apart) and two artists that turn up in the same sitting often are linked (a cosine over the sittings of each). It is a count over your own data, built on your server in a moment and again when plays are added, with no provider and nothing sent anywhere; it needs `discover_from_listening` in `PATCH /me/settings`. The model reads your listening events and the plays of artists you have no music of that an import kept (the artist and the time only). `library` are artists you have that go with what you play and that you did not play for two months; `missing` are artists you played at least three times and have no music of, the ones you play most that also go with what you play now first. Each says which of your artists it goes with. `POST /me/suggestions/wishlist` puts an artist you played on your wishlist by name (it is made without music and without a MusicBrainz id: Sonda never looks one up by name, so it cannot be sent to Lidarr from here). The web app has a "From your listening" tab in Discover.
- **Sharing on Mastodon.** `PUT /me/mastodon` connects your own Mastodon account (the address of your instance and an access token that can only write posts); `GET` shows which account is connected and `DELETE` forgets it and the token. Nothing is connected by default. Once it is, each track of this server that counts as a listen is posted to your account as an unlisted status with the title, artists, album and `#NowPlaying`, and no link: at most one post a minute, never music of a friend. The token stays on the server and is never sent back. If the instance refuses the token on a post (it was revoked or expired), `GET /me/mastodon` says `problem: token_refused`, Settings shows it, and nothing is posted until you connect again with a new token. The address of the instance must be a domain name on https, and the server never connects to a private, loopback or link-local address for it, never follows a redirect and ignores proxy settings.
- **Tags.** `GET /artists/{id}/tags` shows what Last.fm and MusicBrainz say kind of music an artist plays, each with its own weights, and the genres of the artist's own files, read offline. See [External providers](#external-providers).
- **Similar artists.** With Last.fm switched on, `GET /artists/{id}/similar` lists artists similar to an artist, and says which of them you already have. See [External providers](#external-providers). Without it, Sonda makes no outbound network requests at all.

View file

@ -15,6 +15,7 @@ import (
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/about"
"codeberg.org/squidd/sonda/internal/affinity"
httpapi "codeberg.org/squidd/sonda/internal/api/http"
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/catalog"
@ -273,6 +274,7 @@ func run() error {
tasteService := taste.NewService(tasteRepo, taste.Options{MinListen: cfg.Listening.MinListen(), HalfLife: cfg.Taste.HalfLife()})
userSettings := usersettings.NewService(storage.NewUserSettingsRepo(db))
feedbacks := feedback.NewService(storage.NewFeedbackRepo(db))
affinityRepo := storage.NewAffinityRepo(db)
weights := discover.Weights{
Affinity: cfg.Scoring.WeightAffinity, Genre: cfg.Scoring.WeightGenre,
Preference: cfg.Scoring.WeightPreference, Discovery: cfg.Scoring.WeightDiscovery,
@ -312,6 +314,7 @@ func run() error {
Discover: discover.NewService(tasteService, discoveries, tags, tasteRepo, userSettings, feedbacks, weights),
Feedback: feedbacks,
Playlists: playlistSvc,
Suggestions: affinity.NewService(affinityRepo, affinityRepo, feedbacks, affinityRepo, userSettings, cfg.Listening.MinListen()),
SpotifyImport: spotifyimport.NewService(storage.NewCatalogRepo(db), storage.NewListeningRepo(db), playlistSvc, storage.NewSpotifyImportRepo(db), storage.NewOutsidePlayRepo(db), storage.SearchKey, cfg.Listening.MinListen()),
Mixes: mixes.NewService(storage.NewMixRepo(db), tasteService, cfg.Listening.MinListen()),
Smart: smart.NewService(storage.NewSmartRepo(db), tasteService, cfg.Listening.MinListen()),

154
internal/affinity/model.go Normal file
View file

@ -0,0 +1,154 @@
// Package affinity learns which artists a person plays together, from their own listening: the plays are split into
// sittings, and two artists that often turn up in the same sitting are linked. It is a count over the person's own
// data, with no provider and no training service; it can be built again in a moment.
package affinity
import (
"math"
"sort"
"time"
)
// Options are the knobs of the model.
type Options struct {
// Gap is the longest pause inside one sitting: a play that follows the one before by more than this starts another.
Gap time.Duration
// MaxArtists is how many different artists a sitting may have to teach anything: a sitting of a whole library on
// shuffle says nothing about what goes together.
MaxArtists int
// MinTogether is in how many sittings two artists must meet to be linked: once is chance.
MinTogether int
// SeedHalfLife is how long it takes a play to weigh half as much when the artists are ranked by how much they
// are played. It is long, because this is the whole history and not only the last months.
SeedHalfLife time.Duration
}
// DefaultOptions are what Sonda uses.
func DefaultOptions() Options {
return Options{Gap: 30 * time.Minute, MaxArtists: 60, MinTogether: 2, SeedHalfLife: 2 * 365 * 24 * time.Hour}
}
// Play is one play that counts as listened: the artist (as a key that is the same for the same artist wherever the
// play comes from) and when it ended.
type Play struct {
Key string
Name string
At time.Time
}
// Artist is what the model knows of one artist.
type Artist struct {
Key string
Name string
Plays int
Sessions int // the sittings it was in that were used for the links
Last time.Time // its last play
Weight float64 // its plays, each worth 1 and halving every SeedHalfLife
}
// Link says that two artists are played together. A is before B. Score is from 0 to 1: the sittings they share over
// the square root of the product of the sittings of each (a cosine), so a pair that is always together scores 1
// and a pair of which one is everywhere scores low.
type Link struct {
A, B string
Score float64
}
// Model is what was learned.
type Model struct {
Artists []Artist // by key
Links []Link // by A, then B
Plays int // the plays read
Sessions int // the sittings they made
}
type pair struct{ a, b string }
// Build learns from the plays. now is the moment the age of a play is counted to.
func Build(plays []Play, now time.Time, o Options) Model {
sorted := make([]Play, 0, len(plays))
for _, p := range plays {
if p.Key != "" {
sorted = append(sorted, p)
}
}
sort.SliceStable(sorted, func(i, j int) bool { return sorted[i].At.Before(sorted[j].At) })
artists := map[string]*Artist{}
use := func(p Play) *Artist {
a := artists[p.Key]
if a == nil {
a = &Artist{Key: p.Key, Name: p.Name}
artists[p.Key] = a
}
return a
}
sittings := map[string]int{}
together := map[pair]int{}
m := Model{Plays: len(sorted)}
var current map[string]bool
closeSitting := func() {
if len(current) == 0 {
return
}
m.Sessions++
if len(current) > o.MaxArtists {
return
}
keys := make([]string, 0, len(current))
for k := range current {
keys = append(keys, k)
sittings[k]++
}
sort.Strings(keys)
for i := range keys {
for j := i + 1; j < len(keys); j++ {
together[pair{keys[i], keys[j]}]++
}
}
}
var previous time.Time
for i, p := range sorted {
if i == 0 || p.At.Sub(previous) > o.Gap {
closeSitting()
current = map[string]bool{}
}
previous = p.At
current[p.Key] = true
a := use(p)
a.Plays++
if p.At.After(a.Last) {
a.Last = p.At
}
age := now.Sub(p.At)
if age < 0 {
age = 0 // a play dated after now weighs as one that just happened
}
a.Weight += math.Exp2(-float64(age) / float64(o.SeedHalfLife))
}
closeSitting()
for k, n := range sittings {
artists[k].Sessions = n
}
for pr, n := range together {
if n < o.MinTogether {
continue
}
m.Links = append(m.Links, Link{A: pr.a, B: pr.b, Score: float64(n) / math.Sqrt(float64(sittings[pr.a])*float64(sittings[pr.b]))})
}
sort.Slice(m.Links, func(i, j int) bool {
if m.Links[i].A != m.Links[j].A {
return m.Links[i].A < m.Links[j].A
}
return m.Links[i].B < m.Links[j].B
})
m.Artists = make([]Artist, 0, len(artists))
for _, a := range artists {
m.Artists = append(m.Artists, *a)
}
sort.Slice(m.Artists, func(i, j int) bool { return m.Artists[i].Key < m.Artists[j].Key })
return m
}

View file

@ -0,0 +1,151 @@
package affinity
import (
"math"
"testing"
"time"
)
var t0 = time.Date(2023, 3, 1, 9, 0, 0, 0, time.UTC)
// at builds a play of an artist, minutes after t0.
func at(artist string, minutes int) Play {
return Play{Key: artist, Name: artist, At: t0.Add(time.Duration(minutes) * time.Minute)}
}
// sittings are plays of the given artists in order, one minute apart, starting a day after the day before.
func sitting(day int, artists ...string) []Play {
var out []Play
for i, a := range artists {
out = append(out, at(a, day*24*60+i))
}
return out
}
func link(m Model, a, b string) (float64, bool) {
for _, l := range m.Links {
if l.A == a && l.B == b {
return l.Score, true
}
}
return 0, false
}
func TestArtistsPlayedInTheSameSittingAreLinked(t *testing.T) {
var plays []Play
for day := 0; day < 3; day++ {
plays = append(plays, sitting(day, "blur", "oasis")...)
}
m := Build(plays, t0.AddDate(0, 0, 10), DefaultOptions())
score, ok := link(m, "blur", "oasis")
if !ok || math.Abs(score-1) > 1e-9 {
t.Fatalf("link = %v, %v, want 1 (always together)", score, ok)
}
if len(m.Links) != 1 || m.Sessions != 3 || m.Plays != 6 {
t.Errorf("model = %d links, %d sittings, %d plays, want 1, 3, 6", len(m.Links), m.Sessions, m.Plays)
}
}
func TestTheLinkIsTheCosineOfTheSittingsOfTheTwo(t *testing.T) {
var plays []Play
plays = append(plays, sitting(0, "a", "b")...)
plays = append(plays, sitting(1, "a", "b")...)
plays = append(plays, sitting(2, "a")...)
plays = append(plays, sitting(3, "a")...)
plays = append(plays, sitting(4, "a", "c")...) // a is in five sittings, b in two, together twice
m := Build(plays, t0.AddDate(0, 0, 10), DefaultOptions())
want := 2 / math.Sqrt(5*2)
if score, ok := link(m, "a", "b"); !ok || math.Abs(score-want) > 1e-9 {
t.Errorf("a-b = %v, %v, want %v", score, ok, want)
}
if _, ok := link(m, "a", "c"); ok {
t.Error("a and c were together once: not enough to be linked")
}
}
func TestAPauseLongerThanTheGapStartsAnotherSitting(t *testing.T) {
o := DefaultOptions()
gap := int(o.Gap / time.Minute)
together := []Play{at("a", 0), at("b", gap)} // exactly the gap: still one sitting
apart := []Play{at("a", 1000), at("b", 1000+gap+1)} // one minute more: two
m := Build(append(together, apart...), t0.AddDate(0, 0, 10), Options{Gap: o.Gap, MaxArtists: o.MaxArtists, MinTogether: 1, SeedHalfLife: o.SeedHalfLife})
if m.Sessions != 3 {
t.Fatalf("%d sittings, want 3", m.Sessions)
}
if score, ok := link(m, "a", "b"); !ok || math.Abs(score-1/math.Sqrt(2*2)) > 1e-9 {
t.Errorf("a-b = %v, %v, want together in one of the two sittings each is in", score, ok)
}
}
func TestAnArtistRepeatedInASittingCountsOnce(t *testing.T) {
var plays []Play
for day := 0; day < 2; day++ {
plays = append(plays, sitting(day, "a", "a", "a", "b")...)
}
m := Build(plays, t0.AddDate(0, 0, 10), DefaultOptions())
if score, ok := link(m, "a", "b"); !ok || math.Abs(score-1) > 1e-9 {
t.Errorf("a-b = %v, %v, want 1", score, ok)
}
for _, a := range m.Artists {
if a.Key == "a" && (a.Plays != 6 || a.Sessions != 2) {
t.Errorf("a = %+v, want 6 plays in 2 sittings", a)
}
}
}
func TestAMarathonOfManyArtistsTeachesNothing(t *testing.T) {
o := DefaultOptions()
o.MaxArtists = 3
var names []string
for _, n := range []string{"a", "b", "c", "d"} {
names = append(names, n)
}
var plays []Play
for day := 0; day < 3; day++ {
plays = append(plays, sitting(day, names...)...)
}
m := Build(plays, t0.AddDate(0, 0, 10), o)
if len(m.Links) != 0 {
t.Errorf("%d links from sittings of more artists than the limit, want none", len(m.Links))
}
if m.Plays != 12 {
t.Errorf("%d plays, want all 12 still counted", m.Plays)
}
}
func TestRecentPlaysWeighMoreAndTheFutureIsNow(t *testing.T) {
now := t0.AddDate(3, 0, 0)
o := DefaultOptions()
plays := []Play{
{Key: "old", Name: "Old", At: now.AddDate(0, 0, -int(o.SeedHalfLife/(24*time.Hour)))}, // one half-life ago
{Key: "new", Name: "New", At: now},
{Key: "future", Name: "Future", At: now.Add(48 * time.Hour)},
}
m := Build(plays, now, o)
w := map[string]float64{}
for _, a := range m.Artists {
w[a.Key] = a.Weight
}
if math.Abs(w["old"]-0.5) > 0.01 || math.Abs(w["new"]-1) > 1e-9 || math.Abs(w["future"]-1) > 1e-9 {
t.Errorf("weights = %v, want old 0.5, new 1, future 1", w)
}
}
func TestAnArtistKnowsItsNameItsLastPlayAndItsOrderIsStable(t *testing.T) {
plays := []Play{{Key: "b", Name: "Björk", At: t0}, {Key: "a", Name: "Air", At: t0.Add(time.Hour)}, {Key: "b", Name: "BJORK", At: t0.Add(48 * time.Hour)}, {Key: "", Name: "nobody", At: t0}}
m := Build(plays, t0.AddDate(0, 0, 10), DefaultOptions())
if len(m.Artists) != 2 || m.Artists[0].Key != "a" || m.Artists[1].Key != "b" {
t.Fatalf("artists = %+v, want a then b, and none without a key", m.Artists)
}
b := m.Artists[1]
if b.Name != "Björk" || b.Plays != 2 || !b.Last.Equal(t0.Add(48*time.Hour)) {
t.Errorf("b = %+v, want the first name seen, 2 plays and the last time", b)
}
}
func TestNoPlaysMakeAnEmptyModel(t *testing.T) {
m := Build(nil, t0, DefaultOptions())
if len(m.Artists) != 0 || len(m.Links) != 0 || m.Plays != 0 || m.Sessions != 0 {
t.Errorf("model = %+v, want empty", m)
}
}

View file

@ -0,0 +1,414 @@
package affinity
import (
"context"
"errors"
"fmt"
"math"
"sort"
"strings"
"sync"
"time"
"golang.org/x/sync/singleflight"
"codeberg.org/squidd/sonda/internal/feedback"
"codeberg.org/squidd/sonda/internal/usersettings"
)
// Limits and rules.
const (
DefaultLimit = 20
MaxLimit = 50
// seedCount is how many of the most played artists the suggestions start from.
seedCount = 40
// poolCount is how many of the most played artists are considered at all, besides the ones linked to a seed.
poolCount = 300
// recentDays: an artist of the library played in this many days is not "forgotten", so it is not suggested.
recentDays = 60
// minPlays is how many times an artist that is not in the library must have been played to be worth getting.
minPlays = 3
// staleAfter and bigChange say when the model is built again: when plays were added and the last build is older
// than staleAfter, or at once when as many as bigChange were added (an import).
staleAfter = 10 * time.Minute
bigChange = 100
)
// Errors of the service.
var (
// ErrNotAllowed means the listener did not allow Discover to start from their listening.
ErrNotAllowed = errors.New("allow Discover to start from your listening to get suggestions")
// ErrInvalidLimit means a negative limit.
ErrInvalidLimit = errors.New("the limit cannot be negative")
// ErrUnknownArtist means the name is not an artist the listener has played.
ErrUnknownArtist = errors.New("that is not an artist you have played")
)
// Built says when and from what the model was built.
type Built struct {
BuiltAt time.Time
Plays int
Sessions int
Artists int
Links int
}
// LibraryArtist is an artist that exists on this server.
type LibraryArtist struct {
ID int64
Name string
Playable bool // there is music of it to play
}
// Store reads the plays of a person and keeps what was learned from them.
type Store interface {
Plays(ctx context.Context, userID int64, minPlayed time.Duration) ([]Play, error)
PlayCount(ctx context.Context, userID int64, minPlayed time.Duration) (int, error)
Save(ctx context.Context, userID int64, at time.Time, m Model) error
Build(ctx context.Context, userID int64) (Built, bool, error)
Seeds(ctx context.Context, userID int64, limit int) ([]Artist, error)
Artists(ctx context.Context, userID int64, keys []string) (map[string]Artist, error)
Neighbours(ctx context.Context, userID int64, keys []string) ([]Link, error)
}
// Library says which of the artists exist on this server, by search key.
type Library interface {
LibraryArtists(ctx context.Context, keys []string) (map[string]LibraryArtist, error)
}
// Marks reads what the listener decided about artists.
type Marks interface {
Marks(ctx context.Context, userID int64) (map[int64]feedback.Kind, error)
}
// Wishes puts an artist on the wishlist by name and gives its id.
type Wishes interface {
Wishlist(ctx context.Context, userID int64, name string, at time.Time) (int64, error)
}
// Settings reads the listener's own settings.
type Settings interface {
Get(ctx context.Context, userID int64) (usersettings.Settings, error)
}
// Reason is an artist the listener plays, because of which another is suggested.
type Reason struct {
Name string
Score float64
}
// Suggestion is an artist put forward.
type Suggestion struct {
key string
Name string
ArtistID int64 // 0 when no artist of this name exists on this server yet
Score float64
Plays int
LastPlayed time.Time
Wishlisted bool
Because []Reason
}
// Suggestions is the answer.
type Suggestions struct {
// Library are artists the listener has, linked to what they play now, not played for a while.
Library []Suggestion
// Missing are artists the listener plays that they have no music of, the ones they play most and that go with
// what they play now first.
Missing []Suggestion
Built Built
}
// Service learns from the listening of a person and makes suggestions.
type Service struct {
store Store
library Library
marks Marks
wishes Wishes
settings Settings
minListen time.Duration
opts Options
now func() time.Time
flight singleflight.Group
mu sync.Mutex
}
// NewService builds a Service. minListen is how long a play must last to count as a listen.
func NewService(store Store, lib Library, marks Marks, wishes Wishes, settings Settings, minListen time.Duration) *Service {
return &Service{store: store, library: lib, marks: marks, wishes: wishes, settings: settings, minListen: minListen, opts: DefaultOptions(), now: time.Now}
}
// Rebuild learns again from every play of the person, and keeps it.
func (s *Service) Rebuild(ctx context.Context, userID int64) (Built, error) {
v, err, _ := s.flight.Do(fmt.Sprint(userID), func() (any, error) { return s.rebuild(ctx, userID) })
if err != nil {
return Built{}, err
}
return v.(Built), nil
}
func (s *Service) rebuild(ctx context.Context, userID int64) (Built, error) {
plays, err := s.store.Plays(ctx, userID, s.minListen)
if err != nil {
return Built{}, err
}
at := s.now()
m := Build(plays, at, s.opts)
if err := s.store.Save(ctx, userID, at, m); err != nil {
return Built{}, err
}
return Built{BuiltAt: at, Plays: m.Plays, Sessions: m.Sessions, Artists: len(m.Artists), Links: len(m.Links)}, nil
}
// ensure builds the model when there is none or when plays were added since.
func (s *Service) ensure(ctx context.Context, userID int64) (Built, error) {
have, ok, err := s.store.Build(ctx, userID)
if err != nil {
return Built{}, err
}
if !ok {
return s.Rebuild(ctx, userID)
}
n, err := s.store.PlayCount(ctx, userID, s.minListen)
if err != nil {
return Built{}, err
}
changed := n - have.Plays
if changed < 0 {
changed = -changed
}
if changed >= bigChange || (changed > 0 && s.now().Sub(have.BuiltAt) > staleAfter) {
return s.Rebuild(ctx, userID)
}
return have, nil
}
// Suggest makes the suggestions for a person. It needs them to have allowed Discover to start from their listening.
func (s *Service) Suggest(ctx context.Context, userID int64, limit int) (Suggestions, error) {
switch {
case limit < 0:
return Suggestions{}, ErrInvalidLimit
case limit == 0:
limit = DefaultLimit
case limit > MaxLimit:
limit = MaxLimit
}
set, err := s.settings.Get(ctx, userID)
if err != nil {
return Suggestions{}, err
}
if !set.DiscoverFromListening {
return Suggestions{}, ErrNotAllowed
}
build, err := s.ensure(ctx, userID)
if err != nil {
return Suggestions{}, err
}
out := Suggestions{Library: []Suggestion{}, Missing: []Suggestion{}, Built: build}
pool, err := s.store.Seeds(ctx, userID, poolCount)
if err != nil {
return Suggestions{}, err
}
if len(pool) == 0 {
return out, nil
}
seeds := pool[:min(seedCount, len(pool))]
weight := map[string]float64{}
keys := make([]string, 0, len(seeds))
for _, a := range seeds {
weight[a.Key] = math.Log1p(a.Weight)
keys = append(keys, a.Key)
}
links, err := s.store.Neighbours(ctx, userID, keys)
if err != nil {
return Suggestions{}, err
}
type reason struct {
key string
score float64
}
affinity := map[string]float64{}
because := map[string][]reason{}
link := func(seed, other string, score float64) {
w, ok := weight[seed]
if !ok || seed == other {
return
}
affinity[other] += w * score
because[other] = append(because[other], reason{seed, w * score})
}
for _, l := range links {
link(l.A, l.B, l.Score)
link(l.B, l.A, l.Score)
}
stats := map[string]Artist{}
for _, a := range pool {
stats[a.Key] = a
}
var unknown []string
for k := range affinity {
if _, ok := stats[k]; !ok {
unknown = append(unknown, k)
}
}
extra, err := s.store.Artists(ctx, userID, unknown)
if err != nil {
return Suggestions{}, err
}
for k, a := range extra {
stats[k] = a
}
all := make([]string, 0, len(stats))
for k := range stats {
all = append(all, k)
}
sort.Strings(all)
inLibrary, err := s.library.LibraryArtists(ctx, all)
if err != nil {
return Suggestions{}, err
}
marks, err := s.marks.Marks(ctx, userID)
if err != nil {
return Suggestions{}, err
}
names := func(key string) string {
if a, ok := stats[key]; ok {
return a.Name
}
return key
}
make1 := func(key string) Suggestion {
a := stats[key]
sg := Suggestion{key: key, Name: a.Name, Plays: a.Plays, LastPlayed: a.Last}
if la, ok := inLibrary[key]; ok {
sg.ArtistID = la.ID
sg.Name = la.Name
sg.Wishlisted = marks[la.ID] == feedback.Wishlist
}
rs := because[key]
sort.Slice(rs, func(i, j int) bool {
if rs[i].score != rs[j].score {
return rs[i].score > rs[j].score
}
return rs[i].key < rs[j].key
})
for _, r := range rs[:min(3, len(rs))] {
sg.Because = append(sg.Because, Reason{Name: names(r.key), Score: r.score})
}
return sg
}
dismissed := func(key string) bool {
la, ok := inLibrary[key]
return ok && marks[la.ID] == feedback.Dismissed
}
// The artists of the library that go with what is played now and have not been played for a while.
recent := s.now().AddDate(0, 0, -recentDays)
var lib []Suggestion
maxAff := 0.0
for _, key := range all {
la, ok := inLibrary[key]
if !ok || !la.Playable || affinity[key] <= 0 || dismissed(key) || stats[key].Last.After(recent) {
continue
}
maxAff = math.Max(maxAff, affinity[key])
sg := make1(key)
sg.Score = affinity[key]
lib = append(lib, sg)
}
for i := range lib {
lib[i].Score /= maxAff
}
out.Library = top(lib, limit)
// The artists that are played and are not in the library: how much they are played and how well they go with
// what is played now, half and half.
var away []Suggestion
maxOwn, maxAway := 0.0, 0.0
for _, key := range all {
if la, ok := inLibrary[key]; ok && la.Playable {
continue
}
if stats[key].Plays < minPlays || dismissed(key) {
continue
}
maxOwn = math.Max(maxOwn, math.Log1p(stats[key].Weight))
maxAway = math.Max(maxAway, affinity[key])
away = append(away, make1(key))
}
for i := range away {
key := away[i].key
own := math.Log1p(stats[key].Weight) / maxOwn
with := 0.0
if maxAway > 0 {
with = affinity[key] / maxAway
}
away[i].Score = 0.5*own + 0.5*with
}
out.Missing = top(away, limit)
return out, nil
}
// top keeps the best of the suggestions, the highest score first, then the most played, then by name.
func top(in []Suggestion, limit int) []Suggestion {
sort.SliceStable(in, func(i, j int) bool {
a, b := in[i], in[j]
switch {
case a.Score != b.Score:
return a.Score > b.Score
case a.Plays != b.Plays:
return a.Plays > b.Plays
default:
return a.Name < b.Name
}
})
if len(in) > limit {
in = in[:limit]
}
if in == nil {
in = []Suggestion{}
}
return in
}
// AddToWishlist puts an artist the listener has played on their wishlist, by name, and gives its id.
func (s *Service) AddToWishlist(ctx context.Context, userID int64, name string) (int64, error) {
name = strings.TrimSpace(name)
if name == "" {
return 0, ErrUnknownArtist
}
set, err := s.settings.Get(ctx, userID)
if err != nil {
return 0, err
}
if !set.DiscoverFromListening {
return 0, ErrNotAllowed
}
if _, err := s.ensure(ctx, userID); err != nil {
return 0, err
}
found := false
seeds, err := s.store.Seeds(ctx, userID, math.MaxInt32)
if err != nil {
return 0, err
}
for _, a := range seeds {
if a.Name == name {
found = true
name = a.Name
break
}
}
if !found {
return 0, ErrUnknownArtist
}
return s.wishes.Wishlist(ctx, userID, name, s.now())
}

View file

@ -0,0 +1,302 @@
package affinity
import (
"context"
"errors"
"sort"
"testing"
"time"
"codeberg.org/squidd/sonda/internal/feedback"
"codeberg.org/squidd/sonda/internal/usersettings"
)
var now = time.Date(2026, 11, 15, 12, 0, 0, 0, time.UTC)
// memStore keeps the model in memory and counts how often it is built.
type memStore struct {
plays []Play
model Model
built Built
has bool
reads int
}
func (m *memStore) Plays(context.Context, int64, time.Duration) ([]Play, error) {
m.reads++
return m.plays, nil
}
func (m *memStore) PlayCount(context.Context, int64, time.Duration) (int, error) {
return len(m.plays), nil
}
func (m *memStore) Save(_ context.Context, _ int64, at time.Time, model Model) error {
m.model, m.has = model, true
m.built = Built{BuiltAt: at, Plays: model.Plays, Sessions: model.Sessions, Artists: len(model.Artists), Links: len(model.Links)}
return nil
}
func (m *memStore) Build(context.Context, int64) (Built, bool, error) { return m.built, m.has, nil }
func (m *memStore) Seeds(_ context.Context, _ int64, limit int) ([]Artist, error) {
out := append([]Artist(nil), m.model.Artists...)
sort.Slice(out, func(i, j int) bool {
if out[i].Weight != out[j].Weight {
return out[i].Weight > out[j].Weight
}
return out[i].Key < out[j].Key
})
return out[:min(limit, len(out))], nil
}
func (m *memStore) Artists(_ context.Context, _ int64, keys []string) (map[string]Artist, error) {
out := map[string]Artist{}
for _, a := range m.model.Artists {
for _, k := range keys {
if a.Key == k {
out[k] = a
}
}
}
return out, nil
}
func (m *memStore) Neighbours(_ context.Context, _ int64, keys []string) ([]Link, error) {
var out []Link
for _, l := range m.model.Links {
for _, k := range keys {
if l.A == k || l.B == k {
out = append(out, l)
break
}
}
}
return out, nil
}
type fakeLibrary map[string]LibraryArtist
func (f fakeLibrary) LibraryArtists(_ context.Context, keys []string) (map[string]LibraryArtist, error) {
out := map[string]LibraryArtist{}
for _, k := range keys {
if a, ok := f[k]; ok {
out[k] = a
}
}
return out, nil
}
type fakeMarks map[int64]feedback.Kind
func (f fakeMarks) Marks(context.Context, int64) (map[int64]feedback.Kind, error) { return f, nil }
type fakeWishes struct {
names []string
id int64
}
func (f *fakeWishes) Wishlist(_ context.Context, _ int64, name string, _ time.Time) (int64, error) {
f.names = append(f.names, name)
return f.id, nil
}
type fakeSettings struct{ on bool }
func (f fakeSettings) Get(context.Context, int64) (usersettings.Settings, error) {
return usersettings.Settings{DiscoverFromListening: f.on}, nil
}
// ago is a moment this many days before now, and the sitting of artists played one after the other from there.
func ago(days int, artists ...string) []Play {
var out []Play
for i, a := range artists {
out = append(out, Play{Key: a, Name: a, At: now.AddDate(0, 0, -days).Add(time.Duration(i) * time.Minute)})
}
return out
}
type rig struct {
svc *Service
store *memStore
wishes *fakeWishes
marks fakeMarks
}
// newRig plays out this history: three old sittings of radiohead, pixies and canali; two recent ones of radiohead and
// placebo; one play of an artist met once. pixies and placebo are in the library, canali and oneoff are not.
func newRig(on bool) rig {
var plays []Play
for _, d := range []int{200, 199, 198} {
plays = append(plays, ago(d, "radiohead", "pixies", "canali")...)
}
for _, d := range []int{10, 9} {
plays = append(plays, ago(d, "radiohead", "placebo")...)
}
plays = append(plays, ago(150, "oneoff")...)
store := &memStore{plays: plays}
lib := fakeLibrary{
"radiohead": {ID: 1, Name: "Radiohead", Playable: true},
"pixies": {ID: 2, Name: "Pixies", Playable: true},
"placebo": {ID: 3, Name: "Placebo", Playable: true},
}
wishes := &fakeWishes{id: 77}
marks := fakeMarks{}
svc := NewService(store, lib, marks, wishes, fakeSettings{on}, 20*time.Second)
svc.now = func() time.Time { return now }
return rig{svc, store, wishes, marks}
}
func names(in []Suggestion) []string {
out := []string{}
for _, s := range in {
out = append(out, s.Name)
}
return out
}
func equal(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func TestSuggestionsNeedTheListenerToAllowListening(t *testing.T) {
r := newRig(false)
if _, err := r.svc.Suggest(context.Background(), 1, 0); !errors.Is(err, ErrNotAllowed) {
t.Fatalf("err = %v, want ErrNotAllowed", err)
}
if r.store.reads != 0 {
t.Error("a refused request read the plays")
}
if _, err := r.svc.AddToWishlist(context.Background(), 1, "canali"); !errors.Is(err, ErrNotAllowed) {
t.Errorf("wishlist err = %v, want ErrNotAllowed", err)
}
}
func TestTheLibraryListHasTheArtistsThatGoWithWhatIsPlayedAndWereNotPlayedLately(t *testing.T) {
r := newRig(true)
got, err := r.svc.Suggest(context.Background(), 1, 0)
if err != nil {
t.Fatal(err)
}
// placebo goes with radiohead too, but it was played ten days ago; radiohead itself was too.
if !equal(names(got.Library), []string{"Pixies"}) {
t.Fatalf("library = %v, want only Pixies", names(got.Library))
}
p := got.Library[0]
if p.ArtistID != 2 || p.Score != 1 || p.Plays != 3 || !p.LastPlayed.Equal(now.AddDate(0, 0, -198).Add(time.Minute)) {
t.Errorf("Pixies = %+v", p)
}
if len(p.Because) == 0 || p.Because[0].Name == "Pixies" {
t.Errorf("because = %+v, want the artists it goes with", p.Because)
}
}
func TestTheMissingListHasTheArtistsPlayedThatAreNotInTheLibrary(t *testing.T) {
r := newRig(true)
got, err := r.svc.Suggest(context.Background(), 1, 0)
if err != nil {
t.Fatal(err)
}
// oneoff was played once: not worth getting. Radiohead, Pixies and Placebo are in the library.
if !equal(names(got.Missing), []string{"canali"}) {
t.Fatalf("missing = %v, want only canali", names(got.Missing))
}
c := got.Missing[0]
if c.ArtistID != 0 || c.Wishlisted || c.Plays != 3 || c.Score <= 0 || c.Score > 1 {
t.Errorf("canali = %+v", c)
}
}
func TestAMissingArtistThatIsOnTheWishlistSaysSoAndOneNotWantedIsLeftOut(t *testing.T) {
r := newRig(true)
lib := fakeLibrary{
"radiohead": {ID: 1, Name: "Radiohead", Playable: true},
"pixies": {ID: 2, Name: "Pixies", Playable: true},
"canali": {ID: 9, Name: "Giorgio Canali", Playable: false}, // a row without music
}
r.svc.library = lib
r.marks[9] = feedback.Wishlist
got, _ := r.svc.Suggest(context.Background(), 1, 0)
if len(got.Missing) != 1 || got.Missing[0].ArtistID != 9 || !got.Missing[0].Wishlisted || got.Missing[0].Name != "Giorgio Canali" {
t.Fatalf("missing = %+v, want the row 9 on the wishlist", got.Missing)
}
r.marks[9] = feedback.Dismissed
r.marks[2] = feedback.Dismissed
got, _ = r.svc.Suggest(context.Background(), 1, 0)
if len(got.Missing) != 0 || len(got.Library) != 0 {
t.Errorf("after dismissing: library %v, missing %v, want nothing", names(got.Library), names(got.Missing))
}
}
func TestTheModelIsBuiltOnceAndAgainWhenPlaysAreAdded(t *testing.T) {
r := newRig(true)
ctx := context.Background()
for i := 0; i < 3; i++ {
if _, err := r.svc.Suggest(ctx, 1, 0); err != nil {
t.Fatal(err)
}
}
if r.store.reads != 1 {
t.Fatalf("plays read %d times for three requests, want 1", r.store.reads)
}
// A few plays, soon after: the model is left alone.
r.store.plays = append(r.store.plays, ago(0, "radiohead")...)
r.svc.now = func() time.Time { return now.Add(time.Minute) }
_, _ = r.svc.Suggest(ctx, 1, 0)
if r.store.reads != 1 {
t.Errorf("read %d times after one new play a minute later, want 1", r.store.reads)
}
// The same, but the model is old now.
r.svc.now = func() time.Time { return now.Add(time.Hour) }
_, _ = r.svc.Suggest(ctx, 1, 0)
if r.store.reads != 2 {
t.Errorf("read %d times after an hour, want 2", r.store.reads)
}
// An import adds many at once: at once.
for i := 0; i < bigChange; i++ {
r.store.plays = append(r.store.plays, ago(300+i, "radiohead", "pixies")...)
}
_, _ = r.svc.Suggest(ctx, 1, 0)
if r.store.reads != 3 {
t.Errorf("read %d times after a big import, want 3", r.store.reads)
}
}
func TestTheLimitIsAppliedAndBadOnesRefused(t *testing.T) {
r := newRig(true)
if _, err := r.svc.Suggest(context.Background(), 1, -1); !errors.Is(err, ErrInvalidLimit) {
t.Errorf("err = %v, want ErrInvalidLimit", err)
}
got, err := r.svc.Suggest(context.Background(), 1, 1)
if err != nil || len(got.Library) > 1 || len(got.Missing) > 1 {
t.Errorf("limit 1 = %v, %v", got, err)
}
}
func TestNoPlaysGiveEmptyListsAndNotNull(t *testing.T) {
r := newRig(true)
r.store.plays = nil
got, err := r.svc.Suggest(context.Background(), 1, 0)
if err != nil || got.Library == nil || got.Missing == nil || len(got.Library)+len(got.Missing) != 0 {
t.Errorf("got %+v, %v, want two empty lists", got, err)
}
}
func TestOnlyAnArtistThatWasPlayedGoesOnTheWishlist(t *testing.T) {
r := newRig(true)
ctx := context.Background()
// The model is built by the wish itself when no request for suggestions did it.
if _, err := r.svc.AddToWishlist(ctx, 1, "nobody I played"); !errors.Is(err, ErrUnknownArtist) {
t.Errorf("err = %v, want ErrUnknownArtist", err)
}
if _, err := r.svc.AddToWishlist(ctx, 1, " "); !errors.Is(err, ErrUnknownArtist) {
t.Errorf("a blank name: %v, want ErrUnknownArtist", err)
}
id, err := r.svc.AddToWishlist(ctx, 1, "canali")
if err != nil || id != 77 || len(r.wishes.names) != 1 || r.wishes.names[0] != "canali" {
t.Errorf("AddToWishlist = %d, %v, names %v", id, err, r.wishes.names)
}
}

View file

@ -16,6 +16,7 @@ import (
"go.uber.org/zap"
"codeberg.org/squidd/sonda/internal/about"
"codeberg.org/squidd/sonda/internal/affinity"
"codeberg.org/squidd/sonda/internal/auth"
"codeberg.org/squidd/sonda/internal/catalog"
"codeberg.org/squidd/sonda/internal/discover"
@ -149,6 +150,10 @@ func newTestEnvDB(t *testing.T, mutate func(*Deps, *bun.DB)) (http.Handler, *bun
}
}
// Built from the final services, so a test that swaps one gets a Discover that uses it.
if deps.Suggestions == nil {
repo := storage.NewAffinityRepo(db)
deps.Suggestions = affinity.NewService(repo, repo, feedbackSvc, repo, usersettings.NewService(storage.NewUserSettingsRepo(db)), 20*time.Second)
}
if deps.Discover == nil {
deps.Discover = discover.NewService(deps.Taste.(discover.Taster), deps.Discovery.(discover.Similar),
discovery.NewTagService(storage.NewDiscoveryRepo(db), map[string]discovery.TagSource{}, prov, zap.NewNop()),

View file

@ -881,6 +881,21 @@ func TestResponsesMatchTheSpec(t *testing.T) {
c.expect(c.upload("/me/import/spotify", token, export[:len(export)-20]), 400, "a file cut short", "unreadable_file")
c.expect(c.upload("/me/import/spotify", ""), 401, "an import without a token", "unauthorized")
// Suggestions from the listening: refused until it is allowed, then empty for a history with nothing to learn from.
c.expect(c.call("PATCH", "/me/settings", token, map[string]any{"discover_from_listening": false}), 200, "forbid Discover from the listening")
c.expect(c.call("GET", "/me/suggestions", token, nil), 400, "suggestions before they are allowed", "listening_not_allowed")
c.expect(c.call("POST", "/me/suggestions/wishlist", token, map[string]any{"name": "Nobody"}), 400, "a wish before it is allowed", "listening_not_allowed")
c.expect(c.call("PATCH", "/me/settings", token, map[string]any{"discover_from_listening": true}), 200, "allow Discover from the listening")
suggested := c.expect(c.call("GET", "/me/suggestions?limit=5", token, nil), 200, "suggestions")
if lib, _ := suggested["library"].([]any); lib == nil {
c.t.Errorf("suggestions = %v, want empty lists and not null", suggested)
}
c.expect(c.call("GET", "/me/suggestions?limit=-1", token, nil), 400, "a negative limit", "invalid_input")
c.expect(c.call("GET", "/me/suggestions", "", nil), 401, "suggestions without a token", "unauthorized")
c.expect(c.call("POST", "/me/suggestions/wishlist", token, map[string]any{"name": "Nobody I played"}), 404, "an artist not played", "artist_not_played")
c.expect(c.call("POST", "/me/suggestions/wishlist", "", map[string]any{"name": "x"}), 401, "a wish without a token", "unauthorized")
c.expect(c.call("PATCH", "/me/settings", token, map[string]any{"discover_from_listening": false}), 200, "forbid it again")
// ---- duplicate songs ----
c.duplicates(t, token, libID)

View file

@ -0,0 +1,120 @@
package httpapi
import (
"context"
"errors"
"net/http"
"strings"
"time"
"github.com/gin-gonic/gin"
"codeberg.org/squidd/sonda/internal/affinity"
"codeberg.org/squidd/sonda/internal/feedback"
)
// listeningSuggestionService is what the handlers need from affinity.Service.
type listeningSuggestionService interface {
Suggest(ctx context.Context, userID int64, limit int) (affinity.Suggestions, error)
AddToWishlist(ctx context.Context, userID int64, name string) (int64, error)
}
type listeningSuggestionHandlers struct{ svc listeningSuggestionService }
type ownedReasonJSON struct {
Name string `json:"name"`
Score float64 `json:"score"`
}
type listeningSuggestionJSON struct {
Name string `json:"name"`
ArtistID *int64 `json:"artist_id"`
Score float64 `json:"score"`
Plays int `json:"plays"`
LastPlayedAt time.Time `json:"last_played_at"`
Wishlisted bool `json:"wishlisted"`
Because []ownedReasonJSON `json:"because"`
}
type listeningSuggestionsJSON struct {
Library []listeningSuggestionJSON `json:"library"`
Missing []listeningSuggestionJSON `json:"missing"`
Built struct {
At time.Time `json:"at"`
Plays int `json:"plays"`
Sessions int `json:"sessions"`
Artists int `json:"artists"`
Links int `json:"links"`
} `json:"built"`
}
func toListeningSuggestions(in []affinity.Suggestion) []listeningSuggestionJSON {
out := make([]listeningSuggestionJSON, 0, len(in))
for _, s := range in {
j := listeningSuggestionJSON{Name: s.Name, Score: s.Score, Plays: s.Plays, LastPlayedAt: s.LastPlayed.UTC(), Wishlisted: s.Wishlisted, Because: make([]ownedReasonJSON, 0, len(s.Because))}
if s.ArtistID != 0 {
id := s.ArtistID
j.ArtistID = &id
}
for _, r := range s.Because {
j.Because = append(j.Because, ownedReasonJSON{Name: r.Name, Score: r.Score})
}
out = append(out, j)
}
return out
}
// list suggests artists from how the signed-in person listens.
func (h *listeningSuggestionHandlers) list(c *gin.Context) {
limit, ok := queryInt(c, "limit")
if !ok {
return
}
got, err := h.svc.Suggest(c.Request.Context(), currentUserID(c), clampInt(limit))
switch {
case errors.Is(err, affinity.ErrNotAllowed):
writeError(c, http.StatusBadRequest, "listening_not_allowed", err.Error())
return
case errors.Is(err, affinity.ErrInvalidLimit):
writeError(c, http.StatusBadRequest, "invalid_input", err.Error())
return
case err != nil:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
return
}
out := listeningSuggestionsJSON{Library: toListeningSuggestions(got.Library), Missing: toListeningSuggestions(got.Missing)}
out.Built.At, out.Built.Plays, out.Built.Sessions = got.Built.BuiltAt.UTC(), got.Built.Plays, got.Built.Sessions
out.Built.Artists, out.Built.Links = got.Built.Artists, got.Built.Links
c.JSON(http.StatusOK, out)
}
type wishRequest struct {
Name string `json:"name"`
}
// wish puts an artist the person has played on their wishlist, by name.
func (h *listeningSuggestionHandlers) wish(c *gin.Context) {
var req wishRequest
if !bindJSON(c, &req) {
return
}
if strings.TrimSpace(req.Name) == "" {
writeError(c, http.StatusBadRequest, "invalid_input", "name is required")
return
}
id, err := h.svc.AddToWishlist(c.Request.Context(), currentUserID(c), req.Name)
switch {
case err == nil:
c.JSON(http.StatusOK, gin.H{"artist_id": id})
case errors.Is(err, affinity.ErrNotAllowed):
writeError(c, http.StatusBadRequest, "listening_not_allowed", err.Error())
case errors.Is(err, affinity.ErrUnknownArtist):
writeError(c, http.StatusNotFound, "artist_not_played", err.Error())
case errors.Is(err, feedback.ErrUserNotFound):
writeError(c, http.StatusUnauthorized, "unauthorized", "sign in again")
default:
_ = c.Error(err)
writeError(c, http.StatusInternalServerError, "internal", "internal error")
}
}

View file

@ -110,6 +110,9 @@ type spotifyImportService interface {
History(ctx context.Context, userID int64, limit, offset int) (spotifyimport.RunPage, error)
}
// listeningSuggestionService is what the handlers need from affinity.Service.
// (declared in handlers_suggestions.go)
// mixService is what the handlers need from mixes.Service.
type mixService interface {
List(ctx context.Context, userID int64) ([]mixes.Mix, error)
@ -227,6 +230,7 @@ type Deps struct {
Feedback feedbackService
Playlists playlistService
SpotifyImport spotifyImportService
Suggestions listeningSuggestionService
Mixes mixService
Smart smartService
Naming namingService
@ -304,6 +308,9 @@ func NewRouter(d Deps) *gin.Engine {
protected.POST("/listens", l.record)
protected.GET("/listens", l.list)
sg := &listeningSuggestionHandlers{svc: d.Suggestions}
protected.GET("/me/suggestions", sg.list)
protected.POST("/me/suggestions/wishlist", sg.wish)
ts := &tasteHandlers{svc: d.Taste}
protected.GET("/me/taste", ts.taste)
dc := &discoverHandlers{svc: d.Discover}

View file

@ -0,0 +1,174 @@
package httpapi
import (
"fmt"
"net/http"
"testing"
"time"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/storage"
)
const suggestionsPath = "/api/v1/me/suggestions"
// listeningHistory plays this out for the only user: three old sittings of Radiohead, Pixies and (outside the library)
// Giorgio Canali, and two recent ones of Radiohead and Placebo. Pixies and Placebo have a song in the library.
func listeningHistory(t *testing.T, db *bun.DB) {
t.Helper()
seedSong(t, db, "a.flac", "Airbag", "OK Computer", "Radiohead")
seedSong(t, db, "b.flac", "Debaser", "Doolittle", "Pixies")
seedSong(t, db, "c.flac", "Nancy Boy", "Placebo", "Placebo")
var user int64
if err := db.QueryRow("SELECT id FROM users LIMIT 1").Scan(&user); err != nil {
t.Fatal(err)
}
track := func(title string) int64 {
var id int64
if err := db.QueryRow("SELECT id FROM tracks WHERE title = ?", title).Scan(&id); err != nil {
t.Fatal(err)
}
return id
}
n := 0
event := func(title string, at time.Time) {
n++
if _, err := db.Exec("INSERT INTO listening_events (user_id, track_id, event_type, occurred_at, duration_played_ms, idempotency_key) VALUES (?, ?, 'complete', ?, 200000, ?)",
user, track(title), at.Unix(), fmt.Sprintf("k%d", n)); err != nil {
t.Fatal(err)
}
}
away := func(artist string, at time.Time) {
n++
if _, err := db.Exec("INSERT INTO outside_plays (user_id, idempotency_key, artist, artist_key, played_at) VALUES (?, ?, ?, ?, ?)",
user, fmt.Sprintf("o%d", n), artist, storage.SearchKey(artist), at.Unix()); err != nil {
t.Fatal(err)
}
}
now := time.Now()
for _, days := range []int{200, 199, 198} {
at := now.AddDate(0, 0, -days)
event("Airbag", at)
event("Debaser", at.Add(time.Minute))
away("Giorgio Canali", at.Add(2*time.Minute))
}
for _, days := range []int{10, 9} {
at := now.AddDate(0, 0, -days)
event("Airbag", at)
event("Nancy Boy", at.Add(time.Minute))
}
}
func allowListening(t *testing.T, h http.Handler, tok string) {
t.Helper()
if r := do(t, h, "PATCH", "/api/v1/me/settings", tok, map[string]any{"discover_from_listening": true}); r.Code != http.StatusOK {
t.Fatalf("settings: %d %v", r.Code, r.Body)
}
}
func artistNames(body map[string]any, list string) []string {
out := []string{}
items, _ := body[list].([]any)
for _, it := range items {
out = append(out, it.(map[string]any)["name"].(string))
}
return out
}
func TestSuggestionsComeFromHowThePersonListensAndNeedTheirPermission(t *testing.T) {
h, db := newTestEnv(t, func(*Deps) {})
tok := registerAndLogin(t, h)
listeningHistory(t, db)
if r := do(t, h, "GET", suggestionsPath, tok, nil); r.Code != http.StatusBadRequest || errCode(r) != "listening_not_allowed" {
t.Fatalf("without permission: %d %v, want 400 listening_not_allowed", r.Code, r.Body)
}
allowListening(t, h, tok)
r := do(t, h, "GET", suggestionsPath, tok, nil)
if r.Code != http.StatusOK {
t.Fatalf("suggestions: %d %v", r.Code, r.Body)
}
sameKeys(t, "suggestions", r.Body, "library", "missing", "built")
if got := artistNames(r.Body, "library"); len(got) != 1 || got[0] != "Pixies" {
t.Errorf("library = %v, want Pixies (Placebo and Radiohead were played lately)", got)
}
if got := artistNames(r.Body, "missing"); len(got) != 1 || got[0] != "Giorgio Canali" {
t.Errorf("missing = %v, want Giorgio Canali", got)
}
pixies := r.Body["library"].([]any)[0].(map[string]any)
sameKeys(t, "suggestion", pixies, "name", "artist_id", "score", "plays", "last_played_at", "wishlisted", "because")
if pixies["artist_id"] == nil || pixies["score"] != 1.0 || pixies["plays"] != 3.0 || pixies["wishlisted"] != false {
t.Errorf("Pixies = %v", pixies)
}
canali := r.Body["missing"].([]any)[0].(map[string]any)
if canali["artist_id"] != nil || canali["plays"] != 3.0 {
t.Errorf("Canali = %v, want no artist yet and 3 plays", canali)
}
built, _ := r.Body["built"].(map[string]any)
sameKeys(t, "built", built, "at", "plays", "sessions", "artists", "links")
if built["sessions"] != 5.0 || built["artists"] != 4.0 {
t.Errorf("built = %v, want 5 sittings and 4 artists", built)
}
if r := do(t, h, "GET", suggestionsPath+"?limit=-1", tok, nil); r.Code != http.StatusBadRequest || errCode(r) != "invalid_input" {
t.Errorf("a negative limit: %d %v", r.Code, r.Body)
}
if r := do(t, h, "GET", suggestionsPath, "", nil); r.Code != http.StatusUnauthorized {
t.Errorf("without a token: %d, want 401", r.Code)
}
}
func TestAMissingArtistGoesOnTheWishlistByName(t *testing.T) {
h, db := newTestEnv(t, func(*Deps) {})
tok := registerAndLogin(t, h)
listeningHistory(t, db)
allowListening(t, h, tok)
wish := suggestionsPath + "/wishlist"
if r := do(t, h, "POST", wish, tok, map[string]any{"name": "Nobody I played"}); r.Code != http.StatusNotFound || errCode(r) != "artist_not_played" {
t.Errorf("an artist not played: %d %v, want 404 artist_not_played", r.Code, r.Body)
}
if r := do(t, h, "POST", wish, tok, map[string]any{"name": "Giorgio Canali"}); r.Code != http.StatusOK || r.Body["artist_id"] == nil {
t.Fatalf("wish: %d %v", r.Code, r.Body)
}
if r := do(t, h, "POST", wish, tok, map[string]any{"name": "Giorgio Canali"}); r.Code != http.StatusOK {
t.Errorf("wishing twice: %d %v, want 200", r.Code, r.Body)
}
list := do(t, h, "GET", "/api/v1/wishlist", tok, nil)
items, _ := list.Body["items"].([]any)
if len(items) != 1 || items[0].(map[string]any)["artist"].(map[string]any)["name"] != "Giorgio Canali" {
t.Errorf("wishlist = %v, want the artist", list.Body)
}
if got := do(t, h, "GET", suggestionsPath, tok, nil); !got.Body["missing"].([]any)[0].(map[string]any)["wishlisted"].(bool) {
t.Errorf("the suggestion does not say it is on the wishlist: %v", got.Body["missing"])
}
var named int
if err := db.QueryRow("SELECT COUNT(*) FROM artists WHERE name = 'Giorgio Canali' AND mbid IS NULL AND discovered = ?", true).Scan(&named); err != nil || named != 1 {
t.Errorf("%d artists made for it (%v), want one without a MusicBrainz id", named, err)
}
if r := do(t, h, "POST", wish, tok, map[string]any{"nope": 1}); r.Code != http.StatusBadRequest {
t.Errorf("a body that is not valid: %d, want 400", r.Code)
}
if r := do(t, h, "POST", wish, "", map[string]any{"name": "x"}); r.Code != http.StatusUnauthorized {
t.Errorf("without a token: %d, want 401", r.Code)
}
}
func TestAnotherPersonSeesNothingOfMyListening(t *testing.T) {
h, db := newTestEnv(t, func(*Deps) {})
tok := registerAndLogin(t, h)
listeningHistory(t, db)
_, other := addUser(t, h, tok, "other", false)
allowListening(t, h, other)
r := do(t, h, "GET", suggestionsPath, other, nil)
if r.Code != http.StatusOK || len(artistNames(r.Body, "library")) != 0 || len(artistNames(r.Body, "missing")) != 0 {
t.Errorf("another person: %d %v, want empty lists", r.Code, r.Body)
}
if r := do(t, h, "POST", suggestionsPath+"/wishlist", other, map[string]any{"name": "Giorgio Canali"}); r.Code != http.StatusNotFound {
t.Errorf("wishing an artist only someone else played: %d, want 404", r.Code)
}
}

View file

@ -2053,6 +2053,86 @@ paths:
'400': {$ref: '#/components/responses/InvalidInput'}
'401': {$ref: '#/components/responses/Unauthorized'}
/me/suggestions:
get:
tags: [Taste]
operationId: getMySuggestions
summary: Artists suggested by how you listen
description: |
Suggestions learned from your own listening, with no provider and nothing sent anywhere. Sonda splits your
history into sittings (plays less than 30 minutes apart) and links the artists that are often played in the
same sitting; the model is built from your listening events and from the plays of artists you have no music
of that an import kept (see `POST /me/import/spotify`), and is built again when plays are added. It needs
`discover_from_listening` in `PATCH /me/settings` (`listening_not_allowed` without it).
`library` are artists you have, linked to what you play most, that you did not play in the last 60 days,
best first. `missing` are artists you have played at least three times and have no music of, the ones you play
most that also go with what you play now first. Each says which of the artists you play it goes with
(`because`) and has a `score` from 0 to 1 within its list. `artist_id` is null for an artist that does not
exist on this server yet. Artists you dismissed are left out.
security: [{bearerAuth: []}]
parameters:
- name: limit
in: query
description: How many artists in each list. 20 when absent or 0; values above 50 are lowered to 50.
schema: {type: integer, minimum: 0, default: 20}
responses:
'200':
description: The suggestions. Both lists are empty until you have listened to something.
content:
application/json:
schema: {$ref: '#/components/schemas/ListeningSuggestions'}
'400':
description: A negative limit (`invalid_input`), or Discover is not allowed to start from your listening (`listening_not_allowed`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'401': {$ref: '#/components/responses/Unauthorized'}
/me/suggestions/wishlist:
post:
tags: [Taste]
operationId: wishArtistFromSuggestions
summary: Put a suggested artist you do not have on your wishlist
description: |
Puts an artist you have played on your wishlist by its name, for the artists of `missing` that have no
`artist_id`: the artist is made on this server without any music, like the ones a provider names, and has
no MusicBrainz id. The name must be one the model knows from your listening (`artist_not_played`). Doing it
twice keeps the first date. It needs `discover_from_listening` like the suggestions.
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema:
type: object
additionalProperties: false
required: [name]
properties:
name: {type: string}
responses:
'200':
description: The artist is on your wishlist.
content:
application/json:
schema:
type: object
additionalProperties: false
required: [artist_id]
properties:
artist_id: {type: integer}
'400':
description: The body is not valid (`invalid_input`), or Discover is not allowed to start from your listening (`listening_not_allowed`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
'401': {$ref: '#/components/responses/Unauthorized'}
'404':
description: You have not played an artist with this name (`artist_not_played`).
content:
application/json:
schema: {$ref: '#/components/schemas/Error'}
/me/settings:
get:
tags: [Taste]
@ -4866,6 +4946,51 @@ components:
duration_played_ms: {type: integer, format: int64}
counted: {type: boolean, description: Whether the event counts as a listen under the current rule.}
ListeningSuggestion:
type: object
additionalProperties: false
required: [name, artist_id, score, plays, last_played_at, wishlisted, because]
properties:
name: {type: string}
artist_id: {type: integer, nullable: true, description: 'The artist on this server, null when there is none yet.'}
score: {type: number, description: 'From 0 to 1 within its list.'}
plays: {type: integer, description: 'How many times you played it.'}
last_played_at: {type: string, format: date-time}
wishlisted: {type: boolean}
because:
type: array
description: The artists you play that it goes with, the strongest first, at most three.
items:
type: object
additionalProperties: false
required: [name, score]
properties:
name: {type: string}
score: {type: number}
ListeningSuggestions:
type: object
additionalProperties: false
required: [library, missing, built]
properties:
library:
type: array
items: {$ref: '#/components/schemas/ListeningSuggestion'}
missing:
type: array
items: {$ref: '#/components/schemas/ListeningSuggestion'}
built:
type: object
additionalProperties: false
required: [at, plays, sessions, artists, links]
description: When the model was built and from how much.
properties:
at: {type: string, format: date-time}
plays: {type: integer}
sessions: {type: integer}
artists: {type: integer}
links: {type: integer}
Taste:
type: object
additionalProperties: false

View file

@ -0,0 +1,315 @@
package storage
import (
"context"
"database/sql"
"fmt"
"strings"
"time"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/affinity"
"codeberg.org/squidd/sonda/internal/feedback"
)
// AffinityRepo reads the plays of a person and keeps what was learned from them. It implements affinity.Store,
// affinity.Library and affinity.Wishes.
type AffinityRepo struct{ db *bun.DB }
// NewAffinityRepo builds an AffinityRepo.
func NewAffinityRepo(db *bun.DB) *AffinityRepo { return &AffinityRepo{db: db} }
// counted is the condition "this listening event counts as a listen" (see POST /listens), with its one argument.
const counted = "(e.event_type = 'complete' OR e.duration_played_ms >= ?)"
// Plays gives every play of a person that counts as listened, once for each artist credited on it: the listening
// events of the library and the plays kept from imports, leaving out the kept ones that have a listening event now.
func (r *AffinityRepo) Plays(ctx context.Context, userID int64, minPlayed time.Duration) ([]affinity.Play, error) {
var plays []affinity.Play
rows, err := r.db.QueryContext(ctx,
`SELECT a.name, e.occurred_at FROM listening_events e
JOIN track_artists ta ON ta.track_id = e.track_id JOIN artists a ON a.id = ta.artist_id
WHERE e.user_id = ? AND `+counted, userID, minPlayed.Milliseconds())
if err != nil {
return nil, fmt.Errorf("read listens: %w", err)
}
defer rows.Close()
for rows.Next() {
var (
name string
at int64
)
if err := rows.Scan(&name, &at); err != nil {
return nil, err
}
plays = append(plays, affinity.Play{Key: searchKey(name), Name: name, At: time.Unix(at, 0)})
}
if err := rows.Err(); err != nil {
return nil, err
}
away, err := r.db.QueryContext(ctx,
`SELECT o.artist, o.artist_key, o.played_at FROM outside_plays o WHERE o.user_id = ?
AND NOT EXISTS (SELECT 1 FROM listening_events e WHERE e.user_id = o.user_id AND e.idempotency_key = o.idempotency_key)`, userID)
if err != nil {
return nil, fmt.Errorf("read outside plays: %w", err)
}
defer away.Close()
for away.Next() {
var (
name, key string
at int64
)
if err := away.Scan(&name, &key, &at); err != nil {
return nil, err
}
plays = append(plays, affinity.Play{Key: key, Name: name, At: time.Unix(at, 0)})
}
return plays, away.Err()
}
// PlayCount says how many plays Plays would give, without reading them.
func (r *AffinityRepo) PlayCount(ctx context.Context, userID int64, minPlayed time.Duration) (int, error) {
var events, away int
if err := r.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM listening_events e JOIN track_artists ta ON ta.track_id = e.track_id
WHERE e.user_id = ? AND `+counted, userID, minPlayed.Milliseconds()).Scan(&events); err != nil {
return 0, fmt.Errorf("count listens: %w", err)
}
if err := r.db.QueryRowContext(ctx,
`SELECT COUNT(*) FROM outside_plays o WHERE o.user_id = ?
AND NOT EXISTS (SELECT 1 FROM listening_events e WHERE e.user_id = o.user_id AND e.idempotency_key = o.idempotency_key)`, userID).Scan(&away); err != nil {
return 0, fmt.Errorf("count outside plays: %w", err)
}
return events + away, nil
}
// affinityChunk is how many rows one statement inserts: at most seven columns each.
const affinityChunk = 500
type affinityArtistRow struct {
bun.BaseModel `bun:"table:affinity_artists"`
UserID int64 `bun:"user_id,pk"`
ArtistKey string `bun:"artist_key,pk"`
Name string `bun:"name,notnull"`
Plays int `bun:"plays,notnull"`
Sessions int `bun:"sessions,notnull"`
LastPlayedAt int64 `bun:"last_played_at,notnull"`
Weight float64 `bun:"weight,notnull"`
}
type affinityLinkRow struct {
bun.BaseModel `bun:"table:affinity_links"`
UserID int64 `bun:"user_id,pk"`
AKey string `bun:"a_key,pk"`
BKey string `bun:"b_key,pk"`
Score float64 `bun:"score,notnull"`
}
type affinityBuildRow struct {
bun.BaseModel `bun:"table:affinity_builds"`
UserID int64 `bun:"user_id,pk"`
BuiltAt int64 `bun:"built_at,notnull"`
Plays int `bun:"plays,notnull"`
Sessions int `bun:"sessions,notnull"`
Artists int `bun:"artists,notnull"`
Links int `bun:"links,notnull"`
}
// Save replaces what was learned about a person with a new model, in one transaction. The plays kept from imports
// that have a listening event now are removed: the event says it better.
func (r *AffinityRepo) Save(ctx context.Context, userID int64, at time.Time, m affinity.Model) error {
return r.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
for _, table := range []string{"affinity_links", "affinity_artists", "affinity_builds"} {
if _, err := tx.ExecContext(ctx, "DELETE FROM "+table+" WHERE user_id = ?", userID); err != nil {
return fmt.Errorf("clear %s: %w", table, err)
}
}
for start := 0; start < len(m.Artists); start += affinityChunk {
end := min(start+affinityChunk, len(m.Artists))
rows := make([]affinityArtistRow, 0, end-start)
for _, a := range m.Artists[start:end] {
rows = append(rows, affinityArtistRow{UserID: userID, ArtistKey: a.Key, Name: a.Name, Plays: a.Plays, Sessions: a.Sessions, LastPlayedAt: a.Last.Unix(), Weight: a.Weight})
}
if _, err := tx.NewInsert().Model(&rows).Exec(ctx); err != nil {
return fmt.Errorf("insert affinity artists: %w", err)
}
}
for start := 0; start < len(m.Links); start += affinityChunk {
end := min(start+affinityChunk, len(m.Links))
rows := make([]affinityLinkRow, 0, end-start)
for _, l := range m.Links[start:end] {
rows = append(rows, affinityLinkRow{UserID: userID, AKey: l.A, BKey: l.B, Score: l.Score})
}
if _, err := tx.NewInsert().Model(&rows).Exec(ctx); err != nil {
return fmt.Errorf("insert affinity links: %w", err)
}
}
build := &affinityBuildRow{UserID: userID, BuiltAt: at.Unix(), Plays: m.Plays, Sessions: m.Sessions, Artists: len(m.Artists), Links: len(m.Links)}
if _, err := tx.NewInsert().Model(build).Exec(ctx); err != nil {
return fmt.Errorf("insert affinity build: %w", err)
}
if _, err := tx.ExecContext(ctx,
`DELETE FROM outside_plays WHERE user_id = ? AND idempotency_key IN
(SELECT idempotency_key FROM listening_events WHERE user_id = ?)`, userID, userID); err != nil {
return fmt.Errorf("drop outside plays that are listens now: %w", err)
}
return nil
})
}
// Build says when the model of a person was built and from how much; ok is false when it never was.
func (r *AffinityRepo) Build(ctx context.Context, userID int64) (b affinity.Built, ok bool, err error) {
var row affinityBuildRow
err = r.db.NewSelect().Model(&row).Where("user_id = ?", userID).Scan(ctx)
if err == sql.ErrNoRows {
return b, false, nil
}
if err != nil {
return b, false, fmt.Errorf("read affinity build: %w", err)
}
return affinity.Built{BuiltAt: time.Unix(row.BuiltAt, 0), Plays: row.Plays, Sessions: row.Sessions, Artists: row.Artists, Links: row.Links}, true, nil
}
// Seeds gives the artists a person plays the most (by weight), best first.
func (r *AffinityRepo) Seeds(ctx context.Context, userID int64, limit int) ([]affinity.Artist, error) {
var rows []affinityArtistRow
if err := r.db.NewSelect().Model(&rows).Where("user_id = ?", userID).OrderExpr("weight DESC, artist_key").Limit(int64(limit)).Scan(ctx); err != nil {
return nil, fmt.Errorf("read affinity seeds: %w", err)
}
return artistsOf(rows), nil
}
// Artists gives what is known of the artists with these keys.
func (r *AffinityRepo) Artists(ctx context.Context, userID int64, keys []string) (map[string]affinity.Artist, error) {
out := map[string]affinity.Artist{}
for start := 0; start < len(keys); start += affinityChunk {
end := min(start+affinityChunk, len(keys))
var rows []affinityArtistRow
if err := r.db.NewSelect().Model(&rows).Where("user_id = ?", userID).Where("artist_key IN (?)", bun.In(keys[start:end])).Scan(ctx); err != nil {
return nil, fmt.Errorf("read affinity artists: %w", err)
}
for _, a := range artistsOf(rows) {
out[a.Key] = a
}
}
return out, nil
}
func artistsOf(rows []affinityArtistRow) []affinity.Artist {
out := make([]affinity.Artist, 0, len(rows))
for _, row := range rows {
out = append(out, affinity.Artist{Key: row.ArtistKey, Name: row.Name, Plays: row.Plays, Sessions: row.Sessions, Last: time.Unix(row.LastPlayedAt, 0), Weight: row.Weight})
}
return out
}
// Neighbours gives every link that has one of these artists at one end.
func (r *AffinityRepo) Neighbours(ctx context.Context, userID int64, keys []string) ([]affinity.Link, error) {
if len(keys) == 0 {
return nil, nil
}
var rows []affinityLinkRow
if err := r.db.NewSelect().Model(&rows).Where("user_id = ?", userID).
WhereGroup(" AND ", func(q *bun.SelectQuery) *bun.SelectQuery {
return q.Where("a_key IN (?)", bun.In(keys)).WhereOr("b_key IN (?)", bun.In(keys))
}).Scan(ctx); err != nil {
return nil, fmt.Errorf("read affinity links: %w", err)
}
out := make([]affinity.Link, 0, len(rows))
for _, row := range rows {
out = append(out, affinity.Link{A: row.AKey, B: row.BKey, Score: row.Score})
}
return out, nil
}
// LibraryArtists says, for these search keys, which artists exist: the id, and whether there is music to play. When
// several artists share a key the one with music comes first.
func (r *AffinityRepo) LibraryArtists(ctx context.Context, keys []string) (map[string]affinity.LibraryArtist, error) {
out := map[string]affinity.LibraryArtist{}
for start := 0; start < len(keys); start += affinityChunk {
end := min(start+affinityChunk, len(keys))
rows, err := r.db.QueryContext(ctx,
"SELECT a.id, a.name, a.search_key, "+playableArtist("a.id")+" FROM artists a WHERE a.search_key IN (?) ORDER BY a.id",
bun.In(keys[start:end]))
if err != nil {
return nil, fmt.Errorf("find artists: %w", err)
}
for rows.Next() {
var la affinity.LibraryArtist
var key string
if err := rows.Scan(&la.ID, &la.Name, &key, &la.Playable); err != nil {
rows.Close()
return nil, err
}
if have, ok := out[key]; !ok || (la.Playable && !have.Playable) {
out[key] = la
}
}
if err := rows.Err(); err != nil {
rows.Close()
return nil, err
}
rows.Close()
}
return out, nil
}
// Wishlist puts an artist on the wishlist of a person by name: the artist that already has this name (the one with
// music first), or a new one that exists only because of this, like the ones a provider names. It gives the id.
func (r *AffinityRepo) Wishlist(ctx context.Context, userID int64, name string, at time.Time) (int64, error) {
var id int64
err := r.db.RunInTx(ctx, nil, func(ctx context.Context, tx bun.Tx) error {
if err := lockArtistWrites(ctx, tx); err != nil {
return err
}
var one int
if err := tx.QueryRowContext(ctx, "SELECT 1 FROM users WHERE id = ?", userID).Scan(&one); err != nil {
return mapNoRows(err, feedback.ErrUserNotFound, "check user")
}
key := searchKey(name)
rows, err := tx.QueryContext(ctx, "SELECT a.id, "+playableArtist("a.id")+" FROM artists a WHERE a.search_key = ? ORDER BY a.id", key)
if err != nil {
return fmt.Errorf("find artist: %w", err)
}
var playable bool
for rows.Next() {
var cand int64
var p bool
if err := rows.Scan(&cand, &p); err != nil {
rows.Close()
return err
}
if id == 0 || (p && !playable) {
id, playable = cand, p
}
}
if err := rows.Err(); err != nil {
rows.Close()
return err
}
rows.Close()
if id == 0 {
if id, err = insertDiscovered(ctx, tx, strings.TrimSpace(name), ""); err != nil {
return err
}
}
if _, err := tx.ExecContext(ctx,
`INSERT INTO artist_feedback (user_id, artist_id, kind, created_at) VALUES (?, ?, ?, ?)
ON CONFLICT (user_id, artist_id) DO UPDATE SET
created_at = CASE WHEN artist_feedback.kind = excluded.kind THEN artist_feedback.created_at ELSE excluded.created_at END,
kind = excluded.kind`,
userID, id, string(feedback.Wishlist), at.Unix()); err != nil {
return fmt.Errorf("write feedback: %w", err)
}
return nil
})
if err != nil {
return 0, err
}
return id, nil
}

View file

@ -0,0 +1,210 @@
package storage
import (
"context"
"testing"
"time"
"github.com/uptrace/bun"
"codeberg.org/squidd/sonda/internal/affinity"
"codeberg.org/squidd/sonda/internal/library"
"codeberg.org/squidd/sonda/internal/listening"
"codeberg.org/squidd/sonda/internal/spotifyimport"
)
// affinitySong puts a song with one artist in the library and gives the id of its track.
func affinitySong(t *testing.T, db *bun.DB, lib int64, rel, title, artist string) int64 {
t.Helper()
f := library.ParsedFile{
FileInfo: library.FileInfo{RelPath: rel, Size: 100, MTimeUnix: 1},
Metadata: library.Metadata{
Tags: library.Tags{Title: title, Album: "Album of " + artist, AlbumArtist: artist, Artists: []string{artist}},
Properties: library.Properties{Format: "flac", DurationMs: 200_000, SampleRate: 44100, BitRate: 900},
},
}
if _, err := NewLibraryRepo(db).SaveFile(context.Background(), lib, f, 0); err != nil {
t.Fatal(err)
}
var id int64
if err := db.QueryRow("SELECT id FROM tracks WHERE title = ?", title).Scan(&id); err != nil {
t.Fatal(err)
}
return id
}
func TestTheAffinityRepoReadsThePlaysThatCountAndKeepsTheModel(t *testing.T) {
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
ctx := context.Background()
u, err := NewUserRepo(db).CreateUser(ctx, "alice", "h", false)
if err != nil {
t.Fatal(err)
}
other, err := NewUserRepo(db).CreateUser(ctx, "bob", "h", false)
if err != nil {
t.Fatal(err)
}
lib, err := NewLibraryRepo(db).CreateLibrary(ctx, "music", "/music")
if err != nil {
t.Fatal(err)
}
pixies := affinitySong(t, db, lib.ID, "a.flac", "Debaser", "Pixies")
blur := affinitySong(t, db, lib.ID, "b.flac", "Song 2", "Blur")
minListen := 20 * time.Second
listens := NewListeningRepo(db)
ev := func(user, track int64, typ listening.EventType, ms int64, at int64, key string) listening.Event {
return listening.Event{UserID: user, TrackID: track, Type: typ, DurationPlayedMs: ms, OccurredAt: time.Unix(at, 0), IdempotencyKey: key}
}
for _, e := range []listening.Event{
ev(u.ID, pixies, listening.Complete, 1000, 1000, "p1"), // counts: complete
ev(u.ID, pixies, listening.Play, 30_000, 2000, "p2"), // counts: long enough
ev(u.ID, pixies, listening.Play, 5_000, 3000, "p3"), // too short
ev(u.ID, blur, listening.Skip, 1_000, 4000, "p4"), // a skip
ev(other.ID, blur, listening.Complete, 1000, 5000, "p5"),
} {
if _, err := listens.Record(ctx, e); err != nil {
t.Fatal(err)
}
}
outside := NewOutsidePlayRepo(db)
if _, err := outside.RecordOutside(ctx, u.ID, []spotifyimport.OutsidePlay{
{Key: "o1", Artist: "Giorgio Canali", PlayedAt: time.Unix(6000, 0)},
{Key: "p1", Artist: "Pixies", PlayedAt: time.Unix(1000, 0)}, // the same play as the event p1
}); err != nil {
t.Fatal(err)
}
repo := NewAffinityRepo(db)
plays, err := repo.Plays(ctx, u.ID, minListen)
if err != nil {
t.Fatal(err)
}
got := map[string]int{}
for _, p := range plays {
got[p.Name]++
if p.Key != SearchKey(p.Name) {
t.Errorf("key %q of %q, want its search key", p.Key, p.Name)
}
}
if len(plays) != 3 || got["Pixies"] != 2 || got["Giorgio Canali"] != 1 {
t.Errorf("plays = %v (%d), want Pixies twice and Canali once: the short one, the skip, the other person's and the duplicate are out", got, len(plays))
}
if n, err := repo.PlayCount(ctx, u.ID, minListen); err != nil || n != len(plays) {
t.Errorf("PlayCount = %d, %v, want %d", n, err, len(plays))
}
if _, ok, err := repo.Build(ctx, u.ID); err != nil || ok {
t.Errorf("Build before any = %v, %v, want none", ok, err)
}
at := time.Unix(9000, 0)
model := affinity.Model{
Plays: 3, Sessions: 2,
Artists: []affinity.Artist{
{Key: "pixies", Name: "Pixies", Plays: 2, Sessions: 2, Last: time.Unix(2000, 0), Weight: 1.5},
{Key: "blur", Name: "Blur", Plays: 1, Sessions: 1, Last: time.Unix(1000, 0), Weight: 0.5},
{Key: "giorgio canali", Name: "Giorgio Canali", Plays: 1, Sessions: 1, Last: time.Unix(6000, 0), Weight: 3},
},
Links: []affinity.Link{{A: "blur", B: "pixies", Score: 0.5}, {A: "giorgio canali", B: "pixies", Score: 0.25}},
}
if err := repo.Save(ctx, u.ID, at, model); err != nil {
t.Fatal(err)
}
b, ok, err := repo.Build(ctx, u.ID)
if err != nil || !ok || b.Plays != 3 || b.Sessions != 2 || b.Artists != 3 || b.Links != 2 || b.BuiltAt.Unix() != 9000 {
t.Errorf("Build = %+v, %v, %v", b, ok, err)
}
seeds, err := repo.Seeds(ctx, u.ID, 2)
if err != nil || len(seeds) != 2 || seeds[0].Key != "giorgio canali" || seeds[1].Key != "pixies" || seeds[1].Last.Unix() != 2000 {
t.Errorf("Seeds = %+v, %v, want the two heaviest, heaviest first", seeds, err)
}
arts, err := repo.Artists(ctx, u.ID, []string{"blur", "nobody"})
if err != nil || len(arts) != 1 || arts["blur"].Name != "Blur" {
t.Errorf("Artists = %+v, %v", arts, err)
}
links, err := repo.Neighbours(ctx, u.ID, []string{"blur"})
if err != nil || len(links) != 1 || links[0].A != "blur" || links[0].B != "pixies" || links[0].Score != 0.5 {
t.Errorf("Neighbours(blur) = %+v, %v", links, err)
}
if links, _ := repo.Neighbours(ctx, u.ID, []string{"pixies"}); len(links) != 2 {
t.Errorf("Neighbours(pixies) = %+v, want both links, from either end", links)
}
if links, _ := repo.Neighbours(ctx, other.ID, []string{"pixies"}); len(links) != 0 {
t.Errorf("another person's links = %+v, want none", links)
}
if links, err := repo.Neighbours(ctx, u.ID, nil); err != nil || links != nil {
t.Errorf("Neighbours of nobody = %v, %v", links, err)
}
// Building again replaces it, and drops the kept play that has an event now.
if err := repo.Save(ctx, u.ID, at.Add(time.Hour), affinity.Model{Plays: 1}); err != nil {
t.Fatal(err)
}
if b, _, _ := repo.Build(ctx, u.ID); b.Artists != 0 || b.Links != 0 || b.Plays != 1 {
t.Errorf("Build after a smaller model = %+v", b)
}
if seeds, _ := repo.Seeds(ctx, u.ID, 10); len(seeds) != 0 {
t.Errorf("Seeds = %+v, want none after it was replaced", seeds)
}
if got := count(t, db, "SELECT COUNT(*) FROM outside_plays WHERE user_id = ?", u.ID); got != 1 {
t.Errorf("%d kept plays left, want 1 (the one that is a listening event was dropped)", got)
}
})
}
func TestLibraryArtistsSayWhichHaveMusicAndWishlistFindsOrMakesThem(t *testing.T) {
forEachBackend(t, func(t *testing.T, db *bun.DB, _ string) {
ctx := context.Background()
u, err := NewUserRepo(db).CreateUser(ctx, "alice", "h", false)
if err != nil {
t.Fatal(err)
}
lib, err := NewLibraryRepo(db).CreateLibrary(ctx, "music", "/music")
if err != nil {
t.Fatal(err)
}
affinitySong(t, db, lib.ID, "a.flac", "Debaser", "Pixies")
if _, err := db.ExecContext(ctx, "INSERT INTO artists (name, name_key, search_key, discovered) VALUES ('Björk', 'björk', ?, ?)", SearchKey("Björk"), true); err != nil {
t.Fatal(err)
}
repo := NewAffinityRepo(db)
got, err := repo.LibraryArtists(ctx, []string{SearchKey("Pixies"), SearchKey("Björk"), "nobody"})
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || !got[SearchKey("Pixies")].Playable || got[SearchKey("Pixies")].Name != "Pixies" || got[SearchKey("Björk")].Playable || got[SearchKey("Björk")].ID == 0 {
t.Errorf("LibraryArtists = %+v, want Pixies with music, Björk without, nobody not there", got)
}
// The artist that has the name is the one on the wishlist (also written without the accent) and none is made.
id, err := repo.Wishlist(ctx, u.ID, "Bjork", time.Unix(100, 0))
if err != nil || id != got[SearchKey("Björk")].ID {
t.Errorf("Wishlist(Bjork) = %d, %v, want the artist that exists (%d)", id, err, got[SearchKey("Björk")].ID)
}
// A new name makes an artist that nothing names but the mark, with no MusicBrainz id.
id2, err := repo.Wishlist(ctx, u.ID, " Giorgio Canali ", time.Unix(200, 0))
if err != nil || id2 == 0 {
t.Fatalf("Wishlist(new) = %d, %v", id2, err)
}
var name string
var mbidNull, discovered bool
if err := db.QueryRow("SELECT name, mbid IS NULL, discovered FROM artists WHERE id = ?", id2).Scan(&name, &mbidNull, &discovered); err != nil || name != "Giorgio Canali" || !mbidNull || !discovered {
t.Errorf("made artist = %q, no mbid %v, discovered %v, %v", name, mbidNull, discovered, err)
}
again, err := repo.Wishlist(ctx, u.ID, "Giorgio Canali", time.Unix(300, 0))
if err != nil || again != id2 {
t.Errorf("wishing again = %d, %v, want the same artist %d", again, err, id2)
}
var at int64
if err := db.QueryRow("SELECT created_at FROM artist_feedback WHERE user_id = ? AND artist_id = ?", u.ID, id2).Scan(&at); err != nil || at != 200 {
t.Errorf("mark date = %d, %v, want the first date 200", at, err)
}
if got := count(t, db, "SELECT COUNT(*) FROM artists WHERE search_key = ?", SearchKey("Giorgio Canali")); got != 1 {
t.Errorf("%d artists named Canali, want 1", got)
}
if _, err := repo.Wishlist(ctx, 999999, "Whoever", time.Unix(1, 0)); err == nil {
t.Error("a wish of a person that does not exist was accepted")
}
})
}

View file

@ -156,8 +156,8 @@ func TestMigrateCreatesSchemaAndIsIdempotent(t *testing.T) {
if err := db.QueryRow("SELECT COUNT(*) FROM schema_migrations").Scan(&applied); err != nil {
t.Fatal(err)
}
if applied != 31 {
t.Errorf("applied migrations = %d, want 31", applied)
if applied != 32 {
t.Errorf("applied migrations = %d, want 32", applied)
}
})
}

View file

@ -0,0 +1,33 @@
-- PostgreSQL flavour of sqlite/0032_affinity.sql.
CREATE TABLE affinity_builds (
user_id BIGINT PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
built_at BIGINT NOT NULL,
plays INTEGER NOT NULL,
sessions INTEGER NOT NULL,
artists INTEGER NOT NULL,
links INTEGER NOT NULL
);
CREATE TABLE affinity_artists (
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
artist_key TEXT NOT NULL,
name TEXT NOT NULL,
plays INTEGER NOT NULL,
sessions INTEGER NOT NULL,
last_played_at BIGINT NOT NULL,
weight DOUBLE PRECISION NOT NULL,
PRIMARY KEY (user_id, artist_key)
);
CREATE INDEX idx_affinity_artists_weight ON affinity_artists(user_id, weight DESC);
CREATE TABLE affinity_links (
user_id BIGINT NOT NULL REFERENCES users(id) ON DELETE CASCADE,
a_key TEXT NOT NULL,
b_key TEXT NOT NULL,
score DOUBLE PRECISION NOT NULL,
PRIMARY KEY (user_id, a_key, b_key)
);
CREATE INDEX idx_affinity_links_b ON affinity_links(user_id, b_key);

View file

@ -0,0 +1,35 @@
-- What Sonda learned about which artists a person plays together (see internal/affinity), built again from the
-- listening events and the plays of artists outside the library. It is only a cache of that: it can be dropped and
-- built again. Artists are known by their search key, so one that is not in the library has a place too.
CREATE TABLE affinity_builds (
user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
built_at INTEGER NOT NULL, -- unix seconds
plays INTEGER NOT NULL, -- the plays it was built from
sessions INTEGER NOT NULL,
artists INTEGER NOT NULL,
links INTEGER NOT NULL
);
CREATE TABLE affinity_artists (
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
artist_key TEXT NOT NULL,
name TEXT NOT NULL,
plays INTEGER NOT NULL,
sessions INTEGER NOT NULL,
last_played_at INTEGER NOT NULL, -- unix seconds
weight REAL NOT NULL,
PRIMARY KEY (user_id, artist_key)
);
CREATE INDEX idx_affinity_artists_weight ON affinity_artists(user_id, weight DESC);
CREATE TABLE affinity_links (
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
a_key TEXT NOT NULL, -- a_key < b_key
b_key TEXT NOT NULL,
score REAL NOT NULL,
PRIMARY KEY (user_id, a_key, b_key)
);
CREATE INDEX idx_affinity_links_b ON affinity_links(user_id, b_key);

View file

@ -0,0 +1,38 @@
import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'
import { api } from '@/api/client'
import type { components } from '@/api/schema'
export type ListeningSuggestions = components['schemas']['ListeningSuggestions']
export type ListeningSuggestion = components['schemas']['ListeningSuggestion']
const KEY = ['me', 'suggestions']
/**
* The artists suggested by how you listen: the ones you have and have not played lately, and the ones you play and
* have no music of. Everything is worked out on the server from your own listening. It is asked for only when
* `enabled` (when Discover may start from your listening), and the server builds the model on the first call.
*/
export function useListeningSuggestions({ enabled }: { enabled: boolean }) {
return useQuery({
queryKey: KEY,
queryFn: () => api.get<ListeningSuggestions>('/me/suggestions', { limit: 30 }),
enabled,
retry: false,
})
}
/** Puts an artist you have played on your wishlist by its name, and reads the lists again. */
export function useWishPlayedArtist() {
const client = useQueryClient()
return useMutation({
mutationFn: (name: string) => api.post<{ artist_id: number }>('/me/suggestions/wishlist', { name }),
onSuccess: () =>
Promise.all([client.invalidateQueries({ queryKey: KEY }), client.invalidateQueries({ queryKey: ['/wishlist'] }), client.invalidateQueries({ queryKey: ['/discover/dismissed'] })]),
})
}
/** Reads the suggestions again, after the artist was wished or dismissed in another way. */
export function useRefreshListeningSuggestions() {
const client = useQueryClient()
return () => client.invalidateQueries({ queryKey: KEY })
}

View file

@ -1480,6 +1480,59 @@ export interface paths {
patch?: never;
trace?: never;
};
"/me/suggestions": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
/**
* Artists suggested by how you listen
* @description Suggestions learned from your own listening, with no provider and nothing sent anywhere. Sonda splits your
* history into sittings (plays less than 30 minutes apart) and links the artists that are often played in the
* same sitting; the model is built from your listening events and from the plays of artists you have no music
* of that an import kept (see `POST /me/import/spotify`), and is built again when plays are added. It needs
* `discover_from_listening` in `PATCH /me/settings` (`listening_not_allowed` without it).
*
* `library` are artists you have, linked to what you play most, that you did not play in the last 60 days,
* best first. `missing` are artists you have played at least three times and have no music of, the ones you play
* most that also go with what you play now first. Each says which of the artists you play it goes with
* (`because`) and has a `score` from 0 to 1 within its list. `artist_id` is null for an artist that does not
* exist on this server yet. Artists you dismissed are left out.
*/
get: operations["getMySuggestions"];
put?: never;
post?: never;
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/me/suggestions/wishlist": {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
get?: never;
put?: never;
/**
* Put a suggested artist you do not have on your wishlist
* @description Puts an artist you have played on your wishlist by its name, for the artists of `missing` that have no
* `artist_id`: the artist is made on this server without any music, like the ones a provider names, and has
* no MusicBrainz id. The name must be one the model knows from your listening (`artist_not_played`). Doing it
* twice keeps the first date. It needs `discover_from_listening` like the suggestions.
*/
post: operations["wishArtistFromSuggestions"];
delete?: never;
options?: never;
head?: never;
patch?: never;
trace?: never;
};
"/me/settings": {
parameters: {
query?: never;
@ -3257,6 +3310,36 @@ export interface components {
/** @description Whether the event counts as a listen under the current rule. */
counted: boolean;
};
ListeningSuggestion: {
name: string;
/** @description The artist on this server, null when there is none yet. */
artist_id: number | null;
/** @description From 0 to 1 within its list. */
score: number;
/** @description How many times you played it. */
plays: number;
/** Format: date-time */
last_played_at: string;
wishlisted: boolean;
/** @description The artists you play that it goes with, the strongest first, at most three. */
because: {
name: string;
score: number;
}[];
};
ListeningSuggestions: {
library: components["schemas"]["ListeningSuggestion"][];
missing: components["schemas"]["ListeningSuggestion"][];
/** @description When the model was built and from how much. */
built: {
/** Format: date-time */
at: string;
plays: number;
sessions: number;
artists: number;
links: number;
};
};
Taste: {
artists: components["schemas"]["TasteArtist"][];
tags: components["schemas"]["TasteTag"][];
@ -6532,6 +6615,86 @@ export interface operations {
401: components["responses"]["Unauthorized"];
};
};
getMySuggestions: {
parameters: {
query?: {
/** @description How many artists in each list. 20 when absent or 0; values above 50 are lowered to 50. */
limit?: number;
};
header?: never;
path?: never;
cookie?: never;
};
requestBody?: never;
responses: {
/** @description The suggestions. Both lists are empty until you have listened to something. */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["ListeningSuggestions"];
};
};
/** @description A negative limit (`invalid_input`), or Discover is not allowed to start from your listening (`listening_not_allowed`). */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Error"];
};
};
401: components["responses"]["Unauthorized"];
};
};
wishArtistFromSuggestions: {
parameters: {
query?: never;
header?: never;
path?: never;
cookie?: never;
};
requestBody: {
content: {
"application/json": {
name: string;
};
};
};
responses: {
/** @description The artist is on your wishlist. */
200: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": {
artist_id: number;
};
};
};
/** @description The body is not valid (`invalid_input`), or Discover is not allowed to start from your listening (`listening_not_allowed`). */
400: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Error"];
};
};
401: components["responses"]["Unauthorized"];
/** @description You have not played an artist with this name (`artist_not_played`). */
404: {
headers: {
[name: string]: unknown;
};
content: {
"application/json": components["schemas"]["Error"];
};
};
};
};
getMySettings: {
parameters: {
query?: never;

View file

@ -585,6 +585,7 @@
"tabs": {
"label": "Discover sections",
"suggestions": "Suggestions",
"mine": "From your listening",
"wishlist": "Wishlist",
"dismissed": "Not interested"
},
@ -609,6 +610,22 @@
"partial": "Last.fm could not be reached for some artists, so this list may be shorter.",
"busy": "Discover is busy with another request. Try again in a moment.",
"retryIn": "Try again in {{seconds}} s",
"mine": {
"needs": "These suggestions are learned from what you play. Turn on starting Discover from your listening to see them.",
"intro": "Sonda looks at which artists you play in the same sitting, and suggests more of what goes together. It is worked out on your server from your own listening, and nothing is sent anywhere.",
"built": "Learned from {{plays}} plays in {{sessions}} sittings, on {{when}}.",
"nothing": "You have not played anything yet. Play some music, or import your history from Spotify, and the suggestions appear here.",
"libraryTitle": "In your library, not played lately",
"libraryHint": "Artists you have that go with what you play, and that you have not played for two months.",
"libraryNone": "Nothing to bring back right now.",
"missingTitle": "You play them, but you do not have them",
"missingHint": "Artists you have played at least three times and have no music of here, the ones you play most that also go with what you play now first.",
"missingNone": "No artist you play is missing from your library.",
"lastPlayed": "Last played {{when}}",
"plays_one": "{{count}} play",
"plays_other": "{{count}} plays",
"because": "Goes with {{names}}"
},
"marked": {
"hideOwned": "Hide the ones I already have",
"remove": "Remove",

Some files were not shown because too many files have changed in this diff Show more