feat: write the site as a Gemini capsule #12

Merged
squid merged 6 commits from feat/gemini-output into main 2026-11-16 11:55:40 +00:00
13 changed files with 2519 additions and 22 deletions

View file

@ -1,6 +1,6 @@
# motorino
The static site generator of [kmalloc.org](https://kmalloc.org/). It turns Markdown pages, news, templates and language files into plain HTML with no JavaScript. It is made for that one site and is not a general-purpose generator.
The static site generator of [kmalloc.org](https://kmalloc.org/). It turns Markdown pages, news, templates and language files into plain HTML with no JavaScript, and into a Gemini capsule. It is made for that one site and is not a general-purpose generator.
Licence: GPL-3.0-or-later, see `LICENSE`.
@ -9,6 +9,8 @@ Licence: GPL-3.0-or-later, see `LICENSE`.
```
motorino build -site DIR -out DIR build the site in DIR into the output folder (defaults: . and public)
motorino build -drafts ... the same, with the drafts and the posts dated in the future, for a preview
motorino gemini -site DIR -out DIR write the same site as gemtext, for a Gemini server (defaults: . and capsule)
motorino gemini -lang CODE ... the language at the root of the capsule; -drafts and -clean work as for build
motorino build -clean ... empty the output folder first, if it is new, empty or holds .motorino-out
motorino new -site DIR "A title" write news/a-title.md, a draft with the header ready, and print its path
motorino version print the version
@ -80,6 +82,16 @@ Pages: `Title`, `ID` (the same in every language, it links the translations), `D
`{{name}}`, with lowercase letters, digits and underscores, is replaced by a string from `site.json` or the language file. An unknown placeholder stops the build and names the file. Fenced code blocks (found the way CommonMark finds them, in lists and quotes too) are left as written, so an article can show a placeholder there. Anywhere else a backslash before the placeholder (`\{{name}}`) keeps it as written, in a code span too; two backslashes are one literal backslash followed by the value. Code spans are otherwise substituted, which is how a page shows a value in monospace. The id of a heading is not checked against ids that a template writes itself, such as the `news` of the latest news block. `{{news_list}}`, `{{latest_news}}` and `{{categories_list}}` stand alone in a paragraph and expand to generated lists.
## Gemini
`motorino gemini` reads the same site and writes gemtext: the pages of each language, the news once and a `.meta` file for the server. It does not read the templates, the style or the layouts: a page is its title, its text and a few navigation links.
- **Language at the root.** The language served by default sits at the root and the others under `/<code>/`, the default language of the site included. It is set with `-lang`, else `gemini_language` in `site.json`, else the default language of the site. The server must be started with the same default (Agate: `--lang`). It must be a language that has a home page.
- **`.meta`.** The server reads the language of each document from `<out>/.meta`, which it never serves and no page links to. For every language other than the root one there are two lines, `en/*:;lang=en` and `en/**/*:;lang=en`, because a `*` does not cross folders, and the same two lines for `news/` when the news are not in the root language. It is for Agate run with `--central-conf`. A post that names another `Language` than the news has a line of its own, with the path of its file (`news/2026/11/slug.gmi:;lang=it`). The server uses the first line that matches a document, so these come first, then the folders.
- **News.** They exist once, under `/news/`, as in the website: the index, the posts, the categories, the archives and `news/feed.gmi`, a list of `=> /news/2026/11/post.gmi 2026-11-14 Title` lines, newest first, which feed readers subscribe to. Every page ends with the navigation of its language, which has the news. `{{news_blog}}` lists all the posts, without pages of ten.
- **Conversion.** A heading is a heading (three levels), a paragraph is a line, the links of a paragraph, list, quote or table follow it on lines of their own, a list is bullets (a list of only links is link lines), a quote or a note is a quote, code is a preformatted block, a table is a preformatted block of aligned columns, or, when it is wider than 80 columns, a heading for each row (its first cell) with a bullet for each other cell named by its column, a bullet for each row when there is no header or only one column, an image with an alt text is a link line, unless it belongs to the layout (it has the class `logo`, `mark` or `decorative`, or the role `presentation` or `none`: then nothing is written and the file is not copied), and emphasis is dropped. A list item with no text, only an image or a link, has no bullet; a preformatted block in an item follows its bullet. A page has one first level heading: its title; a first heading that says the title is dropped and the others go one level down. The blocks (`{{news_list}}`, `{{latest_news}}` and the others) become generated lines, with the texts of the language file.
- **Addresses.** An address of the site becomes the address of its page in the capsule (`/people.html` is `/en/people.gmi`), and a fragment is dropped. A file listed in `assets` is copied when a page points to it, and only then. `http`, `https`, `mailto`, `gemini`, `xmpp`, `irc`, `ircs`, `matrix`, `bitcoin`, `magnet`, `tel` and `sms` links stay as they are. A path the website serves by other means (`/~squid/`) becomes a link to the website when it starts with one of the comma separated prefixes of `gemini_web_paths` in `site.json`. Any other address that leads nowhere stops the build and names the page.
## Development
```

View file

@ -6,6 +6,7 @@
// Usage:
//
// motorino build [-site DIR] [-out DIR] [-drafts] [-clean]
// motorino gemini [-site DIR] [-out DIR] [-lang CODE] [-drafts] [-clean]
// motorino new [-site DIR] "title"
// motorino version
package main
@ -26,7 +27,7 @@ import (
)
// usage is the text printed for a missing or unknown command.
const usage = "usage: motorino build [-site DIR] [-out DIR] [-drafts] [-clean] | motorino new [-site DIR] \"title\" | motorino version"
const usage = "usage: motorino build [-site DIR] [-out DIR] [-drafts] [-clean] | motorino gemini [-site DIR] [-out DIR] [-lang CODE] [-drafts] [-clean] | motorino new [-site DIR] \"title\" | motorino version"
// main runs the command line and exits with its status code.
func main() {
@ -46,6 +47,8 @@ func run(args []string, stdout, stderr io.Writer) int {
return 0
case "build":
return runBuild(args[1:], stdout, stderr)
case "gemini":
return runGemini(args[1:], stdout, stderr)
case "new":
return runNew(args[1:], stdout, stderr)
default:
@ -95,6 +98,36 @@ func warnStale(stderr io.Writer, out string, stale []string) {
fmt.Fprintf(stderr, "motorino: warning: %d %s in %s were not written by this build, such as %s; build into a new folder or use -clean\n", len(stale), noun, out, stale[0])
}
// runGemini parses the flags of the gemini command and writes the site as a
// Gemini capsule.
func runGemini(args []string, stdout, stderr io.Writer) int {
flags := flag.NewFlagSet("gemini", flag.ContinueOnError)
flags.SetOutput(stderr)
site := flags.String("site", ".", "the folder of the site")
out := flags.String("out", "capsule", "the folder the gemtext is written to")
lang := flags.String("lang", "", "the language at the root of the capsule (default: gemini_language of site.json, then the default language)")
drafts := flags.Bool("drafts", false, "publish the drafts and the posts dated in the future, for a preview")
clean := flags.Bool("clean", false, "empty the output folder first, if it is new, empty or was emptied this way before")
if err := flags.Parse(args); err != nil {
if errors.Is(err, flag.ErrHelp) {
return 0
}
return 2
}
if flags.NArg() > 0 {
fmt.Fprintf(stderr, "motorino: unexpected arguments: %s\n%s\n", strings.Join(flags.Args(), " "), usage)
return 2
}
result, err := build.BuildGemini(build.GeminiOptions{SiteDir: *site, OutDir: *out, Version: version.String(), Drafts: *drafts, Language: *lang, Clean: *clean})
if err != nil {
fmt.Fprintf(stderr, "motorino: %v\n", err)
return 1
}
fmt.Fprintf(stdout, "wrote %d gemtext files into %s\n", result.Pages, *out)
warnStale(stderr, *out, result.Stale)
return 0
}
// runNew parses the flags of the new command and writes the file of a new post.
// The author is the author key of site.json, when it has one.
func runNew(args []string, stdout, stderr io.Writer) int {

View file

@ -188,6 +188,52 @@ func TestRunBuildDraftsFlag(t *testing.T) {
}
}
// TestRunGemini checks the gemini command: the capsule, the language flag and the drafts flag.
func TestRunGemini(t *testing.T) {
dir := miniSite(t)
if err := os.MkdirAll(filepath.Join(dir, "news"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "news", "wip.md"), []byte("Title: Work in progress\nDate: 2026-01-01T12:00:00+0100\nDraft: true\n-\nSoon.\n"), 0o644); err != nil {
t.Fatal(err)
}
var out, errOut bytes.Buffer
capsule := filepath.Join(t.TempDir(), "capsule")
if code := run([]string{"gemini", "-site", dir, "-out", capsule, "-lang", "en"}, &out, &errOut); code != 0 {
t.Fatalf("run(gemini) = %d, stderr %q", code, errOut.String())
}
if want := "gemtext files into " + capsule; !strings.Contains(out.String(), want) {
t.Errorf("output %q does not contain %q", out.String(), want)
}
for _, rel := range []string{"index.gmi", "manifesto.gmi", "people.gmi", "services.gmi", "news/index.gmi", "news/feed.gmi", ".meta"} {
if _, err := os.Stat(filepath.Join(capsule, filepath.FromSlash(rel))); err != nil {
t.Errorf("no %s in the capsule: %v", rel, err)
}
}
if _, err := os.Stat(filepath.Join(capsule, "news", "2026", "01", "wip.gmi")); err == nil {
t.Error("a draft is published without -drafts")
}
preview := filepath.Join(t.TempDir(), "preview")
if code := run([]string{"gemini", "-drafts", "-site", dir, "-out", preview}, &out, &errOut); code != 0 {
t.Fatalf("run(gemini -drafts) = %d, stderr %q", code, errOut.String())
}
if _, err := os.Stat(filepath.Join(preview, "news", "2026", "01", "wip.gmi")); err != nil {
t.Errorf("a draft is missing from the capsule with -drafts: %v", err)
}
}
// TestRunGeminiErrors checks the exit codes: 1 for a failed build, 2 for a mistake in the command line.
func TestRunGeminiErrors(t *testing.T) {
var out, errOut bytes.Buffer
if code := run([]string{"gemini", "-site", miniSite(t), "-out", filepath.Join(t.TempDir(), "c"), "-lang", "xx"}, &out, &errOut); code != 1 || !strings.Contains(errOut.String(), `"xx"`) {
t.Errorf("an unknown language: code %d, stderr %q, want 1 and a message that names it", code, errOut.String())
}
errOut.Reset()
if code := run([]string{"gemini", "extra"}, &out, &errOut); code != 2 || !strings.Contains(errOut.String(), "unexpected arguments") {
t.Errorf("an extra argument: code %d, stderr %q, want 2", code, errOut.String())
}
}
// TestRunBuildWarnsAboutOldFiles checks that old files in the output folder are named in a warning and left alone.
func TestRunBuildWarnsAboutOldFiles(t *testing.T) {
site := miniSite(t)
@ -241,3 +287,27 @@ func TestRunBuildClean(t *testing.T) {
t.Error("-clean removed a file of a folder that is not motorino's")
}
}
// TestRunGeminiOldFilesAndClean checks the warning about old files and -clean for the gemini command.
func TestRunGeminiOldFilesAndClean(t *testing.T) {
site := miniSite(t)
var out, errOut bytes.Buffer
capsule := t.TempDir()
if err := os.WriteFile(filepath.Join(capsule, "old.gmi"), []byte("old"), 0o644); err != nil {
t.Fatal(err)
}
if code := run([]string{"gemini", "-site", site, "-out", capsule}, &out, &errOut); code != 0 || !strings.Contains(errOut.String(), "old.gmi") {
t.Fatalf("run(gemini) = %d, stderr %q, want 0 and a warning that names old.gmi", code, errOut.String())
}
errOut.Reset()
if code := run([]string{"gemini", "-clean", "-site", site, "-out", capsule}, &out, &errOut); code != 1 || !strings.Contains(errOut.String(), ".motorino-out") {
t.Fatalf("run(gemini -clean) into a folder that is not motorino's = %d, stderr %q", code, errOut.String())
}
fresh := filepath.Join(t.TempDir(), "capsule")
for i := 0; i < 2; i++ {
errOut.Reset()
if code := run([]string{"gemini", "-clean", "-site", site, "-out", fresh}, &out, &errOut); code != 0 || errOut.Len() != 0 {
t.Fatalf("run(gemini -clean) %d = %d, stderr %q", i+1, code, errOut.String())
}
}
}

View file

@ -13,6 +13,10 @@ import (
// are expanded they stand for themselves, so a page can hold them.
var blockNames = []string{"news_list", "latest_news", "categories_list", "news_blog", "news_archive"}
// sitePrefix goes before the title of a page that has no title of its own in the
// browser, such as a post.
const sitePrefix = "kmalloc.org: "
// document describes one output page for the shared page frame.
type document struct {
// lang is the language code of the page.
@ -101,7 +105,7 @@ func (b *builder) documentContext(d document) map[string]string {
ctx["doc_q"] = strings.NewReplacer(":", "%3A", "/", "%2F").Replace(docURL)
title := d.title
if d.prefixed {
title = "kmalloc.org: " + title
title = sitePrefix + title
}
ctx["title_html"] = render.EscapeAttr(title)
ctx["page_title"] = render.Escape(d.title)
@ -118,34 +122,45 @@ func (b *builder) documentContext(d document) map[string]string {
return ctx
}
// languageLinks returns the links to the other languages for the menu and the
// alternate links for the head. A page without translations links to the home
// of the first other language that has one.
func (b *builder) languageLinks(d document) (links, alternates string) {
type target struct{ lang, path string }
var twins []target
// langTarget is a page in another language that a page links to.
type langTarget struct{ lang, path string }
// languageTargets returns the pages in the other languages that a page links
// to: its translations, with the address a Switch key gives instead of the
// translation, or, when it has none, the home of the first other language that
// has one.
func (b *builder) languageTargets(d document) []langTarget {
var twins []langTarget
if d.id != "" {
for _, l := range b.cfg.Languages {
if p, ok := b.byKey[pageKey{d.id, l}]; ok && l != d.lang {
twins = append(twins, target{l, p.Path})
twins = append(twins, langTarget{l, p.Path})
}
}
}
if len(twins) == 0 {
for _, l := range b.cfg.Languages {
if p, ok := b.byKey[pageKey{"home", l}]; ok && l != d.lang {
twins = append(twins, target{l, p.Path})
twins = append(twins, langTarget{l, p.Path})
break
}
}
}
for i, t := range twins {
if override, ok := d.switches[t.lang]; ok {
twins[i].path = override
}
}
return twins
}
// languageLinks returns the links to the other languages for the menu and the
// alternate links for the head.
func (b *builder) languageLinks(d document) (links, alternates string) {
twins := b.languageTargets(d)
anchors := make([]string, len(twins))
for i, t := range twins {
href := t.path
if override, ok := d.switches[t.lang]; ok {
href = override
}
anchors[i] = fmt.Sprintf(`<a href="%s" hreflang="%s" lang="%s">%s</a>`, render.EscapeAttr(href), t.lang, t.lang, render.Escape(b.langs[t.lang].Strings["lang_name"]))
anchors[i] = fmt.Sprintf(`<a href="%s" hreflang="%s" lang="%s">%s</a>`, render.EscapeAttr(t.path), t.lang, t.lang, render.Escape(b.langs[t.lang].Strings["lang_name"]))
}
links = strings.Join(anchors, "\n&middot;\n")

501
internal/build/gemini.go Normal file
View file

@ -0,0 +1,501 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package build
import (
"errors"
"fmt"
"os"
"path"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
"git.kmalloc.org/squid/motorino/internal/content"
"git.kmalloc.org/squid/motorino/internal/gemtext"
"git.kmalloc.org/squid/motorino/internal/mdx"
"git.kmalloc.org/squid/motorino/internal/render"
)
// GeminiOptions tell BuildGemini where the site is and where to write the capsule.
type GeminiOptions struct {
// SiteDir is the folder of the site, the same one Build reads.
SiteDir string
// OutDir is the folder the gemtext is written to; it is created when missing.
OutDir string
// Version is the version of motorino.
Version string
// Now is the moment the capsule is built at, as in Options.
Now time.Time
// Drafts publishes the drafts and the posts dated in the future too.
Drafts bool
// Clean empties OutDir before the capsule is written, as Options.Clean does.
Clean bool
// Language is the language served at the root of the capsule, the others
// living under /<code>/. It is the language the server serves by default
// (Agate's --lang). Empty means gemini_language of site.json, then the
// default language of the site.
Language string
}
// GeminiResult summarises a Gemini build.
type GeminiResult struct {
// Pages is the number of gemtext files written.
Pages int
// Stale lists the files that were already in the output folder and that this
// build did not write; see Result.Stale.
Stale []string
}
// gemNews is the address of the news in the capsule. They exist once, whatever
// the language of the root, so it is always /news/.
const gemNews = "/news/"
// metaName is the file that tells the server the language of each folder; the
// server reads it and never serves it.
const metaName = ".meta"
// gemini holds one Gemini build, on top of the loaded site.
type gemini struct {
*builder
// root is the language served at the root.
root string
// pageAddr is the address of every page in the capsule, by ID and language.
pageAddr map[pageKey]string
// addr maps an address of the website to the address in the capsule.
addr map[string]string
// usedAssets lists the assets some page points to: only those are copied.
usedAssets map[string]bool
// gemFiles holds the output, and gemOwners what produces each file.
gemFiles map[string]string
gemOwners map[string]string
}
// BuildGemini reads the site in opts.SiteDir and writes it as a Gemini capsule
// to opts.OutDir: the pages of each language, the news once under /news/ and a
// .meta file for the server. Everything is rendered in memory first, so a
// failure in the content leaves the folder untouched.
func BuildGemini(opts GeminiOptions) (GeminiResult, error) {
b, err := load(Options{SiteDir: opts.SiteDir, OutDir: opts.OutDir, Version: opts.Version, Now: opts.Now, Drafts: opts.Drafts})
if err != nil {
return GeminiResult{}, err
}
g := &gemini{
builder: b, pageAddr: map[pageKey]string{}, addr: map[string]string{}, usedAssets: map[string]bool{},
gemFiles: map[string]string{}, gemOwners: map[string]string{},
}
if g.root, err = g.rootLanguage(opts.Language); err != nil {
return GeminiResult{}, err
}
g.mapAddresses()
for _, p := range b.pages {
if err := g.renderPage(p); err != nil {
return GeminiResult{}, err
}
}
steps := []func() error{g.renderPosts, g.renderCategories, g.renderArchives, g.renderFeed}
for _, step := range steps {
if err := step(); err != nil {
return GeminiResult{}, err
}
}
pages := len(g.gemFiles)
if err := g.publish(metaName, "the language file of the server", g.meta()); err != nil {
return GeminiResult{}, err
}
assets, err := g.assets()
if err != nil {
return GeminiResult{}, err
}
files := make(map[string][]byte, len(g.gemFiles)+len(assets))
for rel, text := range g.gemFiles {
files[rel] = []byte(text)
}
for rel, data := range assets {
files[rel] = data
}
stale, err := writeOutput(opts.OutDir, opts.SiteDir, opts.Clean, files)
if err != nil {
return GeminiResult{}, err
}
return GeminiResult{Pages: pages, Stale: stale}, nil
}
// rootLanguage returns the language at the root of the capsule: the option,
// else gemini_language of site.json, else the default language. It must be a
// language of the site that has a home page.
func (g *gemini) rootLanguage(option string) (string, error) {
code, from := option, "the language option"
if code == "" {
code, from = g.cfg.Strings["gemini_language"], "gemini_language in site.json"
}
if code == "" {
return g.defaultLang, nil
}
if _, ok := g.langs[code]; !ok {
return "", fmt.Errorf("%s: %q is not one of the languages of the site", from, code)
}
if _, ok := g.byKey[pageKey{"home", code}]; !ok {
return "", fmt.Errorf("%s: the language %q has no home page, so it cannot be the root of the capsule", from, code)
}
return code, nil
}
// meta returns the content of the .meta file. The server gives a document the
// language of the first line that matches it, so the lines come from the most
// specific to the most general:
//
// - a line for the file of every post written in another language than the news;
// - for every language other than the one at the root, and for the news when they
// are in another language, two lines for the folder. A glob does not cross
// folders, so each folder has a line for itself and one for what is below it.
func (g *gemini) meta() string {
var out strings.Builder
for _, p := range g.posts {
if p.Language != g.newsLang {
fmt.Fprintf(&out, "%s:;lang=%s\n", gemFile(gemPostAddress(p)), p.Language)
}
}
line := func(folder, lang string) {
fmt.Fprintf(&out, "%s/*:;lang=%s\n%s/**/*:;lang=%s\n", folder, lang, folder, lang)
}
for _, lang := range g.cfg.Languages {
if lang != g.root {
line(lang, lang)
}
}
if g.newsLang != g.root {
line(strings.Trim(gemNews, "/"), g.newsLang)
}
return out.String()
}
// pageGemAddress returns the address in the capsule of a page of lang whose
// address in the website is html: the root language at the root, the others
// under their code, .gmi for .html.
func (g *gemini) pageGemAddress(lang, html string) string {
p := html
if lang != g.defaultLang {
p = strings.TrimPrefix(p, "/"+lang)
}
if strings.HasSuffix(p, ".html") {
p = strings.TrimSuffix(p, ".html") + ".gmi"
}
if lang != g.root {
p = "/" + lang + p
}
return p
}
// gemPostAddress, gemCategoryAddress, gemYearAddress and gemMonthAddress are the
// addresses in the capsule of a post, a category and the archives.
func gemPostAddress(p content.Post) string { return strings.TrimSuffix(p.Path, ".html") + ".gmi" }
func gemCategoryAddress(slug string) string { return gemNews + "category/" + slug + ".gmi" }
func gemYearAddress(year int) string { return fmt.Sprintf("%s%04d/", gemNews, year) }
func gemMonthAddress(year, month int) string {
return fmt.Sprintf("%s%04d/%02d/", gemNews, year, month)
}
// mapAddresses fills the maps from the addresses of the website to those of the capsule.
func (g *gemini) mapAddresses() {
for _, p := range g.pages {
address := g.pageGemAddress(p.Lang, p.Path)
if p.ID == "news" && p.Lang == g.newsLang {
address = gemNews
}
g.pageAddr[pageKey{p.ID, p.Lang}] = address
g.addr[p.Path] = address
}
for _, p := range g.posts {
g.addr[p.Path] = gemPostAddress(p)
}
for _, c := range g.categories {
g.addr[categoryPath(c.Slug)] = gemCategoryAddress(c.Slug)
}
for _, y := range groupArchive(g.posts) {
g.addr[g.yearPath(y.year)] = gemYearAddress(y.year)
for _, m := range y.months {
g.addr[g.monthPath(m.year, m.month)] = gemMonthAddress(m.year, m.month)
}
}
g.addr[g.newsPath+"feed.xml"] = gemNews + "feed.gmi"
}
// publish stores a file of the capsule; source names what produced it.
func (g *gemini) publish(rel, source, text string) error {
if other, taken := g.gemOwners[rel]; taken {
return fmt.Errorf("%s and %s are both published as %s", other, source, rel)
}
g.gemOwners[rel] = source
g.gemFiles[rel] = text
return nil
}
// gemFile returns the file of an address of the capsule: a folder gets index.gmi.
func gemFile(address string) string {
rel := strings.TrimPrefix(address, "/")
if rel == "" || strings.HasSuffix(rel, "/") {
rel += "index.gmi"
}
return rel
}
// scheme matches the start of an address that names a protocol.
var scheme = regexp.MustCompile(`^([A-Za-z][A-Za-z0-9+.-]*):`)
// externalSchemes are the protocols a link may use to leave the capsule.
var externalSchemes = map[string]bool{
"http": true, "https": true, "mailto": true, "gemini": true, "xmpp": true, "irc": true, "ircs": true, "matrix": true,
"bitcoin": true, "magnet": true, "tel": true, "sms": true,
}
// resolve turns an address found in the HTML of the page at base into the
// address to write in the capsule. An address with a protocol stays as it is
// when the protocol is a known one, and is an error otherwise; an address in the
// website is mapped to its page or file, and a file is copied by the end of the
// build. An address that starts with one of the paths of gemini_web_paths in
// site.json is one the website serves by other means: it becomes a link to the
// website. A fragment alone is dropped, and so is the fragment of an address: a
// gemini link cannot point inside a document.
func (g *gemini) resolve(href, base string) (string, error) {
if strings.HasPrefix(href, "#") {
return "", nil
}
if m := scheme.FindStringSubmatch(href); m != nil {
if !externalSchemes[strings.ToLower(m[1])] {
return "", fmt.Errorf("the protocol %q is not supported", m[1])
}
return href, nil
}
if strings.HasPrefix(href, "//") {
return "", errors.New("an address that starts with // is not supported")
}
target, _, _ := strings.Cut(href, "#")
if !strings.HasPrefix(target, "/") {
folder := path.Dir(base)
if strings.HasSuffix(base, "/") {
folder = strings.TrimSuffix(base, "/")
}
joined := path.Join("/", folder, target)
if strings.HasSuffix(target, "/") && joined != "/" {
joined += "/"
}
target = joined
}
if address, ok := g.addr[target]; ok {
return address, nil
}
name := strings.TrimPrefix(target, "/")
for asset := range g.cfg.Assets {
if path.Clean(asset) == name {
g.usedAssets[asset] = true
return target, nil
}
}
for _, prefix := range strings.Split(g.cfg.Strings["gemini_web_paths"], ",") {
if prefix = strings.TrimSpace(prefix); prefix != "" && strings.HasPrefix(target, prefix) {
return g.cfg.Strings["base_url"] + target, nil
}
}
return "", errors.New("no page or file of the site has this address (a path the website serves by other means goes in gemini_web_paths of site.json)")
}
// convert converts the HTML of a page or post to gemtext. base is the address
// of the page in the website, lang its language.
func (g *gemini) convert(name, html, base, lang string) (string, error) {
text, err := gemtext.Convert(html, gemtext.Options{
Link: func(href string) (string, error) { return g.resolve(href, base) },
Block: func(block string) (string, error) { return g.block(block, lang) },
})
if err != nil {
return "", fmt.Errorf("%s: %w", name, err)
}
return strings.TrimRight(text, "\n"), nil
}
// markdown replaces the placeholders of a source, converts it to HTML and puts
// the element <gemblock> where a generated list goes, for convert to expand. A
// list with nothing in it leaves no element, and the heading of an empty list of
// categories or of the archive goes with it.
func (g *gemini) markdown(name, source string, ctx map[string]string) (string, error) {
text, err := mdx.SubstituteMarkdown(name, source, ctx)
if err != nil {
return "", err
}
html, _, err := mdx.Render(text)
if err != nil {
return "", fmt.Errorf("%s: %w", name, err)
}
lang := ctx["lang"]
html, err = replaceBlocks(name, html, func(block string) (string, error) {
text, err := g.block(block, lang)
if err != nil || text == "" {
return "", err
}
return `<gemblock name="` + block + `"></gemblock>`, nil
})
if err != nil {
return "", err
}
return render.Unescape(html), nil
}
// assets reads the files some page points to.
func (g *gemini) assets() (map[string][]byte, error) {
names := make([]string, 0, len(g.usedAssets))
for name := range g.usedAssets {
names = append(names, name)
}
sort.Strings(names)
out := map[string][]byte{}
for _, name := range names {
source := g.cfg.Assets[name]
data, err := os.ReadFile(filepath.Join(g.opts.SiteDir, filepath.FromSlash(source)))
if err != nil {
return nil, fmt.Errorf("asset %q: %w", name, err)
}
rel := path.Clean(name)
if other, taken := g.gemOwners[rel]; taken {
return nil, fmt.Errorf("the asset %q and %s are both published as %s", name, other, rel)
}
out[rel] = data
}
return out, nil
}
// heading is the first level heading of a page.
func heading(title string) string { return "# " + strings.Join(strings.Fields(title), " ") }
// withoutTitle prepares a body converted to gemtext to go under the heading of
// the title. A leading heading that says the title is dropped, since the page
// has it already, and the other first level headings go one level down: a page
// has one.
func withoutTitle(title, body string) string {
lines := strings.Split(body, "\n")
if first, ok := strings.CutPrefix(lines[0], "# "); ok && (first == title || strings.HasSuffix(title, ": "+first)) {
lines = lines[1:]
}
fenced := false
for i, line := range lines {
switch {
case strings.HasPrefix(line, "```"):
fenced = !fenced
case !fenced && strings.HasPrefix(line, "# "):
lines[i] = "#" + line
}
}
return strings.Join(lines, "\n")
}
// navigation returns the links every page ends with: the home and the entries
// of the menu of the language, in the order its file lists them, then the
// other languages. An entry whose page the language lacks is left out.
func (g *gemini) navigation(d document) string {
l := g.langs[d.lang]
var menu []string
if home, ok := g.pageAddr[pageKey{"home", d.lang}]; ok {
label := l.Strings["back"]
if label == "" {
label = g.byKey[pageKey{"home", d.lang}].Title
}
menu = append(menu, gemtext.Link(home, label))
}
for _, key := range l.NavOrder {
address, ok := g.pageAddr[pageKey{key, d.lang}]
if !ok && key == "news" {
address, ok = gemNews, true
}
if ok {
menu = append(menu, gemtext.Link(address, l.Nav[key]))
}
}
parts := []string{strings.Join(menu, "\n")}
var others []string
for _, t := range g.languageTargets(d) {
if address, err := g.resolve(t.path, d.path); err == nil && address != "" {
others = append(others, gemtext.Link(address, g.langs[t.lang].Strings["lang_name"]))
}
}
if len(others) > 0 {
parts = append(parts, strings.Join(others, "\n"))
}
return strings.Join(parts, "\n\n")
}
// translationNote returns the line, and the link to report a problem, that a
// machine translated language puts at the top of its pages; empty for the others.
func (g *gemini) translationNote(lang string) string {
l := g.langs[lang]
note := l.Strings["translation_note"]
if note == "" {
return ""
}
label, issues := l.Strings["translation_note_a"], g.cfg.Strings["issues_url"]
if label == "" || issues == "" {
return gemtext.Text(note)
}
return gemtext.Text(note+" "+label+".") + "\n" + gemtext.Link(issues, label)
}
// compose joins the parts of a page with a blank line between them, leaving out
// the empty ones, and ends the text with a line break.
func compose(parts ...string) string {
var kept []string
for _, part := range parts {
if part = strings.Trim(part, "\n"); part != "" {
kept = append(kept, part)
}
}
return strings.Join(kept, "\n\n") + "\n"
}
// renderPage converts a page and writes it with the navigation.
func (g *gemini) renderPage(p content.Page) error {
d := document{
lang: p.Lang, id: p.ID, nav: p.ID, path: p.Path, title: p.Title,
description: p.Description, switches: p.Switch, source: p.Source,
}
ctx := g.documentContext(d)
version, date := g.cfg.Strings["manifesto_version"], g.cfg.Strings["manifesto_date"]
if p.ManifestoVersion != "" {
version = p.ManifestoVersion
}
if p.ManifestoDate != "" {
date = p.ManifestoDate
}
ctx["manifesto_version"], ctx["manifesto_date"] = render.Escape(version), render.Escape(date)
source := p.Body
if p.Layout == "manifesto" {
var err error
if source, err = mdx.NoEmDash(source); err != nil {
return fmt.Errorf("%s: %w", p.Source, err)
}
}
html, err := g.markdown(p.Source, source, ctx)
if err != nil {
return err
}
body, err := g.convert(p.Source, html, p.Path, p.Lang)
if err != nil {
return err
}
var intro []string
if p.Layout == "manifesto" {
intro = []string{g.manifestoLine(p.Lang, version, date), gemtext.Text(p.Note)}
}
parts := append([]string{heading(p.Title), g.translationNote(p.Lang)}, intro...)
parts = append(parts, withoutTitle(p.Title, body), g.navigation(d))
return g.publish(gemFile(g.pageAddr[pageKey{p.ID, p.Lang}]), p.Source, compose(parts...))
}
// manifestoLine is the line that says the version and the date of the manifesto.
func (g *gemini) manifestoLine(lang, version, date string) string {
line := strings.TrimSpace(g.langs[lang].Strings["version"] + " " + version)
if date != "" {
line += ", " + date
}
return gemtext.Text(line)
}

View file

@ -0,0 +1,249 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package build
import (
"fmt"
"strings"
"git.kmalloc.org/squid/motorino/internal/content"
"git.kmalloc.org/squid/motorino/internal/gemtext"
)
// newsLines returns the lines of a list of posts: a link line for each, with
// its date in front of the title, the format feed readers know, and the
// description under it when asked for.
func newsLines(posts []content.Post, withDescription bool) string {
var lines []string
for _, p := range posts {
lines = append(lines, gemtext.Link(gemPostAddress(p), p.Date.Format(dateOnly)+" "+p.Title))
if withDescription && p.Description != "" {
lines = append(lines, gemtext.Text(p.Description))
}
}
return strings.Join(lines, "\n")
}
// block returns the gemtext of a generated list. The blog list is the whole list
// of the posts: a capsule has no pages of ten.
func (g *gemini) block(name, lang string) (string, error) {
l := g.langs[lang]
empty := gemtext.Text(l.Strings["news_empty"])
switch name {
case "news_list", "news_blog":
if len(g.posts) == 0 {
return empty, nil
}
return newsLines(g.posts, true), nil
case "latest_news":
return g.latestNews(lang), nil
case "categories_list":
lines := make([]string, len(g.categories))
for i, c := range g.categories {
lines[i] = gemtext.Link(gemCategoryAddress(c.Slug), fmt.Sprintf("%s (%d)", c.Name, len(c.posts)))
}
return strings.Join(lines, "\n"), nil
default:
return g.archiveLines(lang), nil
}
}
// latestNews returns the block of the latest news of the home page: a heading
// with the name of the news in the menu, the sentence news_p, the first posts
// and the link to all of them. It is empty without posts.
func (g *gemini) latestNews(lang string) string {
if len(g.posts) == 0 {
return ""
}
l := g.langs[lang]
var lines []string
if name := l.Nav["news"]; name != "" {
lines = append(lines, "#"+heading(name)+"\n")
}
if text := gemtext.Text(l.Strings["news_p"]); text != "" {
lines = append(lines, text)
}
n := newsOnHome
if n > len(g.posts) {
n = len(g.posts)
}
lines = append(lines, newsLines(g.posts[:n], false))
label := l.Strings["news_all"]
if label == "" {
label = l.Nav["news"]
}
return strings.Join(append(lines, gemtext.Link(gemNews, label)), "\n")
}
// archiveLines returns the years that have posts, each followed by its months, or
// nothing when there are none: the list of the news says there is no news.
func (g *gemini) archiveLines(lang string) string {
years := groupArchive(g.posts)
if len(years) == 0 {
return ""
}
var lines []string
for _, y := range years {
lines = append(lines, gemtext.Link(gemYearAddress(y.year), fmt.Sprintf("%d (%d)", y.year, len(y.posts))))
lines = append(lines, g.monthLines(y, lang)...)
}
return strings.Join(lines, "\n")
}
// monthLines returns the link lines of the months of a year.
func (g *gemini) monthLines(y archiveYear, lang string) []string {
lines := make([]string, len(y.months))
for i, m := range y.months {
lines[i] = gemtext.Link(gemMonthAddress(m.year, m.month), fmt.Sprintf("%s (%d)", monthYear(m.year, m.month, g.langs[lang]), len(m.posts)))
}
return lines
}
// newsDocument describes a page of the news for the navigation.
func (g *gemini) newsDocument(lang, path, title, source string) document {
return document{lang: lang, nav: "news", path: path, title: title, source: source}
}
// renderPosts converts every post.
func (g *gemini) renderPosts() error {
for i, p := range g.posts {
if err := g.renderPost(i, p); err != nil {
return err
}
}
return nil
}
// renderPost converts a post: the title, a line with the date, the author, the
// categories and the reading time, the summary when it was written, the text,
// the link to reply on Mastodon, and the links to the older and the newer post
// and to all the news.
func (g *gemini) renderPost(i int, p content.Post) error {
l := g.langs[p.Language]
d := g.newsDocument(p.Language, p.Path, p.Title, p.Source)
ctx := g.documentContext(d)
html, err := g.markdown(p.Source, p.Body, ctx)
if err != nil {
return err
}
body, err := g.convert(p.Source, html, p.Path, p.Language)
if err != nil {
return err
}
parts := []string{longDate(p.Date, l)}
if p.Author != "" {
parts = append(parts, p.Author)
}
if len(p.Categories) > 0 {
names := make([]string, len(p.Categories))
for i, c := range p.Categories {
names[i] = c.Name
}
parts = append(parts, strings.Join(names, ", "))
}
if reading := readingText(p.ReadingMinutes(), l); reading != "" {
parts = append(parts, reading)
}
meta := []string{gemtext.Text(strings.Join(parts, " · "))}
for _, c := range p.Categories {
meta = append(meta, gemtext.Link(gemCategoryAddress(c.Slug), c.Name))
}
summary := ""
if p.HasDescription {
summary = gemtext.Text(p.Description)
}
mastodon := ""
if p.Mastodon != "" {
label := l.Strings["reply_mastodon"]
if label == "" {
return fmt.Errorf("%s: the post has a Mastodon address, but lang/%s.json has no reply_mastodon to label the link", p.Source, p.Language)
}
mastodon = gemtext.Link(p.Mastodon, label)
}
out := compose(heading(p.Title), strings.Join(meta, "\n"), summary, withoutTitle(p.Title, body), mastodon, g.postNavigation(i, l.Code), g.navigation(d))
return g.publish(gemFile(gemPostAddress(p)), p.Source, out)
}
// postNavigation returns the links to the newer and the older post and to all
// the news. A link whose label the language lacks is left out.
func (g *gemini) postNavigation(i int, lang string) string {
l := g.langs[lang]
var lines []string
if label := l.Strings["post_newer"]; label != "" && i > 0 {
lines = append(lines, gemtext.Link(gemPostAddress(g.posts[i-1]), label+": "+g.posts[i-1].Title))
}
if label := l.Strings["post_older"]; label != "" && i+1 < len(g.posts) {
lines = append(lines, gemtext.Link(gemPostAddress(g.posts[i+1]), label+": "+g.posts[i+1].Title))
}
if label := l.Strings["news_all"]; label != "" {
lines = append(lines, gemtext.Link(gemNews, label))
}
return strings.Join(lines, "\n")
}
// renderCategories writes the page of every category.
func (g *gemini) renderCategories() error {
l := g.langs[g.newsLang]
for _, c := range g.categories {
address := gemCategoryAddress(c.Slug)
d := g.newsDocument(g.newsLang, categoryPath(c.Slug), c.Name, fmt.Sprintf("the page of the category %q", c.Name))
title := c.Name
if h := l.Strings["category_h"]; h != "" {
title = h + ": " + c.Name
}
text := compose(heading(title), newsLines(c.posts, true), g.navigation(d))
if err := g.publish(gemFile(address), d.source, text); err != nil {
return err
}
}
return nil
}
// renderArchives writes the page of every year and every month that has posts.
func (g *gemini) renderArchives() error {
lang := g.newsLang
l := g.langs[lang]
page := func(address, htmlPath, name string, posts []content.Post, links []string, source string) error {
title := name
if h := l.Strings["archive_h"]; h != "" {
title = h + ": " + name
}
d := g.newsDocument(lang, htmlPath, name, source)
text := compose(heading(title), strings.Join(links, "\n"), newsLines(posts, true), g.navigation(d))
return g.publish(gemFile(address), source, text)
}
for _, y := range groupArchive(g.posts) {
year := fmt.Sprint(y.year)
if err := page(gemYearAddress(y.year), g.yearPath(y.year), year, y.posts, g.monthLines(y, lang), "the archive of "+year); err != nil {
return err
}
var up []string
if text := l.Strings["archive_up"]; text != "" {
up = []string{gemtext.Link(gemYearAddress(y.year), text+" "+year)}
}
for _, m := range y.months {
name := monthYear(m.year, m.month, l)
if err := page(gemMonthAddress(m.year, m.month), g.monthPath(m.year, m.month), name, m.posts, up, "the archive of "+name); err != nil {
return err
}
}
}
return nil
}
// renderFeed writes the list of the latest posts, newest first, a date before
// each title: the format that Gemini feed readers subscribe to.
func (g *gemini) renderFeed() error {
title := g.byKey[pageKey{"news", g.newsLang}].Title
if !strings.HasPrefix(title, sitePrefix) {
title = sitePrefix + title
}
posts := g.posts
if len(posts) > feedLimit {
posts = posts[:feedLimit]
}
text := compose(heading(title), newsLines(posts, false))
return g.publish(gemFile(gemNews+"feed.gmi"), "the feed", text)
}

View file

@ -0,0 +1,523 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package build
import (
"io/fs"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"testing"
)
// buildGem builds the Gemini output of the site made of files, with the options changed by tune.
func buildGem(t *testing.T, files map[string]string, tune func(*GeminiOptions)) (string, error) {
t.Helper()
out := filepath.Join(t.TempDir(), "capsule")
opts := GeminiOptions{SiteDir: writeFiles(t, files), OutDir: out, Version: "1.2.3", Now: fixedNow, Language: "it"}
if tune != nil {
tune(&opts)
}
_, err := BuildGemini(opts)
return out, err
}
// gemSite is the base site with a services page that has a table and an image, a Spanish news page left out and
// the news index of the English pages.
func gemSite(extra map[string]string) map[string]string {
files := withFiles(map[string]string{
"lang/en.json": blogLangEn, "lang/it.json": blogLangIt,
"content/en/services.md": page("services", "Services", "", "Write to **{{contact_shown}}**.\n\n<table class=\"data\">\n<thead><tr><th>Service</th><th>Kept</th></tr></thead>\n<tbody><tr><td>Mail</td><td>30 days</td></tr></tbody>\n</table>\n\n![The mark of kmalloc](/logo.svg)\n"),
"content/en/news/index.md": page("news", "News", "Layout: bare\n", "# News\n\n{{news_list}}\n\n## Categories\n\n{{categories_list}}\n\n## Archive\n\n{{news_archive}}\n\n[Feed](/news/feed.xml)\n"),
"news/shop.md": "Title: A shop\nAuthor: squid\nDate: 2026-11-14T12:00:00+0100\nCategories: FLUG\nDescription: About the shop.\nMastodon: https://social.kmalloc.org/@squid/115\n-\nThere is a shop. Mail me at squid@kmalloc.org or visit [the site](https://kmalloc.org/).\n",
})
for k, v := range extra {
if v == "" {
delete(files, k)
continue
}
files[k] = v
}
return files
}
// gemFiles returns every file of the output, as slash paths relative to it.
func gemFiles(t *testing.T, out string) []string {
t.Helper()
var files []string
err := filepath.WalkDir(out, func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
rel, _ := filepath.Rel(out, path)
files = append(files, filepath.ToSlash(rel))
return nil
})
if err != nil {
t.Fatal(err)
}
sort.Strings(files)
return files
}
// TestGeminiTree checks which files the capsule has, with Italian at the root.
func TestGeminiTree(t *testing.T) {
out, err := buildGem(t, gemSite(nil), nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
have := map[string]bool{}
for _, f := range gemFiles(t, out) {
have[f] = true
}
for _, want := range []string{
"index.gmi", "manifesto.gmi", "persone.gmi", "servizi.gmi", "cronologia-manifesto.gmi", "progetti.gmi",
"en/index.gmi", "en/manifesto.gmi", "en/people.gmi", "en/services.gmi", "es/manifesto.gmi",
"news/index.gmi", "news/feed.gmi", "news/2026/11/shop.gmi", "news/2026/11/status.gmi", "news/2026/09/gnu-linux-day-2026.gmi",
"news/category/flug.gmi", "news/category/linux-day.gmi", "news/2026/index.gmi", "news/2026/11/index.gmi", "news/2026/09/index.gmi",
".meta", "logo.svg",
} {
if !have[want] {
t.Errorf("the capsule has no %s", want)
}
}
for f := range have {
if strings.HasSuffix(f, ".html") || strings.HasPrefix(f, "it/") || strings.HasPrefix(f, "en/news/") || strings.HasPrefix(f, "es/news/") || strings.HasSuffix(f, "robots.txt") {
t.Errorf("the capsule has %s, which it should not", f)
}
}
}
// TestGeminiRootLanguage checks the language at the root: the option, site.json, then the default language.
func TestGeminiRootLanguage(t *testing.T) {
plain, err := buildGem(t, gemSite(nil), func(o *GeminiOptions) { o.Language = "" })
if err != nil {
t.Fatal(err)
}
mustContain(t, "index.gmi", readOut(t, plain, "index.gmi"), "# kmalloc.org")
if !exists(t, plain, "it/index.gmi") || exists(t, plain, "en/index.gmi") {
t.Error("without a choice the default language is at the root and the others under their code")
}
files := gemSite(map[string]string{"site.json": strings.Replace(baseFiles()["site.json"], `"languages"`, `"gemini_language": "it", "languages"`, 1)})
fromSite, err := buildGem(t, files, func(o *GeminiOptions) { o.Language = "" })
if err != nil {
t.Fatal(err)
}
if !exists(t, fromSite, "en/index.gmi") || exists(t, fromSite, "it/index.gmi") {
t.Error("gemini_language in site.json does not put Italian at the root")
}
for name, tune := range map[string]func(*GeminiOptions){"xx": func(o *GeminiOptions) { o.Language = "xx" }, "es has no home": func(o *GeminiOptions) { o.Language = "es" }} {
if _, err := buildGem(t, gemSite(nil), tune); err == nil {
t.Errorf("language %s: no error", name)
}
}
}
// TestGeminiMeta checks the .meta file that sets the language of the documents.
func TestGeminiMeta(t *testing.T) {
out, err := buildGem(t, gemSite(nil), nil)
if err != nil {
t.Fatal(err)
}
want := "en/*:;lang=en\nen/**/*:;lang=en\nes/*:;lang=es\nes/**/*:;lang=es\nnews/*:;lang=en\nnews/**/*:;lang=en\n"
if got := readOut(t, out, ".meta"); got != want {
t.Errorf(".meta =\n%q\nwant\n%q", got, want)
}
plain, err := buildGem(t, gemSite(nil), func(o *GeminiOptions) { o.Language = "en" })
if err != nil {
t.Fatal(err)
}
want = "it/*:;lang=it\nit/**/*:;lang=it\nes/*:;lang=es\nes/**/*:;lang=es\n"
if got := readOut(t, plain, ".meta"); got != want {
t.Errorf(".meta with English at the root =\n%q\nwant\n%q (the news are in the language of the root: no lines for them)", got, want)
}
for _, f := range gemFiles(t, out) {
if strings.HasSuffix(f, ".gmi") && strings.Contains(readOut(t, out, f), ".meta") {
t.Errorf("%s mentions .meta", f)
}
}
}
// TestGeminiPages checks the headings, the layouts and the navigation of a page.
func TestGeminiPages(t *testing.T) {
out, err := buildGem(t, gemSite(nil), nil)
if err != nil {
t.Fatal(err)
}
services := readOut(t, out, "en/services.gmi")
mustContain(t, "en/services.gmi", services, "# Services\n", "Write to squid (at) kmalloc.org.\n",
"```\nService Kept\n------- -------\nMail 30 days\n```\n", "=> /logo.svg The mark of kmalloc\n")
if strings.Contains(services, "<") || strings.Contains(services, "**") {
t.Errorf("the page has markup left:\n%s", services)
}
mustContain(t, "en/services.gmi navigation", services,
"=> /en/ kmalloc.org\n", "=> /en/manifesto.gmi Manifesto\n", "=> /en/people.gmi People\n", "=> /en/services.gmi Services\n", "=> /news/ News\n",
"=> /servizi.gmi Italiano\n")
manifesto := readOut(t, out, "es/manifesto.gmi")
mustContain(t, "es/manifesto.gmi", manifesto, "# Manifiesto\n", "Version 1.0, 2026-11-12\n", "Traduccion de la version 1.0.\n", "=> /es/manifesto.gmi Manifiesto\n")
home := readOut(t, out, "index.gmi")
mustContain(t, "index.gmi", home, "=> /news/ Tutte le news\n", "=> /news/2026/11/shop.gmi 2026-11-14 A shop\n", "=> /en/ English\n")
if strings.HasPrefix(home, "# ") && !strings.Contains(home, "kmalloc.org") {
t.Errorf("home has no title:\n%s", home)
}
}
// TestGeminiHomesLinkToTheNews checks that every home links to the one copy of the news.
func TestGeminiHomesLinkToTheNews(t *testing.T) {
out, err := buildGem(t, gemSite(nil), nil)
if err != nil {
t.Fatal(err)
}
for _, rel := range []string{"index.gmi", "en/index.gmi"} {
mustContain(t, rel, readOut(t, out, rel), "=> /news/ ", "=> /news/2026/11/shop.gmi 2026-11-14 A shop\n")
}
}
// TestGeminiNews checks a post, the index, a category, the archive and the feed.
func TestGeminiNews(t *testing.T) {
out, err := buildGem(t, blogGem(), nil)
if err != nil {
t.Fatal(err)
}
post := readOut(t, out, "news/2026/11/shop.gmi")
mustContain(t, "post", post, "# A shop\n", "14 November 2026 · squid · FLUG · 1 min read\n", "About the shop.\n",
"There is a shop. Mail me at squid@kmalloc.org or visit the site.\n", "=> https://kmalloc.org/ the site\n",
"=> /news/category/flug.gmi FLUG\n", "=> https://social.kmalloc.org/@squid/115 Reply on Mastodon\n",
"=> /news/2026/11/status.gmi Older: Status page\n", "=> /news/ All news\n", "=> /en/manifesto.gmi Manifesto\n")
if strings.Contains(post, "Newer:") {
t.Error("the newest post has a link to a newer one")
}
gnu := readOut(t, out, "news/2026/09/gnu-linux-day-2026.gmi")
mustContain(t, "older post", gnu, "=> /en/people.gmi people\n", "=> /news/2026/11/status.gmi Newer: Status page\n")
index := readOut(t, out, "news/index.gmi")
mustContain(t, "index", index, "# News\n",
"=> /news/2026/11/shop.gmi 2026-11-14 A shop\nAbout the shop.\n", "=> /news/2026/11/status.gmi 2026-11-12 Status page\n",
"=> /news/category/flug.gmi FLUG (2)\n", "=> /news/2026/ 2026 (3)\n", "=> /news/2026/11/ November 2026 (2)\n", "=> /news/feed.gmi Feed\n")
category := readOut(t, out, "news/category/flug.gmi")
mustContain(t, "category", category, "# Category: FLUG\n", "=> /news/2026/11/shop.gmi 2026-11-14 A shop\n", "=> /news/2026/09/gnu-linux-day-2026.gmi 2026-09-18 GNU/Linux Day 2026\n")
month := readOut(t, out, "news/2026/11/index.gmi")
mustContain(t, "month", month, "# Archive: November 2026\n", "=> /news/2026/ Archive of the year 2026\n", "A shop")
if strings.Contains(month, "GNU/Linux") {
t.Error("the archive of November lists a post of September")
}
year := readOut(t, out, "news/2026/index.gmi")
mustContain(t, "year", year, "# Archive: 2026\n", "=> /news/2026/09/ September 2026 (1)\n", "=> /news/2026/11/ November 2026 (2)\n")
}
// blogGem is the site of the tests of the news pages.
func blogGem() map[string]string { return gemSite(nil) }
// TestGeminiFeed checks the feed: newest first, a date before each title.
func TestGeminiFeed(t *testing.T) {
out, err := buildGem(t, blogGem(), nil)
if err != nil {
t.Fatal(err)
}
feed := readOut(t, out, "news/feed.gmi")
want := "=> /news/2026/11/shop.gmi 2026-11-14 A shop\n=> /news/2026/11/status.gmi 2026-11-12 Status page\n=> /news/2026/09/gnu-linux-day-2026.gmi 2026-09-18 GNU/Linux Day 2026\n"
mustContain(t, "feed", feed, "# kmalloc.org: News\n", want)
if first := strings.Index(feed, "A shop"); first > strings.Index(feed, "Status page") {
t.Error("the feed is not newest first")
}
}
// TestGeminiNeverHasBrokenLinksOrEmptyFiles walks the whole capsule: no empty file, and every
// link that stays in the capsule leads to a file.
func TestGeminiNeverHasBrokenLinksOrEmptyFiles(t *testing.T) {
out, err := buildGem(t, blogGem(), nil)
if err != nil {
t.Fatal(err)
}
link := regexp.MustCompile(`(?m)^=> (\S+)`)
for _, f := range gemFiles(t, out) {
data, err := os.ReadFile(filepath.Join(out, filepath.FromSlash(f)))
if err != nil {
t.Fatal(err)
}
if len(strings.TrimSpace(string(data))) == 0 && f != ".meta" {
t.Errorf("%s is empty", f)
}
if !strings.HasSuffix(f, ".gmi") {
continue
}
for _, m := range link.FindAllStringSubmatch(string(data), -1) {
target := m[1]
if strings.Contains(target, ":") || !strings.HasPrefix(target, "/") {
if !strings.Contains(target, ":") {
t.Errorf("%s has the relative link %s", f, target)
}
continue
}
rel := strings.TrimPrefix(target, "/")
if rel == "" || strings.HasSuffix(rel, "/") {
rel += "index.gmi"
}
if !exists(t, out, rel) {
t.Errorf("%s links to %s, which is not in the capsule", f, target)
}
}
}
}
// TestGeminiLinksMustResolve checks that a link to nothing stops the build and names the page and the address.
func TestGeminiLinksMustResolve(t *testing.T) {
for name, extra := range map[string]map[string]string{
"page": {"content/en/people.md": page("people", "People", "", "See [nowhere](/nowhere.html).\n")},
"post": {"news/broken.md": "Title: Broken\nDate: 2026-11-01T12:00:00+0100\n-\nSee [nowhere](/nowhere.html).\n"},
"unsafe scheme": {"content/en/people.md": page("people", "People", "", "[x](javascript:alert(1))\n")},
} {
t.Run(name, func(t *testing.T) {
_, err := buildGem(t, gemSite(extra), nil)
if err == nil || !strings.Contains(err.Error(), "nowhere.html") && !strings.Contains(err.Error(), "javascript") {
t.Errorf("error = %v, want one that names the address", err)
}
})
}
}
// TestGeminiOnlyReferencedAssets checks that an asset is copied when a page uses it and not otherwise.
func TestGeminiOnlyReferencedAssets(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{"content/en/services.md": page("services", "Services", "", "No image.\n")}), nil)
if err != nil {
t.Fatal(err)
}
if exists(t, out, "logo.svg") {
t.Error("an asset no page uses is copied")
}
}
// TestGeminiDraftsAndFuture checks that hidden posts stay out of the capsule, as in the site.
func TestGeminiDraftsAndFuture(t *testing.T) {
out, err := buildGem(t, gemSite(postNews()), nil)
if err != nil {
t.Fatal(err)
}
for _, f := range gemFiles(t, out) {
if strings.Contains(f, "a-draft") || strings.Contains(f, "in-future") || strings.Contains(f, "secret") {
t.Errorf("the capsule has %s, but the post is hidden", f)
}
}
with, err := buildGem(t, gemSite(postNews()), func(o *GeminiOptions) { o.Drafts = true })
if err != nil {
t.Fatal(err)
}
if !exists(t, with, "news/2026/11/a-draft.gmi") {
t.Error("a build with drafts lacks the draft")
}
}
// TestGeminiBlockMustStandAlone checks the rule of the generated blocks.
func TestGeminiBlockMustStandAlone(t *testing.T) {
_, err := buildGem(t, gemSite(map[string]string{"content/en/people.md": page("people", "People", "", "Text {{news_list}} text.\n")}), nil)
if err == nil || !strings.Contains(err.Error(), "news_list") {
t.Errorf("error = %v, want one about {{news_list}}", err)
}
}
// TestGeminiWebPaths checks that a path the website serves by other means can stay a link to the website.
func TestGeminiWebPaths(t *testing.T) {
files := gemSite(map[string]string{
"site.json": strings.Replace(baseFiles()["site.json"], `"languages"`, `"gemini_web_paths": "/~, /pastes/", "languages"`, 1),
"content/en/people.md": page("people", "People", "", "Pages of the [people](/~squid/) and the [pastes](/pastes/x).\n"),
})
out, err := buildGem(t, files, nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
mustContain(t, "en/people.gmi", readOut(t, out, "en/people.gmi"),
"=> https://kmalloc.org/~squid/ people\n", "=> https://kmalloc.org/pastes/x pastes\n")
if _, err := buildGem(t, gemSite(map[string]string{"content/en/people.md": page("people", "People", "", "[x](/~squid/)\n")}), nil); err == nil {
t.Error("a path of the website is a link when the site did not say so")
}
}
// TestGeminiHeadings checks that a page has one first level heading and that code is left alone.
func TestGeminiHeadings(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{
"content/en/people.md": page("people", "People", "", "# People\n\n# Other\n\n```\n# a comment in code\n```\n"),
}), nil)
if err != nil {
t.Fatal(err)
}
got := readOut(t, out, "en/people.gmi")
mustContain(t, "en/people.gmi", got, "# People\n\n## Other\n\n```\n# a comment in code\n```\n")
if strings.Count(got, "# People") != 1 {
t.Errorf("the title is repeated:\n%s", got)
}
home := readOut(t, out, "en/index.gmi")
mustContain(t, "en/index.gmi", home, "# kmalloc.org\n\n## Technology.\n")
}
// TestGeminiKeepsLinksToOtherProtocols checks the protocols that leave the capsule as they are.
func TestGeminiKeepsLinksToOtherProtocols(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{"content/en/people.md": page("people", "People", "",
"[Mail](mailto:squid@kmalloc.org) [Pay](bitcoin:bc1qxyz) [Chat](xmpp:squid@kmalloc.org) [Capsule](gemini://example.org/)\n")}), nil)
if err != nil {
t.Fatal(err)
}
mustContain(t, "en/people.gmi", readOut(t, out, "en/people.gmi"),
"=> mailto:squid@kmalloc.org Mail\n", "=> bitcoin:bc1qxyz Pay\n", "=> xmpp:squid@kmalloc.org Chat\n", "=> gemini://example.org/ Capsule\n")
}
// TestGeminiTitleThatSaysTheHeading checks that a heading the title already says is not repeated, as in
// the real site where the page title is "kmalloc.org: Services" and the text starts with "# Services".
func TestGeminiTitleThatSaysTheHeading(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{
"content/en/people.md": page("people", "kmalloc.org: People", "", "# People\n\nText.\n"),
}), nil)
if err != nil {
t.Fatal(err)
}
got := readOut(t, out, "en/people.gmi")
if !strings.HasPrefix(got, "# kmalloc.org: People\n\nText.\n") {
t.Errorf("en/people.gmi starts with\n%q", got)
}
}
// TestGeminiFeedTitleIsNotPrefixedTwice checks the feed title when the news page title has the prefix already.
func TestGeminiFeedTitleIsNotPrefixedTwice(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{
"content/en/news/index.md": page("news", "kmalloc.org: News", "", "{{news_list}}\n"),
}), nil)
if err != nil {
t.Fatal(err)
}
if got := readOut(t, out, "news/feed.gmi"); !strings.HasPrefix(got, "# kmalloc.org: News\n") {
t.Errorf("news/feed.gmi starts with %q", got)
}
}
// TestGeminiTranslationNote checks that the note of a machine translated language reads as one sentence.
func TestGeminiTranslationNote(t *testing.T) {
files := gemSite(map[string]string{
"lang/es.json": strings.Replace(baseFiles()["lang/es.json"], `"translation_note": "Traduccion automatica."`, `"translation_note": "Traduccion automatica, avisa si ves un error:"`, 1),
})
out, err := buildGem(t, files, nil)
if err != nil {
t.Fatal(err)
}
mustContain(t, "es/manifesto.gmi", readOut(t, out, "es/manifesto.gmi"),
"Traduccion automatica, avisa si ves un error: avisar.\n=> https://git.kmalloc.org/squid/kmalloc.org/issues avisar\n")
}
// TestGeminiMetaForPostsInAnotherLanguage checks that a post with its own Language has a line of its own in
// .meta, with the real path of its file and before the lines of the folders: the server uses the first line that
// matches a file, and the line of the news folder would hide a line that comes after it.
func TestGeminiMetaForPostsInAnotherLanguage(t *testing.T) {
files := gemSite(map[string]string{
"news/ciao.md": "Title: Ciao\nDate: 2026-11-05T09:00:00+0100\nLanguage: it\n-\nUn testo.\n",
"news/hola.md": "Title: Hola\nDate: 2026-11-04T09:00:00+0100\nLanguage: es\n-\nUn texto.\n",
"news/same.md": "Title: Same\nDate: 2026-11-03T09:00:00+0100\nLanguage: en\n-\nText.\n",
})
out, err := buildGem(t, files, nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
want := "news/2026/11/ciao.gmi:;lang=it\nnews/2026/11/hola.gmi:;lang=es\n" +
"en/*:;lang=en\nen/**/*:;lang=en\nes/*:;lang=es\nes/**/*:;lang=es\nnews/*:;lang=en\nnews/**/*:;lang=en\n"
if got := readOut(t, out, ".meta"); got != want {
t.Errorf(".meta =\n%q\nwant\n%q", got, want)
}
for _, rel := range []string{"news/2026/11/ciao.gmi", "news/2026/11/hola.gmi"} {
if !exists(t, out, rel) {
t.Errorf("the .meta names %s, which is not in the capsule", rel)
}
}
// With English at the root and the news in English too, only the posts in other languages have a line.
plain, err := buildGem(t, files, func(o *GeminiOptions) { o.Language = "en" })
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
want = "news/2026/11/ciao.gmi:;lang=it\nnews/2026/11/hola.gmi:;lang=es\n" +
"it/*:;lang=it\nit/**/*:;lang=it\nes/*:;lang=es\nes/**/*:;lang=es\n"
if got := readOut(t, plain, ".meta"); got != want {
t.Errorf(".meta with English at the root =\n%q\nwant\n%q", got, want)
}
}
// TestGeminiNoNewsAtAll checks that a site without posts says so once, in the news list, and that the empty
// lists leave their headings out.
func TestGeminiNoNewsAtAll(t *testing.T) {
index := "# News\n\n{{news_list}}\n\n## Categories\n\n{{categories_list}}\n\n## Archive\n\n{{news_archive}}\n"
out, err := buildGem(t, gemSite(map[string]string{
"content/en/news/index.md": page("news", "News", "Layout: bare\n", index),
"news/shop.md": "", "news/gnu-linux-day-2026.md": "", "news/status.md": "",
}), nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
got := readOut(t, out, "news/index.gmi")
if n := strings.Count(got, "No news yet."); n != 1 {
t.Errorf("the index says there is no news %d times, want once:\n%s", n, got)
}
for _, heading := range []string{"## Categories", "## Archive"} {
if strings.Contains(got, heading) {
t.Errorf("the index keeps the heading %q of an empty list:\n%s", heading, got)
}
}
mustContain(t, "news/index.gmi", got, "# News\n")
mustContain(t, "news/feed.gmi", readOut(t, out, "news/feed.gmi"), "# kmalloc.org: News\n")
home := readOut(t, out, "index.gmi")
if strings.Contains(home, "Tutte le news") {
t.Errorf("the home links to the news list of a site with no news:\n%s", home)
}
mustContain(t, "index.gmi", home, "# kmalloc.org\n")
}
// TestGeminiOnlyOnePost checks that a single post has no link to a newer or an older one, but has the one to all the news.
func TestGeminiOnlyOnePost(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{"news/gnu-linux-day-2026.md": "", "news/status.md": ""}), nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
post := readOut(t, out, "news/2026/11/shop.gmi")
for _, absent := range []string{"Older", "Newer"} {
if strings.Contains(post, absent) {
t.Errorf("the only post has %q:\n%s", absent, post)
}
}
mustContain(t, "the post", post, "=> /news/ All news\n")
}
// TestGeminiUnusualTitle checks that a title with quotes, an ampersand, a tag and non Latin letters is written as it is.
func TestGeminiUnusualTitle(t *testing.T) {
title := `Say "hi" & <b>go</b> ünï 日本語`
out, err := buildGem(t, gemSite(map[string]string{
"news/odd.md": "Title: " + title + "\nDate: 2026-11-15T09:00:00+0100\nCategories: Odd\n-\nText.\n",
}), nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
mustContain(t, "the post", readOut(t, out, "news/2026/11/odd.gmi"), "# "+title+"\n")
mustContain(t, "news/feed.gmi", readOut(t, out, "news/feed.gmi"), "=> /news/2026/11/odd.gmi 2026-11-15 "+title+"\n")
mustContain(t, "news/index.gmi", readOut(t, out, "news/index.gmi"), "=> /news/2026/11/odd.gmi 2026-11-15 "+title+"\n")
mustContain(t, "the post below", readOut(t, out, "news/2026/11/shop.gmi"), "Newer: "+title)
for _, rel := range []string{"news/2026/11/odd.gmi", "news/index.gmi", "news/category/odd.gmi"} {
if strings.Contains(readOut(t, out, rel), "&lt;") || strings.Contains(readOut(t, out, rel), "&amp;") {
t.Errorf("%s has an HTML escape in the title", rel)
}
}
}
// TestGeminiLeavesTheLogoOut checks that the logo of the home page leaves no link line and no copied file.
func TestGeminiLeavesTheLogoOut(t *testing.T) {
out, err := buildGem(t, gemSite(map[string]string{
"content/en/services.md": page("services", "Services", "", "No image.\n"),
"content/en/index.md": page("home", "kmalloc.org", "Layout: bare\n", "<img class=\"logo\" src=\"/logo.svg\" alt=\"kmalloc\" width=\"112\">\n\n<h1 class=\"hero\">Technology.</h1>\n\n{{latest_news}}\n"),
}), nil)
if err != nil {
t.Fatalf("BuildGemini() error = %v", err)
}
home := readOut(t, out, "en/index.gmi")
if strings.Contains(home, "logo.svg") {
t.Errorf("the home links to the logo:\n%s", home)
}
if !strings.HasPrefix(home, "# kmalloc.org\n\n## Technology.\n") {
t.Errorf("the home starts with %q", home)
}
if exists(t, out, "logo.svg") {
t.Error("the logo is copied, and nothing points to it")
}
}

View file

@ -0,0 +1,37 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package build
import (
"path/filepath"
"strings"
"testing"
)
// init adds the Gemini output to the tests of the output folder in output_test.go.
func init() {
extraBuilders["gemini"] = func(site string) func(string, bool) ([]string, error) {
return func(out string, clean bool) ([]string, error) {
r, err := BuildGemini(GeminiOptions{SiteDir: site, OutDir: out, Version: "1.2.3", Now: fixedNow, Clean: clean, Language: "en"})
return r.Stale, err
}
}
}
// TestGeminiCleanRefusesAFolderThatHoldsTheSite checks that -clean never empties the folder of the site, or one above it.
func TestGeminiCleanRefusesAFolderThatHoldsTheSite(t *testing.T) {
parent := t.TempDir()
site := filepath.Join(parent, "site")
for rel, text := range withFiles(nil) {
put(t, site, rel, text)
}
put(t, parent, motorinoMarker, "x")
for _, out := range []string{site, parent} {
if _, err := BuildGemini(GeminiOptions{SiteDir: site, OutDir: out, Now: fixedNow, Clean: true}); err == nil || !strings.Contains(err.Error(), "site") {
t.Errorf("BuildGemini into %s: error = %v, want one about the site", out, err)
}
}
if !exists(t, site, "site.json") {
t.Error("the site is gone")
}
}

View file

@ -133,12 +133,13 @@ func (b *builder) markdown(name, source, lang string, ctx map[string]string) (st
// codeElement matches a code element, where a placeholder is shown as written.
var codeElement = regexp.MustCompile(`(?s)<code[^>]*>.*?</code>`)
// expandBlocks replaces each paragraph that holds only a block placeholder by
// the generated HTML. An empty list of categories or of the archive goes with the
// heading right before it. A block placeholder inside other text is an error. All
// blocks are checked before any is expanded, so the text of a generated list
// (a news title may say {{latest_news}}) is never taken for a placeholder.
func (b *builder) expandBlocks(name, html, lang string, ctx map[string]string) (string, error) {
// replaceBlocks replaces each paragraph that holds only a block placeholder by
// what replace returns for the name of the block. An empty list of categories or
// of the archive goes with the heading right before it. A block placeholder
// inside other text is an error. All blocks are checked before any is replaced,
// so the text of a generated list (a news title may say {{latest_news}}) is never
// taken for a placeholder.
func replaceBlocks(name, html string, replace func(block string) (string, error)) (string, error) {
outside := codeElement.ReplaceAllString(html, "")
for _, block := range blockNames {
token := "{{" + block + "}}"
@ -151,7 +152,7 @@ func (b *builder) expandBlocks(name, html, lang string, ctx map[string]string) (
if !strings.Contains(html, marker) {
continue
}
generated, err := b.block(block, lang, ctx)
generated, err := replace(block)
if err != nil {
return "", err
}
@ -164,6 +165,11 @@ func (b *builder) expandBlocks(name, html, lang string, ctx map[string]string) (
return html, nil
}
// expandBlocks replaces the block placeholders of a page by the generated HTML.
func (b *builder) expandBlocks(name, html, lang string, ctx map[string]string) (string, error) {
return replaceBlocks(name, html, func(block string) (string, error) { return b.block(block, lang, ctx) })
}
// headedBlocks are the lists a page puts under a heading of its own, which goes
// when the list is empty. The latest news bring their heading with them, and a
// heading before them is the page's.

767
internal/gemtext/gemtext.go Normal file
View file

@ -0,0 +1,767 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Package gemtext turns the HTML of a page into gemtext, the format of the
// Gemini protocol. Gemtext has headings, one-line paragraphs, bullet items,
// quotes, preformatted blocks and links on lines of their own, and nothing
// else: no emphasis, no links inside text, no tables, no images.
package gemtext
import (
"encoding/xml"
"fmt"
"io"
"regexp"
"strconv"
"strings"
"unicode/utf8"
)
// Options tell Convert how to treat the addresses and the generated blocks.
type Options struct {
// Link turns an address found in the HTML (a link or an image) into the
// address to write. An empty result drops the link and keeps its text, as
// for "#section"; an error stops the conversion and names the address. A
// nil Link leaves every address as it is.
Link func(href string) (string, error)
// Block returns the gemtext of a block placeholder, the element
// <gemblock name="...">. A nil Block makes such an element an error.
Block func(name string) (string, error)
}
// node is an element or a piece of text of the parsed HTML.
type node struct {
tag string
text string
attrs map[string]string
kids []*node
}
// link is an address with the text that labels it.
type link struct{ target, label string }
// block is a part of the output: lines of text followed by link lines.
type block struct {
text []string
links []link
}
// converter holds the options of one conversion.
type converter struct{ opts Options }
// Convert converts html, the body of a page, to gemtext. Blocks are separated
// by a blank line; the links of a paragraph, a list, a quote or a table come
// right after it, each on a line of its own, and the output ends with a line
// break unless it is empty.
func Convert(html string, opts Options) (string, error) {
root, err := parse(html)
if err != nil {
return "", err
}
c := &converter{opts: opts}
blocks, err := c.blocks(root.kids)
if err != nil {
return "", err
}
parts := make([]string, 0, len(blocks))
for _, b := range blocks {
if text := b.String(); text != "" {
parts = append(parts, text)
}
}
if len(parts) == 0 {
return "", nil
}
return strings.Join(parts, "\n\n") + "\n", nil
}
// String returns the lines of the block, text first, then the link lines.
func (b block) String() string {
lines := append([]string(nil), b.text...)
seen := map[link]bool{}
for _, l := range b.links {
if seen[l] {
continue
}
seen[l] = true
lines = append(lines, linkLine(l))
}
return strings.Join(lines, "\n")
}
// Text returns plain text as one line of gemtext: white space collapsed, and a
// leading space when the text would be read as another kind of line.
func Text(s string) string { return protect(strings.Join(strings.Fields(s), " ")) }
// Link returns the link line of an address with a label.
func Link(target, label string) string {
return linkLine(link{target, strings.Join(strings.Fields(label), " ")})
}
// linkLine writes a link on its own line.
func linkLine(l link) string {
target := strings.Join(strings.Fields(l.target), "%20")
if l.label == "" || l.label == l.target {
return "=> " + target
}
return "=> " + target + " " + l.label
}
// voidTags are the elements that have no end tag.
var voidTags = map[string]bool{
"area": true, "base": true, "br": true, "col": true, "hr": true, "img": true, "input": true,
"link": true, "meta": true, "param": true, "source": true, "basefont": true, "frame": true, "isindex": true,
}
// parse reads a fragment of HTML into a tree. The decoder is the lenient mode
// of encoding/xml with the HTML void elements and entities: the HTML of a page
// comes from a Markdown renderer and from the author's own raw HTML, which are
// well formed apart from the void elements.
func parse(html string) (*node, error) {
source := "<root>" + html + "</root>"
decoder := xml.NewDecoder(strings.NewReader(source))
decoder.Strict = false
decoder.AutoClose = xml.HTMLAutoClose
decoder.Entity = xml.HTMLEntity
root := &node{tag: "root"}
stack := []*node{root}
for {
token, err := decoder.Token()
if err == io.EOF {
break
}
if err != nil {
return nil, fmt.Errorf("read the HTML: %w", err)
}
top := stack[len(stack)-1]
switch t := token.(type) {
case xml.StartElement:
n := &node{tag: strings.ToLower(t.Name.Local), attrs: map[string]string{}}
for _, a := range t.Attr {
n.attrs[strings.ToLower(a.Name.Local)] = a.Value
}
top.kids = append(top.kids, n)
stack = append(stack, n)
case xml.EndElement:
// The lenient mode closes by itself what the text left open; that is
// a mistake in the page, so an end that the text does not have is an error.
name := strings.ToLower(t.Name.Local)
written := strings.ToLower(source[:decoder.InputOffset()])
if !voidTags[name] && !strings.HasSuffix(written, "</"+name+">") && !strings.HasSuffix(written, "/>") {
return nil, fmt.Errorf("read the HTML: the element <%s> is not closed", name)
}
if len(stack) > 1 {
stack = stack[:len(stack)-1]
}
case xml.CharData:
top.kids = append(top.kids, &node{text: string(t)})
}
}
if len(stack) != 1 || len(root.kids) != 1 {
return nil, fmt.Errorf("read the HTML: the element <%s> is not closed", stack[len(stack)-1].tag)
}
return root.kids[0], nil
}
// blockTags are the elements that start a block of their own.
var blockTags = map[string]bool{
"h1": true, "h2": true, "h3": true, "h4": true, "h5": true, "h6": true, "p": true,
"ul": true, "ol": true, "blockquote": true, "pre": true, "hr": true, "table": true,
"div": true, "section": true, "article": true, "main": true, "header": true, "footer": true,
"nav": true, "aside": true, "figure": true, "figcaption": true, "details": true, "summary": true,
"dl": true, "dt": true, "dd": true, "form": true, "fieldset": true, "address": true, "gemblock": true,
}
// cellTags are the elements of tables and lists whose text is kept apart from the
// text next to it when they are read as part of a line.
var cellTags = map[string]bool{"tr": true, "td": true, "th": true, "li": true}
// skippedTags are the elements that have nothing to read.
var skippedTags = map[string]bool{
"script": true, "style": true, "svg": true, "noscript": true, "template": true,
"head": true, "title": true, "meta": true, "link": true,
}
// skipped reports whether an element is left out: one with nothing to read, or
// one that is hidden.
func skipped(n *node) bool {
_, hidden := n.attrs["hidden"]
return skippedTags[n.tag] || hidden || n.attrs["aria-hidden"] == "true"
}
// blocks converts a list of nodes into blocks: block elements make their own,
// and the text and the inline elements between them make paragraphs.
func (c *converter) blocks(nodes []*node) ([]block, error) {
var out []block
var run []*node
flush := func() error {
if len(run) == 0 {
return nil
}
b, err := c.paragraph(run)
run = nil
if err != nil {
return err
}
out = append(out, b)
return nil
}
for _, n := range nodes {
if n.tag != "" && skipped(n) {
continue
}
if n.tag == "" || !blockTags[n.tag] {
run = append(run, n)
continue
}
if err := flush(); err != nil {
return nil, err
}
got, err := c.element(n)
if err != nil {
return nil, err
}
out = append(out, got...)
}
if err := flush(); err != nil {
return nil, err
}
return out, nil
}
// element converts one block element.
func (c *converter) element(n *node) ([]block, error) {
switch n.tag {
case "h1", "h2", "h3", "h4", "h5", "h6":
return c.heading(n)
case "p", "summary", "figcaption", "dt", "dd":
if strings.Contains(" "+n.attrs["class"]+" ", " note ") || n.attrs["role"] == "note" {
return c.quote(n.kids)
}
b, err := c.paragraph(n.kids)
return []block{b}, err
case "ul", "ol":
b, err := c.list(n)
return []block{b}, err
case "blockquote":
return c.quote(n.kids)
case "pre":
return []block{preformatted(n)}, nil
case "hr":
return nil, nil
case "table":
b, err := c.table(n)
return []block{b}, err
case "gemblock":
return c.generated(n)
default:
return c.blocks(n.kids)
}
}
// generated asks Options.Block for the gemtext of a block placeholder.
func (c *converter) generated(n *node) ([]block, error) {
name := n.attrs["name"]
if c.opts.Block == nil {
return nil, fmt.Errorf("the block %q has no generator", name)
}
text, err := c.opts.Block(name)
if err != nil {
return nil, fmt.Errorf("block %q: %w", name, err)
}
text = strings.Trim(text, "\n")
if text == "" {
return nil, nil
}
return []block{{text: strings.Split(text, "\n")}}, nil
}
// heading converts h1 to h6; gemtext has three levels.
func (c *converter) heading(n *node) ([]block, error) {
text, links, err := c.inline(n.kids, " ")
if err != nil {
return nil, err
}
text = strings.Join(strings.Fields(text), " ")
if text == "" {
return nil, nil
}
level := int(n.tag[1] - '0')
if level > 3 {
level = 3
}
return []block{{text: []string{strings.Repeat("#", level) + " " + text}, links: links}}, nil
}
// paragraph converts a run of text and inline elements. Each line break is a
// line of its own, and the links come after. A paragraph that is only a link
// is only the link line.
func (c *converter) paragraph(nodes []*node) (block, error) {
text, links, err := c.inline(nodes, "\n")
if err != nil {
return block{}, err
}
lines := cleanLines(text)
if len(lines) == 1 && len(links) == 1 && lines[0] == links[0].label {
return block{links: links}, nil
}
for i := range lines {
lines[i] = protect(lines[i])
}
return block{text: lines, links: links}, nil
}
// cleanLines splits text at its line breaks, collapses the white space of every
// line and drops the empty ones.
func cleanLines(text string) []string {
var lines []string
for _, line := range strings.Split(text, "\n") {
if line = strings.Join(strings.Fields(line), " "); line != "" {
lines = append(lines, line)
}
}
return lines
}
// protect keeps a line of text from being read as another kind of line: a
// leading space does not change what it says and gemtext has no escape.
func protect(line string) string {
for _, prefix := range []string{"=>", "#", ">", "```", "* "} {
if strings.HasPrefix(line, prefix) {
return " " + line
}
}
return line
}
// space matches a run of white space, the no-break space included.
var space = regexp.MustCompile(`[\s\x{00a0}]+`)
// inline returns the text of the nodes, the links in them and the addresses of
// their images. lineBreak is what a <br> becomes.
func (c *converter) inline(nodes []*node, lineBreak string) (string, []link, error) {
var text strings.Builder
var links []link
var walk func(n *node) error
walk = func(n *node) error {
if n.tag == "" {
text.WriteString(space.ReplaceAllString(n.text, " "))
return nil
}
if skipped(n) {
return nil
}
switch n.tag {
case "br":
text.WriteString(lineBreak)
return nil
case "img":
alt := strings.Join(strings.Fields(n.attrs["alt"]), " ")
if alt == "" || n.attrs["src"] == "" || decorative(n) {
return nil
}
target, err := c.address(n.attrs["src"])
if err != nil {
return err
}
if target != "" {
links = append(links, link{target, alt})
}
return nil
case "a":
label, inner, err := c.inline(n.kids, " ")
if err != nil {
return err
}
text.WriteString(label)
links = append(links, inner...)
if href := n.attrs["href"]; href != "" {
target, err := c.address(href)
if err != nil {
return err
}
if target != "" {
links = append(links, link{target, strings.Join(strings.Fields(label), " ")})
}
}
return nil
}
if blockTags[n.tag] || cellTags[n.tag] {
text.WriteString(" ")
}
for _, kid := range n.kids {
if err := walk(kid); err != nil {
return err
}
}
if blockTags[n.tag] || cellTags[n.tag] {
text.WriteString(" ")
}
return nil
}
for _, n := range nodes {
if err := walk(n); err != nil {
return "", nil, err
}
}
return text.String(), links, nil
}
// decorativeClasses are the classes of an image that belongs to the layout of the
// page (the logo, a mark, an ornament), not to what the page says.
var decorativeClasses = map[string]bool{"logo": true, "mark": true, "decorative": true}
// decorative reports whether an image is there for the look of the page: it says
// so with a role, or has one of the decorativeClasses. It is left out, a link line
// to it would tell a reader of a capsule nothing.
func decorative(n *node) bool {
if role := n.attrs["role"]; role == "presentation" || role == "none" {
return true
}
for _, class := range strings.Fields(n.attrs["class"]) {
if decorativeClasses[class] {
return true
}
}
return false
}
// address maps an address through Options.Link.
func (c *converter) address(href string) (string, error) {
if c.opts.Link == nil {
return href, nil
}
target, err := c.opts.Link(href)
if err != nil {
return "", fmt.Errorf("the address %q: %w", href, err)
}
return target, nil
}
// item is one bullet of a list, with the preformatted blocks it holds.
type item struct {
text string
links []link
// pre holds the lines of the preformatted blocks of the item, fences included.
pre []string
}
// list converts a list: bullets, with the links after them. A list in which
// every item is only a link is a list of link lines. An item with no text, only
// an image or a link without a label, has no bullet; its links are written all
// the same. A list inside an item is flattened into the same list, and a
// preformatted block in an item follows its bullet. An ordered list numbers its
// items.
func (c *converter) list(n *node) (block, error) {
var items []item
if err := c.items(n, &items); err != nil {
return block{}, err
}
linksOnly := len(items) > 0
var links []link
for _, it := range items {
links = append(links, it.links...)
if len(it.pre) > 0 || len(it.links) != 1 || it.text != it.links[0].label {
linksOnly = false
}
}
if linksOnly {
return block{links: links}, nil
}
var lines []string
number := 0
for _, it := range items {
if it.text != "" {
number++
prefix := "* "
if n.tag == "ol" {
prefix += strconv.Itoa(number) + ". "
}
lines = append(lines, prefix+it.text)
}
lines = append(lines, it.pre...)
}
return block{text: lines, links: links}, nil
}
// items collects the items of a list and of the lists inside it.
func (c *converter) items(list *node, out *[]item) error {
for _, li := range list.kids {
if li.tag != "li" || skipped(li) {
continue
}
var own, nested []*node
var pre []string
for _, kid := range li.kids {
switch {
case kid.tag == "ul" || kid.tag == "ol":
nested = append(nested, kid)
case kid.tag == "pre" && !skipped(kid):
pre = append(pre, preformatted(kid).text...)
default:
own = append(own, kid)
}
}
text, links, err := c.inline(own, " ")
if err != nil {
return err
}
if text = strings.Join(strings.Fields(text), " "); text != "" || len(links) > 0 || len(pre) > 0 {
*out = append(*out, item{text: text, links: links, pre: pre})
}
for _, inner := range nested {
if err := c.items(inner, out); err != nil {
return err
}
}
}
return nil
}
// quote converts the content of a quote: every line starts with "> ", the
// blocks are separated by an empty quote line, and the links come after.
func (c *converter) quote(nodes []*node) ([]block, error) {
inner, err := c.blocks(nodes)
if err != nil {
return nil, err
}
var lines []string
var links []link
for _, b := range inner {
if len(b.text) == 0 {
links = append(links, b.links...)
continue
}
if len(lines) > 0 {
lines = append(lines, ">")
}
for _, line := range b.text {
lines = append(lines, "> "+strings.TrimPrefix(line, " "))
}
links = append(links, b.links...)
}
if len(lines) == 0 && len(links) == 0 {
return nil, nil
}
return []block{{text: lines, links: links}}, nil
}
// preformatted converts a <pre>: its text as it is between two fence lines. A
// line that would close the block early gets a space in front.
func preformatted(n *node) block {
var text strings.Builder
var walk func(n *node)
walk = func(n *node) {
if n.tag == "" {
text.WriteString(n.text)
return
}
if n.tag == "br" {
text.WriteString("\n")
return
}
for _, kid := range n.kids {
walk(kid)
}
}
walk(n)
lines := strings.Split(strings.TrimSuffix(text.String(), "\n"), "\n")
out := []string{"```"}
for _, line := range lines {
if strings.HasPrefix(line, "```") {
line = " " + line
}
out = append(out, strings.TrimRight(line, " \t"))
}
return block{text: append(out, "```")}
}
// table converts a table to a preformatted block of aligned columns, with a
// line under a header row, and the caption as a line of text before it. The
// links of the cells come after the block.
func (c *converter) table(n *node) (block, error) {
var rows [][]string
var links []link
var caption string
headerRow := false
var walk func(n *node) error
walk = func(n *node) error {
for _, kid := range n.kids {
if kid.tag == "" || skipped(kid) {
continue
}
switch kid.tag {
case "caption":
text, inner, err := c.inline(kid.kids, " ")
if err != nil {
return err
}
caption = strings.Join(strings.Fields(text), " ")
links = append(links, inner...)
case "tr":
var cells []string
onlyHeaders := true
for _, cell := range kid.kids {
if cell.tag != "td" && cell.tag != "th" {
continue
}
text, inner, err := c.inline(cell.kids, " ")
if err != nil {
return err
}
cells = append(cells, strings.Join(strings.Fields(text), " "))
links = append(links, inner...)
if cell.tag != "th" {
onlyHeaders = false
}
}
if len(cells) > 0 {
if len(rows) == 0 && onlyHeaders {
headerRow = true
}
rows = append(rows, cells)
}
default:
if err := walk(kid); err != nil {
return err
}
}
}
return nil
}
if err := walk(n); err != nil {
return block{}, err
}
var text []string
if caption != "" {
text = append(text, protect(caption))
}
switch {
case len(rows) == 0:
case tableWidth(rows) <= maxTableWidth:
text = append(text, "```")
text = append(text, alignRows(rows, headerRow)...)
text = append(text, "```")
default:
if caption != "" {
text = append(text, "")
}
text = append(text, records(rows, headerRow)...)
}
return block{text: text, links: links}, nil
}
// maxTableWidth is the widest a table can be and stay a preformatted block: lines
// of a preformatted block are not wrapped by the readers.
const maxTableWidth = 80
// tableWidth returns the width of the rows written as columns two spaces apart.
func tableWidth(rows [][]string) int {
widest := 0
for _, line := range alignRows(rows, false) {
if w := utf8.RuneCountInString(line); w > widest {
widest = w
}
}
return widest
}
// records writes the rows of a wide table as a list. With a header row and
// more than one column each row is a heading, its first cell when it has
// something in it, and a bullet for each other cell that is not empty, named by
// the header of its column. A table of one column is a bullet for each cell, the
// header a line of text before them. Without a header, or with a header and
// nothing else, each row is a bullet with the cells between bars.
func records(rows [][]string, header bool) []string {
columns := 0
for _, r := range rows {
if len(r) > columns {
columns = len(r)
}
}
if header && len(rows) == 1 {
header = false
}
var lines []string
if columns <= 1 {
if header {
lines = append(lines, protect(rows[0][0]))
rows = rows[1:]
}
for _, r := range rows {
lines = append(lines, "* "+strings.Join(r, " | "))
}
return lines
}
if !header {
for _, r := range rows {
lines = append(lines, "* "+strings.Join(r, " | "))
}
return lines
}
names := rows[0]
for i, r := range rows[1:] {
if i > 0 {
lines = append(lines, "")
}
if len(r) > 0 && r[0] != "" {
lines = append(lines, "### "+r[0])
}
for j := 1; j < len(r); j++ {
if r[j] == "" {
continue
}
name := ""
if j < len(names) {
name = names[j] + ": "
}
lines = append(lines, "* "+name+r[j])
}
}
return lines
}
// alignRows writes rows of cells as columns two spaces apart, each as wide as
// its widest cell, with a line of dashes under the first row when it is a header.
func alignRows(rows [][]string, header bool) []string {
columns := 0
for _, r := range rows {
if len(r) > columns {
columns = len(r)
}
}
widths := make([]int, columns)
for _, r := range rows {
for i, cell := range r {
if w := utf8.RuneCountInString(cell); w > widths[i] {
widths[i] = w
}
}
}
write := func(r []string) string {
cells := make([]string, columns)
for i := range cells {
cell := ""
if i < len(r) {
cell = r[i]
}
cells[i] = cell + strings.Repeat(" ", widths[i]-utf8.RuneCountInString(cell))
}
line := strings.TrimRight(strings.Join(cells, " "), " ")
if strings.HasPrefix(line, "```") {
line = " " + line
}
return line
}
var lines []string
for i, r := range rows {
lines = append(lines, write(r))
if i == 0 && header {
dashes := make([]string, columns)
for j := range dashes {
dashes[j] = strings.Repeat("-", widths[j])
}
lines = append(lines, strings.Join(dashes, " "))
}
}
return lines
}

View file

@ -0,0 +1,235 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package gemtext
import (
"errors"
"strings"
"testing"
)
// identity leaves every address as it is.
func identity(href string) (string, error) { return href, nil }
// convert converts html with addresses left as they are and no blocks.
func convert(t *testing.T, html string) string {
t.Helper()
got, err := Convert(html, Options{Link: identity})
if err != nil {
t.Fatalf("Convert(%q) error = %v", html, err)
}
return got
}
// TestConvert checks the gemtext of every kind of element the site uses.
func TestConvert(t *testing.T) {
tests := []struct {
name, html, want string
}{
{"headings", `<h1>Title</h1><h2 id="a">Sub</h2><h4>Deep</h4>`, "# Title\n\n## Sub\n\n### Deep\n"},
{"hero with line breaks", `<h1 class="hero">Technology.<br>Freedom.<br><span class="accent">Dissent.</span></h1>`, "# Technology. Freedom. Dissent.\n"},
{"links go after the paragraph",
`<p>Read the <a href="/a.html">first</a> and <strong>the <a href="https://x.org/b">second</a></strong> page.</p>`,
"Read the first and the second page.\n=> /a.html first\n=> https://x.org/b second\n"},
{"the same link once", `<p><a href="/a">A</a> and <a href="/a">A</a> again.</p>`, "A and A again.\n=> /a A\n"},
{"a paragraph that is only a link", `<p><a href="/x.html">All news</a></p>`, "=> /x.html All news\n"},
{"line breaks", `<p>One<br>Two<br />Three</p>`, "One\nTwo\nThree\n"},
{"whitespace is collapsed", "<p>a\n b\t c</p>", "a b c\n"},
{"entities", `<p>Tom &amp; Jerry&nbsp;&copy; &lt;ok&gt;</p>`, "Tom & Jerry © <ok>\n"},
{"unordered list", `<ul><li>One</li><li>Two <a href="/c">link</a></li></ul>`, "* One\n* Two link\n=> /c link\n"},
{"a list of only links has no bullets", `<ul><li><a href="/a">A</a></li><li><a href="/b">B</a></li></ul>`, "=> /a A\n=> /b B\n"},
{"nested list is flattened", `<ul><li>A<ul><li>A1</li></ul></li><li>B</li></ul>`, "* A\n* A1\n* B\n"},
{"ordered list", `<ol><li>First</li><li>Second</li></ol>`, "* 1. First\n* 2. Second\n"},
{"quote", `<blockquote><p>One</p><p>Two <a href="/x">x</a></p></blockquote>`, "> One\n>\n> Two x\n=> /x x\n"},
{"note is a quote", `<p class="note" role="note">Careful</p>`, "> Careful\n"},
{"preformatted", "<pre><code>line1\n line2\n```\n</code></pre>", "```\nline1\n line2\n ```\n```\n"},
{"table with a header",
`<table class="data"><caption>Data</caption><thead><tr><th>Service</th><th>Kept</th></tr></thead><tbody><tr><td data-label="Service">Mail</td><td>30 days</td></tr><tr><td>Chat</td><td>1 year</td></tr></tbody></table>`,
"Data\n```\nService Kept\n------- -------\nMail 30 days\nChat 1 year\n```\n"},
{"table without a header", `<table><tr><td>a</td><td>bb</td></tr><tr><td>ccc</td><td>d</td></tr></table>`, "```\na bb\nccc d\n```\n"},
{"table links", `<table><tr><td><a href="/m">Mail</a></td><td>x</td></tr></table>`, "```\nMail x\n```\n=> /m Mail\n"},
{"image with a description", `<p><img src="/logo.svg" alt="kmalloc"></p>`, "=> /logo.svg kmalloc\n"},
{"image without a description is skipped", `<p><img src="/logo.svg" alt=""></p>`, ""},
{"lines that look like gemtext are protected", `<p># no</p><p>=&gt; no</p><p>* no</p><p>&gt; no</p>`, " # no\n\n => no\n\n * no\n\n > no\n"},
{"hidden and scripted things are skipped", `<script>x</script><style>y</style><p hidden>z</p><p aria-hidden="true">w</p><svg><path/></svg><p>kept</p>`, "kept\n"},
{"containers are transparent", `<div><section><p>A</p></section></div><div>Loose <em>text</em></div>`, "A\n\nLoose text\n"},
{"time and span are inline", `<p><time datetime="2026-11-14">14 November 2026</time> <span>by</span> squid</p>`, "14 November 2026 by squid\n"},
{"empty input", ``, ""},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := convert(t, tt.html); got != tt.want {
t.Errorf("Convert() =\n%q\nwant\n%q", got, tt.want)
}
})
}
}
// TestConvertLinkMapping checks that Link rewrites and drops addresses, and that its error stops the conversion.
func TestConvertLinkMapping(t *testing.T) {
link := func(href string) (string, error) {
switch {
case strings.HasPrefix(href, "#"):
return "", nil
case href == "/broken.html":
return "", errors.New("no such page")
}
target, _, _ := strings.Cut(href, "#")
return strings.Replace(target, ".html", ".gmi", 1), nil
}
got, err := Convert(`<p>See <a href="/a.html#x">a</a> and <a href="#top">top</a>.</p>`, Options{Link: link})
if err != nil || got != "See a and top.\n=> /a.gmi a\n" {
t.Errorf("Convert() = %q, %v", got, err)
}
_, err = Convert(`<p><a href="/broken.html">x</a></p>`, Options{Link: link})
if err == nil || !strings.Contains(err.Error(), "/broken.html") || !strings.Contains(err.Error(), "no such page") {
t.Errorf("error = %v, want one that names the address and the cause", err)
}
_, err = Convert(`<p><img src="/broken.html" alt="pic"></p>`, Options{Link: link})
if err == nil {
t.Error("a broken image address is not an error")
}
}
// TestConvertBlocks checks the placeholders of generated blocks.
func TestConvertBlocks(t *testing.T) {
block := func(name string) (string, error) {
if name != "news_list" {
return "", errors.New("unknown block " + name)
}
return "=> /a 2026-01-01 A\n=> /b 2025-12-31 B", nil
}
got, err := Convert(`<p>intro</p><gemblock name="news_list"></gemblock><p>end</p>`, Options{Link: identity, Block: block})
if err != nil || got != "intro\n\n=> /a 2026-01-01 A\n=> /b 2025-12-31 B\n\nend\n" {
t.Errorf("Convert() = %q, %v", got, err)
}
if _, err := Convert(`<gemblock name="other"></gemblock>`, Options{Link: identity, Block: block}); err == nil {
t.Error("an unknown block is not an error")
}
if _, err := Convert(`<gemblock name="news_list"></gemblock>`, Options{Link: identity}); err == nil {
t.Error("a block without a Block function is not an error")
}
}
// TestConvertRejectsBrokenHTML checks that markup that cannot be read is an error.
func TestConvertRejectsBrokenHTML(t *testing.T) {
if _, err := Convert(`<p>unclosed`, Options{Link: identity}); err == nil || !strings.Contains(err.Error(), "HTML") {
t.Errorf("error = %v, want one about the HTML", err)
}
}
// TestTextAndLink checks the helpers that write single lines.
func TestTextAndLink(t *testing.T) {
if got := Text(" two \n words "); got != "two words" {
t.Errorf("Text() = %q", got)
}
if got := Text("# not a heading"); got != " # not a heading" {
t.Errorf("Text() = %q, want a leading space", got)
}
if got := Link("/a.gmi", " A page "); got != "=> /a.gmi A page" {
t.Errorf("Link() = %q", got)
}
if got := Link("/a.gmi", ""); got != "=> /a.gmi" {
t.Errorf("Link() without a label = %q", got)
}
}
// TestConvertWideTable checks that a table that does not fit in 80 columns becomes a list of records, a
// heading for the first cell of each row and a bullet for each other cell, named by its column.
func TestConvertWideTable(t *testing.T) {
long := strings.Repeat("word ", 20) + "end"
html := `<table><caption>Data</caption><thead><tr><th>Service</th><th>Kept</th><th>Seen</th></tr></thead>` +
`<tbody><tr><td><a href="/m">Mail</a></td><td>` + long + `</td><td>all</td></tr><tr><td>Chat</td><td></td><td>some</td></tr></tbody></table>`
want := "Data\n\n### Mail\n* Kept: " + long + "\n* Seen: all\n\n### Chat\n* Seen: some\n=> /m Mail\n"
if got := convert(t, html); got != want {
t.Errorf("Convert() =\n%q\nwant\n%q", got, want)
}
noHeader := `<table><tr><td>a</td><td>` + long + `</td></tr><tr><td>b</td><td>c</td></tr></table>`
if got, want := convert(t, noHeader), "* a | "+long+"\n* b | c\n"; got != want {
t.Errorf("Convert() without a header =\n%q\nwant\n%q", got, want)
}
}
// TestConvertListItems checks the items that have no text, more than one paragraph or a preformatted block.
func TestConvertListItems(t *testing.T) {
tests := []struct {
name, html, want string
}{
{"an item that is only an image has no bullet", `<ul><li><img src="/a.png" alt="pic"></li><li>Text</li></ul>`, "* Text\n=> /a.png pic\n"},
{"an item that is a link without text has no bullet", `<ul><li><a href="/x"></a></li><li>Text</li></ul>`, "* Text\n=> /x\n"},
{"an item of only an image in a list of only images", `<ul><li><img src="/a.png" alt="A"></li><li><img src="/b.png" alt="B"></li></ul>`, "=> /a.png A\n=> /b.png B\n"},
{"paragraphs of an item do not stick", `<ul><li><p>First</p><p>Second</p></li></ul>`, "* First Second\n"},
{"cells of a table in an item do not stick", `<ul><li><table><tr><td>a</td><td>b</td></tr></table></li></ul>`, "* a b\n"},
{"a preformatted block in an item stays a block", "<ul><li>Run:<pre><code>x\n y</code></pre></li><li>Next</li></ul>", "* Run:\n```\nx\n y\n```\n* Next\n"},
{"a preformatted block that is all of an item", "<ul><li><pre>code</pre></li></ul>", "```\ncode\n```\n"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := convert(t, tt.html); got != tt.want {
t.Errorf("Convert() =\n%q\nwant\n%q", got, tt.want)
}
})
}
}
// TestConvertAddressWithSpaces checks that a space in an address does not end the address of a link line.
func TestConvertAddressWithSpaces(t *testing.T) {
if got, want := convert(t, `<p><a href="/a b/c d.html">t</a></p>`), "=> /a%20b/c%20d.html t\n"; got != want {
t.Errorf("Convert() = %q, want %q", got, want)
}
if got, want := Link("/a b", "x"), "=> /a%20b x"; got != want {
t.Errorf("Link() = %q, want %q", got, want)
}
}
// TestConvertWideTableEdges checks the wide tables with no title for a row, with a header only and with one column.
func TestConvertWideTableEdges(t *testing.T) {
long := strings.Repeat("word ", 20) + "end"
tests := []struct {
name, html, want string
}{
{"a row with an empty first cell has no heading",
`<table><tr><th>Service</th><th>Kept</th></tr><tr><td></td><td>` + long + `</td></tr><tr><td>Chat</td><td>1 year</td></tr></table>`,
"* Kept: " + long + "\n\n### Chat\n* Kept: 1 year\n"},
{"a header and nothing else keeps the header",
`<table><thead><tr><th>` + long + `</th><th>Seen</th></tr></thead></table>`,
"* " + long + " | Seen\n"},
{"one wide column with a header",
`<table><tr><th>Name</th></tr><tr><td>` + long + `</td></tr><tr><td>short</td></tr></table>`,
"Name\n* " + long + "\n* short\n"},
{"one wide column without a header",
`<table><tr><td>` + long + `</td></tr><tr><td>short</td></tr></table>`,
"* " + long + "\n* short\n"},
{"one narrow column", `<table><tr><th>Name</th></tr><tr><td>a</td></tr></table>`, "```\nName\n----\na\n```\n"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := convert(t, tt.html); got != tt.want {
t.Errorf("Convert() =\n%q\nwant\n%q", got, tt.want)
}
})
}
}
// TestConvertDecorativeImages checks that the images of the layout leave no link line and those of the content do.
func TestConvertDecorativeImages(t *testing.T) {
tests := []struct {
name, html, want string
}{
{"a logo", `<img class="logo" src="/logo.svg" alt="kmalloc" width="112"><p>Text</p>`, "Text\n"},
{"a logo among classes", `<p><img class="big mark" src="/m.svg" alt="mark"> Text</p>`, "Text\n"},
{"a decorative image", `<p><img class="decorative" src="/d.png" alt="swirl"> Text</p>`, "Text\n"},
{"role presentation", `<p><img role="presentation" src="/d.png" alt="swirl"> Text</p>`, "Text\n"},
{"role none", `<p><img role="none" src="/d.png" alt="swirl"> Text</p>`, "Text\n"},
{"an image of the content", `<p><img class="figure" src="/chart.png" alt="Sales by month"></p>`, "=> /chart.png Sales by month\n"},
{"an image that has no class", `<p><img src="/chart.svg" alt="A diagram"></p>`, "=> /chart.svg A diagram\n"},
{"a class that only contains the word", `<p><img class="logotype" src="/l.png" alt="Type"></p>`, "=> /l.png Type\n"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := convert(t, tt.html); got != tt.want {
t.Errorf("Convert() =\n%q\nwant\n%q", got, tt.want)
}
})
}
}

View file

@ -35,6 +35,8 @@ type Lang struct {
Strings map[string]string
// Nav maps a menu entry to its label in this language.
Nav map[string]string
// NavOrder lists the menu entries in the order the file writes them.
NavOrder []string
}
// shownKeys are the address keys that get a harvest-proof "_shown" twin.
@ -144,9 +146,35 @@ func loadLang(dir, code string) (Lang, error) {
if err := decodeField(raw, name, "nav", &lang.Nav); err != nil {
return Lang{}, err
}
order, err := keyOrder(raw["nav"])
if err != nil {
return Lang{}, fmt.Errorf("%s: nav: %w", name, err)
}
lang.NavOrder = order
return lang, nil
}
// keyOrder returns the keys of a JSON object in the order they are written.
func keyOrder(object json.RawMessage) ([]string, error) {
decoder := json.NewDecoder(strings.NewReader(string(object)))
if _, err := decoder.Token(); err != nil {
return nil, err
}
var keys []string
for decoder.More() {
key, err := decoder.Token()
if err != nil {
return nil, err
}
keys = append(keys, key.(string))
var skipped json.RawMessage
if err := decoder.Decode(&skipped); err != nil {
return nil, err
}
}
return keys, nil
}
// readObject reads a JSON file that holds one object and returns its fields
// undecoded. Errors name the file.
func readObject(dir, name string) (map[string]json.RawMessage, error) {

View file

@ -184,3 +184,24 @@ func TestLoadNeedsLanguages(t *testing.T) {
t.Errorf("Load() error = %v, want it to mention languages", err)
}
}
// TestLoadKeepsTheOrderOfTheMenu checks that the entries of nav are listed in the
// order the language file writes them, which a map cannot tell.
func TestLoadKeepsTheOrderOfTheMenu(t *testing.T) {
files := validFiles()
files["lang/en.json"] = `{"lang_name": "English", "nav": {"manifesto": "Manifesto", "people": "People", "news": "News", "services": "Services"}}`
_, langs, err := Load(writeSite(t, files))
if err != nil {
t.Fatalf("Load() error = %v", err)
}
want := []string{"manifesto", "people", "news", "services"}
got := langs["en"].NavOrder
if len(got) != len(want) {
t.Fatalf("NavOrder = %v, want %v", got, want)
}
for i := range want {
if got[i] != want[i] {
t.Fatalf("NavOrder = %v, want %v", got, want)
}
}
}