Building this site

silasjelley.com is a static website of a mish-mash of my documents and the public tip of a much bigger pile of documents that I write, read and search every day, and the software that builds it, which I’ve named Mosaic.

Most of what I use Mosaic for is not reflected at all in what you can see from this site. Earlier versions of this page described a single Python script that turned some Markdown files into HTML. That script gradually grew more opinionated and gargantuan, went through a half a dozen rewrites, three document markup languages, and eventually arrived at the (much more) complex, but ultimately much more useful (to me) system that it is today. This page is an attempt to describe how it all works, and records some of the more significant forks in the decision tree that got it here.

Summary

I keep one big folder of plain text documents on my computer. In it you’ll find¹ my journal going back to 2017, my notes and essays, a couple of thousand quotes and references to things I’ve read, watched and listened to, and the pages of this site. Beside it sits a second pile of records: photos, audio, GPS tracks from walks and rides, my ledger and so on.

Every document says, in a little block of metadata at the top, who is allowed to see it. Most of the time I’m just writing and reading in my own private workspace, and the question of who can see what doesn’t come up. Every so often I run mos site publish in a terminal, and the documents that are marked public are compiled into web pages, checked very thoroughly, and pushed to a computer somewhere on the internet, from which anyone with a browser can read them. Today that’s you.

TLDR: I hit buttons, you hit buttons, magic happens and somehow you see the words that I made when I hit the buttons. There’s a bunch of pipes and computers in between, but magic does a lot of the heavy lifting.

A system of documents

The whole thing rests on a handful of simple ideas.

Everything is a document. A journal entry, an essay, a quote from a book, the page listing my feeds, this page: each one is a single Typst file. At the time of writing there are about ten thousand of them, adding up to a couple million words. Photos, audio, PDFs and other files aren’t documents themselves, but each one gets an entry in a list (a manifest) that gives it the same kind of metadata a document has.

Everything has a permanent ID. Each document and file has a UUID, and documents link to each other by a shortcode of the first eight characters of it rather than by URL or filename. That means I can rename a file, move it to another folder or change its title, and nothing breaks. Even the file’s location on disk doesn’t mean anything: new notes are just named after the moment they were created, like 2026/10/22/054500.typ.

Private by default (with strict guarentees) A good chunk of the code of this is dedicated to correctness, to making sure unexpected things don’t happen. And a good part of that is set to the task of making sure nothing gets out that shouldn’t, and shouts loudly at me for any missteps. Only an explicit and verified access: "public", and a few other conditions allows you to read this page. This explicitness gives me the peace of mind to maintain one holistic and consistent system without fear of leakage, rather than maintaining two parallel and independent systems and the frustrating duplication that would necessitate.

Rich and bi-directional links between documents Every link from one document to another also creates a backlink pointing in the other direction. In this way a connected graph of objects is formed, allowing for the discovery of otherwise hidden connections and patterns. This is further enriched by the use of semantic embeddings to draw connections between documents that don’t have any links between them.

The site is just one view of the pile. The core of the system is a simple composable API of objects. When I’m reading and writing in Mosaic, I see everything, and in a totally different way than this site; but I also have other “views” into the system that cut a more narrow line, for when I want to focus on something in particular, e.g. I have a whole app dedicated exclusively to enriching the metadata attached to external references/quotes — that view does not see anything but those resources.

The website is just yet another “view”, the result of mos site publish cutting it all down to that subset of documents and assets that I’m happy for other people to see, then wrapping it up in a bundle of manilla CSS, and pushing it to my server.

Who has access

Two fields in each document’s metadata decide where it ends up.

• private: only me. These never end up on the site, no matter what. This is the default.
• protected: people I’ve given a password to. These are published under /protected/ behind a login.
• public: anyone.

presentation says how it shows up for the people who can see it:

• listed: a normal page that turns up in lists, feeds, search and backlinks. This is the default.
• unlisted: a real page with a real URL, but it isn’t advertised anywhere. You need the link.
• embedded: no page of its own at all. It only exists to be quoted or embedded inside other documents.

All nine combinations are allowed, and the build checks that nothing leaks between them. A public page that tries to quote a private note will cause a build failure and go nowhere.

Inputs

The documents themselves

Every document starts with the same import (my own custom Typst functions) and a block of metadata, followed by the text itself. Here’s roughly what the top of this page looks like:

#import "@local/mosaic:0.1.0": *

#metadata((
  created: "2022-04-24T12:00:00+12:00",
  uid: "ba4574ce-516e-4b42-ac32-ba7c06e41e1a",
  slug: "build",
  access: "public",
  presentation: "listed",
  title: "Building this site",
  category: "meta",
  secondary: "misc",
)) <meta>

silasjelley.com is a static website of a mish-mash...

The metadata is a plain Typst dictionary, and holds the keys to Mosaic’s ability to make the system useful to me. These are the ones that do most of the work:

• uid: A universally unique identifier version 4. The first eight characters must also be unique across my documents and assets, so I can use a short version in links without making a guessing game of it.
• slug: The path on the web server, deliberately unrelated to the filename and the title. I keep it unchanged once published so other people’s links keep working; Mosaic checks that no two things claim the same path.
• category: and secondary: Two levels of loose grouping, like meta and misc here. I can make up new ones as I go, and Mosaic checks they don’t conflict.
• created: When the document was first written, including the time and timezone offset. Like the UID, it stays put.
• available: and updated: When it was ready for reading, and when it last changed in a meaningful way. available falls back to created; updated falls back to available, then created.
• access: and presentation: Who can see it, and how it shows up. See above. Leaving them out means private.
• title: and creator: What it’s called and who made it. A document without a title has it filled in from the source or the date automatically.
• tags: Arbitrary labels, useful for filtering and for collections, like ("wandering", "draft"). These mustn’t repeat the category or secondary. Feeds are generated for them.
• source: and via: Where something came from and how I found it. For a quote from a book that I found in somebody else’s essay, the book is the source and the essay is the via. Both use the same set of fields, listed below.

This is a fixed vocabulary: an unknown key is an error, including inside source, via and location. Optional text can be empty, as can a tuple or dictionary, but none isn’t a metadata value here. To clear an optional date, parent or cover image, the key is removed altogether.

Everything else about a document is worked out when it’s needed rather than written down: its URL, its word count, which other documents it links to and which link back to it.

The body is ordinary Typst plus a small library of my own functions for the things a website needs: site-link to link to another document by its ID, img and gallery for pictures, site-quote and site-embed to pull in bits of other documents, collection to list documents matching some criteria, and helpers for audio, video, PDFs, asides, poems and collapsible sections.

Markup

I should write a lot about text markup languages because it’s something of a passion of mine, but precisely because of that I get stuck when I come to write about it because I feel like there is so much to say. So instead I will say it very briefly:

The earliest versions of the “document system” (~6 years ago) were unstructured plain text, sometimes accumulating in documents many thousands of words long. From there markdown was the natural evolution, formatted, but not especially structured. Eventually, after being endlessly frustrated by the under specification of markdown, I began migrating to Djot after reading John McFarlane’s Beyond Markdown.

Djot was very nice for a while, it gave me pleasure to write in a well specified markup, and I made extensive use of its extensibility to write my own custom shortcuts — the early version of a system of transclusion, bi-directional linking, and graph building. But, as with markdown before it, the greatest pleasure was in migrating to something better. Djot’s final gift was that by being better far more internally consistent, translating its defined syntax to what would come next went much more smoothly than migrating from markdown to Djot had prior.

Typist’s syntax was love at first sight for me, but I had to wait a long time before I could actually use it. With the tease of native HTML export on the roadmap (even in an experimental state) I didn’t feel like developing my own Typst to HTML renderer. But Typst’s HTML export took quite a long time. Once it had the earliest experimental support in the mainline I began working it into my system. True to form for Martin Haug & Laurenz Mädje though, even the “experimental” release was pretty polished, and it has churned very little since.

Finally I was writing the markup I’d always dreamed of! (Yes, I dream of markup…)

Who knows what the future holds, but given the thoughtful and pragmatic development style of the Typst team, I’m inclined to say that Typst is the last markup format I will ever need².

That’s enough about markup for now.

Files

All my  150,000 images (plus 60,000 RAW versions) and  12,000 videos live in ~/images, ~/raws, and ~/videos and are represented in the system by entries in “manifests” (one entry per file, grouped in the case of e.g. RAW > JPEG versions). These manifests are JSONLines files, so each entry is on it’s own line — useful as it allows parallelised ingest for the larger manifests.

The same is true for ~/audio recordings, scans, fonts, GPS tracks and so on. Each one is listed in a manifest file with its UID, title, slug, access level, where it came from and so on, along with format specific metadata. The same “private unless I say otherwise” rule applies, so a photo can’t sneak onto the site just because a now public page tried to show it.

Putting documents together

Lists that keep themselves up to date

The metadata earns its keep when I want to gather things together. Rather than maintain a page of links by hand, I can ask for a collection:

#collection(
  include_: "journal",
  filter_: "favourite=true",
  style: "title",
  order: "reverse-chronological",
  limit: 10,
)

That gives me the ten most recent favourites from my journal, ordered by their available dates. Mark another entry as a favourite and it joins the list the next time the page is built (or instantly in my local system). The list lives in an ordinary document, so I can put an introduction above it, pictures around it, or several different lists on the same page.

include_ and exclude_ pick categories, secondary groupings or tags; filter_ narrows things down by metadata, including nested fields like source.creator. I can show just titles, summaries, or whole bodies, and sort alphabetically or put the oldest entries first. On the public site these lists only draw from public, listed documents. The same query in my personal workspace can see the private pile too, so what it returns depends on who is reading it.

Write it once, use it elsewhere

Quotes get their own documents because I often want the same passage in more than one place. The words live in one file, along with their source and perhaps a via; and then I can reference/transclude that document from anywhere in the system. If I write:

#site-quote("d1b4e585") // This one's disabled so you can see the syntax)

Then Mosaic brings in the passage and its attribution together:

For want of a Nail the Shoe was lost; for want of a Shoe the Horse was lost; and for want of a Horse the Rider was lost, being overtaken and slain by the Enemy, all for want of Care about a Horse-shoe Nail.

— Benjamin Franklin, Poor Richard Improved, 1758

If I fix a typo or find the missing page number, the next build picks that up everywhere the quote appears.

site-embed does a similar job for a document’s full content, without wrapping it as a quotation (useful for assembling a larger document from several smaller/atomic ones) while site-cite makes a footnote citation when I only need the reference. A quote marked presentation: "embedded" doesn’t need a web page of its own to be useful this way, though most of the quotes I save are also listed publicly at my quotes page. It still needs the right access level: putting a private quote inside a public essay doesn’t magically make it public. The check follows references inside the included document too, so an image or another quote can’t slip through by being one step removed, 6 degrees of Kevin Bacon Transcluded Documents style.

This also keeps references contained, and therefore semantically useful. A link inside an embedded passage belongs to the document that actually contains it, rather than becoming a newly authored link from every page that borrows the passage.

Writing and reading

Day to day, I mostly use Mosaic as a place to read and write rather than as a site builder. The command is mos:

• mos open starts a little web app on my own computer that can see every document, private ones included. It has one search box, a reader, an editor with a live preview, and a panel showing what links to what. While I do spend a lot of time in the app, especially when I’m doing anything visual, I spend most of the rest of the time on my computer in a terminal, so it was always an essential design requirement that all the functionality of Mosaic be equally accessible from the app and the terminal.

• mos new creates a new document with a fresh ID and a timestamped filename.
• mos add adds a file to the right manifest.
• mos fmt tidies documents into a consistent shape (without ever changing how they render).
• mos check looks for broken links, bad metadata, duplicate IDs and other mischief across the whole pile.
• mos search and mos related search everything from the terminal, the latter loading the embedding model for semantic search.

Editing without getting in the way

The workspace lets me edit the words beside a live preview, with the metadata in a separate drawer. When I insert a link or an image, I can search for it by name and let the editor fill in its short ID. But these are still ordinary files: I can open one in another text editor, and Mosaic watches for changes.

That means two editors can have different ideas about what a note currently says. Before saving, Mosaic checks that the file is still the version I opened. If something else has changed it, my unsaved words stay in the editor and I can compare the two versions, reload the file, or explicitly keep mine. Of course, concurrent editing isn’t a major concern for a personal knowledge management (PKM) system.

Saving is explicit by default; I can turn on autosave in the browser’s settings. The browser also keeps a local draft of unsaved work and offers to restore it when I reopen the note. When creating a note with mos new in an external editor, a failed save would leave the edited draft on disk and tell me where it is.

A note doesn’t have to be finished before I can save it. Broken references or an unfinished body can stay on disk with diagnostics; publishing is where those problems have to be resolved. Bad metadata or a duplicate identity still blocks the save, because those would make it unclear what the document even is.

Finding things again

Search works two ways at once. There’s ordinary keyword search, and there’s search by meaning: a small language model running entirely on my own computer turns passages into numbers, so that a search for “being lost in the mountains” can find a journal entry that never uses any of those words but which captures that feeling. I also embed my image library, so that same query might return a photo from a foggy day in the Tararua mountains in New Zealand, with both the content of the image as well as any description I’ve added to it being fed into the embeddings. None of this leaves my machine, and none of it goes into the public site; the search box on this site is a plain keyword search over public pages only³.

The workspace search box and mos search understand the same queries. For example:

mos search '#journal after:2025 before:2026 lost in the mountains'

Here #journal narrows things to that category, and the dates pick entries with an available date from the start of 2025 up to, but not including, the start of 2026. A tag works the same way as the category: if I’ve tagged things walking, I can add #walking to narrow it further. The remaining words are what I want to find, by wording or meaning. Quoted phrases must actually occur, a minus sign excludes a word or tag, and access:private would narrow the results to private documents. mos search --lexical asks for keyword search alone.

Sometimes I don’t know what to search for at all. The Related notes view starts from the saved document I’m reading and suggests others with similar content, even when I’ve never linked them together. That’s a different kind of wandering from following backlinks, and often the more surprising one.

What gets kept

The Typst files, asset manifests and original photos, recordings and other files are the things I’ve actually written or collected. They’re the source of truth. Titles, tags, permissions and attribution live there, where I can read and edit them without Mosaic.

There is also a private SQLite database holding things the computer worked out: extracted text, the numbers used for semantic search, generated image descriptions and the details of how those descriptions were made. mos index update refreshes the search data from saved files; unsaved editor buffers don’t go into it. Image descriptions are a separate, deliberate job, mos index describe, so searching or indexing doesn’t silently start captioning every photo. Machine descriptions stay in the database, without rewriting my manifests or granting anything permission to appear on the site.

Most of this derived data can be rebuilt, though that takes time and generating a description again needn’t produce the same words. If I want to keep the exact descriptions I’ve already generated, the database belongs in my backups too. The build’s cached HTML and resized images are less precious: losing those just makes the next build do more work. Keeping that distinction clear makes it much easier to know what I need to look after.

Printing and sharing a PDF

The same Typst source can also become a PDF:

mos pdf meta/build.typ --audience public -o /tmp/build.pdf

The audience is deliberate: public can use public resources, protected can also use protected ones, and personal can use everything. Mosaic checks the references and gives the compiler only the files that audience is allowed to see, just as it does for the website. The export stays outside the site’s publishable output.

This is a typeset version of the document, rather than a screenshot of its web page. Links, pictures, quotes and citations have PDF versions, and disclosures print their full contents instead of hiding them behind a click. There are limits: collections currently show a placeholder in PDFs while I decide how they should behave in a non-interative media, and interactive web features don’t become interactive paper. For a piece of writing I want to print or send as a file, though, I don’t need to maintain a second copy of the words.

Building

Building the full public site is one command, mos site build, and takes about a minute (the local app is always “live” as it incrementally rebuilds with any change) In roughly this order, it:

1. Reads every document and manifest and checks the lot, so that one broken file or link anywhere stops the build rather than producing a slightly wrong site.
2. Works out which documents and files the public is allowed to see.
3. Copies only those into a temporary folder, along with my Typst library, and compiles each page from there. Typst literally cannot see my private notes while it builds the public site, so it can’t accidentally include them. Protected pages get their own pass in the same way.
4. Wraps the compiled pages in the templates, and fills in the generated bits: lists of documents, backlinks, feeds, a sitemap, statistics and a plain list of every external link on the site.
5. Converts images to smaller JPEG and AVIF versions, and copies over the other public files.
6. Builds the search index with Pagefind, from public listed pages only.
7. Checks the finished site: every file must be accounted for, every link must go somewhere it’s allowed to go, and nothing private may appear anywhere in it.

Compiled pages and converted images are cached, so after the first build only the things that changed are redone. I don’t keep a copy of the built site lying around: it’s built in memory, checked, published (or not) and thrown away. What I do keep is an inventory of what was built, so that the publish step can refuse to send anything that has changed since it was checked.

Publishing

mos site publish does a build and then sends it to my server. It’s more paranoid about this than it strictly needs to be, which is the point:

• Before sending anything it asks my web server (Caddy) how it’s configured, and refuses to publish unless the password protection on /protected is definitely in place.
• Each publish goes into a brand new folder on the server. Files that haven’t changed are reused rather than sent again, so a typical publish only uploads a handful of files.
• Once everything has arrived, the server checks the files against the inventory, then a single switch points the site at the new folder. You either see the old site or the new one, never a half-uploaded mix.
• The last few versions are kept around in case I need to roll back.

The server itself is a small machine running NixOS, and its configuration (including all the old URL redirects, which only ever get added to) lives in its own repository⁴.

Decisions along the way

A rough history of the bigger forks in the road:

• Markdown to Djot to Typst. Typst is a real typesetting language, so the same source can become a web page or a properly typeset PDF, and the little functions I need (galleries, quotes, maps, citations) can be written in the language itself rather than bolted on with search and replace.
• TOML to Typst metadata. Once the documents were Typst, it made sense for the metadata to be Typst too, rather than a block of a different language glued to the top.
• Paths to IDs. Linking by ID instead of by URL or filename means I can rename and reorganise freely without breaking anything.
• A published date to explicit access. The old system decided whether to publish something by checking whether a published date was set. That’s a brittle system to gate whether your diary goes on the internet. Now everything is private unless it says otherwise, and the build refuses to guess.
• Website to system of documents. The site used to it’s own thing, distince from my journals etc. Now all of my notes, journals, images, and other media are brought together in a single system that also happens to be able to output my website.
• Still static. Through all of it, the output is still a folder of plain HTML files served by an ordinary web server. It’s fast, cheap, and will happily outlive whatever I use to build it.

Anyhow, that ought to be enough going on for now, but reach out if you have any questions.