SEP 35: Incremental builds
Status: Partially implemented.
Date: 2023-04-20
Shallow and full builds are fully functional, the so called 'normal' build that
works with a partial cache but can still rebuild where new interlinks are found
(unlike 'shallow) is proving harder to get right. Fully restoring state between
incremental builds is requiring a few cross cutting changes to build functions
in terms of state passing.
At time of writing (2023-04-11) a fresh build of the site takes 1.4 seconds. Almost all of that time is spent templating documents (using Jinja templating). With incremental builds I estimate this could be brought down to ~400 milliseconds.
2023-04-20: shallow builds as implemented currently yield ~250ms builds, faster than I expected and ~150ms of that is in loading imports.
The motivation is two-fold and not directly prompted by speed. Re-writing hundreds of documents every time a single one of them changes is just wasteful. Further, I want to be happy leaving a filesystem watcher running that rebuilds the site locally every time a file is modified. Without incremental builds this could mean hundreds of files being wastefully recreated constantly.
Many moons ago I implemented a simple hash-based incremental check into the build system. It was good for a while, but as deeper relationships between documents emerged it proved too primitive and I had to disable it.
A new incremental build system would need to account for documents touched by changes in other documents, even if there had been no changes to that document directly.
Incremental builds should (probably) work on the basis of last access time.
