https://silasjelley.com/feeds/notes/ /icon/48.ico Silas Jelley reply@silasjelley.com 2025-11-06T14:02:46Z tag:silasjelley.com,2020-08-20:1c1c8fae-67a6-4bf0-88c8-a6fd2988d395 2025-11-06T14:31:46Z 2025-11-06T14:31:46Z <p>I wanted a more flexible way to show lists of related posts. The old way involved adding a collection block to a page’s front matter, like this:</p> <div class="code-block" data-lang="toml"><div class="code-header"><span class="code-lang">toml</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="toml">[<span style="color: #4b69c6">collection</span>]<br/><span style="color: #4b69c6">include</span> = [<span style="color: #198810">"</span><span style="color: #198810">essays</span><span style="color: #198810">"</span>]</code></pre></div> <p>This worked, but it had big limitations: you could only have one collection, and it always appeared at the bottom of the page. The new <code>\#collection()</code> shortcode lets me drop collections (as many as I want) anywhere in any document, directly from the body of the document. I also added much improved sorting and filtering options.</p> <p>So, TLDR, to show the 5 most recent quotes I’ve saved that are tagged favourite:</p> <div class="code-block"><div class="code-header"><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code>\#collection(include_: "quotes", limit: 5, filter_: "tags=favourite")</code></pre></div> <p>Which would render as:</p> <section data-pagefind-ignore class="collection-section"> <article class="collection title"> <header> <h2><a href="/2026/01/30/154005">Anne Frank, The Diary of a Young Girl, Penguin Books (2008), p. 249-250, 1947</a></h2><time datetime="2026-01-30 15:40:05"> <a href="/2026/01/30/154005"> Jan 30, 2026</a> </time></header> </article> <article class="collection title"> <header> <h2><a href="/2026/01/28/214251">Anne Morriss, Starbucks &#34;The Way I See It&#34; cup #76, 2009</a></h2><time datetime="2026-01-28 21:42:51"> <a href="/2026/01/28/214251"> Jan 28, 2026</a> </time></header> </article> <article class="collection title"> <header> <h2><a href="/2026/01/04/210609">William H. Whyte, Is Anybody Listening?, 2014</a></h2><time datetime="2026-01-04 21:06:09"> <a href="/2026/01/04/210609"> Jan 4, 2026</a> </time></header> </article> <article class="collection title"> <header> <h2><a href="/2025/11/13/112802">Geoff Manaugh, Seer, BLDGBLOG, 2025</a></h2><time datetime="2025-11-13 11:28:02"> <a href="/2025/11/13/112802"> Nov 13, 2025</a> </time></header> </article> <article class="collection title"> <header> <h2><a href="/2025/11/04/193308">Craig Mod, Day 6: Kiso-Fukushima to Narai, Between Two Mountains, 2025</a></h2><time datetime="2025-11-04 19:33:08"> <a href="/2025/11/04/193308"> Nov 4, 2025</a> </time></header> </article> </section> <p>My first effort had a circular dependency problem: how can you generate a page that needs a list of other pages, when those other pages might not have been generated yet? The solution is necessarily two-pass, and so while sharing its syntax with the other shortcodes, it does not run until later in the build.</p> <ol> <li value="1"> <p>First Pass: When the site generator sees a <code>\#collection()</code> shortcode, it doesn’t build the list right away. Instead, it leaves a unique placeholder and makes a note of the document and the parameters that were used.</p> </li> <li value="2"> <p>Second Pass: After all the pages have had their main content generated, a final step loops over the documents it made a note of.</p> </li> </ol> <p>It now has all the data it needs to build the collection, so it renders the list into HTML and swaps it with the placeholder.</p> <p>Despite being two pass, it’s pretty efficient as it only re-visits the pages that actually need it.</p> <p>Collections exclude the current page unless the <code>include_self=True</code> boolean is passed, such as for a series table of contents.</p> <p>Collections can contain documents that themselves contain collections, even wehn rendering the whole child document (e.g. <code>style="body"</code>). This is achieved through iterative.</p> <hr/> <p>Filtering:</p> <ul> <li> <p><code>include="[tags]"</code>: Includes documents matching a comma-separated list of categories or tags.</p> <ul> <li><code>\#collection(include_: "essays, wandering")</code></li> </ul> </li> <li> <p><code>exclude="[tags]"</code>: Excludes documents from the included set.</p> </li> </ul> <p>k</p> <ul> <li> <p><code>\#collection(include_: "essays", exclude_: "personal")</code></p> </li> <li> <p><code>filter="[filters]"</code>: Filters documents by metadata fields. Supports . for nested fields (e.g., available.year) and checks for list membership (e.g., tags=…).</p> <ul> <li><code>\#collection(filter_: "favourite=True")</code></li> <li><code>\#collection(filter_: "creator=Silas Jelley, available.year=2024")</code></li> <li><code>\#collection(include_: "notes", filter_: "tags=python")</code></li> </ul> </li> </ul> <p>Sorting &amp; Slicing:</p> <ul> <li> <p><code>order="[order]"</code>: Sets the sort order.</p> <ul> <li>reverse-chronological (default), chronological, alphabetical.</li> <li><code>\#collection(include_: "pearls", order: "alphabetical")</code></li> </ul> </li> <li> <p><code>limit="[int]"</code>: Limits the number of documents displayed.</p> <ul> <li> <p><code>\#collection(include_: "main", limit: 5)</code></p> </li> <li> <p><code>offset="[int]"</code>: Skips the first N documents.</p> <ul> <li><code>\#collection(include_: "main", limit: "5,", offset: 5)</code></li> </ul> </li> <li> <p>Paginate collections: <code>\#collection(include_: "photos", limit: "10,", offset: 20)</code> shows the third page of photos if you’re showing 10 per page.</p> </li> </ul> </li> </ul> <p>Display Styles:</p> <ul> <li> <p><code>style="[style]"</code>: Controls the appearance.</p> <ul> <li>title (default): A simple list of linked titles and dates.</li> <li>summary: Titles, dates, and a short summary of the content.</li> <li>body: The full rendered HTML body of each document.</li> <li><code>\#collection(include_: "changelogs", style: "body")</code></li> </ul> </li> <li> <p><code>numbered=True</code>: Renders a title style collection as a numbered list (&lt;ol&gt;).</p> <ul> <li><code>\#collection(include_: "series-a", order: "chronological", numbered: true)</code></li> </ul> </li> <li> <p><code>show_date=False</code>: Hides the date in title and summary styles.</p> <ul> <li><code>\#collection(include_: "essays", show-date: false)</code></li> </ul> </li> </ul> <p>Behavior:</p> <ul> <li> <p><code>include_self=True</code>: Includes the current document in the list if it matches the filters. The default (False) is to exclude it to prevent circular references.</p> <ul> <li><code>\#collection(include_: "series-a", include-self: true)</code></li> </ul> </li> </ul> <hr/> <p>This change completely removes the old front matter based collection system.</p> 2:31pm on November 6, 2025 from Atyrau, Atyrau oblysy, Kazakhstan<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Add @collection() shortcode”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:b6863e79-27a9-4bec-b49e-7412959b99f3 2025-10-23T20:54:46Z 2025-10-23T20:54:46Z <p>One of the outputs of this site is <a href="/links.txt">a plaintext file containing all the external links</a> I reference across the site. I noticed recently that a couple of the Wikipedia links in there were incorrect. Wikipedia is one of the few websites that uses parentheses “()” in URLs, and my naive link extracting regex would finish looking at the first closing parenthesis because it took that to mean it was inside <a href="https://htmlpreview.github.io/?https://github.com/jgm/djot/blob/master/doc/syntax.html#link">a Djot format link</a> and shouldn’t include the bracket.</p> <p>The result, a URL like <code>https://en.wikipedia.org/wiki/Cardinality_(data_modeling)</code> would lose it’s last parenthesis and be rendered invalid.</p> <p>I first concocted a bit of clever parsing to handle this, and then I decided that the most robust thing to do would be throw away that cleverness and fail loudly if any such URL was found. Now, if I save a URL with unencoded parentheses into one of my markup files, it’ll raise a <code>ValueError</code> and print a helpful error like so:</p> <div class="code-block"><div class="code-header"><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code> ERROR: Document contains URL with improperly encoded characters:<br/> Title: We shape our software; thereafter our software shapes us<br/> File: /home/silas/notes/essays/software-shapes-us.md<br/><br/> Replace with:<br/> #link("https://en.wikipedia.org/wiki/Ontology_%28information_science%29")[ontlogies]</code></pre></div> <p>(I’ve hidden the link that includes the link in the error output, else that would raise an error here :D)</p> <p>This involved changes to <code>extract_external_links()</code> and <code>process_documents()</code>.</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">extract_external_links</span>(<br/> text: <span style="color: #4b69c6">str</span>, site, status<br/>) -&gt; Tuple[List, Optional[Tuple[<span style="color: #4b69c6">str</span>, <span style="color: #4b69c6">str</span>]]]:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Extract external links from text.</span><br/><br/><span style="color: #74747c"> Returns:</span><br/><span style="color: #74747c"> Tuple of (sorted external links list, optional (link_text, url) tuple if problematic link found)</span><br/><span style="color: #74747c"> </span><span style="color: #74747c">"""</span><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Pattern that captures markdown links with URLs containing unencoded parens</span><br/> markdown_link_pattern <span style="color: #d73948">=</span> <span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #1d6c76">\[</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #1d6c76">\]</span><span style="color: #b60157">]</span><span style="color: #d73948">+</span><span style="color: #198810">)</span><span style="color: #1d6c76">\]</span><span style="color: #1d6c76">\(</span><span style="color: #198810">(</span><span style="color: #198810">https</span><span style="color: #d73948">?</span><span style="color: #198810">://</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">)</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #1d6c76">\(</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">)</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #1d6c76">\)</span><span style="color: #198810">)</span><span style="color: #1d6c76">\)</span><span style="color: #198810">"</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Check for markdown links with unencoded parentheses in URL</span><br/> problematic_links <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">findall</span>(markdown_link_pattern, text)<br/> <span style="color: #d73948">if</span> problematic_links:<br/> <span style="color: #d73948">return</span> [], problematic_links[<span style="color: #b60157">0</span>] <span style="color: #74747c">#</span><span style="color: #74747c"> Return first problematic link</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Pattern that handles URLs (now assumes parens are encoded)</span><br/> url_pattern <span style="color: #d73948">=</span> (<br/> <span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">https</span><span style="color: #d73948">?</span><span style="color: #198810">://</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #b60157">[</span><span style="color: #b60157">-</span><span style="color: #b60157">\w</span><span style="color: #b60157">.</span><span style="color: #b60157">]</span><span style="color: #d73948">|</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">%</span><span style="color: #b60157">[</span><span style="color: #b60157">\d</span><span style="color: #b60157">a-f</span><span style="color: #b60157">A-F</span><span style="color: #b60157">]</span><span style="color: #d73948">{2}</span><span style="color: #198810">)</span><span style="color: #198810">)</span><span style="color: #d73948">+</span><span style="color: #198810">"</span> <span style="color: #74747c">#</span><span style="color: #74747c"> domain</span><br/> <span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">/</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">\s</span><span style="color: #1d6c76">\"</span><span style="color: #b60157">'&lt;&gt;)</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">"</span> <span style="color: #74747c">#</span><span style="color: #74747c"> path (no unencoded parens allowed)</span><br/> )<br/> matches <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">findall</span>(url_pattern, text)<br/><br/> external_links <span style="color: #d73948">=</span> <span style="color: #4b69c6">set</span>()<br/> <span style="color: #d73948">for</span> url <span style="color: #d73948">in</span> matches:<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Clean up any trailing punctuation that might have been caught</span><br/> url <span style="color: #d73948">=</span> url.<span style="color: #4b69c6">rstrip</span>(<span style="color: #198810">"</span><span style="color: #198810">.,;:!?</span><span style="color: #198810">"</span>)<br/><br/> parsed_url <span style="color: #d73948">=</span> <span style="color: #4b69c6">urlparse</span>(url)<br/> <span style="color: #d73948">if</span> parsed_url.netloc.<span style="color: #4b69c6">lower</span>() <span style="color: #d73948">!=</span> <span style="color: #198810">"</span><span style="color: #198810">silasjelley.com</span><span style="color: #198810">"</span>:<br/> external_links.<span style="color: #4b69c6">add</span>(url)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Only add to site.links if not a draft</span><br/> <span style="color: #d73948">if</span> status <span style="color: #d73948">!=</span> <span style="color: #198810">"</span><span style="color: #198810">draft</span><span style="color: #198810">"</span>:<br/> site.links[<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>].<span style="color: #4b69c6">add</span>(url)<br/><br/> <span style="color: #d73948">return</span> <span style="color: #4b69c6">sorted</span>(external_links), <span style="color: #d73948">None</span><br/><br/><br/><span style="color: #d73948">async</span> <span style="color: #d73948">def</span> <span style="color: #4b69c6">process_document</span>(<br/> filepath: Path, site: SiteMetadata<br/>) -&gt; Tuple[<span style="color: #4b69c6">str</span>, DocumentMetadata]:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Process a document file and return its UID and metadata.</span><span style="color: #74747c">"""</span><br/><br/> <span style="color: #d73948">with</span> <span style="color: #4b69c6">open</span>(filepath, <span style="color: #198810">"</span><span style="color: #198810">rb</span><span style="color: #198810">"</span>) <span style="color: #d73948">as</span> f:<br/> <span style="color: #d73948">try</span>:<br/> parsed_toml <span style="color: #d73948">=</span> tomllib.<span style="color: #4b69c6">load</span>(f)<br/> <span style="color: #d73948">except</span>:<br/> logger.<span style="color: #4b69c6">error</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Error while processing document: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">filepath</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">import</span> sys<br/><br/> sys.<span style="color: #4b69c6">exit</span>(<span style="color: #b60157">1</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> The UID is now the top-level table name</span><br/> uid <span style="color: #d73948">=</span> parsed_toml[<span style="color: #198810">"</span><span style="color: #198810">uid</span><span style="color: #198810">"</span>]<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Process metadata into DocumentMetadata instance</span><br/> document <span style="color: #d73948">=</span> <span style="color: #4b69c6">preprocess_metadata</span>(filepath, parsed_toml)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Extract external links from the plain text content</span><br/> <span style="color: #d73948">try</span>:<br/> plain_text <span style="color: #d73948">=</span> (<br/> document.content.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">+</span> <span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #198810">"</span><br/> <span style="color: #d73948">+</span> document.source.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">url</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">+</span> <span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #198810">"</span><br/> <span style="color: #d73948">+</span> document.via.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">url</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>)<br/> )<br/><br/> status <span style="color: #d73948">=</span> document.status <span style="color: #d73948">if</span> document.status <span style="color: #d73948">else</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> external_links, problematic <span style="color: #d73948">=</span> <span style="color: #4b69c6">extract_external_links</span>(plain_text, site, status)<br/><br/> <span style="color: #d73948">if</span> problematic:<br/> link_text, url <span style="color: #d73948">=</span> problematic<br/> encoded_url <span style="color: #d73948">=</span> url.<span style="color: #4b69c6">replace</span>(<span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">%28</span><span style="color: #198810">"</span>).<span style="color: #4b69c6">replace</span>(<span style="color: #198810">"</span><span style="color: #198810">)</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">%29</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">raise</span> <span style="color: #4b69c6">ValueError</span>(<br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #1d6c76">\n</span><span style="color: #1d6c76">\n</span><span style="color: #198810"> URL contains unencoded parentheses in document:</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> Title: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">document</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">title</span><span style="color: #8b41b1">}</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> File: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">filepath</span><span style="color: #8b41b1">}</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> Link: #link(</span><span style="color: #198810">"</span>{url}<span style="color: #198810">"</span><span style="color: #198810">)[</span><span style="color: #b60157">{</span><span style="color: #b60157">link_text</span><span style="color: #b60157">}</span><span style="color: #198810">]</span><span style="color: #1d6c76">\n</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> Replace with:</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> #link(</span><span style="color: #198810">"</span>{encoded_url}<span style="color: #198810">"</span><span style="color: #198810">)[</span><span style="color: #b60157">{</span><span style="color: #b60157">link_text</span><span style="color: #b60157">}</span><span style="color: #198810">]</span><span style="color: #1d6c76">\n</span><span style="color: #1d6c76">\n</span><span style="color: #198810">"</span><br/> )<br/><br/> document.links[<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>] <span style="color: #d73948">=</span> external_links<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">KeyError</span>:<br/> logger.<span style="color: #4b69c6">warn</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">KeyError while compiling external links from </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">document</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">filepath</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">pass</span><br/><br/> <span style="color: #d73948">return</span> uid, document<br/></code></pre></div> 8:54pm on October 23, 2025 from Astrakhan, Astrakhan Oblast, Russia<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Fail build on unencoded parentheses in URLs”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:b21b87ee-3bbc-48a6-8d26-e614c8ceca74 2025-10-23T19:23:56Z 2025-10-23T19:23:56Z <p>I figured it made good sense to see if there were any old skeletons lingering in stylesheet before I switch to inlining the core of my styles into the page head. Easily done with PurgeCSS so long as you can stomach another Node package in your filesystem.</p> <p><code>npx</code> makes it trivial.</p> <div class="code-block" data-lang="sh"><div class="code-header"><span class="code-lang">sh</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="sh"><span style="color: #4b69c6">npx</span> purgecss --css /tmp/silasjelley.com/styles.css --content /tmp/silasjelley.com/<span style="color: #d73948">*</span><span style="color: #d73948">*</span>.html --output /tmp/purge.css</code></pre></div> <p>Diffing the two stylesheets (I use <a href="https://meldmerge.org/">Meld</a>) showed up a handful of old CSS id’s and classes that I was no longer using, shaving a little off that all importing render blocking cascade.</p> 7:23pm on October 23, 2025 from Astrakhan, Astrakhan Oblast, Russia<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Trim redundant CSS with PurgeCSS”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:27362b37-54ed-40b3-981b-a3ea059bbc0b 2025-10-23T19:09:13Z 2025-10-23T19:09:13Z <p>Added the right incantation to get fonts to pre-load, reducing request chaining by one hop.</p> <p>Note, <code>crossorigin</code> is required even though it’s from the same domain, else the browser will not use them and redownload them when the stylesheet arrives.</p> <div class="code-block" data-lang="html"><div class="code-header"><span class="code-lang">html</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="html"> \&lt;<span style="color: #4b69c6">link</span> <span style="color: #8b41b1">rel</span>=<span style="color: #198810">"</span><span style="color: #198810">preload</span><span style="color: #198810">"</span> <span style="color: #8b41b1">href</span>=<span style="color: #198810">"</span><span style="color: #198810">/library/misc/fonts/Inter/Regular.woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">as</span>=<span style="color: #198810">"</span><span style="color: #198810">font</span><span style="color: #198810">"</span> <span style="color: #8b41b1">type</span>=<span style="color: #198810">"</span><span style="color: #198810">font/woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">crossorigin</span>&gt;<br/> \&lt;<span style="color: #4b69c6">link</span> <span style="color: #8b41b1">rel</span>=<span style="color: #198810">"</span><span style="color: #198810">preload</span><span style="color: #198810">"</span> <span style="color: #8b41b1">href</span>=<span style="color: #198810">"</span><span style="color: #198810">/library/misc/fonts/Inter/Bold.woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">as</span>=<span style="color: #198810">"</span><span style="color: #198810">font</span><span style="color: #198810">"</span> <span style="color: #8b41b1">type</span>=<span style="color: #198810">"</span><span style="color: #198810">font/woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">crossorigin</span>&gt;<br/> \&lt;<span style="color: #4b69c6">link</span> <span style="color: #8b41b1">rel</span>=<span style="color: #198810">"</span><span style="color: #198810">preload</span><span style="color: #198810">"</span> <span style="color: #8b41b1">href</span>=<span style="color: #198810">"</span><span style="color: #198810">/library/misc/fonts/Source_Serif/Regular.woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">as</span>=<span style="color: #198810">"</span><span style="color: #198810">font</span><span style="color: #198810">"</span> <span style="color: #8b41b1">type</span>=<span style="color: #198810">"</span><span style="color: #198810">font/woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">crossorigin</span>&gt;<br/> \&lt;<span style="color: #4b69c6">link</span> <span style="color: #8b41b1">rel</span>=<span style="color: #198810">"</span><span style="color: #198810">preload</span><span style="color: #198810">"</span> <span style="color: #8b41b1">href</span>=<span style="color: #198810">"</span><span style="color: #198810">/library/misc/fonts/Source_Serif/Italic.woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">as</span>=<span style="color: #198810">"</span><span style="color: #198810">font</span><span style="color: #198810">"</span> <span style="color: #8b41b1">type</span>=<span style="color: #198810">"</span><span style="color: #198810">font/woff2</span><span style="color: #198810">"</span> <span style="color: #8b41b1">crossorigin</span>&gt;</code></pre></div> 7:09pm on October 23, 2025 from Astrakhan, Astrakhan Oblast, Russia<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Preloading fonts to reduce request chaining”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:da507ef4-6d86-43fd-95f8-61028987ab05 2025-10-23T18:39:53Z 2025-10-23T18:39:53Z <p>Turns out AVIF images, which I serve to all modern browsers visiting my site, weren’t included in the cache TTL that I use for all my static assets, oops.</p> <p>Here’s the updated cache control header for my caddy webserver:</p> <div class="code-block" data-lang="caddyfile"><div class="code-header"><span class="code-lang">caddyfile</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="caddyfile"># Cache Control<br/># The 'file' directive ensures that cache-control is only set for files that actually exist<br/><br/>@static {<br/> file<br/> path *.avif *.ico *.css *.js *.gif *.jpg *.jpeg *.png *.svg *.woff *.woff2 *.gpx<br/>}<br/>header @static Cache-Control max-age=5184000<br/></code></pre></div> 6:39pm on October 23, 2025 from Astrakhan, Astrakhan Oblast, Russia<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Cache all image assets in Caddy with a Cache-Control directive”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:444e7809-f476-449e-919c-778f7bb0baee 2025-10-21T23:01:31Z 2025-10-21T23:01:31Z <p>html5validator is turning up a whole littany of validation errors. There all pretty minor, but I like for <a href="https://en.wikipedia.org/wiki/Robustness_principle">everything I emit to be well formed</a>.</p> <div class="code-block" data-lang="sh"><div class="code-header"><span class="code-lang">sh</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="sh"><span style="color: #4b69c6">uvx</span> html5validator --root /tmp/silasjelley.com</code></pre></div> <p>Non exhaustively:</p> <ul> <li> <p>Figcaption apparently invalid in blockquotes, have replaced with the &lt;cite&gt; element across src/{attribution,shortcodes} (quotes), templates/{collection,default}</p> </li> <li> <p>Tweaked tests/html_validation.py to ignore the unnamed &lt;meta charset&gt; declaration as this is correct HTML5 as far as I know.</p> </li> <li> <p>Added alt text to tracking pixel to suppress that warning</p> </li> <li> <p>Now stripping the &lt;p&gt; and &lt;/p&gt; opening and closing tags from the Jotdown/Djot formatted attribution string, which I’m glad to have done anyway.</p> </li> <li> <p>Found and removed an erroneous/left-over trailing &lt;/ul&gt; tag from template/default</p> </li> <li> <p>Removed invalid &lt;hr&gt; from &lt;summary&gt; element in template/base</p> </li> <li> <p>Fixed alert template using paragraph inside a &lt;span&gt;</p> </li> <li> <p>Fixed formatting of datetime attribute on some &lt;time&gt; elements in templates/collection</p> </li> <li> <p>Fixed @aside(…) shortcodes not adding datetime to the &lt;time&gt; element</p> </li> <li> <p>Fixed an issue where my external links finder regex was being over greedy</p> <p>The regex, <code>r"(https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+(?:/[^)\s]*)?)"</code> would match some html elements, eg: ‘<a href="https://player.vimeo.com/api/player.js">https://player.vimeo.com/api/player.js</a>“&gt;&lt;/script&gt;‘ and a few other edge cases.</p> <p>Have replaced with a more robust regex: <code>r"https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+(?:/[^\s\"'&lt;&gt;)]*)?(?=[\s\"'&lt;&gt;)]|\$)"</code></p> <p>During this checking of external links via the /links.txt file in the output_dir I noticed a minor privacy leak in that external links in draft documents were leaking to the /links.txt file. Not major, but now patched.</p> </li> <li> <p>Fixed a sneaky HTML escaping issue in templates/collection (<code>{{ item.content.html|striptags|truncate(210) }}</code>) where the use of striptags can, inadvertently, actually create tags, ie if item.content includes for example “&lt;code&gt;&amp;lt;head&amp;gt;&lt;/code&gt;”, then once the &lt;code&gt; tags have been stripped, a &lt;head&gt; reaches the final output.</p> </li> <li> <p>Fixed the technically invalid use of an unamed &lt;meta&gt; property in the &lt;head&gt; for pagefind’s default image (the favicon) by simply shifting it to the favicon link itself:</p> <p>Previously: &lt;meta data-pagefind-default-meta=“image[content]” content=“/favicon.png” /&gt; Now: &lt;link rel=“icon” type=“image/png” href=“/favicon.png” data-pagefind-default-meta=“image[href]” /&gt;</p> </li> <li> <p>Numerous fixes in my source texts in  /notes, such as:</p> <ul> <li>Having tags closing in the wrong order: &lt;pre&gt;&lt;code&gt;…&lt;/pre&gt;&lt;/code&gt;, things that browsers will tolerate fine, but not correct</li> <li>One case of an audio tag not being closed</li> <li>Replaced an obselete “frameborder” attribute on an iframe with border:none style</li> <li>Cleaned up URLs where trailing ‘?ref=…’ parameters had been left on</li> <li>Moved invalid &lt;style&gt; blocks inside body into inline styles</li> <li>Replace Djot inline raw &lt;details&gt; elements with block raw to prevent them being wrapping in &lt;p&gt; tags</li> </ul> </li> </ul> 11:01pm on October 21, 2025 from Astrakhan, Astrakhan Oblast, Russia<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Validating HTML with html5validator”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:58d17e36-c7fb-47fd-8ecf-ddf4217b8367 2025-09-25T00:09:56Z 2025-09-25T00:09:56Z <p>Fixed a bug where code blocks were inadvertently being double subtracted from <code>site.words.self</code> (the wordcount of my own written prose), where they should only have been subtracted once.</p> 12:09am on September 25, 2025 from Appledore, South West England, United Kingdom<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“September 25, 2025 12.09AM”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:6f697035-19d7-46df-81c4-5466cb0cb803 2025-09-24T03:40:06Z 2025-09-24T03:40:06Z <p>Took a simple approach to speeding up rebuilds of the site that I’ve been meaning to crack on with for a while. Ripped out the old output_html function and replaced it with one that natively supports incremental builds.</p> <p>May tweak the state that gets hashed for each document if the cache doesn’t stay as warm as it can, but initial results show lots of cache hits, with the <code>Generate Hypertext</code> build step on a typical rebuild having been cut from around 10 seconds, to 300 milliseconds!</p> <p>That’s more than half the total build time gone in most cases :)</p> <p>Here’s the bulk of the code:</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">import</span> json<br/><span style="color: #d73948">import</span> logging<br/><span style="color: #d73948">from</span> dataclasses <span style="color: #d73948">import</span> asdict<br/><span style="color: #d73948">from</span> hashlib <span style="color: #d73948">import</span> md5<br/><span style="color: #d73948">from</span> pathlib <span style="color: #d73948">import</span> Path<br/><span style="color: #d73948">from</span> typing <span style="color: #d73948">import</span> Dict, Any, TYPE_CHECKING<br/><br/><span style="color: #74747c">#</span><span style="color: #74747c"> Import types only for type hints, avoiding circular imports</span><br/><span style="color: #d73948">if</span> TYPE_CHECKING:<br/> <span style="color: #d73948">from</span> jinja2 <span style="color: #d73948">import</span> Environment<br/> <span style="color: #d73948">from</span> build <span style="color: #d73948">import</span> DocumentMetadata, AssetMetadata, SiteMetadata<br/><br/>logger <span style="color: #d73948">=</span> logging.<span style="color: #4b69c6">getLogger</span>(<span style="color: #4b69c6">__name__</span>)<br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">calculate_content_hash</span>(<br/> page: <span style="color: #198810">"</span><span style="color: #198810">DocumentMetadata</span><span style="color: #198810">"</span>, collections: Dict, site: <span style="color: #198810">"</span><span style="color: #198810">SiteMetadata</span><span style="color: #198810">"</span><br/>) -&gt; <span style="color: #4b69c6">str</span>:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Calculate a hash representing all content that could affect this page's output.</span><span style="color: #74747c">"""</span><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Convert page data to string, excluding volatile fields like backlinks</span><br/> page_dict <span style="color: #d73948">=</span> <span style="color: #4b69c6">asdict</span>(page)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Remove volatile fields that change frequently but don't affect content</span><br/> volatile_fields <span style="color: #d73948">=</span> [<span style="color: #198810">"</span><span style="color: #198810">links</span><span style="color: #198810">"</span>] <span style="color: #74747c">#</span><span style="color: #74747c"> Remove if backlinks change shouldn't trigger rebuild</span><br/> <span style="color: #d73948">for</span> field <span style="color: #d73948">in</span> volatile_fields:<br/> page_dict.<span style="color: #4b69c6">pop</span>(field, <span style="color: #d73948">None</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Include relevant collection data (for pages that use collections)</span><br/> collection_data <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">if</span> page.collection.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">include</span><span style="color: #198810">"</span>):<br/> <span style="color: #d73948">for</span> include <span style="color: #d73948">in</span> page.collection[<span style="color: #198810">"</span><span style="color: #198810">include</span><span style="color: #198810">"</span>]:<br/> <span style="color: #d73948">if</span> include <span style="color: #d73948">in</span> collections:<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Hash just the UIDs and titles of collection items to detect changes</span><br/> collection_items <span style="color: #d73948">=</span> [<br/> (item.uid, item.title, item.updated.<span style="color: #4b69c6">isoformat</span>())<br/> <span style="color: #d73948">for</span> item <span style="color: #d73948">in</span> collections[include][:<span style="color: #b60157">20</span>]<br/> ] <span style="color: #74747c">#</span><span style="color: #74747c"> Limit to avoid huge hashes</span><br/> collection_data <span style="color: #d73948">+=</span> <span style="color: #4b69c6">str</span>(collection_items)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Include site metadata that might affect rendering</span><br/> site_data <span style="color: #d73948">=</span> {<br/> <span style="color: #198810">"</span><span style="color: #198810">stylesheet_hash</span><span style="color: #198810">"</span>: site.stylesheet_hash,<br/> <span style="color: #198810">"</span><span style="color: #198810">name</span><span style="color: #198810">"</span>: site.name,<br/> <span style="color: #198810">"</span><span style="color: #198810">url</span><span style="color: #198810">"</span>: site.url,<br/> <span style="color: #198810">"</span><span style="color: #198810">baseurl</span><span style="color: #198810">"</span>: site.baseurl,<br/> }<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Combine all data and hash</span><br/> combined_data <span style="color: #d73948">=</span> json.<span style="color: #4b69c6">dumps</span>(<br/> [page_dict, collection_data, site_data], sort_keys<span style="color: #d73948">=</span><span style="color: #d73948">True</span>, default<span style="color: #d73948">=</span><span style="color: #4b69c6">str</span><br/> )<br/> <span style="color: #d73948">return</span> <span style="color: #4b69c6">md5</span>(combined_data.<span style="color: #4b69c6">encode</span>()).<span style="color: #4b69c6">hexdigest</span>()<br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">load_build_cache</span>(output_dir: Path) -&gt; Dict[<span style="color: #4b69c6">str</span>, Dict[<span style="color: #4b69c6">str</span>, Any]]:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Load the build cache from previous run.</span><span style="color: #74747c">"""</span><br/> cache_file <span style="color: #d73948">=</span> output_dir <span style="color: #d73948">/</span> <span style="color: #198810">"</span><span style="color: #198810">.build_cache.json</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">if</span> cache_file.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">try</span>:<br/> <span style="color: #d73948">with</span> <span style="color: #4b69c6">open</span>(cache_file, <span style="color: #198810">"</span><span style="color: #198810">r</span><span style="color: #198810">"</span>) <span style="color: #d73948">as</span> f:<br/> <span style="color: #d73948">return</span> json.<span style="color: #4b69c6">load</span>(f)<br/> <span style="color: #d73948">except</span> (json.JSONDecodeError, <span style="color: #4b69c6">IOError</span>):<br/> logger.<span style="color: #4b69c6">warning</span>(<span style="color: #198810">"</span><span style="color: #198810">Could not load build cache, rebuilding all files</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">return</span> {}<br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">save_build_cache</span>(cache: Dict[<span style="color: #4b69c6">str</span>, Dict[<span style="color: #4b69c6">str</span>, Any]], output_dir: Path):<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Save the build cache for next run.</span><span style="color: #74747c">"""</span><br/> cache_file <span style="color: #d73948">=</span> output_dir <span style="color: #d73948">/</span> <span style="color: #198810">"</span><span style="color: #198810">.build_cache.json</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">try</span>:<br/> <span style="color: #d73948">with</span> <span style="color: #4b69c6">open</span>(cache_file, <span style="color: #198810">"</span><span style="color: #198810">w</span><span style="color: #198810">"</span>) <span style="color: #d73948">as</span> f:<br/> json.<span style="color: #4b69c6">dump</span>(cache, f)<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">IOError</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">warning</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Could not save build cache: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">needs_rebuild</span>(<br/> page: <span style="color: #198810">"</span><span style="color: #198810">DocumentMetadata</span><span style="color: #198810">"</span>,<br/> collections: Dict,<br/> site: <span style="color: #198810">"</span><span style="color: #198810">SiteMetadata</span><span style="color: #198810">"</span>,<br/> output_path: Path,<br/> cache: Dict[<span style="color: #4b69c6">str</span>, Dict[<span style="color: #4b69c6">str</span>, Any]],<br/> template_file: <span style="color: #4b69c6">str</span>,<br/> env: <span style="color: #198810">"</span><span style="color: #198810">Environment</span><span style="color: #198810">"</span>,<br/>) -&gt; <span style="color: #4b69c6">bool</span>:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Determine if a page needs to be rebuilt.</span><span style="color: #74747c">"""</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Always rebuild if output doesn't exist</span><br/> <span style="color: #d73948">if</span> <span style="color: #d73948">not</span> output_path.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/><br/> page_key <span style="color: #d73948">=</span> page.uid<br/> current_hash <span style="color: #d73948">=</span> <span style="color: #4b69c6">calculate_content_hash</span>(page, collections, site)<br/><br/> <span style="color: #d73948">if</span> page_key <span style="color: #d73948">not</span> <span style="color: #d73948">in</span> cache:<br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/><br/> cached_data <span style="color: #d73948">=</span> cache[page_key]<br/><br/> <span style="color: #d73948">if</span> cached_data.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">content_hash</span><span style="color: #198810">"</span>) <span style="color: #d73948">!=</span> current_hash:<br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Check template modification time</span><br/> <span style="color: #d73948">try</span>:<br/> template_path <span style="color: #d73948">=</span> env.loader.<span style="color: #4b69c6">get_source</span>(env, template_file)[<span style="color: #b60157">1</span>]<br/> <span style="color: #d73948">if</span> template_path:<br/> template_mtime <span style="color: #d73948">=</span> <span style="color: #4b69c6">Path</span>(template_path).<span style="color: #4b69c6">stat</span>().st_mtime<br/> <span style="color: #d73948">if</span> cached_data.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">template_mtime</span><span style="color: #198810">"</span>, <span style="color: #b60157">0</span>) <span style="color: #d73948">&lt;</span> template_mtime:<br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">Exception</span>:<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> If we can't check template mtime, rebuild to be safe</span><br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Check source file modification time</span><br/> source_mtime <span style="color: #d73948">=</span> page.filepath.<span style="color: #4b69c6">stat</span>().st_mtime<br/> <span style="color: #d73948">if</span> cached_data.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">source_mtime</span><span style="color: #198810">"</span>, <span style="color: #b60157">0</span>) <span style="color: #d73948">&lt;</span> source_mtime:<br/> <span style="color: #d73948">return</span> <span style="color: #d73948">True</span><br/><br/> <span style="color: #d73948">return</span> <span style="color: #d73948">False</span><br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">output_html_incremental</span>(<br/> assets: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #198810">"</span><span style="color: #198810">AssetMetadata</span><span style="color: #198810">"</span>],<br/> documents: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #198810">"</span><span style="color: #198810">DocumentMetadata</span><span style="color: #198810">"</span>],<br/> collections: Dict[<span style="color: #4b69c6">str</span>, Any],<br/> site: <span style="color: #198810">"</span><span style="color: #198810">SiteMetadata</span><span style="color: #198810">"</span>,<br/> env: <span style="color: #198810">"</span><span style="color: #198810">Environment</span><span style="color: #198810">"</span>,<br/> output_dir: Path,<br/> build_page_collection_func, <span style="color: #74747c">#</span><span style="color: #74747c"> Function to build page collections</span><br/>) -&gt; <span style="color: #d73948">None</span>:<br/> logger.<span style="color: #4b69c6">info</span>(<span style="color: #198810">"</span><span style="color: #198810">Generating Hypertext (incremental)</span><span style="color: #198810">"</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Load build cache</span><br/> cache <span style="color: #d73948">=</span> <span style="color: #4b69c6">load_build_cache</span>(output_dir)<br/> new_cache <span style="color: #d73948">=</span> {}<br/><br/> built_count <span style="color: #d73948">=</span> <span style="color: #b60157">0</span><br/> skipped_count <span style="color: #d73948">=</span> <span style="color: #b60157">0</span><br/><br/> <span style="color: #d73948">for</span> key, page <span style="color: #d73948">in</span> documents.<span style="color: #4b69c6">items</span>():<br/> template_file <span style="color: #d73948">=</span> page.layout<br/> output_path <span style="color: #d73948">=</span> output_dir <span style="color: #d73948">/</span> page.slug <span style="color: #d73948">/</span> <span style="color: #198810">"</span><span style="color: #198810">index.html</span><span style="color: #198810">"</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Check if rebuild is needed</span><br/> <span style="color: #d73948">if</span> <span style="color: #d73948">not</span> <span style="color: #4b69c6">needs_rebuild</span>(<br/> page, collections, site, output_path, cache, template_file, env<br/> ):<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Copy cache entry to new cache</span><br/> new_cache[key] <span style="color: #d73948">=</span> cache[key]<br/> skipped_count <span style="color: #d73948">+=</span> <span style="color: #b60157">1</span><br/> logger.<span style="color: #4b69c6">debug</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> SKIP: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">page</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">filepath</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">continue</span><br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Rebuild the page</span><br/> template <span style="color: #d73948">=</span> env.<span style="color: #4b69c6">get_template</span>(template_file)<br/> collection <span style="color: #d73948">=</span> <span style="color: #4b69c6">build_page_collection_func</span>(page, collections)<br/><br/> output <span style="color: #d73948">=</span> template.<span style="color: #4b69c6">render</span>(<br/> documents<span style="color: #d73948">=</span>documents,<br/> assets<span style="color: #d73948">=</span>assets,<br/> collections<span style="color: #d73948">=</span>collections,<br/> collection<span style="color: #d73948">=</span>collection,<br/> page<span style="color: #d73948">=</span><span style="color: #4b69c6">asdict</span>(page),<br/> site<span style="color: #d73948">=</span>site,<br/> )<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Ensure output directory exists</span><br/> output_path.parent.<span style="color: #4b69c6">mkdir</span>(parents<span style="color: #d73948">=</span><span style="color: #d73948">True</span>, exist_ok<span style="color: #d73948">=</span><span style="color: #d73948">True</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Write the file</span><br/> <span style="color: #d73948">with</span> <span style="color: #4b69c6">open</span>(output_path, <span style="color: #198810">"</span><span style="color: #198810">w</span><span style="color: #198810">"</span>) <span style="color: #d73948">as</span> f:<br/> f.<span style="color: #4b69c6">write</span>(output)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Update cache</span><br/> current_hash <span style="color: #d73948">=</span> <span style="color: #4b69c6">calculate_content_hash</span>(page, collections, site)<br/> template_mtime <span style="color: #d73948">=</span> <span style="color: #b60157">0</span><br/> <span style="color: #d73948">try</span>:<br/> template_path <span style="color: #d73948">=</span> env.loader.<span style="color: #4b69c6">get_source</span>(env, template_file)[<span style="color: #b60157">1</span>]<br/> <span style="color: #d73948">if</span> template_path:<br/> template_mtime <span style="color: #d73948">=</span> <span style="color: #4b69c6">Path</span>(template_path).<span style="color: #4b69c6">stat</span>().st_mtime<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">Exception</span>:<br/> <span style="color: #d73948">pass</span><br/><br/> new_cache[key] <span style="color: #d73948">=</span> {<br/> <span style="color: #198810">"</span><span style="color: #198810">content_hash</span><span style="color: #198810">"</span>: current_hash,<br/> <span style="color: #198810">"</span><span style="color: #198810">template_mtime</span><span style="color: #198810">"</span>: template_mtime,<br/> <span style="color: #198810">"</span><span style="color: #198810">source_mtime</span><span style="color: #198810">"</span>: page.filepath.<span style="color: #4b69c6">stat</span>().st_mtime,<br/> <span style="color: #198810">"</span><span style="color: #198810">output_path</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">str</span>(output_path),<br/> }<br/><br/> built_count <span style="color: #d73948">+=</span> <span style="color: #b60157">1</span><br/> logger.<span style="color: #4b69c6">debug</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> BUILD: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">page</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">filepath</span><span style="color: #8b41b1">}</span><span style="color: #198810"> &gt;&gt; </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">output_path</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Save updated cache</span><br/> <span style="color: #4b69c6">save_build_cache</span>(new_cache, output_dir)<br/><br/> logger.<span style="color: #4b69c6">info</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Built: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">built_count</span><span style="color: #8b41b1">}</span><span style="color: #198810">, Skipped: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">skipped_count</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/><br/><br/><span style="color: #d73948">def</span> <span style="color: #4b69c6">clear_build_cache</span>(output_dir: Path) -&gt; <span style="color: #d73948">None</span>:<br/> <span style="color: #74747c">"""</span><span style="color: #74747c">Clear the build cache to force a full rebuild.</span><span style="color: #74747c">"""</span><br/> cache_file <span style="color: #d73948">=</span> output_dir <span style="color: #d73948">/</span> <span style="color: #198810">"</span><span style="color: #198810">.build_cache.json</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">if</span> cache_file.<span style="color: #4b69c6">exists</span>():<br/> cache_file.<span style="color: #4b69c6">unlink</span>()<br/> logger.<span style="color: #4b69c6">info</span>(<span style="color: #198810">"</span><span style="color: #198810">Build cache cleared</span><span style="color: #198810">"</span>)</code></pre></div> 3:40am on September 24, 2025 from Appledore, South West England, United Kingdom<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Added incremental generation of HTML output”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:ddcfd972-3c36-4bc6-848c-1d3841c3692c 2025-09-23T14:45:34Z 2025-09-23T14:45:34Z <aside class="aside"> <p>PENDING:</p> <ul> <li>[ ] Implement <code>@terminal/shell/command</code></li> <li>[x] Implement <code>@embed</code>and replace <code>aside::</code> and <code>import::</code> substitutions/imports with <code>@embed</code></li> <li>[ ] Make assets support backlinks so that <code>@link</code> can be used for assets too, then review all uses of <code>@val("id.slug")</code> and replace with <code>@link</code> where possible</li> <li>[x] Finish migrating all instances of old <code>link::</code> and <code>slug:</code> syntax (slug is now <code>@val("\$uid.slug")</code>)</li> <li>[x] Remove last of the legacy substitution code</li> <li>[ ] Ultimately I would love to get rid of all regex based substitution schemes and instead parse my documents into an AST and then walk the tree, altering nodes directly, but I’m a ways away from that yet.</li> </ul> </aside> <p>I want a better design for the shortcodes I use throughout the site.</p> <p>The current “design” is captured in the regexes used to extract them:</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"> REF_LINK_RE <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">compile</span>(<span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">!</span><span style="color: #d73948">?</span><span style="color: #1d6c76">\[</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #1d6c76">\]</span><span style="color: #b60157">]</span><span style="color: #d73948">*?</span><span style="color: #198810">)</span><span style="color: #1d6c76">\]</span><span style="color: #198810">(</span><span style="color: #1d6c76">\(</span><span style="color: #198810">(</span><span style="color: #b60157">.</span><span style="color: #d73948">*?</span><span style="color: #198810">::</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">)</span><span style="color: #b60157">]</span><span style="color: #d73948">+</span><span style="color: #198810">)</span><span style="color: #1d6c76">\)</span><span style="color: #198810">)</span><span style="color: #198810">"</span>)<br/> REF_SLUG_RE <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">compile</span>(<span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #d73948">?</span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">!`</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810">slug::</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #b60157">a-z</span><span style="color: #b60157">A-Z</span><span style="color: #b60157">0-9</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">#</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">)</span><span style="color: #b60157">\s</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">)</span><span style="color: #198810">"</span>)<br/> REF_TITLE_RE <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">compile</span>(<span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #1d6c76">\{</span><span style="color: #1d6c76">\{</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810">title::</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #b60157">a-z</span><span style="color: #b60157">A-Z</span><span style="color: #b60157">0-9</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">#</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">}</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #1d6c76">\}</span><span style="color: #1d6c76">\}</span><span style="color: #198810">)</span><span style="color: #198810">"</span>)<br/> REF_CITE_RE <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">compile</span>(<span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #1d6c76">\{</span><span style="color: #1d6c76">\{</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810">cite::</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #b60157">a-z</span><span style="color: #b60157">A-Z</span><span style="color: #b60157">0-9</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">#</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">}</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #1d6c76">\}</span><span style="color: #1d6c76">\}</span><span style="color: #198810">)</span><span style="color: #198810">"</span>)<br/> REF_IMPT_RE <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">compile</span>(<br/> <span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #1d6c76">\{</span><span style="color: #1d6c76">\{</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">aside</span><span style="color: #d73948">|</span><span style="color: #198810">import</span><span style="color: #198810">)</span><span style="color: #198810">::</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #b60157">[</span><span style="color: #b60157">a-z</span><span style="color: #b60157">A-Z</span><span style="color: #b60157">0-9</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">#</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">}</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">)</span><span style="color: #198810">(</span><span style="color: #198810"> </span><span style="color: #d73948">*</span><span style="color: #1d6c76">\}</span><span style="color: #1d6c76">\}</span><span style="color: #198810">)</span><span style="color: #198810">"</span><br/> )<br/><br/> text <span style="color: #d73948">=</span> page.content.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">if</span> text:<br/> text <span style="color: #d73948">=</span> <span style="color: #4b69c6">replace_import_references</span>(text, REF_IMPT_RE, merged_data, key, page)<br/> text <span style="color: #d73948">=</span> <span style="color: #4b69c6">replace_cite_references</span>(text, REF_CITE_RE, merged_data)<br/> text <span style="color: #d73948">=</span> <span style="color: #4b69c6">replace_title_references</span>(text, REF_TITLE_RE, merged_data)<br/> text <span style="color: #d73948">=</span> <span style="color: #4b69c6">replace_slug_references</span>(text, REF_SLUG_RE, merged_data)<br/> text <span style="color: #d73948">=</span> <span style="color: #4b69c6">process_reference_links</span>(text, REF_LINK_RE, merged_data, key)<br/> page.content[<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>] <span style="color: #d73948">=</span> text.<span style="color: #4b69c6">strip</span>()</code></pre></div> <p>For example, <code>::Author](quote::\$UID)</code> would be replaced with a quote embed of the provided uid. This design was meant to closely follow the semantics of markdown/Djot links. As my document system expands, and I move further away from existing markup syntax, I want a design that generalises better, provides more legibility and extensibility, and has support for parameters.</p> <p>TLDR: <code>@keyword(option="value")</code> or something similar.</p> <p>Core Idea: A single sigil, @, initiates a “function” with a clear name and parameters.</p> <p>Syntax &amp; Examples:</p> <ul> <li> <p>The Function Call: <code>@command(id, ...)</code> or <code>@command{...}</code></p> </li> <li> <p>Links: <code>@link("a1b2c3d4", fragment: "...")[]</code></p> <ul> <li>Could use positional params for brevity: <code>@link("a1b2c3d4")[]</code></li> </ul> </li> <li> <p>Images/Media:</p> <ul> <li><code>@img("a1b2c3d4", alt: "...")</code></li> <li><code>@video("a1b2c3d4")</code></li> </ul> </li> <li> <p>Inline Data: A dedicated command to retrieve values.</p> <ul> <li>The title is <code>@val("a1b2c3d4.title")</code>.</li> <li>Find it at <code>@val("a1b2c3d4.slug")</code>.</li> </ul> </li> <li> <p>Formatted Content &amp; Transclusion: Each has a clear, named command.</p> <div class="code-block"><div class="code-header"><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code>@cite("a1b2c3d4")<br/>@quote("a1b2c3d4")<br/>@embed("a1b2c3d4")<br/>@ gallery{<br/> @img("b2c3d4e5")<br/> @img("f6")<br/>} (Block-form for wrapping content)</code></pre></div> </li> </ul> <hr/> <ul> <li><strong>Key/value parameters vs positional arguments?</strong>.<br/></li> </ul> <p>Positional arguments can be more terse, but like short-opts (vs long-opts) they are less legible and they bake in assumptions. A key/value system is more verbose but removes ambiguity and makes changing defaults, and adding or deprecating arguments much safer.</p> <ul> <li>There should be a syntax for escaping/skipping a shortcode, when I want the actual shortcode to remain in a document.</li> </ul> 2:45pm on September 23, 2025 from Appledore, South West England, United Kingdom<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“SEP 64: Redesign shortcodes”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:3661f564-715a-4ced-a2ca-3c9c8556d17a 2025-01-15T12:14:47Z 2025-01-15T12:14:47Z <p>For a while I’ve been using justified text to give blockquotes on the site a distinct visual language, I like it, even though algorithmically justified text on the web is far from perfect, with certain viewport widths yielding ugly rivers in a text block.</p> <p>Last night I came across a post from Tyler Sticka — <a href="https://cloudfour.com/thinks/justified-text-better-than-expected/">Justified Text: Better Than Expected?</a> — that introduced me to a couple of CSS properties that improve rendering,</p> <div class="code-block" data-lang="css"><div class="code-header"><span class="code-lang">css</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="css"><span style="color: #4b69c6">blockquote</span> {<br/> <span style="color: #4b69c6">hyphens</span>: <span style="color: #4b69c6">auto</span>;<br/> hyphenate-limit-chars: <span style="color: #b60157">7</span>;<br/>}</code></pre></div> <p>Here’s what I use for blockquotes now,</p> <div class="code-block" data-lang="css"><div class="code-header"><span class="code-lang">css</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="css"><span style="color: #4b69c6">blockquote</span> {<br/> <span style="color: #4b69c6">padding</span>: <span style="color: #b60157">0</span> <span style="color: #b60157">1</span><span style="color: #d73948">em</span> <span style="color: #b60157">0</span> <span style="color: #b60157">2</span><span style="color: #d73948">em</span>;<br/> <span style="color: #4b69c6">margin</span>: <span style="color: #b60157">1</span><span style="color: #b60157">.</span><span style="color: #b60157">4</span><span style="color: #d73948">em</span> <span style="color: #b60157">0</span>;<br/> <span style="color: #4b69c6">position</span>: <span style="color: #4b69c6">relative</span>;<br/> <span style="color: #4b69c6">color</span>: <span style="color: #4b69c6">var</span>(<span style="color: #4b69c6">--</span><span style="color: #4b69c6">blockquote-foreground</span>);<br/>}<br/><br/><span style="color: #4b69c6">blockquote</span> <span style="color: #4b69c6">p</span> {<br/> <span style="color: #4b69c6">text-align</span>: <span style="color: #4b69c6">justify</span>;<br/> <span style="color: #4b69c6">hyphens</span>: <span style="color: #4b69c6">auto</span>;<br/> hyphenate-limit-chars: <span style="color: #b60157">7</span>;<br/> <span style="color: #4b69c6">margin</span>: <span style="color: #b60157">1</span><span style="color: #d73948">em</span> <span style="color: #b60157">0</span> <span style="color: #b60157">0</span> <span style="color: #b60157">0</span>;<br/>}<br/><span style="color: #4b69c6">blockquote</span> <span style="color: #4b69c6">p</span> + <span style="color: #4b69c6">p</span> {<br/> <span style="color: #4b69c6">text-indent</span>: <span style="color: #b60157">2</span><span style="color: #d73948">em</span>;<br/> <span style="color: #4b69c6">margin</span>: <span style="color: #b60157">0</span>;<br/>}</code></pre></div> 12:14pm on January 15, 2025 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Improved rendering of justified text”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:f544d737-8000-408d-b32d-86428a349e4d 2024-12-16T09:54:32Z 2024-12-16T09:54:30Z <p>In continuing to ratchet up the strictness of the build system I landed on the concept of dataclasses, which are new to this novice programmer, though the concept of a schema for validation is not.</p> <p>Anyway, I landed on creating one dataclass for Documents and another for Assets.</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #301414">@</span><span style="color: #301414">dataclass</span><br/><span style="color: #d73948">class</span> <span style="color: #4b69c6">DocumentMetadata</span>:<br/> filepath: Path<br/> uid: <span style="color: #4b69c6">str</span><br/> slug: <span style="color: #4b69c6">str</span><br/> title: <span style="color: #4b69c6">str</span><br/> primary: <span style="color: #4b69c6">str</span><br/> secondary: <span style="color: #4b69c6">str</span><br/> available: datetime.datetime<br/> created: datetime.datetime<br/> updated: datetime.datetime<br/> creator: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> note: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> favourite: <span style="color: #4b69c6">bool</span> <span style="color: #d73948">=</span> <span style="color: #d73948">False</span><br/> parent: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> description: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> layout: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> TEMPLATE_DEFAULT<br/> source: Dict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/> via: Dict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/> location: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">continent</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">country</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">region</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">city</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">note</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">lat</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">int</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">lng</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">int</span>,<br/> }<br/> )<br/> collection: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">style</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">title</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">order</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">chronological</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">include</span><span style="color: #198810">"</span>: [],<br/> }<br/> )<br/> attribution: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">djot</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">html</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> }<br/> )<br/> media: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">application/toml</span><span style="color: #198810">"</span><br/> words: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">lines</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">words</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> },<br/> <span style="color: #198810">"</span><span style="color: #198810">references</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> }<br/> )<br/> status: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> links: LinksDict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">internal</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> <span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> <span style="color: #198810">"</span><span style="color: #198810">backlinks</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> }<br/> )<br/> options: List[<span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">list</span>)<br/> tags: List[<span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">list</span>)<br/> content: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/><br/> <span style="color: #d73948">def</span> <span style="color: #4b69c6">__post_init__</span>(self):<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Validate links dictionary structure</span><br/> required_link_types <span style="color: #d73948">=</span> {<span style="color: #198810">"</span><span style="color: #198810">internal</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">backlinks</span><span style="color: #198810">"</span>}<br/> <span style="color: #d73948">if</span> (<br/> <span style="color: #d73948">not</span> <span style="color: #4b69c6">isinstance</span>(<span style="color: #d73948">self</span>.links, <span style="color: #4b69c6">dict</span>)<br/> <span style="color: #d73948">or</span> <span style="color: #4b69c6">set</span>(<span style="color: #d73948">self</span>.links.<span style="color: #4b69c6">keys</span>()) <span style="color: #d73948">!=</span> required_link_types<br/> ):<br/> <span style="color: #d73948">raise</span> <span style="color: #4b69c6">ValueError</span>(<br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">links must be a dictionary with exactly these keys: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">required_link_types</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span><br/> )<br/> <span style="color: #d73948">for</span> key <span style="color: #d73948">in</span> <span style="color: #d73948">self</span>.links:<br/> <span style="color: #d73948">if</span> <span style="color: #d73948">not</span> <span style="color: #4b69c6">isinstance</span>(<span style="color: #d73948">self</span>.links[key], <span style="color: #4b69c6">set</span>):<br/> <span style="color: #d73948">self</span>.links[key] <span style="color: #d73948">=</span> <span style="color: #4b69c6">set</span>(<span style="color: #d73948">self</span>.links[key])<br/></code></pre></div> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #301414">@</span><span style="color: #301414">dataclass</span><br/><span style="color: #d73948">class</span> <span style="color: #4b69c6">AssetMetadata</span>:<br/> filepath: Path<br/> media: <span style="color: #4b69c6">str</span><br/> uid: <span style="color: #4b69c6">str</span><br/> slug: <span style="color: #4b69c6">str</span><br/> title: <span style="color: #4b69c6">str</span><br/> available: datetime.datetime<br/> created: datetime.datetime<br/> updated: datetime.datetime<br/> creator: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> note: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> favourite: <span style="color: #4b69c6">bool</span> <span style="color: #d73948">=</span> <span style="color: #d73948">False</span><br/> source: Dict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/> via: Dict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/> <span style="color: #4b69c6">hash</span>: <span style="color: #4b69c6">str</span> <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">"</span><br/> output_width: <span style="color: #4b69c6">int</span> <span style="color: #d73948">=</span> <span style="color: #b60157">0</span><br/> output_height: <span style="color: #4b69c6">int</span> <span style="color: #d73948">=</span> <span style="color: #b60157">0</span><br/> location: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">continent</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">country</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">region</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">city</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">note</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">lat</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">int</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">lng</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">int</span>,<br/> }<br/> )<br/> attribution: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">djot</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">html</span><span style="color: #198810">"</span>: <span style="color: #198810">"</span><span style="color: #198810">"</span>,<br/> }<br/> )<br/> words: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">lines</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">words</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> },<br/> <span style="color: #198810">"</span><span style="color: #198810">references</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> }<br/> )<br/> links: LinksDict <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">internal</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> <span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> <span style="color: #198810">"</span><span style="color: #198810">backlinks</span><span style="color: #198810">"</span>: <span style="color: #4b69c6">list</span>(),<br/> }<br/> )<br/> tags: List[<span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">list</span>)<br/> content: Dict[<span style="color: #4b69c6">str</span>, <span style="color: #4b69c6">str</span>] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(default_factory<span style="color: #d73948">=</span><span style="color: #4b69c6">dict</span>)<br/><br/> <span style="color: #d73948">def</span> <span style="color: #4b69c6">__post_init__</span>(self):<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Validate links dictionary structure</span><br/> required_link_types <span style="color: #d73948">=</span> {<span style="color: #198810">"</span><span style="color: #198810">internal</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">backlinks</span><span style="color: #198810">"</span>}<br/> <span style="color: #d73948">if</span> (<br/> <span style="color: #d73948">not</span> <span style="color: #4b69c6">isinstance</span>(<span style="color: #d73948">self</span>.links, <span style="color: #4b69c6">dict</span>)<br/> <span style="color: #d73948">or</span> <span style="color: #4b69c6">set</span>(<span style="color: #d73948">self</span>.links.<span style="color: #4b69c6">keys</span>()) <span style="color: #d73948">!=</span> required_link_types<br/> ):<br/> <span style="color: #d73948">raise</span> <span style="color: #4b69c6">ValueError</span>(<br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">links must be a dictionary with exactly these keys: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">required_link_types</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span><br/> )<br/> <span style="color: #d73948">for</span> key <span style="color: #d73948">in</span> <span style="color: #d73948">self</span>.links:<br/> <span style="color: #d73948">if</span> <span style="color: #d73948">not</span> <span style="color: #4b69c6">isinstance</span>(<span style="color: #d73948">self</span>.links[key], <span style="color: #4b69c6">set</span>):<br/> <span style="color: #d73948">self</span>.links[key] <span style="color: #d73948">=</span> <span style="color: #4b69c6">set</span>(<span style="color: #d73948">self</span>.links[key])<br/></code></pre></div> <p>This turned up a couple of oversights in metadata for existing documents, and also encouraged me to restructure the final built metadata, see backlinks and wordcounting.</p> <p>Speaking of backlinks, I coupled this work with my ongoing effort to make the building of the site completely deterministic. The builds already are deterministic in every way that matters. Same inputs make for the same output, except in one way: because documents are built asynchronously, it has been possible for the order of backlink references to change between builds. This doesn’t alter any functionality or correctness but does mean that the order of the backlinks in the hidden build summary of each page could change, and this makes diffing over the whole site to spot regressions more noisy, as many differences were not really differences at all.</p> <p>Solving this turned out to be easy once I realised that a few pages kept seeing the order shift because they referenced pages that had the same publish (<code>available</code>) date, which is the key for the sort order. With that remedied, determinism is assured.</p> <details><summary>For a glimpse at the surface area of the changes, click here to see a diff from implementing the first dataclass, though more changes have landed since.</summary><div class="code-block" data-lang="diff"><div class="code-header"><span class="code-lang">diff</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="diff"> # Imports<br/> from collections import Counter<br/><span style="color: #d73948">-</span><span style="color: #d73948">from dataclasses import dataclass, field</span><br/><span style="color: #198810">+</span><span style="color: #198810">from dataclasses import dataclass, field, asdict</span><br/> from hashlib import md5<br/> from pathlib import Path<br/> from shutil import copyfile<br/> from subprocess import run<br/> from typing import List, Dict, Any, Tuple<br/><span style="color: #198810">+</span><span style="color: #198810">from typing_extensions import TypedDict</span><br/> import asyncio<br/> import datetime<br/> import logging<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-73,11 +74,11</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">class SiteMetadata:</span></strong><br/> words: Dict[str, int] = field(<br/> default_factory=lambda: {"drafts": 0, "references": 0, "self": 0}<br/> )<br/><span style="color: #d73948">-</span><span style="color: #d73948"> links: Dict[str, set] = field(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> links: Dict[str, Any] = field(</span><br/> default_factory=lambda: {<br/><span style="color: #d73948">-</span><span style="color: #d73948"> "internal": set(),</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "internal": list(),</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "backlinks": list(),</span><br/> "external": set(),<br/><span style="color: #d73948">-</span><span style="color: #d73948"> "backlinks": set(),</span><br/> }<br/> )<br/> pagecount: int = 0<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-91,9 +92,15</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">class SiteMetadata:</span></strong><br/> slug_to_title_lookup: Dict[str, str] = field(default_factory=dict)<br/><br/><br/><span style="color: #198810">+</span><span style="color: #198810">class LinksDict(TypedDict):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> internal: list[str]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> external: list[str]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> backlinks: list[str]</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><br/> \@dataclass<br/> class DocumentMetadata:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> filename: Path</span><br/><span style="color: #198810">+</span><span style="color: #198810"> filepath: Path</span><br/> uid: str<br/> slug: str<br/> created: datetime.datetime<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-102,20 +109,78</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">class DocumentMetadata:</span></strong><br/> title: str<br/> primary: str<br/> secondary: str<br/><span style="color: #198810">+</span><span style="color: #198810"> creator: str = ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> note: str = ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> favourite: bool = False</span><br/><span style="color: #198810">+</span><span style="color: #198810"> parent: str = ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> description: str = ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> layout: str = TEMPLATE_DEFAULT</span><br/><span style="color: #198810">+</span><span style="color: #198810"> source: Dict = field(default_factory=dict)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> via: Dict = field(default_factory=dict)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> location: Dict = field(default_factory=dict)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> collection: Dict[str, Any] = field(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> default_factory=lambda: {</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "style": "title",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "order": "chronological",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "include": [],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> }</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> attribution: Dict[str, str] = field(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> default_factory=lambda: {</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "plain": "",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "djot": "",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "html": "",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> }</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> media: str = "application/toml"</span><br/> words: int = 0<br/><span style="color: #d73948">-</span><span style="color: #d73948"> links: Dict[str, set] = field(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> status: str = ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> links: LinksDict = field(</span><br/> default_factory=lambda: {<br/><span style="color: #d73948">-</span><span style="color: #d73948"> "internal": set(),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "external": set(),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "backlinks": set(),</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "internal": list(),</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "external": list(),</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "backlinks": list(),</span><br/> }<br/> )<br/> options: List[str] = field(default_factory=list)<br/> tags: List[str] = field(default_factory=list)<br/><span style="color: #d73948">-</span><span style="color: #d73948"> interlinks: List[str] = field(default_factory=list)</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> backlinks: List[str] = field(default_factory=list)</span><br/> content: Dict[str, str] = field(default_factory=dict)<br/><span style="color: #d73948">-</span><span style="color: #d73948"> html: str = ""</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> def __post_init__(self):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Validate required string fields are not empty</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # for field_name in ["uid", "slug", "title", "primary", "secondary"]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # value = getattr(self, field_name)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # if not isinstance(value, str) or not value.strip():</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # raise ValueError(f"\n\n{self}\n\n{field_name} {value} must be a non-empty string"\n)</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Validate filepath is a Path object</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if not isinstance(self.filepath, Path):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> self.filepath = Path(self.filepath)</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ## Validate datetime fields</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # for field_name in ["created", "updated", "available"]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # value = getattr(self, field_name)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # if not isinstance(value, datetime.datetime):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # raise ValueError(f"{field_name} must be a datetime object")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # # Ensure timezone is None</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # setattr(self, field_name, value.replace(tzinfo=None))</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Validate words is non-negative</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if not isinstance(self.words, int) or self.words &lt; 0:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> raise ValueError("words must be a non-negative integer")</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Validate links dictionary structure</span><br/><span style="color: #198810">+</span><span style="color: #198810"> required_link_types = {"internal", "external", "backlinks"}</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if (</span><br/><span style="color: #198810">+</span><span style="color: #198810"> not isinstance(self.links, dict)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> or set(self.links.keys()) != required_link_types</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> raise ValueError(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> f"links must be a dictionary with exactly these keys: {required_link_types}"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for key in self.links:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if not isinstance(self.links[key], set):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> self.links[key] = set(self.links[key])</span><br/><br/><br/> def init_site():<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-132,12 +197,47</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def init_site():</span></strong><br/> )<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def load_assets():</span><br/><span style="color: #198810">+</span><span style="color: #198810">def preprocess_asset_metadata(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> uid: str, asset_data: Dict[str, Any], manifest_path: Path</span><br/><span style="color: #198810">+</span><span style="color: #198810">) -&gt; Dict[str, Any]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """Preprocess asset metadata to ensure it meets DocumentMetadata requirements."""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed = asset_data.copy()</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Handle dates</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for date_field in ["created", "updated", "available"]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if isinstance(processed.get(date_field), str):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed[date_field] = _parse_date(processed[date_field])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> elif isinstance(processed.get(date_field), datetime.datetime):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed[date_field] = processed[date_field].replace(tzinfo=None)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> else:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed[date_field] = datetime.datetime.now()</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Set required fields with defaults if not present</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed.setdefault("uid", uid)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed.setdefault("primary", "asset")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed.setdefault("secondary", processed["media"])</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> return processed</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810">def load_assets() -&gt; Dict[str, DocumentMetadata]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """Load asset manifests and convert them to DocumentMetadata instances."""</span><br/> assets = {}<br/> asset_manifests = list(ASSET_DIR.glob("manifests/*.toml"))<br/><span style="color: #198810">+</span><br/> for manifest in asset_manifests:<br/> with open(manifest, "rb") as f:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> assets.update(tomllib.load(f))</span><br/><span style="color: #198810">+</span><span style="color: #198810"> manifest_data = tomllib.load(f)</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for uid, asset_data in manifest_data.items():</span><br/><span style="color: #198810">+</span><span style="color: #198810"> try:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed_data = preprocess_asset_metadata(uid, asset_data, manifest)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed_data["filepath"] = ASSET_DIR / processed_data["filepath"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> assets[uid] = DocumentMetadata(**processed_data)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> except Exception as e:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> print(f"Error processing asset {uid} from {manifest}: {str(e)}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> continue</span><br/><span style="color: #198810">+</span><br/> return assets<br/><br/><br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-176,7 +276,7</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def get_files() -&gt; List[Path]:</span></strong><br/> return [f for f in NOTES_DIR.glob("**/*.md") if "available = " in f.read_text()]<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def extract_external_links(text: str) -&gt; List:</span><br/><span style="color: #198810">+</span><span style="color: #198810">def extract_external_links(text: str, site) -&gt; List:</span><br/> url_pattern = r"(https?://(?:[-\w.]|(?:%[\da-fA-F]{2}))+(?:/[^)\s]*)?)"<br/> matches = re.findall(url_pattern, text)<br/><br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-186,18 +286,21</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def extract_external_links(text: str) -&gt; List:</span></strong><br/> parsed_url = urlparse(url)<br/> if parsed_url.netloc.lower() != "silasjelley.com":<br/> external_links.add(url)<br/><span style="color: #198810">+</span><span style="color: #198810"> site.links["external"].add(url)</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> return list(external_links)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> return sorted(external_links)</span><br/><br/><br/> async def process_document(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> filename: Path, site: SiteMetadata</span><br/><span style="color: #d73948">-</span><span style="color: #d73948">) -&gt; Tuple[str, Dict[str, Any]]:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> with open(filename, "rb") as f:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> filepath: Path, site: SiteMetadata</span><br/><span style="color: #198810">+</span><span style="color: #198810">) -&gt; Tuple[str, DocumentMetadata]:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """Process a document file and return its UID and metadata."""</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> with open(filepath, "rb") as f:</span><br/> try:<br/> parsed_toml = tomllib.load(f)<br/> except:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> print(filename)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> print(filepath)</span><br/> import sys<br/><br/> sys.exit(1)<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-205,52 +308,48</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def process_document(</span></strong><br/> # The UID is now the top-level table name<br/> uid = parsed_toml["uid"]<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Preprocess metadata (assuming this function exists and works with the new format)</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> document = preprocess_metadata(filename, parsed_toml)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Process metadata into DocumentMetadata instance</span><br/><span style="color: #198810">+</span><span style="color: #198810"> document = preprocess_metadata(filepath, parsed_toml)</span><br/><br/> # Calculate and update word counts<br/> try:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> document["words"] = len(document["content"]["plain"].split())</span><br/><span style="color: #198810">+</span><span style="color: #198810"> document.words = len(document.content.get("plain", "").split())</span><br/> except:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> document["words"] = 0</span><br/><span style="color: #198810">+</span><span style="color: #198810"> document.words = 0</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if document.get("status", "") == "draft":</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.words["drafts"] += document["words"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if document.status == "draft":</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site.words["drafts"] += document.words</span><br/> else:<br/> try:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> document["source"]["words"] = len(document["source"]["text"].split())</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.words["references"] += document["source"]["words"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> source_words = len(document.source.get("text", "").split())</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site.words["references"] += source_words</span><br/> except KeyError:<br/> pass<br/> try:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.words["self"] += document["words"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site.words["self"] += document.words</span><br/> except:<br/> pass<br/><br/> # Extract external links from the plain text content<br/> try:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> plain_text = document.get("content", {}).get("plain", "") + " "</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> plain_text += document.get("source", {}).get("url", "") + " "</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> plain_text += document.get("via", {}).get("url", "") + " "</span><br/><span style="color: #d73948">-</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> external_links = extract_external_links(plain_text)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> plain_text = (</span><br/><span style="color: #198810">+</span><span style="color: #198810"> document.content.get("plain", "")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + " "</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + document.source.get("url", "")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + " "</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + document.via.get("url", "")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Store the external links in document["links"]["external"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> document["links"] = {</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "internal": set(),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "external": set(),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "backlinks": set(),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> }</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> document["links"]["external"].update(external_links)</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.links["external"].update(external_links)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> external_links = extract_external_links(plain_text, site)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> document.links["external"] = external_links</span><br/> except KeyError:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> print(f"KeyError while compiling external links from {document['filename']}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> print(f"KeyError while compiling external links from {document.filepath}")</span><br/> pass<br/><br/> return uid, document<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">async def ingest_documents(site: SiteMetadata) -&gt; Dict[str, Dict[str, Any]]:</span><br/><span style="color: #198810">+</span><span style="color: #198810">async def ingest_documents(site: SiteMetadata) -&gt; Dict[str, Any]:</span><br/> logger.info("Ingesting files")<br/> file_list = get_files()<br/> documents = {}<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-258,11 +357,9</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def ingest_documents(site: SiteMetadata) -&gt; Dict[str, Dict[str, Any]]:</span></strong><br/> slug_to_uid_lookup = {}<br/> site_primaries = set()<br/> site_secondaries = set()<br/><span style="color: #d73948">-</span><span style="color: #d73948"> site_series = set()</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> tags = set()</span><br/> uuid_collision_lookup = []<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> tasks = [process_document(filename, site) for filename in file_list]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> tasks = [process_document(filepath, site) for filepath in file_list]</span><br/> results = await asyncio.gather(*tasks)<br/> site.words["total"] = (<br/> site.words["drafts"] + site.words["references"] + site.words["self"]<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-270,13 +367,11</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def ingest_documents(site: SiteMetadata) -&gt; Dict[str, Dict[str, Any]]:</span></strong><br/><br/> for uid, doc in results:<br/> documents[uid] = doc<br/><span style="color: #d73948">-</span><span style="color: #d73948"> slug_to_title_lookup[doc["slug"]] = doc["title"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> slug_to_uid_lookup[doc["slug"]] = uid</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site_primaries.add(doc["primary"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site_secondaries.add(doc["secondary"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if "series" in doc:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site_series.add(doc["series"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> tags.update(doc.get("tags") or [])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> slug_to_title_lookup[doc.slug] = doc.title</span><br/><span style="color: #198810">+</span><span style="color: #198810"> slug_to_uid_lookup[doc.slug] = uid</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site_primaries.add(doc.primary)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site_secondaries.add(doc.secondary)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site.tags += doc.tags</span><br/> uuid_collision_lookup.append(uid)<br/><br/> site.slug_to_uid_lookup = slug_to_uid_lookup<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-284,7 +379,6</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def ingest_documents(site: SiteMetadata) -&gt; Dict[str, Dict[str, Any]]:</span></strong><br/> check_uuid_collisions(uuid_collision_lookup)<br/> site.primaries = list(site_primaries)<br/> site.secondaries = list(site_secondaries)<br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.tags = list(tags)</span><br/> site.pagecount = len(documents)<br/><br/> logger.info(f"Ingested {site.pagecount} files")<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-373,14 +467,14</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def process_image_parallel(input_data: Tuple[Path, Path, int, str]) -&gt; None:</span></strong><br/><br/><br/> def process_assets(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> assets: Dict[str, Dict[str, Any]], asset_dir: Path, output_dir: Path</span><br/><span style="color: #198810">+</span><span style="color: #198810"> assets: Dict[str, DocumentMetadata], asset_dir: Path, output_dir: Path</span><br/> ) -&gt; None:<br/> logger.info("Processing assets")<br/> manifest_images = []<br/><br/> for asset_identifier, asset_metadata in assets.items():<br/><span style="color: #d73948">-</span><span style="color: #d73948"> source_path = Path(asset_metadata["filepath"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> output_path = output_dir / asset_metadata["slug"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> source_path = Path(asset_metadata.filepath)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> output_path = output_dir / asset_metadata.slug</span><br/> os.makedirs(output_path.parent, exist_ok=True)<br/><br/> if not source_path.exists():<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-422,74 +516,52</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def _parse_date(date_str: str) -&gt; datetime.datetime:</span></strong><br/> return datetime.datetime.strptime(date_str, "%Y-%m-%d").replace(tzinfo=None)<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def preprocess_metadata(filename: Path, metadata: Dict[str, Any]) -&gt; Dict[str, Any]:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> """Preprocesses metadata for a document, ensuring required fields exist and formatting data."""</span><br/><span style="color: #d73948">-</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["filename"] = filename</span><br/><span style="color: #d73948">-</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Validate required fields</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> required_fields = ["uid", "slug", "available", "created", "primary", "secondary"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> missing_fields = [field for field in required_fields if field not in metadata]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if missing_fields:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> raise ValueError(</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> f"[ERROR] Document missing {', '.join(missing_fields)}\n {filename}"</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> )</span><br/><span style="color: #d73948">-</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Set default values</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata.setdefault("updated", metadata["created"])</span><br/><span style="color: #198810">+</span><span style="color: #198810">def preprocess_metadata(filepath: Path, metadata: Dict[str, Any]) -&gt; DocumentMetadata:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """Preprocesses metadata for a document and converts it to a DocumentMetadata instance."""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Create a working copy to avoid modifying the input</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed = metadata.copy()</span><br/><br/> # Parse date fields<br/> for date_field in ["created", "updated", "available"]:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> if date_field in metadata:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if isinstance(metadata[date_field], str):</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata[date_field] = _parse_date(metadata[date_field])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> elif isinstance(metadata[date_field], datetime.datetime):</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata[date_field] = metadata[date_field].replace(tzinfo=None)</span><br/><span style="color: #d73948">-</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Process source information</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if "source" in metadata:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if "via" in metadata:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata.update(</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> process_source_information(metadata["source"], metadata["via"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> )</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> else:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata.update(process_source_information(metadata["source"], {}))</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if isinstance(processed.get(date_field), str):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed[date_field] = _parse_date(processed[date_field])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> elif isinstance(processed.get(date_field), datetime.datetime):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed[date_field] = processed[date_field].replace(tzinfo=None)</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Set default updated time if not provided</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed.setdefault("updated", processed.get("created"))</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Process source information if present</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if "source" in processed:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["attribution"] = process_source_information(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["source"], processed.get("via", {})</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> else:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["attribution"] = {}</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["source"] = {}</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if "via" not in processed:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["via"] = {}</span><br/><br/> # Determine title<br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["title"] = (</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata.get("title")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> or metadata.get("attribution", {}).get("plain")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> or metadata["created"].strftime("%B %e, %Y %-I.%M%p")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["title"] = (</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed.get("title")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> or processed.get("attribution", {}).get("plain")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> or processed["created"].strftime("%B %e, %Y %-I.%M%p")</span><br/> )<br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Ensure title and slug are strings</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["title"] = str(metadata["title"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if metadata.get("status") == "draft":</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["slug"] = "drafts/" + metadata["uid"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["parent"] = "a26221ae-c742-4b73-8dc6-7f8807456a1b"</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> else:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["slug"] = str(metadata["slug"])</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Initialize interlinks, backlinks, and tags</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["interlinks"] = set()</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["backlinks"] = set()</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["tags"] = metadata.get("tags") or []</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Handle draft status</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if processed.get("status") == "draft":</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["slug"] = f"drafts/{processed['uid']}"</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> # Strip whitespace from plain content</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> try:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["content"]["plain"] = metadata["content"]["plain"].strip()</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> except KeyError:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> pass</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> try:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> metadata["source"]["text"] = metadata["source"]["text"].strip()</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> except KeyError:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> pass</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Add filepath as it's required but comes from function parameter</span><br/><span style="color: #198810">+</span><span style="color: #198810"> processed["filepath"] = filepath</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> return metadata</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Create and return DocumentMetadata instance</span><br/><span style="color: #198810">+</span><span style="color: #198810"> return DocumentMetadata(**processed)</span><br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def process_source_information(</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> source: Dict[str, Any], via</span><br/><span style="color: #d73948">-</span><span style="color: #d73948">) -&gt; Dict[str, Dict[str, str]]:</span><br/><span style="color: #198810">+</span><span style="color: #198810">def process_source_information(source: Dict[str, Any], via) -&gt; Dict[str, str]:</span><br/> creator = source.get("creator") or source.get("director")<br/> title = source.get("title") or (<br/> " ♫ " + str(source.get("track"))<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-587,13 +659,11</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def process_source_information(</span></strong><br/> partsvia = f" (#link("{escape_url(via["url"]")[via]}))"<br/><br/> return {<br/><span style="color: #d73948">-</span><span style="color: #d73948"> "attribution": {</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "plain": f"{speaker}{', '.join(partsplain + partsshared)}",</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "djot": f"{speaker}{', '.join(partsdjot + partsshared)}",</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "html": format_rich_attribution(</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> " — " + f"{speaker}{', '.join(partshtml + partsshared) + partsvia}"</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> ),</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> }</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "plain": f"{speaker}{', '.join(partsplain + partsshared)}",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "djot": f"{speaker}{', '.join(partsdjot + partsshared)}",</span><br/><span style="color: #198810">+</span><span style="color: #198810"> "html": format_rich_attribution(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> " — " + f"{speaker}{', '.join(partshtml + partsshared) + partsvia}"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ),</span><br/> }<br/><br/><br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-619,8 +689,8</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def check_uuid_collisions(uuid_list):</span></strong><br/><br/><br/> def insert_substitutions(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents: Dict[str, Dict[str, Any]],</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> assets: Dict[str, Dict[str, Any]],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents: Dict[str, DocumentMetadata],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> assets: Dict[str, DocumentMetadata],</span><br/> site: SiteMetadata,<br/> ) -&gt; None:<br/> logger.info("Performing substitutions")<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-635,53 +705,56</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def insert_substitutions(</span></strong><br/> merged_data = {*documents, *assets}<br/><br/> for key, page in documents.items():<br/><span style="color: #d73948">-</span><span style="color: #d73948"> logger.debug(f" {key}, {page['title'][:40]}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> logger.debug(f" {key}, {page.title[:40]}")</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> text = page.get("content", {}).get("plain")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text = page.content.get("plain")</span><br/> if text:<br/> text = replace_import_references(text, REF_IMPT_RE, merged_data, key, page)<br/> text = replace_cite_references(text, REF_CITE_RE, merged_data)<br/> text = replace_title_references(text, REF_TITLE_RE, merged_data)<br/> text = replace_slug_references(text, REF_SLUG_RE, merged_data)<br/> text = process_reference_links(text, REF_LINK_RE, merged_data, key)<br/><span style="color: #d73948">-</span><span style="color: #d73948"> page["content"]["plain"] = text.strip()</span><br/><span style="color: #198810">+</span><span style="color: #198810"> page.content["plain"] = text.strip()</span><br/><br/><br/> def replace_slug_references(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> text: str, regex: re.Pattern, merged_data: Dict[str, Dict[str, Any]]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text: str, regex: re.Pattern, merged_data: Dict[str, DocumentMetadata]</span><br/> ) -&gt; str:<br/> for match in regex.finditer(text):<br/> ref_type, ref_short_id = match.groups()<br/> full_match = match.group(0)<br/> ref_id = next((k for k in merged_data if k.startswith(ref_short_id)), None)<br/> if ref_id:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> replacement = f"/{merged_data[ref_id]['slug']}"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> try:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> replacement = f"/{merged_data[ref_id].slug}"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> except AttributeError:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> replacement = f"/{merged_data[ref_id].slug}"</span><br/> text = text.replace(full_match, replacement)<br/> return text<br/><br/><br/> def replace_title_references(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> text: str, regex: re.Pattern, merged_data: Dict[str, Dict[str, Any]]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text: str, regex: re.Pattern, merged_data: Dict[str, DocumentMetadata]</span><br/> ) -&gt; str:<br/> for match in regex.finditer(text):<br/> opening, ref_type, ref_short_id, comment, closing = match.groups()<br/> full_match = match.group(0)<br/> ref_id = next((k for k in merged_data if k.startswith(ref_short_id)), None)<br/> if ref_id:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> replacement = merged_data[ref_id]["title"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> replacement = merged_data[ref_id].title</span><br/> text = text.replace(full_match, replacement)<br/> return text<br/><br/><br/> def replace_cite_references(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> text: str, regex: re.Pattern, merged_data: Dict[str, Dict[str, Any]]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text: str, regex: re.Pattern, merged_data: Dict[str, DocumentMetadata]</span><br/> ) -&gt; str:<br/> for match in regex.finditer(text):<br/> opening, ref_type, ref_short_id, comment, closing = match.groups()<br/> full_match = match.group(0)<br/> ref_id = next((k for k in merged_data if k.startswith(ref_short_id)), None)<br/> if ref_id:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> replacement = f"[{merged_data[ref_id]["attribution"]["djot"]}] (/{merged_data[ref_id]["slug"]})"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> replacement = f"[{merged_data[ref_id].attribution["djot"]}] (/{merged_data[ref_id].slug})"</span><br/> text = text.replace(full_match, replacement)<br/> return text<br/><br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-689,29 +762,32</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def replace_cite_references(</span></strong><br/> def replace_import_references(<br/> text: str,<br/> regex: re.Pattern,<br/><span style="color: #d73948">-</span><span style="color: #d73948"> merged_data: Dict[str, Dict[str, Any]],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> merged_data: Dict[str, DocumentMetadata],</span><br/> key: str,<br/><span style="color: #d73948">-</span><span style="color: #d73948"> page: Dict,</span><br/><span style="color: #198810">+</span><span style="color: #198810"> page: DocumentMetadata,</span><br/> ) -&gt; str:<br/> for match in regex.finditer(text):<br/> opening, ref_type, ref_short_id, comment, closing = match.groups()<br/> full_match = match.group(0)<br/> ref_id = next((k for k in merged_data if k.startswith(ref_short_id)), None)<br/> if ref_id:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_text = merged_data[ref_id]["content"]["plain"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_text = merged_data[ref_id].content["plain"]</span><br/> if ref_type == "import::":<br/> replacement = ref_text<br/> elif ref_type == "aside::":<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_title = merged_data[ref_id]["title"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_slug = merged_data[ref_id]["slug"]</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_location = merged_data[ref_id].get("location", "")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> location_string = (</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> " ⚕ "</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> + ref_location.get("city")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> + ", "</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> + ref_location.get("country")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> or ""</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_title = merged_data[ref_id].title</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_slug = merged_data[ref_id].slug</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_location = merged_data[ref_id].location</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if ref_location:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> location_string = (</span><br/><span style="color: #198810">+</span><span style="color: #198810"> " ⚕ "</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + ref_location.get("city")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + ", "</span><br/><span style="color: #198810">+</span><span style="color: #198810"> + ref_location.get("country")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> or ""</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> else:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> location_string = ""</span><br/> replacement = (<br/> f'{{.aside}}\n{':'*78}\n'<br/> f'{ref_text}\\\n'<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-722,18 +798,22</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def replace_import_references(</span></strong><br/> )<br/> else:<br/> raise ValueError(f"Unrecognised reference type: {ref_type}")<br/><span style="color: #d73948">-</span><span style="color: #d73948"> if not page.get("status", "") == "draft":</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> merged_data[ref_id]["backlinks"].add(key)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if not page.status == "draft":</span><br/><span style="color: #198810">+</span><span style="color: #198810"> merged_data[ref_id].links["backlinks"].add(key)</span><br/> text = text.replace(full_match, replacement)<br/> return text<br/><br/><br/> def process_reference_links(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> text: str, regex: re.Pattern, merged_data: Dict[str, Dict[str, Any]], key: str</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text: str, regex: re.Pattern, merged_data: Dict[str, DocumentMetadata], key: str</span><br/> ) -&gt; str:<br/> for ref_text_match, _, ref_type, ref_short_id in regex.findall(text):<br/> match = f"#link("{ref_type}{ref_short_id}")[{ref_text_match}]"<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_id = next((k for k in merged_data if k.startswith(ref_short_id)), None)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_id = next(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> (k for k in merged_data.keys() if k.startswith(ref_short_id)), None</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if ref_id is None:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> print(f"No match found for {ref_short_id}")</span><br/><br/> if not ref_id:<br/> raise ValueError(<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-746,16 +826,16</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def process_reference_links(</span></strong><br/> )<br/><br/> ref_text = get_reference_text(ref_text_match, merged_data[ref_id])<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_slug = f"/{merged_data[ref_id]['slug']}"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_slug = f"/{merged_data[ref_id].slug}"</span><br/><br/> if ref_type == "link::":<br/> try:<br/> # Double quotes within a page title are escaped so that they don't break the HTML 'title' element<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_title = f"{merged_data[ref_id]['title']} | {merged_data[ref_id]['created'].strftime('%B %Y')}".replace(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_title = f"{merged_data[ref_id].title} | {merged_data[ref_id].created.strftime('%B %Y')}".replace(</span><br/> '"', '\\"'<br/> )<br/> except KeyError:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_title = merged_data[ref_id]["title"].replace('"', '\\"')</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_title = merged_data[ref_id].title.replace('"', '\\"')</span><br/> replacement = f'#link("{ref_slug}")[{ref_text}]{{title="{ref_title}"}}'<br/> elif ref_type == "img::":<br/> replacement = f"#link("{ref_slug}")[{ref_text}]"<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-778,18 +858,18</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def process_reference_links(</span></strong><br/> return text<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def get_reference_text(ref_text_match: str, ref_data: Dict) -&gt; str:</span><br/><span style="color: #198810">+</span><span style="color: #198810">def get_reference_text(ref_text_match: str, ref_data) -&gt; str:</span><br/> if ref_text_match.startswith("::") or ref_text_match == "":<br/><span style="color: #d73948">-</span><span style="color: #d73948"> return ref_data.get("title")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> return ref_data.title</span><br/> return ref_text_match<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def create_quote_replacement(ref_data: Dict, ref_slug: str) -&gt; str:</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_src = ref_data["attribution"]["djot"]</span><br/><span style="color: #198810">+</span><span style="color: #198810">def create_quote_replacement(ref_data: DocumentMetadata, ref_slug: str) -&gt; str:</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_src = ref_data.attribution["djot"]</span><br/> try:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> ref_text = ref_data["source"]["text"].replace("\n\n", "\n&gt; \n&gt; ").strip()</span><br/><span style="color: #198810">+</span><span style="color: #198810"> ref_text = ref_data.source["text"].replace("\n\n", "\n&gt; \n&gt; ").strip()</span><br/> except:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> print(f"Error creating quote replacement: {ref_data["uid"]}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> print(f"Error creating quote replacement: {ref_data.uid}")</span><br/> import sys<br/><br/> sys.exit()<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-808,10 +888,10</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">Your browser does not support the video tag.</span></strong><br/> def generate_html(documents):<br/> logger.info("Generating HTML")<br/> for key, page in documents.items():<br/><span style="color: #d73948">-</span><span style="color: #d73948"> if page.get("content", {}).get("plain"):</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> page["content"]["html"] = run_jotdown(page["content"]["plain"])</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if page.get("source", {}).get("text"):</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> page["source"]["html"] = run_jotdown(page["source"]["text"])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if page.content.get("plain"):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> page.content["html"] = run_jotdown(page.content["plain"])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if page.source.get("text"):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> page.source["html"] = run_jotdown(page.source["text"])</span><br/><br/><br/> class LedgerLexer(RegexLexer):<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-841,7 +921,7</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">class LedgerLexer(RegexLexer):</span></strong><br/> }<br/><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948">def highlight_code(code: str, language: str = None) -&gt; str:</span><br/><span style="color: #198810">+</span><span style="color: #198810">def highlight_code(code: str, language: str) -&gt; str:</span><br/> """<br/> Highlight code using Pygments with specified or guessed language.<br/> """<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-921,20 +1001,17</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def build_backlinks(documents, site):</span></strong><br/> FOOTNOTE_LINK_URL_RE = re.compile(r"\[.+?\]:\s\/(.*)", re.DOTALL)<br/> interlink_count = 0<br/> for key, page in documents.items():<br/><span style="color: #d73948">-</span><span style="color: #d73948"> if (</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> "nobacklinks" in page.get("options", "")</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> or page.get("status", "") == "draft"</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> ):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if "nobacklinks" in page.options or page.status == "draft":</span><br/> continue<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> logger.debug(page["filename"])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> logger.debug(page.filepath)</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> text = page.get("content", {}).get("plain")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> text = page.content.get("plain")</span><br/> # Skip if no main content<br/> if not text:<br/> continue<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> interlinks = set(documents[key]["interlinks"])</span><br/><span style="color: #198810">+</span><span style="color: #198810"> interlinks = set(documents[key].links["internal"])</span><br/><br/> combined_refs = INLINE_LINK_RE.findall(text) + FOOTNOTE_LINK_URL_RE.findall(<br/> text<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-950,11 +1027,28</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def build_backlinks(documents, site):</span></strong><br/> continue<br/> logger.warning(f"\nKeyError in {page['title']} ({key}): {slug}")<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents[key]["interlinks"] = list(interlinks)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[key].links["internal"] = interlinks</span><br/> for interlink_key in interlinks:<br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents[interlink_key]["backlinks"].add(key)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[interlink_key].links["backlinks"].add(key)</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> return interlink_count</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for key, page in documents.items():</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Sort interlinks based on published dates</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[key].links["internal"] = sorted(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[key].links["internal"],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> key=lambda x: documents[x].available,</span><br/><span style="color: #198810">+</span><span style="color: #198810"> reverse=True, # Most recent first</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><span style="color: #198810"> # Sort backlinks based on published dates</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[key].links["backlinks"] = sorted(</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents[key].links["backlinks"],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> key=lambda x: documents[x].available,</span><br/><span style="color: #198810">+</span><span style="color: #198810"> reverse=True, # Most recent first</span><br/><span style="color: #198810">+</span><span style="color: #198810"> )</span><br/><span style="color: #198810">+</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """</span><br/><span style="color: #198810">+</span><span style="color: #198810"> TODO: REMOVE SITE.BACKLINKS in favour a 'stats' or 'count' (templates will need updating</span><br/><span style="color: #198810">+</span><span style="color: #198810"> """</span><br/><span style="color: #198810">+</span><span style="color: #198810"> site.backlinks += interlink_count</span><br/><br/><br/> def should_ignore_slug(slug):<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-966,7 +1060,7</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def should_ignore_slug(slug):</span></strong><br/><br/><br/> def build_collections(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents: Dict[str, Dict[str, Any]], site: SiteMetadata</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents: Dict[str, DocumentMetadata], site: SiteMetadata</span><br/> ) -&gt; Tuple[Dict[str, List[Dict[str, Any]]], List[Dict[str, Any]]]:<br/> collections = {<br/> primary: []<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-978,24 +1072,24</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def build_collections(</span></strong><br/> sitemap = []<br/><br/> for key, page in sorted(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents.items(), key=lambda k_v: k_v[1]["available"], reverse=True</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents.items(), key=lambda k_v: k_v[1].available, reverse=True</span><br/> ):<br/><span style="color: #d73948">-</span><span style="color: #d73948"> if page.get("status", "") == "draft":</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if page.status == "draft":</span><br/> collections["cd68b918-ac5f-4d6c-abb5-a55a0318846b"].append(page)<br/> continue<br/><span style="color: #d73948">-</span><span style="color: #d73948"> elif "nofeed" in page.get("options", []):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> elif "nofeed" in page.options:</span><br/> sitemap.append(page)<br/> continue<br/> else:<br/> sitemap.append(page)<br/> collections["everything"].append(page)<br/><span style="color: #d73948">-</span><span style="color: #d73948"> collections[page["primary"]].append(page)</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> collections[page["secondary"]].append(page)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> collections[page.primary].append(page)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> collections[page.secondary].append(page)</span><br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> for tag in page.get("tags", []):</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for tag in page.tags:</span><br/> collections[tag].append(page)<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> if page["secondary"] in [</span><br/><span style="color: #198810">+</span><span style="color: #198810"> if page.secondary in [</span><br/> "essays",<br/> "wandering",<br/> "rambling",<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-1008,8 +1102,8</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def build_collections(</span></strong><br/><br/><br/> def output_html(<br/><span style="color: #d73948">-</span><span style="color: #d73948"> assets: Dict[str, Dict[str, Any]],</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> documents: Dict[str, Dict[str, Any]],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> assets: Dict[str, DocumentMetadata],</span><br/><span style="color: #198810">+</span><span style="color: #198810"> documents: Dict[str, DocumentMetadata],</span><br/> collections: Dict[str, List[Dict[str, Any]]],<br/> site: SiteMetadata,<br/> env: Environment,<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-1018,7 +1112,7</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def output_html(</span></strong><br/> logger.info("Generating Hypertext")<br/><br/> for key, page in documents.items():<br/><span style="color: #d73948">-</span><span style="color: #d73948"> template_file = page.get("layout", TEMPLATE_DEFAULT)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> template_file = page.layout</span><br/> template = env.get_template(template_file)<br/><br/> collection = build_page_collection(page, collections)<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-1028,28 +1122,29</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">def output_html(</span></strong><br/> assets=assets,<br/> collections=collections,<br/> collection=collection,<br/><span style="color: #d73948">-</span><span style="color: #d73948"> page=page,</span><br/><span style="color: #198810">+</span><span style="color: #198810"> page=asdict(page),</span><br/> site=site,<br/> )<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> output_path = output_dir / page["slug"] / "index.html"</span><br/><span style="color: #198810">+</span><span style="color: #198810"> output_path = output_dir / page.slug / "index.html"</span><br/> output_path.parent.mkdir(parents=True, exist_ok=True)<br/><br/> with open(output_path, "w") as f:<br/> f.write(output)<br/><br/><span style="color: #d73948">-</span><span style="color: #d73948"> logger.debug(f" {page['filename']} &gt;&gt; {output_path}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> logger.debug(f" {page.filepath} &gt;&gt; {output_path}")</span><br/><br/><br/> def build_page_collection(page, collections):<br/> try:<br/> collection = [<br/> item<br/><span style="color: #d73948">-</span><span style="color: #d73948"> for include in page["collection"]["include"]</span><br/><span style="color: #198810">+</span><span style="color: #198810"> for include in page.collection["include"]</span><br/> for item in collections[include]<br/> ]<br/><span style="color: #d73948">-</span><span style="color: #d73948"> return sorted(collection, key=lambda x: x["available"], reverse=True)</span><br/><span style="color: #198810">+</span><span style="color: #198810"> return sorted(collection, key=lambda x: x.available, reverse=True)</span><br/> except KeyError:<br/><span style="color: #198810">+</span><span style="color: #198810"> print(f"Failed collection for {page.filepath}")</span><br/> return []<br/><br/><br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-1126,8 +1221,7</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def main():</span></strong><br/> generate_html(documents)<br/><br/> # Build backlinks and collections<br/><span style="color: #d73948">-</span><span style="color: #d73948"> interlink_count = build_backlinks(documents, site)</span><br/><span style="color: #d73948">-</span><span style="color: #d73948"> site.backlinks += interlink_count</span><br/><span style="color: #198810">+</span><span style="color: #198810"> build_backlinks(documents, site)</span><br/> collections, sitemap = build_collections(documents, site)<br/><br/> # Output HTML, feeds, and sitemap<br/><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">-1142,7 +1236,8</span><span style="color: #8b41b1"> </span><span style="color: #8b41b1">@@</span><span style="color: #8b41b1"> </span><strong><span style="color: #4b69c6">async def main():</span></strong><br/> logger.info("Build complete!")<br/> logger.info(f"Pages: {site.pagecount}")<br/> logger.info(f"Words: {site.words["total"]}")<br/><span style="color: #d73948">-</span><span style="color: #d73948"> logger.info(f"Interlinks: {interlink_count}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> logger.info(f"Internal links: {site.backlinks}")</span><br/><span style="color: #198810">+</span><span style="color: #198810"> logger.info(f"External links: {len(site.links["external"])}")</span><br/><br/><br/> if __name__ == "__main__":</code></pre></div></details> 9:54am on December 16, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Add dataclasses to ratchet up the strictness”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:875afe1a-3179-41ff-88df-8344816e6722 2024-12-15T17:25:27Z 2024-12-15T17:25:27Z <p>CSS is pure mayhem, but whatever, it’s what we’ve got. For a long time I’ve had reasonably good responsive image galleries. Really the last remaining roadblocks has been with those galleries where I’ve decided I need to vary the width of images <em>for aesthetic reasons</em>…</p> <p>Well, today I improved the style behaviour in half of those cases.</p> <p>The only caveat to this set of styles is that you have to be careful when using the <code>wide-first</code> or <code>wide-last</code> classes as they can leave an orphan image out in space if you don’t consider how many (odd or even) images are in the gallery or if they are of very different aspect ratios. But besides those I’ve found this to be pretty rock solid, if a little verbose.</p> <p>Here’s the relevant cascade,</p> <div class="code-block" data-lang="css"><div class="code-header"><span class="code-lang">css</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="css">root {<br/> <span style="color: #4b69c6">--</span><span style="color: #4b69c6">body-width</span>: <span style="color: #b60157">820</span><span style="color: #d73948">px</span>;<br/> <span style="color: #4b69c6">--</span><span style="color: #4b69c6">gallery-spacing</span>: clamp(<span style="color: #b60157">2</span><span style="color: #d73948">px</span>, <span style="color: #b60157">4</span><span style="color: #d73948">px</span>, <span style="color: #b60157">1</span><span style="color: #d73948">rem</span>);<br/> <span style="color: #4b69c6">--</span><span style="color: #4b69c6">gallery-transition</span>: <span style="color: #b60157">0</span><span style="color: #b60157">.</span><span style="color: #b60157">3</span><span style="color: #d73948">s</span> <span style="color: #4b69c6">ease-in-out</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> {<br/> <span style="color: #4b69c6">display</span>: <span style="color: #4b69c6">grid</span>;<br/> gap: <span style="color: #4b69c6">var</span>(<span style="color: #4b69c6">--</span><span style="color: #4b69c6">gallery-spacing</span>);<br/> <span style="color: #4b69c6">width</span>: <span style="color: #b60157">100</span><span style="color: #d73948">%</span>;<br/> <span style="color: #4b69c6">margin</span>: <span style="color: #b60157">2</span><span style="color: #d73948">rem</span> <span style="color: #b60157">0</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">img</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">picture</span> {<br/> <span style="color: #4b69c6">width</span>: <span style="color: #b60157">100</span><span style="color: #d73948">%</span>;<br/> <span style="color: #4b69c6">height</span>: <span style="color: #b60157">100</span><span style="color: #d73948">%</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">img</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">picture</span> <span style="color: #4b69c6">img</span> {<br/> <span style="color: #4b69c6">width</span>: <span style="color: #b60157">100</span><span style="color: #d73948">%</span>;<br/> <span style="color: #4b69c6">height</span>: <span style="color: #b60157">100</span><span style="color: #d73948">%</span>;<br/> <span style="color: #4b69c6">object-fit</span>: <span style="color: #4b69c6">cover</span>;<br/> <span style="color: #4b69c6">border-radius</span>: <span style="color: #4b69c6">var</span>(<span style="color: #4b69c6">--</span><span style="color: #4b69c6">image-radius</span>);<br/> <span style="color: #4b69c6">transition</span>: <span style="color: #4b69c6">transform</span> <span style="color: #4b69c6">var</span>(<span style="color: #4b69c6">--</span><span style="color: #4b69c6">gallery-transition</span>);<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">img</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">hover</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">picture</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">hover</span> <span style="color: #4b69c6">img</span> {<br/> <span style="color: #4b69c6">transform</span>: <span style="color: #4b69c6">scale</span>(<span style="color: #b60157">1</span><span style="color: #b60157">.</span><span style="color: #b60157">01</span>);<br/> <span style="color: #4b69c6">cursor</span>: <span style="color: #4b69c6">zoom-in</span>;<br/>}<br/><br/><span style="color: #74747c">/*</span><span style="color: #74747c"> Dynamic columns based on number of children </span><span style="color: #74747c">*/</span><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">1</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">1</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">1</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">2</span>)),<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">2</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">1</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">2</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>);<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">1</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">3</span>)),<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">2</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">2</span>)),<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">3</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">1</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">3</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>);<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #d73948">n</span> <span style="color: #b60157">+ 4</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #4b69c6">auto-fit</span>, <span style="color: #4b69c6">minmax</span>(<span style="color: #b60157">250</span><span style="color: #d73948">px</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>));<br/>}<br/><br/><span style="color: #74747c">/*</span><span style="color: #74747c"> Layout variants </span><span style="color: #74747c">*/</span><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">one-wide</span> {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #b60157">1</span><span style="color: #d73948">fr</span> <span style="color: #d73948">!important</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">two-wide</span> {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">2</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>) <span style="color: #d73948">!important</span>;<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">four-wide</span> {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #4b69c6">auto-fit</span>, <span style="color: #4b69c6">minmax</span>(<span style="color: #b60157">185</span><span style="color: #d73948">px</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>));<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">wide-first</span> &gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">first-child</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">wide-last</span> &gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">last-child</span> {<br/> <span style="color: #4b69c6">grid-column</span>: <span style="color: #b60157">1</span> / <span style="color: #b60157">-1</span>;<br/> <span style="color: #4b69c6">max-height</span>: <span style="color: #b60157">80</span><span style="color: #d73948">vh</span>;<br/>}<br/><br/><span style="color: #d73948">@</span><span style="color: #d73948">media</span> (<span style="color: #4b69c6">max-width</span>: <span style="color: #b60157">768</span><span style="color: #d73948">px</span>) {<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">1</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">3</span>)),<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">2</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">2</span>)),<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">3</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">1</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">2</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>);<br/> }<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">four-wide</span>,<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #d73948">n</span> <span style="color: #b60157">+ 4</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">2</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>);<br/> }<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #b60157">3</span>)<span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-last-child</span>(<span style="color: #b60157">1</span>)) &gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">last-child</span> {<br/> <span style="color: #4b69c6">grid-column</span>: <span style="color: #4b69c6">span</span> <span style="color: #b60157">2</span>;<br/> }<br/>}<br/><br/><span style="color: #d73948">@</span><span style="color: #d73948">media</span> (<span style="color: #4b69c6">max-width</span>: <span style="color: #b60157">480</span><span style="color: #d73948">px</span>) {<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>;<br/> }<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">four-wide</span>,<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">has</span>(&gt; <span style="color: #8b41b1">:</span><span style="color: #8b41b1">nth-child</span>(<span style="color: #d73948">n</span> <span style="color: #b60157">+ 4</span>)) {<br/> <span style="color: #4b69c6">grid-template-columns</span>: <span style="color: #4b69c6">repeat</span>(<span style="color: #b60157">2</span>, <span style="color: #b60157">1</span><span style="color: #d73948">fr</span>);<br/> }<br/><br/> <span style="color: #74747c">/*</span><span style="color: #74747c"> Disable hover effects on touch devices </span><span style="color: #74747c">*/</span><br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">img</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">hover</span>,<br/> <span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span> &gt; <span style="color: #4b69c6">picture</span><span style="color: #8b41b1">:</span><span style="color: #8b41b1">hover</span> <span style="color: #4b69c6">img</span> {<br/> <span style="color: #4b69c6">transform</span>: <span style="color: #4b69c6">none</span>;<br/> }<br/><br/>}<br/><br/><span style="color: #74747c">/*</span><span style="color: #74747c"> Full viewport width variant </span><span style="color: #74747c">*/</span><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">full-viewport</span> {<br/> <span style="color: #4b69c6">width</span>: <span style="color: #b60157">100</span><span style="color: #d73948">vw</span>;<br/> <span style="color: #4b69c6">margin-left</span>: <span style="color: #4b69c6">calc</span>(<span style="color: #b60157">50</span><span style="color: #d73948">%</span> <span style="color: #d73948">-</span> <span style="color: #b60157">50</span><span style="color: #d73948">vw</span>);<br/> <span style="color: #4b69c6">margin-right</span>: <span style="color: #4b69c6">calc</span>(<span style="color: #b60157">50</span><span style="color: #d73948">%</span> <span style="color: #d73948">-</span> <span style="color: #b60157">50</span><span style="color: #d73948">vw</span>);<br/>}<br/><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">full-viewport</span> <span style="color: #4b69c6">img</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">full-viewport</span> <span style="color: #4b69c6">picture</span> {<br/> <span style="color: #4b69c6">transform</span>: <span style="color: #4b69c6">none</span> <span style="color: #d73948">!important</span>;<br/> <span style="color: #4b69c6">border-radius</span>: <span style="color: #b60157">0</span>;<br/>}<br/><br/><span style="color: #74747c">/*</span><span style="color: #74747c"> Square aspect ratio class </span><span style="color: #74747c">*/</span><br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">square-items</span> &gt; <span style="color: #4b69c6">img</span>,<br/><span style="color: #8b41b1">.</span><span style="color: #8b41b1">gallery</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">square-items</span> &gt; <span style="color: #4b69c6">picture</span> <span style="color: #4b69c6">img</span> {<br/> aspect-ratio: <span style="color: #b60157">1</span>;<br/>}<br/></code></pre></div> 5:25pm on December 15, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“CSS for responsive image galleries at all viewport widths”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:88862f11-7c81-450f-b8b7-9aa0fd261e21 2024-12-15T10:27:56Z 2024-12-15T10:27:56Z <p>I’m including more and larger code snippets in documents throughout the site — such as this one — and I didn’t like that these were being counted amongst the ‘wordcount’ of the site. I’ve always distinguished between my words and reference words — when you see a quote in one of my documents, that isn’t counted in my own wordcount — but now it’s time do distinguish more granularly between <em>my prose</em> and <em>my code</em>.</p> <p>The first change was to the <code>SiteMetadata</code> dataclass.</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #301414">@</span><span style="color: #301414">dataclass</span><br/><span style="color: #d73948">class</span> <span style="color: #4b69c6">SiteMetadata</span>:<br/><span style="color: #d73948">...</span><br/> words: Dict[<span style="color: #4b69c6">str</span>, Any] <span style="color: #d73948">=</span> <span style="color: #4b69c6">field</span>(<br/> default_factory<span style="color: #d73948">=</span><span style="color: #d73948">lambda</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">drafts</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>: {<br/> <span style="color: #198810">"</span><span style="color: #198810">lines</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> <span style="color: #198810">"</span><span style="color: #198810">words</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> },<br/> <span style="color: #198810">"</span><span style="color: #198810">references</span><span style="color: #198810">"</span>: <span style="color: #b60157">0</span>,<br/> }<br/> )<br/><span style="color: #d73948">...</span></code></pre></div> <p>The same block was also added to the <code>DocumentMetadata</code> dataclass, just without the <code>drafts</code> key.</p> <p>The logic for tallying up the totals isn’t very elegant, I consider this a first draft, but it works. First the total document is summed,</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"> <span style="color: #d73948">for</span> key, page <span style="color: #d73948">in</span> documents.<span style="color: #4b69c6">items</span>():<br/> prose_wordcount <span style="color: #d73948">=</span> <span style="color: #4b69c6">len</span>(page.content.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>).<span style="color: #4b69c6">split</span>())<br/> references_wordcount <span style="color: #d73948">=</span> <span style="color: #4b69c6">len</span>(page.source.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">text</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>).<span style="color: #4b69c6">split</span>())<br/> <span style="color: #d73948">if</span> page.status <span style="color: #d73948">==</span> <span style="color: #198810">"</span><span style="color: #198810">draft</span><span style="color: #198810">"</span>:<br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> prose_wordcount<br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">drafts</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> prose_wordcount<br/> <span style="color: #d73948">else</span>:<br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> prose_wordcount<br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> prose_wordcount<br/><br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">references</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> references_wordcount<br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">references</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> references_wordcount<br/><br/> logger.<span style="color: #4b69c6">debug</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">key</span><span style="color: #8b41b1">}</span><span style="color: #198810">, </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">page</span><span style="color: #8b41b1">.</span><span style="color: #8b41b1">title</span><span style="color: #8b41b1">[</span><span style="color: #8b41b1">:</span><span style="color: #b60157">40</span><span style="color: #8b41b1">]</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)</code></pre></div> <p>And then, during the syntax highlighting, the code lines are tallied up and then subtracted from the prose lines.</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">def</span> <span style="color: #4b69c6">save_code_block</span>(match):<br/> leading_space <span style="color: #d73948">=</span> match.<span style="color: #4b69c6">group</span>(<span style="color: #b60157">1</span>)<br/> raw_html_marker <span style="color: #d73948">=</span> match.<span style="color: #4b69c6">group</span>(<span style="color: #b60157">2</span>)<br/> language <span style="color: #d73948">=</span> match.<span style="color: #4b69c6">group</span>(<span style="color: #b60157">3</span>)<br/> code <span style="color: #d73948">=</span> match.<span style="color: #4b69c6">group</span>(<span style="color: #b60157">4</span>).<span style="color: #4b69c6">rstrip</span>()<br/> trailing_space <span style="color: #d73948">=</span> match.<span style="color: #4b69c6">group</span>(<span style="color: #b60157">5</span>)<br/> code_words <span style="color: #d73948">=</span> <span style="color: #4b69c6">len</span>(code.<span style="color: #4b69c6">split</span>())<br/> code_lines <span style="color: #d73948">=</span> <span style="color: #4b69c6">len</span>(code.<span style="color: #4b69c6">splitlines</span>())<br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">lines</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> code_lines<br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">words</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> code_words<br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">lines</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> code_lines<br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">code</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">words</span><span style="color: #198810">"</span>] <span style="color: #d73948">+=</span> code_words<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Remove the wordcount of codeblocks from the prose wordcounts</span><br/> page.words[<span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>] <span style="color: #d73948">-=</span> code_words<br/> site.words[<span style="color: #198810">"</span><span style="color: #198810">self</span><span style="color: #198810">"</span>] <span style="color: #d73948">-=</span> code_words<br/><br/> <span style="color: #d73948">...</span></code></pre></div> <p>The result is exactly what I wanted, but the method isn’t super elegant.</p> <p>Including this post, the current counts are as follows,</p> <div class="code-block" data-lang="console"><div class="code-header"><span class="code-lang">console</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="console">222,852 words<br/><br/> 136,784 of my own published words<br/> 21,441 words of unpublished drafts<br/> 64,627 words of quotes and reference material<br/> 6,659 lines of code<br/> 24,182 words of code</code></pre></div> 10:27am on December 15, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Distinguishing between prose and code when counting words”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:12b3458c-71f1-497d-a431-74f0a7f79a60 2024-12-11T09:49:29Z 2024-12-11T09:49:29Z <p>The <a href="/stats" title="Stats | June 2022">stats page</a> has long shown a count of the number of unique external links referenced on the site. Before this change the stats showed 483 unique external links, the relevant code is contained in the snippet below,</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">def</span> <span style="color: #4b69c6">extract_external_links</span>(text: <span style="color: #4b69c6">str</span>) -&gt; List:<br/> url_pattern <span style="color: #d73948">=</span> <span style="color: #d73948">r</span><span style="color: #198810">"</span><span style="color: #198810">(</span><span style="color: #198810">https</span><span style="color: #d73948">?</span><span style="color: #198810">://</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #b60157">[</span><span style="color: #b60157">-</span><span style="color: #b60157">\w</span><span style="color: #b60157">.</span><span style="color: #b60157">]</span><span style="color: #d73948">|</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">%</span><span style="color: #b60157">[</span><span style="color: #b60157">\d</span><span style="color: #b60157">a-f</span><span style="color: #b60157">A-F</span><span style="color: #b60157">]</span><span style="color: #d73948">{2}</span><span style="color: #198810">)</span><span style="color: #198810">)</span><span style="color: #d73948">+</span><span style="color: #198810">(</span><span style="color: #198810">?:</span><span style="color: #198810">/</span><span style="color: #b60157">[</span><span style="color: #d73948">^</span><span style="color: #b60157">)</span><span style="color: #b60157">\s</span><span style="color: #b60157">]</span><span style="color: #d73948">*</span><span style="color: #198810">)</span><span style="color: #d73948">?</span><span style="color: #198810">)</span><span style="color: #198810">"</span><br/> matches <span style="color: #d73948">=</span> re.<span style="color: #4b69c6">findall</span>(url_pattern, text)<br/><br/> external_links <span style="color: #d73948">=</span> <span style="color: #4b69c6">set</span>()<br/> <span style="color: #d73948">for</span> url <span style="color: #d73948">in</span> matches:<br/> parsed_url <span style="color: #d73948">=</span> <span style="color: #4b69c6">urlparse</span>(url)<br/> <span style="color: #d73948">if</span> parsed_url.netloc.<span style="color: #4b69c6">lower</span>() <span style="color: #d73948">!=</span> <span style="color: #198810">"</span><span style="color: #198810">silasjelley.com</span><span style="color: #198810">"</span>:<br/> external_links.<span style="color: #4b69c6">add</span>(url)<br/><br/> <span style="color: #d73948">return</span> <span style="color: #4b69c6">list</span>(external_links)<br/><br/><span style="color: #d73948">...</span><br/><br/><span style="color: #d73948">try</span>:<br/> plain_text <span style="color: #d73948">=</span> document[<span style="color: #198810">"</span><span style="color: #198810">content</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>]<br/> external_links <span style="color: #d73948">=</span> <span style="color: #4b69c6">extract_external_links</span>(plain_text)<br/><br/> document[<span style="color: #198810">"</span><span style="color: #198810">links</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>].<span style="color: #4b69c6">update</span>(external_links)<br/> site.links[<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>].<span style="color: #4b69c6">update</span>(external_links)<br/><span style="color: #d73948">except</span> <span style="color: #4b69c6">KeyError</span>:<br/> <span style="color: #d73948">pass</span></code></pre></div> <p>Up to now I’ve only included links within the main body of a page (<code>content["plain"]</code>), but my document format has evolved since then with more and more links being structured into the <code>[source]</code> and <code>[via]</code> tables.</p> <p>First I wanted to grasp what kind of a difference this would make so, as usual, I started with a shell pipeline search through the output of the build script,</p> <div class="code-block" data-lang="sh"><div class="code-header"><span class="code-lang">sh</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="sh"><span style="color: #4b69c6">rg</span> --no-filename \<br/> --only-matching \<br/> --no-line-number \<br/> <span style="color: #198810">'</span><span style="color: #198810">http.*?://[^ \[\]"&amp;&lt;&gt;)]*</span><span style="color: #198810">'</span> \<br/> <span style="color: #d73948">*</span><span style="color: #d73948">*</span>/<span style="color: #d73948">*</span>.html <span style="color: #d73948">|</span> \<br/> <span style="color: #4b69c6">sed</span> \<br/> -e <span style="color: #198810">"</span><span style="color: #198810">s/'.*//</span><span style="color: #198810">"</span> \<br/> -e <span style="color: #198810">'</span><span style="color: #198810">/silasjelley\.com/d</span><span style="color: #198810">'</span> \<br/> -e <span style="color: #198810">'</span><span style="color: #198810">s,/$,,</span><span style="color: #198810">'</span> \<br/> -e <span style="color: #198810">'</span><span style="color: #198810">s/#.*//</span><span style="color: #198810">'</span> <span style="color: #d73948">|</span> \<br/> <span style="color: #4b69c6">sort</span> <span style="color: #d73948">|</span> \<br/> <span style="color: #4b69c6">uniq</span> <span style="color: #d73948">|</span> \<br/> <span style="color: #4b69c6">wc</span> -l</code></pre></div> <p>The regex for link matching was pretty coarse so I tidied/normalised the output and removed internal links with some <code>sed</code> patterns, before sorting, de-duping, and counting (<code> sort | uniq | wc -l </code>).</p> <p>The result: 907 unique external links, almost double the 483 contained purely in the main body content.</p> <p>Now that number is slightly inflated as it includes links to external APIs such as <code>https://api.maptiler.com/maps/topo-v2/{z}/{x}/{y}.png?key=APIKEY</code> for <a href="/map" title="Map of my world | July 2023">my walk map</a>, so our final number should be slightly lower.</p> <p>This change includes the <code>[source]</code> and <code>[via]</code> tables when searching for links,</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">try</span>:<br/> plain_text <span style="color: #d73948">=</span> document.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">content</span><span style="color: #198810">"</span>, {}).<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">plain</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>) <span style="color: #d73948">+</span> <span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #198810">"</span><br/> plain_text <span style="color: #d73948">+=</span> document.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">source</span><span style="color: #198810">"</span>, {}).<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">url</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>) <span style="color: #d73948">+</span> <span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #198810">"</span><br/> plain_text <span style="color: #d73948">+=</span> document.<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">via</span><span style="color: #198810">"</span>, {}).<span style="color: #4b69c6">get</span>(<span style="color: #198810">"</span><span style="color: #198810">url</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">"</span>) <span style="color: #d73948">+</span> <span style="color: #198810">"</span><span style="color: #198810"> </span><span style="color: #198810">"</span><br/><br/> external_links <span style="color: #d73948">=</span> <span style="color: #4b69c6">extract_external_links</span>(plain_text)<br/><br/> document[<span style="color: #198810">"</span><span style="color: #198810">links</span><span style="color: #198810">"</span>][<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>].<span style="color: #4b69c6">update</span>(external_links)<br/> site.links[<span style="color: #198810">"</span><span style="color: #198810">external</span><span style="color: #198810">"</span>].<span style="color: #4b69c6">update</span>(external_links)<br/><span style="color: #d73948">except</span> <span style="color: #4b69c6">KeyError</span>:<br/> <span style="color: #4b69c6">print</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">KeyError while compiling external links from </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">document</span><span style="color: #8b41b1">[</span><span style="color: #198810">'</span><span style="color: #198810">filename</span><span style="color: #198810">'</span><span style="color: #8b41b1">]</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">pass</span></code></pre></div> <p>After the change, the stats reflect 869 unique external links.</p> 9:49am on December 11, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Include all external URLs in site statistics”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:5f39dd4e-8ae8-47d3-a3ff-120f9ce3d0c9 2024-12-04T18:16:03Z 2024-12-04T18:16:03Z <p>Up to now, image references, eg. <code>#link("img:​:5f39dd4e")[::ImageAlt]</code> have functioned like so:</p> <ul> <li>The source section (between parentheses) is substituted directly from the asset manifest, in this case referenced by the first 8-chars of its UUIDv4.</li> <li>If the Alt-text [between square brackets] begins with ‘::’, it is also replaced by the description from the corresponding asset manifest, else it is left as is.</li> </ul> <p>This has worked well, but I’ve been wanting to add AVIF support in order to take advantage of its better compression performance. AVIF is not universally supported, so a fallback (eg. JPEG) is necessary. The HTML <code>\&lt;picture&gt;</code> element is well suited to this. The Djot image syntax has been bypassed, image references are now directly replaced with a final HTML snippet like so,</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python">replacement <span style="color: #d73948">=</span> <span style="color: #d73948">f</span><span style="color: #198810">"""</span><span style="color: #198810">`​`​`=html</span><br/><span style="color: #198810">\&lt;</span><span style="color: #198810">picture&gt;</span><br/><span style="color: #198810">\&lt;</span><span style="color: #198810">source srcset="</span><span style="color: #8b41b1">{</span><span style="color: #4b69c6">Path</span><span style="color: #8b41b1">(</span><span style="color: #8b41b1">ref_slug</span><span style="color: #8b41b1">)</span><span style="color: #8b41b1">.</span><span style="color: #4b69c6">with_suffix</span><span style="color: #8b41b1">(</span><span style="color: #198810">"</span><span style="color: #198810">.avif</span><span style="color: #198810">"</span><span style="color: #8b41b1">)</span><span style="color: #8b41b1">}</span><span style="color: #198810">" type="image/avif" /&gt;</span><br/><span style="color: #198810">\&lt;</span><span style="color: #198810">img alt="</span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">ref_text</span><span style="color: #8b41b1">}</span><span style="color: #198810">" src="</span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">ref_slug</span><span style="color: #8b41b1">}</span><span style="color: #198810">"&gt;</span><br/><span style="color: #198810">\&lt;</span><span style="color: #198810">/picture&gt;</span><br/><span style="color: #198810">`​`​`</span><span style="color: #198810">"""</span></code></pre></div> <p>The actual AVIF creation support was added using <code>pillow_avif-plugin</code>, a plugin for the Pillow library I was already using for compressing JPEGs. Most of the relevant code is in the <code>process_image_parallel</code> function,</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">def</span> <span style="color: #4b69c6">process_image_parallel</span>(input_data: Tuple[Path, Path, <span style="color: #4b69c6">int</span>, <span style="color: #4b69c6">str</span>]) -&gt; <span style="color: #d73948">None</span>:<br/> workaround_import <span style="color: #d73948">=</span> pillow_avif.AvifImagePlugin<br/> input_image, output_path, output_width, uid <span style="color: #d73948">=</span> input_data<br/> lock_path <span style="color: #d73948">=</span> output_path.<span style="color: #4b69c6">with_suffix</span>(<span style="color: #198810">"</span><span style="color: #198810">.lock</span><span style="color: #198810">"</span>)<br/> lock <span style="color: #d73948">=</span> <span style="color: #4b69c6">FileLock</span>(<span style="color: #4b69c6">str</span>(lock_path))<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Define AVIF output path</span><br/> avif_output_path <span style="color: #d73948">=</span> output_path.<span style="color: #4b69c6">with_suffix</span>(<span style="color: #198810">"</span><span style="color: #198810">.avif</span><span style="color: #198810">"</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Check if AVIF support is available</span><br/> avif_available <span style="color: #d73948">=</span> <span style="color: #198810">"</span><span style="color: #198810">AVIF</span><span style="color: #198810">"</span> <span style="color: #d73948">in</span> Image.SAVE<br/><br/> <span style="color: #d73948">try</span>:<br/> <span style="color: #d73948">with</span> lock:<br/> <span style="color: #d73948">if</span> output_path.<span style="color: #4b69c6">exists</span>() <span style="color: #d73948">and</span> avif_output_path.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">return</span><br/><br/> os.<span style="color: #4b69c6">makedirs</span>(output_path.parent, exist_ok<span style="color: #d73948">=</span><span style="color: #d73948">True</span>)<br/><br/> <span style="color: #d73948">with</span> Image.<span style="color: #4b69c6">open</span>(input_image) <span style="color: #d73948">as</span> im:<br/> original_format <span style="color: #d73948">=</span> im.format<br/> im <span style="color: #d73948">=</span> ImageOps.<span style="color: #4b69c6">exif_transpose</span>(im)<br/> output_height <span style="color: #d73948">=</span> <span style="color: #4b69c6">int</span>(im.size[<span style="color: #b60157">1</span>] <span style="color: #d73948">*</span> (output_width <span style="color: #d73948">/</span> im.size[<span style="color: #b60157">0</span>]))<br/><br/> <span style="color: #d73948">with</span> im.<span style="color: #4b69c6">resize</span>(<br/> (output_width, output_height), Image.Resampling.LANCZOS<br/> ) <span style="color: #d73948">as</span> output_image:<br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Save JPEG version</span><br/> <span style="color: #d73948">if</span> (<br/> original_format <span style="color: #d73948">!=</span> <span style="color: #198810">"</span><span style="color: #198810">JPEG</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">and</span> <span style="color: #4b69c6">str</span>(output_path).<span style="color: #4b69c6">endswith</span>(<span style="color: #198810">"</span><span style="color: #198810">jpg</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">and</span> output_image.mode <span style="color: #d73948">in</span> (<span style="color: #198810">"</span><span style="color: #198810">RGBA</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">P</span><span style="color: #198810">"</span>)<br/> ):<br/> output_image <span style="color: #d73948">=</span> output_image.<span style="color: #4b69c6">convert</span>(<span style="color: #198810">"</span><span style="color: #198810">RGB</span><span style="color: #198810">"</span>)<br/><br/> output_image.<span style="color: #4b69c6">save</span>(output_path, quality<span style="color: #d73948">=</span><span style="color: #b60157">85</span>, optimize<span style="color: #d73948">=</span><span style="color: #d73948">True</span>)<br/><br/> <span style="color: #74747c">#</span><span style="color: #74747c"> Save AVIF version only if support is available</span><br/> <span style="color: #d73948">if</span> avif_available:<br/> <span style="color: #d73948">try</span>:<br/> <span style="color: #d73948">if</span> output_image.mode <span style="color: #d73948">in</span> (<span style="color: #198810">"</span><span style="color: #198810">RGBA</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">P</span><span style="color: #198810">"</span>):<br/> avif_image <span style="color: #d73948">=</span> output_image.<span style="color: #4b69c6">convert</span>(<span style="color: #198810">"</span><span style="color: #198810">RGB</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">else</span>:<br/> avif_image <span style="color: #d73948">=</span> output_image.<span style="color: #4b69c6">copy</span>()<br/><br/> avif_image.<span style="color: #4b69c6">save</span>(<br/> avif_output_path,<br/> format<span style="color: #d73948">=</span><span style="color: #198810">"</span><span style="color: #198810">AVIF</span><span style="color: #198810">"</span>,<br/> quality<span style="color: #d73948">=</span><span style="color: #b60157">60</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Lower quality for better compression, still maintains good visual quality</span><br/> speed<span style="color: #d73948">=</span><span style="color: #b60157">5</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Slowest speed = best compression (0 is slowest, 10 is fastest)</span><br/> bits<span style="color: #d73948">=</span><span style="color: #b60157">10</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Use 10-bit color depth for better quality-to-size ratio</span><br/> compress_level<span style="color: #d73948">=</span><span style="color: #b60157">8</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Highest compression level (range 0-8)</span><br/> color_space<span style="color: #d73948">=</span><span style="color: #198810">"</span><span style="color: #198810">bt709</span><span style="color: #198810">"</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Use YUV BT.709 color space</span><br/> chroma<span style="color: #d73948">=</span><span style="color: #b60157">0</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> 4:4:4 chroma sampling (0=4:4:4, 1=4:2:0, 2=4:2:2)</span><br/> num_threads<span style="color: #d73948">=</span><span style="color: #b60157">0</span>, <span style="color: #74747c">#</span><span style="color: #74747c"> Use all available CPU threads for encoding</span><br/> )<br/> logger.<span style="color: #4b69c6">debug</span>(<br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Processed image: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810"> -&gt; </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">output_path</span><span style="color: #8b41b1">}</span><span style="color: #198810"> and </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">avif_output_path</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span><br/> )<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">Exception</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">error</span>(<br/> <span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Error saving AVIF version of </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810">: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span><br/> )<br/> <span style="color: #d73948">else</span>:<br/> logger.<span style="color: #4b69c6">error</span>(<br/> <span style="color: #198810">"</span><span style="color: #198810">AVIF support not available. Skipping AVIF conversion.</span><span style="color: #198810">"</span><br/> )<br/> logger.<span style="color: #4b69c6">debug</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Processed image: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810"> -&gt; </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">output_path</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/><br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">OSError</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">error</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">OS error processing </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810">: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">Exception</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">error</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Error processing </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810">: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">finally</span>:<br/> <span style="color: #d73948">if</span> lock_path.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">try</span>:<br/> lock_path.<span style="color: #4b69c6">unlink</span>()<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">OSError</span>:<br/> <span style="color: #d73948">pass</span></code></pre></div> <p>It was also necessary to update my gallery/lightbox JavaScript code, as the previous version was loading the JPEG fallback when the lightbox opened, wasting bandwidth.</p> <div class="code-block" data-lang="js"><div class="code-header"><span class="code-lang">js</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="js"><span style="color: #d73948">class</span> <span style="color: #4b69c6">Lightbox</span> {<br/> <span style="color: #4b69c6">constructor</span>(galleryElement) {<br/> <span style="color: #d73948">this</span>.galleryElement <span style="color: #d73948">=</span> galleryElement;<br/> <span style="color: #d73948">this</span>.galleryItems <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.galleryElement.<span style="color: #4b69c6">querySelectorAll</span>(<span style="color: #198810">'</span><span style="color: #198810">img, picture</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">=</span> <span style="color: #b60157">0</span>;<br/> <span style="color: #d73948">this</span>.overlay <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.content <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.image <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.caption <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.closeButton <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.prevButton <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.nextButton <span style="color: #d73948">=</span> <span style="color: #d73948">null</span>;<br/> <span style="color: #d73948">this</span>.isOpen <span style="color: #d73948">=</span> <span style="color: #d73948">false</span>;<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">init</span>();<br/> <span style="color: #d73948">this</span>.imageCount <span style="color: #d73948">=</span> galleryElement.<span style="color: #4b69c6">querySelectorAll</span>(<span style="color: #198810">'</span><span style="color: #198810">img</span><span style="color: #198810">'</span>).length;<br/> }<br/><br/> <span style="color: #4b69c6">init</span>() {<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">createLightbox</span>();<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">addEventListeners</span>();<br/> }<br/><br/> <span style="color: #4b69c6">createLightbox</span>() {<br/> <span style="color: #d73948">this</span>.overlay <span style="color: #d73948">=</span> <span style="color: #4b69c6">document</span>.<span style="color: #4b69c6">createElement</span>(<span style="color: #198810">'</span><span style="color: #198810">div</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.overlay.className <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">lightbox-overlay</span><span style="color: #198810">'</span>;<br/> <span style="color: #d73948">this</span>.overlay.innerHTML <span style="color: #d73948">=</span> <span style="color: #198810">`</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">div class="lightbox-content"&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">img src="" alt="" class="lightbox-image"&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">p class="lightbox-caption"&gt;</span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">/p&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">button class="lightbox-prev"&gt;&amp;lt;</span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">/button&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">button class="lightbox-next"&gt;&amp;gt;</span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">/button&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">/div&gt;</span><br/><span style="color: #198810"> </span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">button class="lightbox-close"&gt;&amp;times;</span><span style="color: #1d6c76">\&lt;</span><span style="color: #198810">/button&gt;</span><br/><span style="color: #198810"> </span><span style="color: #198810">`</span>;<br/> <span style="color: #4b69c6">document</span>.body.<span style="color: #4b69c6">appendChild</span>(<span style="color: #d73948">this</span>.overlay);<br/><br/> <span style="color: #d73948">this</span>.content <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-content</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.image <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-image</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.caption <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-caption</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.closeButton <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-close</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.prevButton <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-prev</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">this</span>.nextButton <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">.lightbox-next</span><span style="color: #198810">'</span>);<br/> }<br/><br/> <span style="color: #4b69c6">getImageDetails</span>(element) {<br/> <span style="color: #d73948">if</span> (element.tagName.<span style="color: #4b69c6">toLowerCase</span>() <span style="color: #d73948">===</span> <span style="color: #198810">'</span><span style="color: #198810">picture</span><span style="color: #198810">'</span>) {<br/> <span style="color: #d73948">const</span> img <span style="color: #d73948">=</span> element.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">img</span><span style="color: #198810">'</span>);<br/> <span style="color: #d73948">return</span> {<br/> <span style="color: #74747c">//</span><span style="color: #74747c"> currentSrc gives us the source that the browser actually loaded</span><br/> src: img.currentSrc <span style="color: #d73948">||</span> img.src,<br/> alt: img.alt<br/> };<br/> } <span style="color: #d73948">else</span> {<br/> <span style="color: #d73948">return</span> {<br/> src: element.currentSrc <span style="color: #d73948">||</span> element.src,<br/> alt: element.alt<br/> };<br/> }<br/> }<br/><br/> <span style="color: #4b69c6">addEventListeners</span>() {<br/> <span style="color: #d73948">this</span>.galleryItems.<span style="color: #4b69c6">forEach</span>((item, index) <span style="color: #d73948">=&gt;</span> {<br/> <span style="color: #74747c">//</span><span style="color: #74747c"> For picture elements, we need to attach the listener to the img inside</span><br/> <span style="color: #d73948">const</span> clickTarget <span style="color: #d73948">=</span> item.tagName.<span style="color: #4b69c6">toLowerCase</span>() <span style="color: #d73948">===</span> <span style="color: #198810">'</span><span style="color: #198810">picture</span><span style="color: #198810">'</span><br/> <span style="color: #d73948">?</span> item.<span style="color: #4b69c6">querySelector</span>(<span style="color: #198810">'</span><span style="color: #198810">img</span><span style="color: #198810">'</span>)<br/> <span style="color: #d73948">:</span> item;<br/><br/> clickTarget.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">click</span><span style="color: #198810">'</span>, (e) <span style="color: #d73948">=&gt;</span> {<br/> e.<span style="color: #4b69c6">preventDefault</span>();<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">openLightbox</span>(index);<br/> });<br/> });<br/><br/> <span style="color: #d73948">this</span>.overlay.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">click</span><span style="color: #198810">'</span>, (e) <span style="color: #d73948">=&gt;</span> {<br/> <span style="color: #d73948">if</span> (<span style="color: #d73948">!</span><span style="color: #d73948">this</span>.content.<span style="color: #4b69c6">contains</span>(e.target) <span style="color: #d73948">||</span> e.target <span style="color: #d73948">===</span> <span style="color: #d73948">this</span>.content) {<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">closeLightbox</span>();<br/> }<br/> });<br/><br/> <span style="color: #d73948">this</span>.closeButton.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">click</span><span style="color: #198810">'</span>, () <span style="color: #d73948">=&gt;</span> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">closeLightbox</span>());<br/><br/> <span style="color: #4b69c6">document</span>.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">keydown</span><span style="color: #198810">'</span>, (e) <span style="color: #d73948">=&gt;</span> {<br/> <span style="color: #d73948">if</span> (<span style="color: #d73948">this</span>.isOpen) {<br/> <span style="color: #d73948">if</span> (e.key <span style="color: #d73948">===</span> <span style="color: #198810">'</span><span style="color: #198810">Escape</span><span style="color: #198810">'</span>) <span style="color: #d73948">this</span>.<span style="color: #4b69c6">closeLightbox</span>();<br/> <span style="color: #d73948">if</span> (e.key <span style="color: #d73948">===</span> <span style="color: #198810">'</span><span style="color: #198810">ArrowLeft</span><span style="color: #198810">'</span>) <span style="color: #d73948">this</span>.<span style="color: #4b69c6">showPrevious</span>();<br/> <span style="color: #d73948">if</span> (e.key <span style="color: #d73948">===</span> <span style="color: #198810">'</span><span style="color: #198810">ArrowRight</span><span style="color: #198810">'</span>) <span style="color: #d73948">this</span>.<span style="color: #4b69c6">showNext</span>();<br/> }<br/> });<br/><br/> <span style="color: #4b69c6">window</span>.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">popstate</span><span style="color: #198810">'</span>, () <span style="color: #d73948">=&gt;</span> {<br/> <span style="color: #d73948">if</span> (<span style="color: #d73948">this</span>.isOpen) {<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">closeLightbox</span>(<span style="color: #d73948">false</span>);<br/> }<br/> });<br/><br/> <span style="color: #d73948">this</span>.prevButton.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">click</span><span style="color: #198810">'</span>, () <span style="color: #d73948">=&gt;</span> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">showPrevious</span>());<br/> <span style="color: #d73948">this</span>.nextButton.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">click</span><span style="color: #198810">'</span>, () <span style="color: #d73948">=&gt;</span> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">showNext</span>());<br/> }<br/><br/> <span style="color: #4b69c6">openLightbox</span>(index) {<br/> <span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">=</span> index;<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">updateLightboxContent</span>();<br/> <span style="color: #d73948">this</span>.overlay.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">block</span><span style="color: #198810">'</span>;<br/> <span style="color: #4b69c6">document</span>.body.style.overflow <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">hidden</span><span style="color: #198810">'</span>;<br/> <span style="color: #d73948">this</span>.isOpen <span style="color: #d73948">=</span> <span style="color: #d73948">true</span>;<br/> history.<span style="color: #4b69c6">pushState</span>({ lightboxOpen: <span style="color: #d73948">true</span> }, <span style="color: #198810">'</span><span style="color: #198810">'</span>);<br/> }<br/><br/> <span style="color: #4b69c6">closeLightbox</span>(pushState <span style="color: #d73948">=</span> <span style="color: #d73948">true</span>) {<br/> <span style="color: #d73948">this</span>.overlay.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">none</span><span style="color: #198810">'</span>;<br/> <span style="color: #4b69c6">document</span>.body.style.overflow <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">'</span>;<br/> <span style="color: #d73948">this</span>.isOpen <span style="color: #d73948">=</span> <span style="color: #d73948">false</span>;<br/> <span style="color: #d73948">if</span> (pushState) {<br/> history.<span style="color: #4b69c6">pushState</span>({ lightboxOpen: <span style="color: #d73948">false</span> }, <span style="color: #198810">'</span><span style="color: #198810">'</span>);<br/> }<br/> }<br/><br/><span style="color: #4b69c6">updateLightboxContent</span>() {<br/> <span style="color: #d73948">const</span> currentItem <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.galleryItems[<span style="color: #d73948">this</span>.currentIndex];<br/> <span style="color: #d73948">const</span> { src, alt } <span style="color: #d73948">=</span> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">getImageDetails</span>(currentItem);<br/><br/> <span style="color: #d73948">this</span>.image.src <span style="color: #d73948">=</span> src;<br/> <span style="color: #d73948">this</span>.image.alt <span style="color: #d73948">=</span> alt;<br/> <span style="color: #d73948">this</span>.caption.textContent <span style="color: #d73948">=</span> alt;<br/><br/> <span style="color: #74747c">//</span><span style="color: #74747c"> Use imageCount instead of galleryItems.length</span><br/> <span style="color: #d73948">if</span> (<span style="color: #d73948">this</span>.imageCount <span style="color: #d73948">&lt;=</span> <span style="color: #b60157">1</span>) {<br/> <span style="color: #d73948">this</span>.prevButton.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">none</span><span style="color: #198810">'</span>;<br/> <span style="color: #d73948">this</span>.nextButton.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">none</span><span style="color: #198810">'</span>;<br/> } <span style="color: #d73948">else</span> {<br/> <span style="color: #d73948">this</span>.prevButton.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">block</span><span style="color: #198810">'</span>;<br/> <span style="color: #d73948">this</span>.nextButton.style.display <span style="color: #d73948">=</span> <span style="color: #198810">'</span><span style="color: #198810">block</span><span style="color: #198810">'</span>;<br/> }<br/>}<br/><br/> <span style="color: #4b69c6">showPrevious</span>() {<br/> <span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">=</span> (<span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">-</span> <span style="color: #b60157">1</span> <span style="color: #d73948">+</span> <span style="color: #d73948">this</span>.galleryItems.length) <span style="color: #d73948">%</span> <span style="color: #d73948">this</span>.galleryItems.length;<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">updateLightboxContent</span>();<br/> }<br/><br/> <span style="color: #4b69c6">showNext</span>() {<br/> <span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">=</span> (<span style="color: #d73948">this</span>.currentIndex <span style="color: #d73948">+</span> <span style="color: #b60157">1</span>) <span style="color: #d73948">%</span> <span style="color: #d73948">this</span>.galleryItems.length;<br/> <span style="color: #d73948">this</span>.<span style="color: #4b69c6">updateLightboxContent</span>();<br/> }<br/>}<br/><br/><span style="color: #74747c">//</span><span style="color: #74747c"> Initialize the lightbox for each gallery</span><br/><span style="color: #4b69c6">document</span>.<span style="color: #4b69c6">addEventListener</span>(<span style="color: #198810">'</span><span style="color: #198810">DOMContentLoaded</span><span style="color: #198810">'</span>, () <span style="color: #d73948">=&gt;</span> {<br/> <span style="color: #d73948">const</span> galleries <span style="color: #d73948">=</span> <span style="color: #4b69c6">document</span>.<span style="color: #4b69c6">querySelectorAll</span>(<span style="color: #198810">'</span><span style="color: #198810">.gallery</span><span style="color: #198810">'</span>);<br/> galleries.<span style="color: #4b69c6">forEach</span>(gallery <span style="color: #d73948">=&gt;</span> <span style="color: #d73948">new</span> Lightbox(gallery));<br/>});</code></pre></div> <p>The result: higher image quality (raised image default width from 1400px to 1600px) and <strong>half</strong> the file-size, eg. <a href="/nonsense" title="Some of the nonsense from my old loaf | May 2022">my nonsense page</a> has been cut from 7.8MB transferred to load, to 3.45MB. Or, to compare only the actual images, 6.8MB to 2.94MB, a 57% saving. At higher quality! Madness.</p> <p>Coupled with the previous change —<span style="white-space: pre-wrap"> </span><a href="/2024/11/17/175502" title="Convert heavy PNGs to JPEG where appropriate | November 2024">Convert heavy PNGs to JPEG where appropriate</a> — that page has been cut from 15.9 to 3.45MB. Happy with that.</p> <aside class="aside "> <p>Small problem with the new image reference solution: classes specified in Djot are no longer passed through as the Djot rendering is bypassed. Most images on the site are wrapped in a ‘gallery’ div, so I can still class that, but it does create a small annoyance for bare images.</p> <time datetime="2024-12-05 10:08:23" class="smallcaps"><a href="/2024/12/05/100823">December 5, 2024 10.08AM ⚕ Vancouver, Canada</a></time> </aside> 6:16pm on December 4, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Switch image references to use picture elements and AVIF encoded files”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:8d5230e3-f7ce-43c0-9fe4-e79b2071fd2b 2024-11-19T20:31:27Z 2024-11-19T20:31:27Z <p>A very minor change to resolve a bit of acute visual nit-picking. The space in eg, ‘p. 56’, ‘Ch. 7’, ‘Vol. 9’, or ‘(3rd edition)’, created by my attribution logic, could cause the two parts to appear on separate lines in a long attribution string or a narrow viewport. Switching to a non-breaking space ‘ ‘ (visually indistinguishable from a normal space) means they will always appear side by side.</p> <p>Unicode codepoint U+00A0, commonly abbreviated as NBSP.</p> 8:31pm on November 19, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Switched to non-breaking spaces in Chapter, Volume, Edition, and Page attribution”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:5f25baf8-9305-4a85-af66-6ae111ee7cdd 2024-11-17T17:55:02Z 2024-11-17T17:55:02Z <p>While viewing my <a href="/nonsense" title="Some of the nonsense from my old loaf | May 2022">nonsense page</a> with the network tab of the developer tools open I noticed the page was transferring a total of 15.8MB to load. The page loads hundreds of posts so this isn’t completely absurd, but it felt excessive. Turns out most of the weight was coming from a handful of PNG posters and screenshots.</p> <p>In the general case, JPEGs are smaller than PNGs, but I didn’t want to hardcode conversion of all PNGs because there are times when lossless or transparent images are wanted.</p> <p>I toyed around with a couple of implementations. My first prototype added a <code>convert_to</code> key to the asset manifests, but in the end I decided to simply convert on the basis of the output_path suffix being different than the input_path suffix in a manifest entry, allowing every stage of the build to seamlessly pick up the correct path to reference.</p> <p>Switching those few heavy-weight PNGs to JPEG output cut the page weight from 15.8MB down to 5.18MB.</p> <p>The most egregious offender was <a href="/2022/06/10/212140" title="Princess Mononoke, my first Ghibli">Princess Mononoke, my first Ghibli</a>“) coming in at 2.8MB initially, down to 240kB after conversion to JPEG. Shown below, the only change required in a manifest entry is to change the suffix/file-extension.</p> <div class="code-block" data-lang="toml"><div class="code-header"><span class="code-lang">toml</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="toml">[<span style="color: #4b69c6">27991f33-f40f-48ab-9690-0003db7b2038</span>]<br/><span style="color: #4b69c6">type</span> = <span style="color: #198810">"</span><span style="color: #198810">poster</span><span style="color: #198810">"</span><br/><span style="color: #4b69c6">title</span> = <span style="color: #198810">"</span><span style="color: #198810">The poster for the film Blade Runner (1982)</span><span style="color: #198810">"</span><br/><span style="color: #4b69c6">filepath</span> = <span style="color: #198810">"</span><span style="color: #198810">/home/silas/library/images/posters/films/1982_Blade-Runner.png</span><span style="color: #198810">"</span><br/><span style="color: #4b69c6">slug</span> = <span style="color: #198810">"</span><span style="color: #198810">library/images/posters/films/1982_Blade-Runner.jpg</span><span style="color: #198810">"</span></code></pre></div> <p>The only change I needed to make to the build script was to check the colour-space when converting to JPEG and switch it to RGB in the case that the source colour-space was either ‘RGBA’ or ‘P’:</p> <div class="code-block" data-lang="python"><div class="code-header"><span class="code-lang">python</span><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code data-lang="python"><span style="color: #d73948">def</span> <span style="color: #4b69c6">process_image_parallel</span>(input_data: Tuple[Path, Path, <span style="color: #4b69c6">int</span>]) -&gt; <span style="color: #d73948">None</span>:<br/> input_image, output_path, output_width <span style="color: #d73948">=</span> input_data<br/> lock_path <span style="color: #d73948">=</span> output_path.<span style="color: #4b69c6">with_suffix</span>(<span style="color: #198810">"</span><span style="color: #198810">.lock</span><span style="color: #198810">"</span>)<br/> lock <span style="color: #d73948">=</span> <span style="color: #4b69c6">FileLock</span>(<span style="color: #4b69c6">str</span>(lock_path))<br/><br/> <span style="color: #d73948">try</span>:<br/> <span style="color: #d73948">with</span> lock:<br/> <span style="color: #d73948">if</span> output_path.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">return</span><br/><br/> os.<span style="color: #4b69c6">makedirs</span>(output_path.parent, exist_ok<span style="color: #d73948">=</span><span style="color: #d73948">True</span>)<br/><br/> <span style="color: #d73948">with</span> Image.<span style="color: #4b69c6">open</span>(input_image) <span style="color: #d73948">as</span> im:<br/> original_format <span style="color: #d73948">=</span> im.format<br/> im <span style="color: #d73948">=</span> ImageOps.<span style="color: #4b69c6">exif_transpose</span>(im)<br/> output_height <span style="color: #d73948">=</span> <span style="color: #4b69c6">int</span>(im.size[<span style="color: #b60157">1</span>] <span style="color: #d73948">*</span> (output_width <span style="color: #d73948">/</span> im.size[<span style="color: #b60157">0</span>]))<br/><br/> <span style="color: #d73948">with</span> im.<span style="color: #4b69c6">resize</span>(<br/> (output_width, output_height), Image.Resampling.LANCZOS<br/> ) <span style="color: #d73948">as</span> output_image:<br/> <span style="color: #d73948">if</span> (<br/> original_format <span style="color: #d73948">!=</span> <span style="color: #198810">"</span><span style="color: #198810">JPEG</span><span style="color: #198810">"</span><br/> <span style="color: #d73948">and</span> <span style="color: #4b69c6">str</span>(output_path).<span style="color: #4b69c6">endswith</span>(<span style="color: #198810">"</span><span style="color: #198810">jpg</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">and</span> output_image.mode <span style="color: #d73948">in</span> (<span style="color: #198810">"</span><span style="color: #198810">RGBA</span><span style="color: #198810">"</span>, <span style="color: #198810">"</span><span style="color: #198810">P</span><span style="color: #198810">"</span>)<br/> ):<br/> output_image <span style="color: #d73948">=</span> output_image.<span style="color: #4b69c6">convert</span>(<span style="color: #198810">"</span><span style="color: #198810">RGB</span><span style="color: #198810">"</span>)<br/><br/> output_image.<span style="color: #4b69c6">save</span>(output_path, quality<span style="color: #d73948">=</span><span style="color: #b60157">85</span>, optimize<span style="color: #d73948">=</span><span style="color: #d73948">True</span>)<br/><br/> logger.<span style="color: #4b69c6">debug</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Processed image: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810"> -&gt; </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">output_path</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/><br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">OSError</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">error</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">OS error processing </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810">: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">Exception</span> <span style="color: #d73948">as</span> e:<br/> logger.<span style="color: #4b69c6">error</span>(<span style="color: #d73948">f</span><span style="color: #198810">"</span><span style="color: #198810">Error processing </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">input_image</span><span style="color: #8b41b1">}</span><span style="color: #198810">: </span><span style="color: #8b41b1">{</span><span style="color: #8b41b1">e</span><span style="color: #8b41b1">}</span><span style="color: #198810">"</span>)<br/> <span style="color: #d73948">finally</span>:<br/> <span style="color: #d73948">if</span> lock_path.<span style="color: #4b69c6">exists</span>():<br/> <span style="color: #d73948">try</span>:<br/> lock_path.<span style="color: #4b69c6">unlink</span>()<br/> <span style="color: #d73948">except</span> <span style="color: #4b69c6">OSError</span>:<br/> <span style="color: #d73948">pass</span></code></pre></div> 5:55pm on November 17, 2024 from Vancouver, British Columbia, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Convert heavy PNGs to JPEG where appropriate”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:5a0d3cbb-f865-49ce-8b97-2a33a61296f8 2024-11-15T13:27:17Z 2024-11-15T13:27:17Z <p>Added import references, restoring and improving the functionality of the ‘transclude’ reference that I previously deprecated and removed.</p> <p>Has two modes, both of which support comments.</p> <ul> <li><code>import</code>, transcludes the content of the referenced document as is, without any additional markup or visual cue.</li> <li><code>aside</code>, creates an aside-type block that also includes an outbound link.</li> </ul> <p>Both support an optional comment at the reference-site that is discarded during substitution.</p> <p>The following snippet (minus the backslash),</p> <div class="code-block"><div class="code-header"><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code>{{ aside::501108be // getting arrested again }\}</code></pre></div> <p>Would be substituted with this markup,</p> <div class="code-block"><div class="code-header"><button aria-label="Copy code" class="code-copy" type="button">Copy</button></div><pre><code>{{ aside::501108be // getting arrested again }}</code></pre></div> <p>Which would then be rendered into the following later in the build,</p> <p>{{ aside::501108be</p> <p>Both forms of import create a backlink in the referenced document, but only <code>aside</code> creates a visible outward link at the reference-site. This allows documents to be assembled seamlessly where that is a better fit than a call-out, but generally I expect I’ll use the <code>aside</code> mode more often, as I generally prefer for connections to be bi-directional.</p> 1:27pm on November 15, 2024 from Calgary, Alberta, Canada<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“Import referenced documents at build time”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:c5a2e4c4-e7c5-4dcb-94b6-2add1517bd5d 2024-11-06T11:23:32Z 2024-11-06T11:23:32Z <p>Currently a given reference may optionally include a <code>url</code> key, with a single URL as its value. The idea of a single canonical URL for every resource is somewhat limiting.</p> <p>Replacing the current key/value <code>url</code> with an array, <code>urls</code>, has the potential to make preservation more adaptable and complete. Example, I don’t subscribe to the New York Times, so when something/someone points me to an article there, I use &lt;archive.is&gt; to view it. So that’s two URLs related to that resource.</p> <p>The most significant implementation question seems to be whether the new <code>urls</code> key should contain an Array, or a Dict.</p> <p>A simple array would force meaning into the order of the array, eg. the attribution constructor would have to index from the beginning or the end of the array, taking the first or the last available value. That feels clumsy.</p> <p>Using a Dict would allow picking URLs on the basis of semantic keys, eg. an &lt;archive.is&gt; URLs could be under <code>archive_is</code>, a Quote Investigator link could be under <code>quote_in</code> OR in a nested <code>references</code> array etc. Would have to explore whether I really want to ‘hardcode’ such sites.</p> 11:23am on November 6, 2024 from Istanbul, Marmara, Turkey<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“SEP 63: Multiple URLs for source records”">Reply via email</a></p> tag:silasjelley.com,2020-08-20:d70b1778-afa3-4291-bf0f-2555ecb5d3dd 2024-11-05T13:59:30Z 2024-11-05T13:59:30Z <p>Pavlos posed the question as I hitch hiked back to my line after detouring back to Albania to see Jess, but it had slipped from my mind until just now when I overheard someone at the hostel ask, “What’s your favourite question?”</p> 1:59pm on November 5, 2024 from Istanbul, Marmara, Turkey<p><a href="mailto:reply@silasjelley.com?subject=Reply%20to:%20“What question would you like to be asked?”">Reply via email</a></p>