# rss_item

guid: https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#atom-entries
title: Anthropic's new Citations API
author: Simon Willison
url: https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#atom-entries
feedurl: https://simonwillison.net/atom/entries/
content_mime_type: text/html
content:
<p>Here's a new API-only feature from Anthropic that requires quite a bit of assembly in order to unlock the value: <a href="https://www.anthropic.com/news/introducing-citations-api">Introducing Citations on the Anthropic API</a>. Let's talk about what this is and why it's interesting.</p>

<ul>
  <li><a href="https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#citations-for-rag">Citations for Retrieval Augmented Generation</a></li>
  <li><a href="https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#trying-out-the-new-api-with-uv-run">Trying out the new API with uv run</a></li>
  <li><a href="https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#rendering-the-citations">Rendering the citations</a></li>
  <li><a href="https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#now-i-need-to-design-an-abstraction-layer-for-llm">Now I need to design an abstraction layer for LLM</a></li>
  <li><a href="https://simonwillison.net/2025/Jan/24/anthropics-new-citations-api/#anthropic-s-strategy-contrasted-with-openai">Anthropic's strategy contrasted with OpenAI</a></li>
</ul>

<h4 id="citations-for-rag">Citations for Retrieval Augmented Generation</h4>

<p>The core of the <a href="https://simonwillison.net/tags/rag/">Retrieval Augmented Generation</a> (RAG) pattern is to take a user's question, retrieve portions of documents that might be relevant to that question and then answer the question by including those text fragments in the context provided to the LLM.</p>
<p>This usually works well, but there is still a risk that the model may answer based on other information from its training data (sometimes OK) or hallucinate entirely incorrect details (definitely bad).</p>
<p>The <em>best</em> way to help mitigate these risks is to support the answer with citations that incorporate direct quotations from the underlying source documents. This even acts as a form of fact-checking: the user can confirm that the quoted text did indeed come from those documents, helping provide relatively robust protection against hallucinated details resulting in incorrect answers.</p>
<p>Actually building a system that does this can be quite tricky. Matt Yeung described a pattern for this he called <a href="https://mattyyeung.github.io/deterministic-quoting">Deterministic Quoting</a> last April, where answers are accompanied by direct quotations from the source documents that are guaranteed to be copied across and not lossily transformed by the model.</p>
<p>This is a great idea, but actually building it requires some quite sophisticated prompt engineering and complex implementation code.</p>
<p>Claude's new <a href="https://docs.anthropic.com/en/docs/build-with-claude/citations">Citations API</a> mechanism handles the difficult parts of this for you. You still need to implement most of RAG - identifying potentially relevant documents, then feeding that content in as part of the prompt - but Claude's API will then do the difficult work of extracting relevant citations and including them in the response that it sends back to you.</p>
<h4 id="trying-out-the-new-api-with-uv-run">Trying out the new API with uv run</h4>
<p>I tried the API out using Anthropic's Python client library, which was <a href="https://github.com/anthropics/anthropic-sdk-python/commit/67aa83e5d589f6afad5fbc8bd2e616cc71a80a29">just updated</a> to support the citations API.</p>
<p>I ran a scratch Python 3.13 interpreter with that package using <a href="https://docs.astral.sh/uv/">uv run</a> like this (after first setting the necessary <code>ANTHROPIC_API_KEY</code> environment variable using <a href="https://llm.datasette.io/en/stable/help.html#llm-keys-get-help">llm keys get</a>):</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> ANTHROPIC_API_KEY=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-s"><span class="pl-pds">$(</span>llm keys get claude<span class="pl-pds">)</span></span><span class="pl-pds">"</span></span>
uv run --with anthropic --python 3.13 python</pre></div>
<p>Python 3.13 has <a href="https://docs.python.org/3/whatsnew/3.13.html#a-better-interactive-interpreter">a nicer interactive interpreter</a> which you can more easily paste code into. Using <code>uv run</code> like this gives me an environment with that package pre-installed without me needing to setup a virtual environment as a separate step.</p>
<p>Then I ran the following code, adapted from <a href="https://docs.anthropic.com/en/docs/build-with-claude/citations">Anthropic's example</a>. The <a href="https://gist.github.com/simonw/9fbb3c2e2c40c181727e497e358fd7ce">text.txt Gist</a> contains text I copied out from my <a href="https://simonwillison.net/2024/Dec/31/llms-in-2024/">Things we learned about LLMs in 2024</a> post.</p>
<pre><span class="pl-k">import</span> <span class="pl-s1">urllib</span>.<span class="pl-s1">request</span>
<span class="pl-k">import</span> <span class="pl-s1">json</span>

<span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-s">'https://gist.githubusercontent.com/simonw/9fbb3c2e2c40c181727e497e358fd7ce/raw/6ac20704f5a46b567b774b07fd633a74944bab2b/text.txt'</span>
<span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s1">urllib</span>.<span class="pl-c1">request</span>.<span class="pl-c1">urlopen</span>(<span class="pl-s1">url</span>).<span class="pl-c1">read</span>().<span class="pl-c1">decode</span>(<span class="pl-s">'utf-8'</span>)

<span class="pl-k">import</span> <span class="pl-s1">anthropic</span>

<span class="pl-s1">client</span> <span class="pl-c1">=</span> <span class="pl-s1">anthropic</span>.<span class="pl-c1">Anthropic</span>()

<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">client</span>.<span class="pl-c1">messages</span>.<span class="pl-c1">create</span>(
    <span class="pl-s1">model</span><span class="pl-c1">=</span><span class="pl-s">"claude-3-5-sonnet-20241022"</span>,
    <span class="pl-s1">max_tokens</span><span class="pl-c1">=</span><span class="pl-c1">1024</span>,
    <span class="pl-s1">messages</span><span class="pl-c1">=</span>[
        {
            <span class="pl-s">"role"</span>: <span class="pl-s">"user"</span>,
            <span class="pl-s">"content"</span>: [
                {
                    <span class="pl-s">"type"</span>: <span class="pl-s">"document"</span>,
                    <span class="pl-s">"source"</span>: {
                        <span class="pl-s">"type"</span>: <span class="pl-s">"text"</span>,
                        <span class="pl-s">"media_type"</span>: <span class="pl-s">"text/plain"</span>,
                        <span class="pl-s">"data"</span>: <span class="pl-s1">text</span>,
                    },
                    <span class="pl-s">"title"</span>: <span class="pl-s">"My Document"</span>,
                    <span class="pl-s">"context"</span>: <span class="pl-s">"This is a trustworthy document."</span>,
                    <span class="pl-s">"citations"</span>: {<span class="pl-s">"enabled"</span>: <span class="pl-c1">True</span>}
                },
                {
                    <span class="pl-s">"type"</span>: <span class="pl-s">"text"</span>,
                    <span class="pl-s">"text"</span>: <span class="pl-s">"What were the top trends?"</span>
                }
            ]
        }
    ]
)
<span class="pl-en">print</span>(<span class="pl-s1">json</span>.<span class="pl-c1">dumps</span>(<span class="pl-s1">response</span>.<span class="pl-c1">to_dict</span>(), <span class="pl-s1">indent</span><span class="pl-c1">=</span><span class="pl-c1">2</span>))</pre>
<p>The JSON output from that starts like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>msg_01P3zs4aYz2Baebumm4Fejoi<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"content"</span>: [
    {
      <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>Based on the document, here are the key trends in AI/LLMs from 2024:<span class="pl-cce">\n\n</span>1. Breaking the GPT-4 Barrier:<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
      <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>
    },
    {
      <span class="pl-ent">"citations"</span>: [
        {
          <span class="pl-ent">"cited_text"</span>: <span class="pl-s"><span class="pl-pds">"</span>I<span class="pl-cce">\u2019</span>m relieved that this has changed completely in the past twelve months. 18 organizations now have models on the Chatbot Arena Leaderboard that rank higher than the original GPT-4 from March 2023 (GPT-4-0314 on the board)<span class="pl-cce">\u2014</span>70 models in total.<span class="pl-cce">\n\n</span><span class="pl-pds">"</span></span>,
          <span class="pl-ent">"document_index"</span>: <span class="pl-c1">0</span>,
          <span class="pl-ent">"document_title"</span>: <span class="pl-s"><span class="pl-pds">"</span>My Document<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"end_char_index"</span>: <span class="pl-c1">531</span>,
          <span class="pl-ent">"start_char_index"</span>: <span class="pl-c1">288</span>,
          <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>char_location<span class="pl-pds">"</span></span>
        }
      ],
      <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>The GPT-4 barrier was completely broken, with 18 organizations now having models that rank higher than the original GPT-4 from March 2023, with 70 models in total surpassing it.<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>
    },
    {
      <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-cce">\n\n</span>2. Increased Context Lengths:<span class="pl-cce">\n</span><span class="pl-pds">"</span></span>,
      <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>
    },</pre></div>
<p>Here's <a href="https://gist.github.com/simonw/022d082ccfd636256f72150df344335e">the full response</a>.</p>
<p>This format is pretty interesting! It's the standard Claude format but those <code>"content"</code> blocks now include an optional additional <code>"citations"</code> key which contains a list of relevant citation extracts that support the claim in the <code>"text"</code> block.</p>
<h4 id="rendering-the-citations">Rendering the citations</h4>
<p>Eyeballing the JSON output wasn't particularly fun. I wanted a very quick tool to help me see that output in a more visual way.</p>
<p>A trick I've been using a lot recently is that LLMs like Claude are <em>really</em> good at writing code to turn arbitrary JSON shapes like this into a more human-readable format.</p>
<p>I fired up my <a href="https://simonwillison.net/2024/Dec/19/one-shot-python-tools/#custom-instructions">Artifacts project</a>, pasted in the above JSON and prompted it like this:</p>
<blockquote>
<p>Build a tool where I can paste JSON like this into a textarea and the result will be rendered in a neat way - it should should intersperse text with citations, where each citation has the cited_text rendered in a blockquote</p>
</blockquote>
<p>It helped me <a href="https://tools.simonwillison.net/render-claude-citations">build this tool</a> (<a href="https://gist.github.com/simonw/85bd050908486de36b078c8c7d01e903">follow-up prompt here</a>), which lets you paste in JSON and produces a rendered version of the text:</p>
<p><img src="https://static.simonwillison.net/static/2025/render-citations-artifact.jpg" alt="Render Claude Citations tool. Paste a JSON response from Claude below to render it with citations. JSON is shown, then a Render Message button, then an iframe containing the rendered text." style="max-width: 100%;" /></p>
<h4 id="now-i-need-to-design-an-abstraction-layer-for-llm">Now I need to design an abstraction layer for LLM</h4>
<p>I'd like to upgrade my <a href="https://llm.datasette.io/">LLM</a> tool and <a href="https://github.com/simonw/llm-claude-3">llm-claude-3</a> plugin to include support for this new feature... but doing so is going to be relatively non-trivial.</p>
<p>The problem is that LLM currently bakes in an assumption that all LLMs respond with a stream of text.</p>
<p>With citations, this is no longer true! Claude is now returning chunks of text that aren't just a plain string - they are annotated with citations, which need to be stored and processed somehow by the LLM library.</p>
<p>This isn't the only edge-case of this type. DeepSeek recently released their Reasoner API which has a similar problem: it can return two different types of text, one showing reasoning text and one showing final content. I <a href="https://gist.github.com/simonw/a5ca117dd0325c93a5b1f5a18c4a9e34">described those differences here</a>.</p>
<p>I've opened a design issue to tackle this challenge in the LLM repository: <a href="https://github.com/simonw/llm/issues/716">Design an abstraction for responses that are not just a stream of text</a>.</p>
<h4 id="anthropic-s-strategy-contrasted-with-openai">Anthropic's strategy contrasted with OpenAI</h4>
<p>Another interesting aspect of this release is how it helps illustrate a strategic difference between Anthropic and OpenAI.</p>
<p>OpenAI are increasingly behaving like a consumer products company. They just made a big splash with their <a href="https://simonwillison.net/2025/Jan/23/introducing-operator/">Operator</a> browser-automation agent system - a much more polished, consumer-product version of Anthropic's own <a href="https://simonwillison.net/2025/Jan/23/introducing-operator/">Computer Use</a> demo from a few months ago.</p>
<p>Meanwhile, Anthropic are clearly focused much more on the developer / "enterprise" market. This Citations feature is API-only and directly addresses a specific need that developers trying to build reliable RAG systems on top of their platform may not even have realized they had.</p>


guid: https://simonwillison.net/2025/Jan/24/selfish-open-source/#atom-entries
title: A selfish personal argument for releasing code as Open Source
author: Simon Willison
url: https://simonwillison.net/2025/Jan/24/selfish-open-source/#atom-entries
feedurl: https://simonwillison.net/atom/entries/
content_mime_type: text/html
content:
<p>I'm the guest for the most recent episode of the Real Python podcast with Christopher Bailey, talking about <a href="https://realpython.com/podcasts/rpp/236/">Using LLMs for Python Development</a>. We covered a <em>lot</em> of other topics as well - most notably my relationship with Open Source development over the years.</p>
<p>At <a href="https://realpython.com/podcasts/rpp/236/#t=332">5m32s</a> I presented what I think is the best version yet of my selfish personal argument for why it makes sense to default to releasing code as Open Source:</p>
<blockquote>
<p>I didn't really get heavily back into open source until about maybe six years ago when I'd been working for a big company in the US, and I got frustrated that all of the code I was writing, I'd never get to use again.</p>
<p>I realized that one of the best things about open source software is that you can solve a problem once and then you can slap an open source license on that solution and you will <em>never</em> have to solve that problem ever again, no matter who's employing you in the future.</p>
<p>It's a sneaky way of solving a problem permanently.</p>
<p>Once I realized that I started open sourcing everything, like pretty much every piece of code I've written in the past six years, I've open sourced purely as a defense against me losing access to that code in the future.</p>
<p>Because I've written loads of code for employers that I don't get to use anymore - and how many times do you want to reinvent things?</p>
<p>I like to say that my interest in open source is actually really selfish. I figured something out. I never want to have to do this work ever again.</p>
<p>If I slap a license on it, write documentation for it so that I can remember what it does and write unit tests for it so it's easy for me to keep it working in the future, that's entirely beneficial to me.</p>
</blockquote>
<p>The rest of the episode was a really great conversation - other topics we covered included:</p>
<ul>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=244">4m40s</a>: My first ever significant open source project - a PHP XML-RPC library that ended up in WordPress twenty years ago</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=608">10m08s</a>: Benefits I've gained from starting a blog 22+ years ago</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=1334">22m14s</a>: How to get started using LLMs to write Python</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=2215">36m55s</a>: My workflow for using LLMs for code - for both the experimental research work (I called this the "Mise en place phase") and the follow-up where I actually write the finished code</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=3314">55m14s</a>: Why an SVG of a pelican riding a bicycle?</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=3468">57m48s</a>: Why saying "do it better" actually works!</li>
<li>
<a href="https://realpython.com/podcasts/rpp/236/#t=3624">1h0m24s</a>: Cooking with LLMs! How to get a weirdly tasty guacamole recipe</li>
<li><a href="https://realpython.com/podcasts/rpp/236/#t=4132">1h08m52s</a>: My latest thoughts on local models</li>
</ul>


guid: https://simonwillison.net/2025/Jan/31/o3-mini/#atom-entries
title: OpenAI o3-mini, now available in LLM
author: Simon Willison
url: https://simonwillison.net/2025/Jan/31/o3-mini/#atom-entries
feedurl: https://simonwillison.net/atom/entries/
content_mime_type: text/html
content:
<p>OpenAI's <a href="https://openai.com/index/openai-o3-mini/">o3-mini is out today</a>. As with other o-series models it's a slightly difficult one to evaluate - we now need to decide if a prompt is best run using GPT-4o, o1, o3-mini or (if we have access) o1 Pro.</p>
<p>Confusing matters further, the benchmarks in <a href="https://openai.com/index/o3-mini-system-card/">the o3-mini system card</a> (PDF) aren't a universal win for o3-mini across all categories. It generally benchmarks higher than GPT-4o and o1 but not across everything.</p>
<p>The biggest win for o3-mini is on the Codeforces ELO competitive programming benchmark, which I think is <a href="https://arxiv.org/abs/2501.01257">described by this 2nd January 2025 paper</a>, with the following scores:</p>
<ul>
<li>o3-mini (high) 2130</li>
<li>o3-mini (medium) 2036</li>
<li>o1 1891</li>
<li>o3-mini (low) 1831</li>
<li>o1-mini 1650</li>
<li>o1-preview 1258</li>
<li>GPT-4o 900</li>
</ul>
<p>Weirdly, that GPT-4o score was in an older copy of the System Card PDF which has been replaced by an updated document that doesn't mention Codeforces ELO scores at all.</p>
<p>One note from the System Card that stood out for me concerning intended applications of o3-mini for OpenAI themselves:</p>
<blockquote>
<p>We also plan to allow users to use o3-mini to search the internet and summarize the results in ChatGPT. We expect o3-mini to be a useful and safe model for doing this, especially given its performance on the jailbreak and instruction hierarchy evals detailed in Section 4 below.</p>
</blockquote>
<p>This is notable because the existing o1 models on ChatGPT have not yet had access to their web search tool - despite the mixture of search and "reasoning" models having very clear benefits.</p>
<p>o3-mini does not and <a href="https://twitter.com/nikunjhanda/status/1885415728624656481">will not</a> support vision. We will have to wait for future OpenAI reasoning models for that.</p>
<p>I released <a href="https://llm.datasette.io/en/stable/changelog.html#v0-21">LLM 0.21</a> with support for the new model, plus its <code>-o reasoning_effort high</code> (or <code>medium</code> or <code>low</code>) option for tweaking the reasoning effort - details <a href="https://github.com/simonw/llm/issues/728">in this issue</a>.</p>
<p>Note that the new model is currently only available for <a href="https://platform.openai.com/docs/guides/rate-limits/usage-tiers#tier-3-rate-limits">Tier 3</a> and higher users, which requires you to have spent at least $100 on the API.</p>
<p>o3-mini <a href="https://openai.com/api/pricing/">is priced</a> at $1.10/million input tokens, $4.40/million output tokens - less than half the price of GPT-4o (currently $2.50/$10) and massively cheaper than o1 ($15/$60). The GPT-4o comparison isn't quite as simple as that though, as o3-mini's invisible reasoning tokens still count towards the output tokens you get charged for.</p>
<p>I tried using it to summarize <a href="https://news.ycombinator.com/item?id=42890627">this conversation about o3-mini on Hacker News</a>, using <a href="https://til.simonwillison.net/llms/claude-hacker-news-themes#user-content-adding-a--m-model-option">my hn-summary.sh script</a>.</p>

<div class="highlight highlight-source-shell"><pre>hn-summary.sh 42890627 -o o3-mini</pre></div>

<p>Here's <a href="https://gist.github.com/simonw/09e5922be0cbb85894cf05e6d75ae050">the result</a> - it used 18,936 input tokens and 2,905 output tokens for a total cost of 3.3612 cents.</p>

<p>o3-mini (and o1-mini) are text-only models: they don't accept image inputs. The full o1 API model can accept images in the same way as GPT-4o.</p>

<p>Another characteristic worth noting is o3-mini's token output limit - the measure of how much text it can output in one go.  That's 100,000 tokens, compared to 16,000 for GPT-4o and just 8,000 for both DeepSeek R1 and Claude 3.5.</p>

<p>Invisible "reasoning tokens" come out of the same budget, so it's likely not possible to have it output the full 100,000.</p>

<p>The model accepts up to 200,000 tokens of input, an improvement on GPT-4o's 128,000.</p>

<p>An application where output limits really matter is translation between human languages, where the output can realistically be expected to have a similar length to the input. It will be interesting seeing how well o3-mini works for that, especially given its low price.</p>
<p>Update: Here's a <a href="https://news.ycombinator.com/item?id=42894215#42895610">fascinating comment</a> on this by professional translator Tom Gally on Hacker News:</p>
<blockquote>
<p>I just did a test in which both R1 and o3-mini got worse at translation in the latter half of a long text. [...]</p>
<p>An initial comparison of the output suggested that, while R1 didn’t seem bad, o3-mini produced a writing style closer to what I asked for in the prompt—smoother and more natural English.
But then I noticed that the output length was 5,855 characters for R1, 9,052 characters for o3-mini, and 11,021 characters for my own polished version. Comparing the three translations side-by-side with the original Japanese, I discovered that R1 had omitted entire paragraphs toward the end of the speech, and that o3-mini had switched to a strange abbreviated style (using slashes instead of “and” between noun phrases, for example) toward the end as well. The vanilla versions of ChatGPT, Claude, and Gemini that I ran the same prompt and text through a month ago had had none of those problems.</p>
</blockquote>


guid: https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#atom-entries
title: Using pip to install a Large Language Model that's under 100MB
author: Simon Willison
url: https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#atom-entries
feedurl: https://simonwillison.net/atom/entries/
content_mime_type: text/html
content:
<p>I just released <a href="https://github.com/simonw/llm-smollm2">llm-smollm2</a>, a new plugin for <a href="https://llm.datasette.io/">LLM</a> that bundles a quantized copy of the <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct">SmolLM2-135M-Instruct</a> LLM inside of the Python package.</p>
<p>This means you can now <code>pip install</code> a full LLM!</p>

<p>If you're already using <a href="https://llm.datasette.io/">LLM</a> you can install it like this:</p>
<div class="highlight highlight-source-shell"><pre>llm install llm-smollm2</pre></div>
<p>Then run prompts like this:</p>
<div class="highlight highlight-source-shell"><pre>llm -m SmolLM2 <span class="pl-s"><span class="pl-pds">'</span>Are dogs real?<span class="pl-pds">'</span></span></pre></div>
<p>(New favourite test prompt for tiny models, courtesy of <a href="https://bsky.app/profile/timfduffy.com/post/3lhknvwijps2o">Tim Duffy</a>. <a href="https://gist.github.com/simonw/a71d83cd8c6caa49cec4a9f296c267df">Here's the result</a>).</p>
<p>If you don't have LLM yet first <a href="https://llm.datasette.io/en/stable/setup.html">follow these installation instructions</a>, or <code>brew install llm</code> or <code>pipx install llm</code> or <code>uv tool install llm</code> depending on your preferred way of getting your Python tools.</p>
<p>If you have <a href="https://github.com/astral-sh/uv">uv</a> setup you don't need to install anything at all! The following command will spin up an ephemeral environment, install the necessary packages and start a chat session with the model all in one go:</p>
<div class="highlight highlight-source-shell"><pre>uvx --with llm-smollm2 llm chat -m SmolLM2</pre></div>

<p><img src="https://static.simonwillison.net/static/2025/smol.gif" alt="$ uvx --with llm-smollm2 llm chat -m SmolLM2
Chatting with SmolLM2
Type 'exit' or 'quit' to exit
Type '!multi' to enter multiple lines, then '!end' to finish
> tell me a long story about a dog
The story of a dog is a tale of love, loss, and the unspoken rules of the universe. It begins with a small, timid, and timid dog named Max. Max was born into a loving family, but his life was forever changed by a cruel and cruel owner. He was left to fend for himself, and he was left to fend for himself.

Max's owner, a kind and gentle soul named Elena, was a cruel and cruel one. She had no patience, no love, and she would do anything to keep Max away from her. Max was left to fend for himself, and he was left to fend for himself.

Max's owner, Elena, was determined to make Max happy. She took Max to a beautiful and beautiful place, where Max was allowed to roam freely. Max was allowed to roam freely, and Max was allowed to roam freely. [Then repeats that sentence many times]" style="max-width: 100%;" /></p>

<ul>
  <li><a href="https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#finding-a-tiny-model">Finding a tiny model</a></li>
  <li><a href="https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#building-the-plugin">Building the plugin</a></li>
  <li><a href="https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#packaging-the-plugin">Packaging the plugin</a></li>
  <li><a href="https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#publishing-to-pypi">Publishing to PyPI</a></li>
  <li><a href="https://simonwillison.net/2025/Feb/7/pip-install-llm-smollm2/#is-the-model-any-good-">Is the model any good?</a></li>
</ul>

<h4 id="finding-a-tiny-model">Finding a tiny model</h4>
<p>The fact that the model is almost exactly 100MB is no coincidence: that's the <a href="https://pypi.org/help/#file-size-limit">default size limit</a> for a Python package that can be uploaded to the Python Package Index (PyPI).</p>
<p>I <a href="https://bsky.app/profile/simonwillison.net/post/3lhklqd62jc2x">asked on Bluesky</a> if anyone had seen a just-about-usable GGUF model that was under 100MB, and Artisan Loaf <a href="https://bsky.app/profile/artisanloaf.bsky.social/post/3lhklumfhvs2r">pointed me</a> to <a href="https://huggingface.co/HuggingFaceTB/SmolLM2-135M-Instruct">SmolLM2-135M-Instruct</a>.</p>
<p>I ended up using <a href="https://huggingface.co/QuantFactory/SmolLM2-135M-Instruct-GGUF/tree/main">this quantization</a> by <a href="https://huggingface.co/QuantFactory">QuantFactory</a> just because it was the first sub-100MB model I tried that worked.</p>
<p>Trick for finding quantized models: Hugging Face has a neat "model tree" feature in the side panel of their model pages, which includes links to relevant quantized models. I find most of my GGUFs using that feature.</p>
<p><img src="https://static.simonwillison.net/static/2025/hugging-face-model-tree.jpg" alt="Model tree for HuggingFaceTB/SmolLM2-135M-Instruct. 60 Quantizations, 6 adapters, 80 finetunes, 1 merge." style="max-width: 100%;" /></p>
<h4 id="building-the-plugin">Building the plugin</h4>
<p>I first tried the model out using Python and the <a href="https://github.com/abetlen/llama-cpp-python">llama-cpp-python</a> library like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run --with llama-cpp-python python</pre></div>
<p>Then:</p>
<pre><span class="pl-k">from</span> <span class="pl-s1">llama_cpp</span> <span class="pl-k">import</span> <span class="pl-v">Llama</span>
<span class="pl-k">from</span> <span class="pl-s1">pprint</span> <span class="pl-k">import</span> <span class="pl-s1">pprint</span>
<span class="pl-s1">llm</span> <span class="pl-c1">=</span> <span class="pl-en">Llama</span>(<span class="pl-s1">model_path</span><span class="pl-c1">=</span><span class="pl-s">"SmolLM2-135M-Instruct.Q4_1.gguf"</span>)
<span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">llm</span>.<span class="pl-c1">create_chat_completion</span>(<span class="pl-s1">messages</span><span class="pl-c1">=</span>[
    {<span class="pl-s">"role"</span>: <span class="pl-s">"user"</span>, <span class="pl-s">"content"</span>: <span class="pl-s">"Hi"</span>}
])
<span class="pl-en">pprint</span>(<span class="pl-s1">output</span>)</pre>
<p>This gave me the output I was expecting:</p>
<pre>{<span class="pl-s">'choices'</span>: [{<span class="pl-s">'finish_reason'</span>: <span class="pl-s">'stop'</span>,
              <span class="pl-s">'index'</span>: <span class="pl-c1">0</span>,
              <span class="pl-s">'logprobs'</span>: <span class="pl-c1">None</span>,
              <span class="pl-s">'message'</span>: {<span class="pl-s">'content'</span>: <span class="pl-s">'Hello! How can I assist you today?'</span>,
                          <span class="pl-s">'role'</span>: <span class="pl-s">'assistant'</span>}}],
 <span class="pl-s">'created'</span>: <span class="pl-c1">1738903256</span>,
 <span class="pl-s">'id'</span>: <span class="pl-s">'chatcmpl-76ea1733-cc2f-46d4-9939-90efa2a05e7c'</span>,
 <span class="pl-s">'model'</span>: <span class="pl-s">'SmolLM2-135M-Instruct.Q4_1.gguf'</span>,
 <span class="pl-s">'object'</span>: <span class="pl-s">'chat.completion'</span>,
 <span class="pl-s">'usage'</span>: {<span class="pl-s">'completion_tokens'</span>: <span class="pl-c1">9</span>, <span class="pl-s">'prompt_tokens'</span>: <span class="pl-c1">31</span>, <span class="pl-s">'total_tokens'</span>: <span class="pl-c1">40</span>}}</pre>
<p>But it also <em>spammed</em> my terminal with a huge volume of debugging output - which started like this:</p>
<pre><code>llama_model_load_from_file_impl: using device Metal (Apple M2 Max) - 49151 MiB free
llama_model_loader: loaded meta data with 33 key-value pairs and 272 tensors from SmolLM2-135M-Instruct.Q4_1.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = llama
</code></pre>
<p>And then continued for more than <a href="https://gist.github.com/simonw/9ef7acd836b1cc40c14686eae4dca340">500 lines</a>!</p>
<p>I've had this problem with <code>llama-cpp-python</code> and <code>llama.cpp</code> in the past, and was sad to find that the documentation still doesn't have a great answer for how to avoid this.</p>
<p>So I turned to the just released <a href="https://simonwillison.net/2025/Feb/5/gemini-2/">Gemini 2.0 Pro (Experimental)</a>, because I know it's a strong model with a long input limit.</p>
<p>I ran the entire <code>llama-cpp-python</code> codebase through it like this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">cd</span> /tmp
git clone https://github.com/abetlen/llama-cpp-python
<span class="pl-c1">cd</span> llama-cpp-python
files-to-prompt -e py <span class="pl-c1">.</span> -c <span class="pl-k">|</span> llm -m gemini-2.0-pro-exp-02-05 \
  <span class="pl-s"><span class="pl-pds">'</span>How can I prevent this library from logging any information at all while it is running - no stderr or anything like that<span class="pl-pds">'</span></span></pre></div>
<p>Here's <a href="https://gist.github.com/simonw/20476c2c6f7604df2994212cebfafef4#response">the answer I got back</a>. It recommended setting the logger to <code>logging.CRITICAL</code>, passing <code>verbose=False</code> to the constructor and, most importantly, using the following context manager to suppress all output:</p>
<pre><span class="pl-k">from</span> <span class="pl-s1">contextlib</span> <span class="pl-k">import</span> <span class="pl-s1">contextmanager</span>, <span class="pl-s1">redirect_stderr</span>, <span class="pl-s1">redirect_stdout</span>

<span class="pl-en">@<span class="pl-s1">contextmanager</span></span>
<span class="pl-k">def</span> <span class="pl-en">suppress_output</span>():
    <span class="pl-s">"""</span>
<span class="pl-s">    Suppresses all stdout and stderr output within the context.</span>
<span class="pl-s">    """</span>
    <span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s1">os</span>.<span class="pl-c1">devnull</span>, <span class="pl-s">"w"</span>) <span class="pl-k">as</span> <span class="pl-s1">devnull</span>:
        <span class="pl-k">with</span> <span class="pl-en">redirect_stdout</span>(<span class="pl-s1">devnull</span>), <span class="pl-en">redirect_stderr</span>(<span class="pl-s1">devnull</span>):
            <span class="pl-k">yield</span></pre>
<p>This worked! It turned out most of the output came from initializing the <code>LLM</code> class, so I wrapped that like so:</p>
<pre><span class="pl-k">with</span> <span class="pl-en">suppress_output</span>():
    <span class="pl-s1">model</span> <span class="pl-c1">=</span> <span class="pl-en">Llama</span>(<span class="pl-s1">model_path</span><span class="pl-c1">=</span><span class="pl-s1">self</span>.<span class="pl-c1">model_path</span>, <span class="pl-s1">verbose</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre>
<p>Proof of concept in hand I set about writing the plugin. I started with my <a href="https://github.com/simonw/llm-plugin">simonw/llm-plugin</a> cookiecutter template:</p>
<div class="highlight highlight-source-shell"><pre>uvx cookiecutter gh:simonw/llm-plugin</pre></div>
<pre><code>  [1/6] plugin_name (): smollm2
  [2/6] description (): SmolLM2-135M-Instruct.Q4_1 for LLM
  [3/6] hyphenated (smollm2): 
  [4/6] underscored (smollm2): 
  [5/6] github_username (): simonw
  [6/6] author_name (): Simon Willison
</code></pre>
<p>The <a href="https://github.com/simonw/llm-smollm2/blob/0.1.1/llm_smollm2/__init__.py">rest of the plugin</a> was mostly borrowed from my existing <a href="https://github.com/simonw/llm-gguf/blob/0.2/llm_gguf.py">llm-gguf</a> plugin, updated based on the latest README for the <code>llama-cpp-python</code> project.</p>
<p>There's more information on building plugins in <a href="https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html">the tutorial on writing a plugin</a></p>
<h4 id="packaging-the-plugin">Packaging the plugin</h4>
<p>Once I had that working the last step was to figure out how to package it for PyPI. I'm never quite sure of the best way to bundle a binary file in a Python package, especially one that uses a <code>pyproject.toml</code> file... so I dumped a copy of my existing <code>pyproject.toml</code> file into o3-mini-high and prompted:</p>
<blockquote>
<p>Modify this to bundle a SmolLM2-135M-Instruct.Q4_1.gguf file inside the package. I don't want to use hatch or a manifest or anything, I just want to use setuptools.</p>
</blockquote>
<p>Here's <a href="https://chatgpt.com/share/67a59122-67c8-8006-9be4-29f8419343ad">the shared transcript</a> - it gave me exactly what I wanted. I bundled it by adding this to the end of the <code>toml</code> file:</p>
<div class="highlight highlight-source-toml"><pre>[<span class="pl-en">tool</span>.<span class="pl-en">setuptools</span>.<span class="pl-en">package-data</span>]
<span class="pl-smi">llm_smollm2</span> = [<span class="pl-s"><span class="pl-pds">"</span>SmolLM2-135M-Instruct.Q4_1.gguf<span class="pl-pds">"</span></span>]</pre></div>
<p>Then dropping that <code>.gguf</code> file into the <code>llm_smollm2/</code> directory and putting my plugin code in <code>llm_smollm2/__init__.py</code>.</p>
<p>I tested it locally by running this:</p>
<div class="highlight highlight-source-shell"><pre>python -m pip install build
python -m build</pre></div>
<p>I fired up a fresh virtual environment and ran <code>pip install ../path/to/llm-smollm2/dist/llm_smollm2-0.1-py3-none-any.whl</code> to confirm that the package worked as expected.</p>
<h4 id="publishing-to-pypi">Publishing to PyPI</h4>
<p>My cookiecutter template comes with <a href="https://github.com/simonw/llm-smollm2/blob/main/.github/workflows/publish.yml">a GitHub Actions workflow</a> that publishes the package to PyPI when a new release is created using the GitHub web interface. Here's the relevant YAML:</p>
<div class="highlight highlight-source-yaml"><pre>  <span class="pl-ent">deploy</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">needs</span>: <span class="pl-s">[test]</span>
    <span class="pl-ent">environment</span>: <span class="pl-s">release</span>
    <span class="pl-ent">permissions</span>:
      <span class="pl-ent">id-token</span>: <span class="pl-s">write</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v4</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v5</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s"><span class="pl-pds">"</span>3.13<span class="pl-pds">"</span></span>
        <span class="pl-ent">cache</span>: <span class="pl-s">pip</span>
        <span class="pl-ent">cache-dependency-path</span>: <span class="pl-s">pyproject.toml</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pip install setuptools wheel build</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Build</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        python -m build</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Publish</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">pypa/gh-action-pypi-publish@release/v1</span></pre></div>
<p>This runs after the <code>test</code> job has passed. It uses the <a href="https://github.com/pypa/gh-action-pypi-publish">pypa/gh-action-pypi-publish</a> Action to publish to PyPI - I wrote more about how that works <a href="https://til.simonwillison.net/pypi/pypi-releases-from-github">in this TIL</a>.</p>
<h4 id="is-the-model-any-good-">Is the model any good?</h4>
<p>This one really isn't! It's not really surprising but it turns out 94MB really isn't enough space for a model that can do anything useful.</p>
<p>It's <em>super</em> fun to play with, and I continue to maintain that small, weak models are a great way to help build a mental model of how this technology actually works.</p>
<p>That's not to say SmolLM2 isn't a fantastic model family. I'm running the smallest, most restricted version here. <a href="https://huggingface.co/blog/smollm">SmolLM - blazingly fast and remarkably powerful</a> describes the full model family - which comes in 135M, 360M, and 1.7B sizes. The larger versions are a whole lot more capable.</p>
<p>If anyone can figure out something genuinely useful to do with the 94MB version I'd love to hear about it.</p>


guid: tag:til.simonwillison.net,2020-04-30:hugo_basic.md
title: The most basic possible Hugo site
author: Simon Willison
url: https://til.simonwillison.net/hugo/basic
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>With <a href="https://gist.github.com/simonw/6f7b6a40713b36749da845065985bb28">Claude's help</a> I figured out what I think is the most basic version of a static site generated using <a href="https://gohugo.io/" rel="nofollow">Hugo</a>.</p>
<p>I wanted a base template that set out a common layout, then two example of pages that used that layout to render content with a custom title.</p>
<p>This is my first time ever trying out Hugo so it's quite possible there's an even simpler approach, but this is what I got to.</p>
<div class="markdown-heading"><h2 class="heading-element">The layouts</h2><a id="user-content-the-layouts" class="anchor" aria-label="Permalink: The layouts" href="#the-layouts"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Hugo uses a directory called <code>layouts</code> to define its templates. I created three files in this directory:</p>
<div class="markdown-heading"><h3 class="heading-element">layouts/_default/baseof.html</h3><a id="user-content-layouts_defaultbaseofhtml" class="anchor" aria-label="Permalink: layouts/_default/baseof.html" href="#layouts_defaultbaseofhtml"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-c1">&lt;!DOCTYPE html<span class="pl-kos">&gt;</span></span>
<span class="pl-kos">&lt;</span><span class="pl-ent">html</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">head</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">title</span><span class="pl-kos">&gt;</span>{{ .Title }}<span class="pl-kos">&lt;/</span><span class="pl-ent">title</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">head</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">body</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">nav</span><span class="pl-kos">&gt;</span>
        <span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">/</span>"<span class="pl-kos">&gt;</span>Home<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span>
        <span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">/about</span>"<span class="pl-kos">&gt;</span>About<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">nav</span><span class="pl-kos">&gt;</span>
    {{ .Content }}
    <span class="pl-kos">&lt;</span><span class="pl-ent">footer</span><span class="pl-kos">&gt;</span>
        <span class="pl-kos">&lt;</span><span class="pl-ent">p</span><span class="pl-kos">&gt;</span>My Site Footer<span class="pl-kos">&lt;/</span><span class="pl-ent">p</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">footer</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">body</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">html</span><span class="pl-kos">&gt;</span></pre></div>
<div class="markdown-heading"><h3 class="heading-element">layouts/_default/single.html</h3><a id="user-content-layouts_defaultsinglehtml" class="anchor" aria-label="Permalink: layouts/_default/single.html" href="#layouts_defaultsinglehtml"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-text-html-basic"><pre>{{ define "main" }}
    {{ .Content }}
{{ end }}</pre></div>
<div class="markdown-heading"><h3 class="heading-element">layouts/index.html</h3><a id="user-content-layoutsindexhtml" class="anchor" aria-label="Permalink: layouts/index.html" href="#layoutsindexhtml"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-text-html-basic"><pre>{{ define "main" }}
    {{ .Content }}
{{ end }}</pre></div>
<p>Note that <code>index.html</code> and <code>single.html</code> are identical. I'm not sure why Hugo needed both - the <code>index.html</code> file is used for the homepage.</p>
<div class="markdown-heading"><h2 class="heading-element">The content</h2><a id="user-content-the-content" class="anchor" aria-label="Permalink: The content" href="#the-content"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I created a <code>content</code> directory with two files in it:</p>
<div class="markdown-heading"><h3 class="heading-element">content/_index.md</h3><a id="user-content-content_indexmd" class="anchor" aria-label="Permalink: content/_index.md" href="#content_indexmd"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-text-md"><pre><span class="pl-s">---</span>
<span class="pl-ent">title</span>: <span class="pl-s">Welcome</span>
<span class="pl-s">---</span>
<span class="pl-mh"># <span class="pl-en">Welcome to my site</span></span>

This is my homepage content in Markdown.</pre></div>
<div class="markdown-heading"><h3 class="heading-element">content/about.html</h3><a id="user-content-contentabouthtml" class="anchor" aria-label="Permalink: content/about.html" href="#contentabouthtml"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-text-html-basic"><pre>---
title: About
---
<span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>About Us<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">p</span><span class="pl-kos">&gt;</span>This is the about page, written directly in HTML.<span class="pl-kos">&lt;/</span><span class="pl-ent">p</span><span class="pl-kos">&gt;</span></pre></div>
<p>Note that both of these have frontmatter at the top to define the title. This seems to be the right pattern for combining HTML content and Markdown content in the same project.</p>
<p>According to the <a href="https://gohugo.io/content-management/organization/#index-pages-_indexmd" rel="nofollow">Hugo documentation</a>:</p>
<blockquote>
<p><code>_index.md</code> has a special role in Hugo. It allows you to add front matter and content to <code>home</code>, <code>section</code>, <code>taxonomy</code>, and <code>term</code> pages.</p>
</blockquote>
<p>Since I'm not using the <code>taxonomy</code> and <code>term</code> features yet this probably isn't relevant to me.</p>
<div class="markdown-heading"><h2 class="heading-element">Getting Hugo working</h2><a id="user-content-getting-hugo-working" class="anchor" aria-label="Permalink: Getting Hugo working" href="#getting-hugo-working"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Here's a Bash script that, when run, will create the directories and files listed above:</p>
<div class="highlight highlight-source-shell"><pre>mkdir hugo-site
<span class="pl-c1">cd</span> hugo-site
mkdir -p layouts/_default content
cat <span class="pl-k">&gt;</span> layouts/_default/baseof.html <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span></span>
<span class="pl-s">&lt;!DOCTYPE html&gt;</span>
<span class="pl-s">&lt;html&gt;</span>
<span class="pl-s">&lt;head&gt;</span>
<span class="pl-s">    &lt;title&gt;{{ .Title }}&lt;/title&gt;</span>
<span class="pl-s">&lt;/head&gt;</span>
<span class="pl-s">&lt;body&gt;</span>
<span class="pl-s">    &lt;nav&gt;</span>
<span class="pl-s">        &lt;a href="/"&gt;Home&lt;/a&gt;</span>
<span class="pl-s">        &lt;a href="/about"&gt;About&lt;/a&gt;</span>
<span class="pl-s">    &lt;/nav&gt;</span>
<span class="pl-s">    {{ .Content }}</span>
<span class="pl-s">    &lt;footer&gt;</span>
<span class="pl-s">        &lt;p&gt;My Site Footer&lt;/p&gt;</span>
<span class="pl-s">    &lt;/footer&gt;</span>
<span class="pl-s">&lt;/body&gt;</span>
<span class="pl-s">&lt;/html&gt;</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
cat <span class="pl-k">&gt;</span> layouts/_default/single.html <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span></span>
<span class="pl-s">{{ define "main" }}</span>
<span class="pl-s">    {{ .Content }}</span>
<span class="pl-s">{{ end }}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
cat <span class="pl-k">&gt;</span> layouts/index.html <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span></span>
<span class="pl-s">{{ define "main" }}</span>
<span class="pl-s">    {{ .Content }}</span>
<span class="pl-s">{{ end }}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
cat <span class="pl-k">&gt;</span> content/_index.md <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span></span>
<span class="pl-s">---</span>
<span class="pl-s">title: Welcome</span>
<span class="pl-s">---</span>
<span class="pl-s"># Welcome to my site</span>
<span class="pl-s"></span>
<span class="pl-s">This is my homepage content in Markdown.</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
cat <span class="pl-k">&gt;</span> content/about.html <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span></span>
<span class="pl-s">---</span>
<span class="pl-s">title: About</span>
<span class="pl-s">---</span>
<span class="pl-s">&lt;h1&gt;About Us&lt;/h1&gt;</span>
<span class="pl-s">&lt;p&gt;This is the about page, written directly in HTML.&lt;/p&gt;</span>
<span class="pl-s"><span class="pl-k">EOF</span></span></pre></div>
<p>Having created these files (in a <code>hugo-site</code> folder) we can initialize a Hugo site like this:</p>
<div class="highlight highlight-source-shell"><pre>hugo new site <span class="pl-c1">.</span> --force</pre></div>
<p>The <code>--force</code> flag is needed because we are creating the site in a directory that already contains files.</p>
<p>Now run this to preview the site in a browser:</p>
<div class="highlight highlight-source-shell"><pre>hugo server</pre></div>
<p>And visit <code>http://localhost:1313/</code> to see the site.</p>
<p>The About page is now located at <code>/about/</code> and the homepage is at <code>/</code>.</p>
<div class="markdown-heading"><h2 class="heading-element">Building the site for deployment</h2><a id="user-content-building-the-site-for-deployment" class="anchor" aria-label="Permalink: Building the site for deployment" href="#building-the-site-for-deployment"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Running this command builds a static version of the site in the <code>public</code> directory:</p>
<div class="highlight highlight-source-shell"><pre>hugo build</pre></div>
<p>When I ran it I got this non-fatal warning:</p>
<pre><code>WARN  found no layout file for "html" for kind "taxonomy": You should create a template file which matches Hugo Layouts Lookup Rules for this combination.
</code></pre>
<p>Since I wasn't using the <code>taxonomy</code> feature I added the following to the <code>hugo.conf</code> file that had been generated for me:</p>
<div class="highlight highlight-source-toml"><pre><span class="pl-smi">disableKinds</span> = [<span class="pl-s"><span class="pl-pds">'</span>taxonomy<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>term<span class="pl-pds">'</span></span>]</pre></div>
<p>After this, <code>hugo build</code> ran without warnings.</p>
<p>The <code>public</code> folder contained these files:</p>
<pre><code>public
public/index.html
public/index.xml
public/about
public/about/index.html
public/sitemap.xml
</code></pre>
<p><code>index.yml</code> is an RSS 2.0 feed, and <code>sitemap.xml</code> is a <a href="https://www.sitemaps.org/" rel="nofollow">Sitemap</a>.</p>
<p>Both of these had <code>example.org</code> hard-coded into them, e.g. <code>sitemap.xml</code> looked like this:</p>
<div class="highlight highlight-text-xml"><pre>&lt;?<span class="pl-ent">xml</span><span class="pl-e"> version</span>=<span class="pl-s"><span class="pl-pds">"</span>1.0<span class="pl-pds">"</span></span><span class="pl-e"> encoding</span>=<span class="pl-s"><span class="pl-pds">"</span>utf-8<span class="pl-pds">"</span></span><span class="pl-e"> standalone</span>=<span class="pl-s"><span class="pl-pds">"</span>yes<span class="pl-pds">"</span></span>?&gt;
&lt;<span class="pl-ent">urlset</span> <span class="pl-e">xmlns</span>=<span class="pl-s"><span class="pl-pds">"</span>http://www.sitemaps.org/schemas/sitemap/0.9<span class="pl-pds">"</span></span>
  <span class="pl-e">xmlns</span><span class="pl-e">:</span><span class="pl-e">xhtml</span>=<span class="pl-s"><span class="pl-pds">"</span>http://www.w3.org/1999/xhtml<span class="pl-pds">"</span></span>&gt;
  &lt;<span class="pl-ent">url</span>&gt;
    &lt;<span class="pl-ent">loc</span>&gt;https://example.org/about/&lt;/<span class="pl-ent">loc</span>&gt;
  &lt;/<span class="pl-ent">url</span>&gt;&lt;<span class="pl-ent">url</span>&gt;
    &lt;<span class="pl-ent">loc</span>&gt;https://example.org/&lt;/<span class="pl-ent">loc</span>&gt;
  &lt;/<span class="pl-ent">url</span>&gt;
&lt;/<span class="pl-ent">urlset</span>&gt;</pre></div>
<p>The fix for that is to update <code>hugo.toml</code> to set the domain name where the site will be deployed:</p>
<div class="highlight highlight-source-toml"><pre><span class="pl-smi">baseURL</span> = <span class="pl-s"><span class="pl-pds">'</span>https://example.simonwillison.net/<span class="pl-pds">'</span></span>
<span class="pl-smi">languageCode</span> = <span class="pl-s"><span class="pl-pds">'</span>en-us<span class="pl-pds">'</span></span>
<span class="pl-smi">title</span> = <span class="pl-s"><span class="pl-pds">'</span>My example site<span class="pl-pds">'</span></span>
<span class="pl-smi">disableKinds</span> = [<span class="pl-s"><span class="pl-pds">'</span>taxonomy<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>term<span class="pl-pds">'</span></span>]</pre></div>
<p>Running <code>hugo build</code> rewrites the <code>sitemap.xml</code> and <code>index.xml</code> files to use the correct domain name.</p>
<p>I haven't actually deployed a Hugo site yet, but I expect I'll try this using either <a href="https://pages.github.com/">GitHub Pages</a> or <a href="https://www.netlify.com/" rel="nofollow">Netlify</a> or <a href="https://pages.cloudflare.com/" rel="nofollow">Cloudflare Pages</a> soon.</p>


guid: tag:til.simonwillison.net,2020-04-30:llms_prompt-gemini.md
title: Running prompts against images, PDFs, audio and video with Google Gemini
author: Simon Willison
url: https://til.simonwillison.net/llms/prompt-gemini
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I'm still working towards adding multi-modal support to my <a href="https://llm.datasette.io/" rel="nofollow">LLM</a> tool. In the meantime, here are notes on running prompts against images and PDFs and audio and video files from the command-line using the <a href="https://ai.google.dev/gemini-api" rel="nofollow">Google Gemini</a> family of models.</p>
<blockquote>
<p><strong>Update</strong>: I integrated the research from this TIL into my LLM tool, which can now run multi-modal prompts against Gemini like this:</p>
<div class="highlight highlight-source-shell"><pre>llm -m gemini-1.5-flash <span class="pl-s"><span class="pl-pds">"</span>describe this image<span class="pl-pds">"</span></span> -a image.jpg</pre></div>
<p>See <a href="https://simonwillison.net/2024/Oct/29/llm-multi-modal/#atom-everything" rel="nofollow">You can now run prompts against images, audio and video in your terminal using LLM</a> for details.</p>
</blockquote>
<div class="markdown-heading"><h2 class="heading-element">Using curl</h2><a id="user-content-using-curl" class="anchor" aria-label="Permalink: Using curl" href="#using-curl"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Here's the initial recipe I figured out using <code>curl</code>.</p>
<p>The Gemini models take a JSON document sent via POST that looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"contents"</span>: [
    {
      <span class="pl-ent">"role"</span>: <span class="pl-s"><span class="pl-pds">"</span>user<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"parts"</span>: [
        {
          <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>Extract text from this image<span class="pl-pds">"</span></span>
        },
        {
          <span class="pl-ent">"inlineData"</span>: {
            <span class="pl-ent">"data"</span>: <span class="pl-s"><span class="pl-pds">"</span>... base 64 encoded image data ...<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"mimeType"</span>: <span class="pl-s"><span class="pl-pds">"</span>image/png<span class="pl-pds">"</span></span>
          }
        }
      ]
    }
  ]
}</pre></div>
<p>So the first challenge is to construct that document, including the base64 encoded image.</p>
<p>On macOS you can encode a file using <code>base64 -i image.png</code>. On other platforms you may not need the <code>-i</code> option.</p>
<p>So we can create the JSON document like this:</p>
<div class="highlight highlight-source-shell"><pre>cat <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span> &gt; input.json</span>
<span class="pl-s">{</span>
<span class="pl-s">  "contents": [</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "role": "user",</span>
<span class="pl-s">      "parts": [</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "text": "Extract text from this image"</span>
<span class="pl-s">        },</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "inlineData": {</span>
<span class="pl-s">            "data": "<span class="pl-s"><span class="pl-pds">$(</span>base64 -i image.png<span class="pl-pds">)</span></span>",</span>
<span class="pl-s">            "mimeType": "image/png"</span>
<span class="pl-s">          }</span>
<span class="pl-s">        }</span>
<span class="pl-s">      ]</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span></pre></div>
<p>This creates a <code>input.json</code> file containing the base64 encoded image, ready to be sent to the Gemini API.</p>
<p>Now we can send it using <code>curl</code>:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> GOOGLE_API_KEY=<span class="pl-s"><span class="pl-pds">'</span>... your key here ...<span class="pl-pds">'</span></span>

curl -s <span class="pl-s"><span class="pl-pds">"</span>https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-8b-latest:generateContent?key=<span class="pl-smi">$GOOGLE_API_KEY</span><span class="pl-pds">"</span></span> \
  -H <span class="pl-s"><span class="pl-pds">'</span>Content-Type: application/json<span class="pl-pds">'</span></span> \
  -X POST \
  -d @input.json</pre></div>
<p>The model name goes in the URL - here I'm using <code>gemini-1.5-flash-8b-latest</code>, Google's cheapest and fastest model.</p>
<p>Model values you can use are:</p>
<ul>
<li>
<code>gemini-1.5-flash-8b-latest</code> - the cheapest and fastest model, $0.04/million input tokens, 0.001 cents per image</li>
<li>
<code>gemini-1.5-flash-latest</code> - the one in the middle, $0.07/million input tokens, 0.0019 cents per image</li>
<li>
<code>gemini-1.5-pro-latest</code> - the most powerful model, $1.25/million input tokens, 0.0323 cents per image</li>
</ul>
<p>It's hard to overestimate how <em>cheap</em> these models are. An input image is charged at 258 tokens. That means the price per image processed is measured in fraction of a cent - those numbers above really are correct, an image even through Gemini Pro will cost less than 1/30th of a cent, and the other two models are even cheaper.</p>
<p>You get charged for output tokens too, which vary depending on the length of the response. Use <a href="https://tools.simonwillison.net/llm-prices" rel="nofollow">my LLM pricing calculator</a> to explore those.</p>
<p>The output of a prompt includes a usage section that shows you exactly how many tokens you spent. Here's example output for the prompt "extract text from this image" against this image:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/b0e18d6e-eca5-4a7a-bed8-7ffb0f0d9c68"><img src="https://github.com/user-attachments/assets/b0e18d6e-eca5-4a7a-bed8-7ffb0f0d9c68" alt="Rough handwriting black marker on white card, it reads Example handwriting Let's try this out" style="max-width: 100%;"></a></p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"candidates"</span>: [
    {
      <span class="pl-ent">"content"</span>: {
        <span class="pl-ent">"parts"</span>: [
          {
            <span class="pl-ent">"text"</span>: <span class="pl-s"><span class="pl-pds">"</span>Example handwriting<span class="pl-cce">\n</span>Let's try this out<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-ent">"role"</span>: <span class="pl-s"><span class="pl-pds">"</span>model<span class="pl-pds">"</span></span>
      },
      <span class="pl-ent">"finishReason"</span>: <span class="pl-s"><span class="pl-pds">"</span>STOP<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"safetyRatings"</span>: [
        {
          <span class="pl-ent">"category"</span>: <span class="pl-s"><span class="pl-pds">"</span>HARM_CATEGORY_HATE_SPEECH<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"probability"</span>: <span class="pl-s"><span class="pl-pds">"</span>NEGLIGIBLE<span class="pl-pds">"</span></span>
        },
        {
          <span class="pl-ent">"category"</span>: <span class="pl-s"><span class="pl-pds">"</span>HARM_CATEGORY_DANGEROUS_CONTENT<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"probability"</span>: <span class="pl-s"><span class="pl-pds">"</span>NEGLIGIBLE<span class="pl-pds">"</span></span>
        },
        {
          <span class="pl-ent">"category"</span>: <span class="pl-s"><span class="pl-pds">"</span>HARM_CATEGORY_HARASSMENT<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"probability"</span>: <span class="pl-s"><span class="pl-pds">"</span>NEGLIGIBLE<span class="pl-pds">"</span></span>
        },
        {
          <span class="pl-ent">"category"</span>: <span class="pl-s"><span class="pl-pds">"</span>HARM_CATEGORY_SEXUALLY_EXPLICIT<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"probability"</span>: <span class="pl-s"><span class="pl-pds">"</span>NEGLIGIBLE<span class="pl-pds">"</span></span>
        }
      ],
      <span class="pl-ent">"avgLogprobs"</span>: <span class="pl-c1">-0.000025986179631824296</span>
    }
  ],
  <span class="pl-ent">"usageMetadata"</span>: {
    <span class="pl-ent">"promptTokenCount"</span>: <span class="pl-c1">264</span>,
    <span class="pl-ent">"candidatesTokenCount"</span>: <span class="pl-c1">9</span>,
    <span class="pl-ent">"totalTokenCount"</span>: <span class="pl-c1">273</span>
  },
  <span class="pl-ent">"modelVersion"</span>: <span class="pl-s"><span class="pl-pds">"</span>gemini-1.5-flash-8b-001<span class="pl-pds">"</span></span>
}</pre></div>
<p>Total cost: 0.0011 cents.</p>
<div class="markdown-heading"><h2 class="heading-element">Using a Bash script</h2><a id="user-content-using-a-bash-script" class="anchor" aria-label="Permalink: Using a Bash script" href="#using-a-bash-script"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I got Claude to write me a script to automate this process. Here's how you can use it:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> GOOGLE_API_KEY=<span class="pl-s"><span class="pl-pds">'</span>... your key here ...<span class="pl-pds">'</span></span>

prompt-gemini <span class="pl-s"><span class="pl-pds">'</span>extract text from this image<span class="pl-pds">'</span></span> example-handwriting.jpg</pre></div>
<p>It accepts PNG, JPG, GIF or PDF files, automatically sending the correct <code>mimeType</code> to the API. Note that PDFs with multiple pages are charged differently - I tried a 19 page PDF and it cost 12842 tokens, suggesting around 675 tokens per page.</p>
<p>You can also add a <code>-m</code> option to specify a different model:</p>
<div class="highlight highlight-source-shell"><pre>prompt-gemini <span class="pl-s"><span class="pl-pds">'</span>extract text from this image<span class="pl-pds">'</span></span> example-handwriting.jpg -m pro</pre></div>
<p>Shortcuts <code>pro</code>, <code>flash</code> and <code>8b</code> are supported - it defaults to the cheapest 8b model.</p>
<p>By default it outputs the full JSON response, so you can see things like the <code>"usageMetadata"</code> block. To output just the raw returned text add <code>-r</code>:</p>
<div class="highlight highlight-source-shell"><pre>prompt-gemini <span class="pl-s"><span class="pl-pds">'</span>extract text from this image<span class="pl-pds">'</span></span> example-handwriting.jpg -r</pre></div>
<pre><code>Example handwriting
Let's try this out
</code></pre>
<p>Here's the script - save it somewhere on your path and run <code>chmod 755 prompt-gemini</code> to make it executable:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>

<span class="pl-c"><span class="pl-c">#</span> Check if GOOGLE_API_KEY is set</span>
<span class="pl-k">if</span> [ <span class="pl-k">-z</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$GOOGLE_API_KEY</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
    <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Error: GOOGLE_API_KEY environment variable is not set<span class="pl-pds">"</span></span> <span class="pl-k">&gt;&amp;2</span>
    <span class="pl-c1">exit</span> 1
<span class="pl-k">fi</span>

<span class="pl-c"><span class="pl-c">#</span> Default model and options</span>
model=<span class="pl-s"><span class="pl-pds">"</span>8b<span class="pl-pds">"</span></span>
prompt=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>
image_file=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>
jq_filter=<span class="pl-s"><span class="pl-pds">"</span>.<span class="pl-pds">"</span></span>

<span class="pl-c"><span class="pl-c">#</span> Parse arguments</span>
<span class="pl-k">while</span> [[ <span class="pl-smi">$#</span> <span class="pl-k">-gt</span> 0 ]]<span class="pl-k">;</span> <span class="pl-k">do</span>
    <span class="pl-k">case</span> <span class="pl-smi">$1</span> <span class="pl-k">in</span>
        -m)
            model=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$2</span><span class="pl-pds">"</span></span>
            <span class="pl-c1">shift</span> 2
            ;;
        -r)
            jq_filter=<span class="pl-s"><span class="pl-pds">"</span>.candidates[0].content.parts[0].text<span class="pl-pds">"</span></span>
            <span class="pl-c1">shift</span>
            ;;
        <span class="pl-k">*</span>)
            <span class="pl-k">if</span> [ <span class="pl-k">-z</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$prompt</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
                prompt=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$1</span><span class="pl-pds">"</span></span>
            <span class="pl-k">elif</span> [ <span class="pl-k">-z</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$image_file</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
                image_file=<span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$1</span><span class="pl-pds">"</span></span>
            <span class="pl-k">fi</span>
            <span class="pl-c1">shift</span>
            ;;
    <span class="pl-k">esac</span>
<span class="pl-k">done</span>

<span class="pl-c"><span class="pl-c">#</span> Validate prompt</span>
<span class="pl-k">if</span> [ <span class="pl-k">-z</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$prompt</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
    <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Error: No prompt provided<span class="pl-pds">"</span></span> <span class="pl-k">&gt;&amp;2</span>
    <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Usage: prompt-gemini <span class="pl-cce">\"</span>prompt<span class="pl-cce">\"</span> [image_file] [-m model] [-r]<span class="pl-pds">"</span></span> <span class="pl-k">&gt;&amp;2</span>
    <span class="pl-c1">exit</span> 1
<span class="pl-k">fi</span>

<span class="pl-c"><span class="pl-c">#</span> Map model names to full model strings</span>
<span class="pl-k">case</span> <span class="pl-smi">$model</span> <span class="pl-k">in</span>
    <span class="pl-s"><span class="pl-pds">"</span>8b<span class="pl-pds">"</span></span>|<span class="pl-s"><span class="pl-pds">"</span>flash-8b<span class="pl-pds">"</span></span>)
        model_string=<span class="pl-s"><span class="pl-pds">"</span>gemini-1.5-flash-8b-latest<span class="pl-pds">"</span></span>
        ;;
    <span class="pl-s"><span class="pl-pds">"</span>flash<span class="pl-pds">"</span></span>)
        model_string=<span class="pl-s"><span class="pl-pds">"</span>gemini-1.5-flash-latest<span class="pl-pds">"</span></span>
        ;;
    <span class="pl-s"><span class="pl-pds">"</span>pro<span class="pl-pds">"</span></span>)
        model_string=<span class="pl-s"><span class="pl-pds">"</span>gemini-1.5-pro-latest<span class="pl-pds">"</span></span>
        ;;
    <span class="pl-k">*</span>)
        model_string=<span class="pl-s"><span class="pl-pds">"</span>gemini-1.5-<span class="pl-smi">$model</span><span class="pl-pds">"</span></span>
        ;;
<span class="pl-k">esac</span>

<span class="pl-c"><span class="pl-c">#</span> Create temporary file</span>
temp_file=<span class="pl-s"><span class="pl-pds">$(</span>mktemp<span class="pl-pds">)</span></span>
<span class="pl-c1">trap</span> <span class="pl-s"><span class="pl-pds">'</span>rm -f "$temp_file"<span class="pl-pds">'</span></span> EXIT

<span class="pl-c"><span class="pl-c">#</span> Determine mime type if image file is provided</span>
<span class="pl-k">if</span> [ <span class="pl-k">-n</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$image_file</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
    <span class="pl-k">if</span> [ <span class="pl-k">!</span> <span class="pl-k">-f</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$image_file</span><span class="pl-pds">"</span></span> ]<span class="pl-k">;</span> <span class="pl-k">then</span>
        <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Error: File '<span class="pl-smi">$image_file</span>' not found<span class="pl-pds">"</span></span> <span class="pl-k">&gt;&amp;2</span>
        <span class="pl-c1">exit</span> 1
    <span class="pl-k">fi</span>

    <span class="pl-c"><span class="pl-c">#</span> Get file extension and convert to lowercase</span>
    ext=<span class="pl-s"><span class="pl-pds">$(</span>echo <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">${image_file<span class="pl-k">##*</span>.}</span><span class="pl-pds">"</span></span> <span class="pl-k">|</span> tr <span class="pl-s"><span class="pl-pds">'</span>[:upper:]<span class="pl-pds">'</span></span> <span class="pl-s"><span class="pl-pds">'</span>[:lower:]<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span>
    
    <span class="pl-k">case</span> <span class="pl-smi">$ext</span> <span class="pl-k">in</span>
        png)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>image/png<span class="pl-pds">"</span></span>
            ;;
        jpg|jpeg)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>image/jpeg<span class="pl-pds">"</span></span>
            ;;
        gif)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>image/gif<span class="pl-pds">"</span></span>
            ;;
        pdf)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>application/pdf<span class="pl-pds">"</span></span>
            ;;
        mp3)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>audio/mp3<span class="pl-pds">"</span></span>
            ;;
        mp4)
            mime_type=<span class="pl-s"><span class="pl-pds">"</span>video/mp4<span class="pl-pds">"</span></span>
            ;;
        <span class="pl-k">*</span>)
            <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Error: Unsupported file type .<span class="pl-smi">$ext</span><span class="pl-pds">"</span></span> <span class="pl-k">&gt;&amp;2</span>
            <span class="pl-c1">exit</span> 1
            ;;
    <span class="pl-k">esac</span>

    <span class="pl-c"><span class="pl-c">#</span> Create JSON with image data</span>
    cat <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span> &gt; "<span class="pl-smi">$temp_file</span>"</span>
<span class="pl-s">{</span>
<span class="pl-s">  "contents": [</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "role": "user",</span>
<span class="pl-s">      "parts": [</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "text": "<span class="pl-smi">$prompt</span>"</span>
<span class="pl-s">        },</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "inlineData": {</span>
<span class="pl-s">            "data": "<span class="pl-s"><span class="pl-pds">$(</span>base64 -i <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$image_file</span><span class="pl-pds">"</span></span><span class="pl-pds">)</span></span>",</span>
<span class="pl-s">            "mimeType": "<span class="pl-smi">$mime_type</span>"</span>
<span class="pl-s">          }</span>
<span class="pl-s">        }</span>
<span class="pl-s">      ]</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
<span class="pl-k">else</span>
    <span class="pl-c"><span class="pl-c">#</span> Create JSON without image data</span>
    cat <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span> &gt; "<span class="pl-smi">$temp_file</span>"</span>
<span class="pl-s">{</span>
<span class="pl-s">  "contents": [</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "role": "user",</span>
<span class="pl-s">      "parts": [</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "text": "<span class="pl-smi">$prompt</span>"</span>
<span class="pl-s">        }</span>
<span class="pl-s">      ]</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>
<span class="pl-k">fi</span>

<span class="pl-c"><span class="pl-c">#</span> Make API request with jq filter</span>
curl -s <span class="pl-s"><span class="pl-pds">"</span>https://generativelanguage.googleapis.com/v1beta/models/<span class="pl-smi">$model_string</span>:generateContent?key=<span class="pl-smi">$GOOGLE_API_KEY</span><span class="pl-pds">"</span></span> \
    -H <span class="pl-s"><span class="pl-pds">'</span>Content-Type: application/json<span class="pl-pds">'</span></span> \
    -X POST \
    -d @<span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$temp_file</span><span class="pl-pds">"</span></span> <span class="pl-k">|</span> jq <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$jq_filter</span><span class="pl-pds">"</span></span> -r</pre></div>
<div class="markdown-heading"><h2 class="heading-element">How I got Claude to write the Bash script</h2><a id="user-content-how-i-got-claude-to-write-the-bash-script" class="anchor" aria-label="Permalink: How I got Claude to write the Bash script" href="#how-i-got-claude-to-write-the-bash-script"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Here's the prompt I fed to Claude to create this, starting with the Bash + <code>curl</code> recipe I had already figured out:</p>
<blockquote>
<div class="highlight highlight-source-shell"><pre>cat <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOF</span> &gt; input.json</span>
<span class="pl-s">{</span>
<span class="pl-s">  "contents": [</span>
<span class="pl-s">    {</span>
<span class="pl-s">      "role": "user",</span>
<span class="pl-s">      "parts": [</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "text": "Extract text from this imaage"</span>
<span class="pl-s">        },</span>
<span class="pl-s">        {</span>
<span class="pl-s">          "inlineData": {</span>
<span class="pl-s">            "data": "<span class="pl-s"><span class="pl-pds">$(</span>base64 -i output_0.png<span class="pl-pds">)</span></span>",</span>
<span class="pl-s">            "mimeType": "image/png"</span>
<span class="pl-s">          }</span>
<span class="pl-s">        }</span>
<span class="pl-s">      ]</span>
<span class="pl-s">    }</span>
<span class="pl-s">  ]</span>
<span class="pl-s">}</span>
<span class="pl-s"><span class="pl-k">EOF</span></span>

curl <span class="pl-s"><span class="pl-pds">"</span>https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-8b-latest:generateContent?key=<span class="pl-smi">$GOOGLE_API_KEY</span><span class="pl-pds">"</span></span> \
  -H <span class="pl-s"><span class="pl-pds">'</span>Content-Type: application/json<span class="pl-pds">'</span></span> \
  -X POST \
  -d @input.json <span class="pl-k">|</span> jq</pre></div>
<p>Turn this into a Bash script that runs like this:</p>
<div class="highlight highlight-source-shell"><pre>prompt-gemini <span class="pl-s"><span class="pl-pds">"</span>this is the prompt<span class="pl-pds">"</span></span>
prompt-gemini <span class="pl-s"><span class="pl-pds">"</span>This is the prompt<span class="pl-pds">"</span></span> blah.png
prompt-gemini <span class="pl-s"><span class="pl-pds">"</span>This is the prompt<span class="pl-pds">"</span></span> blah.pdf
prompt-gemini <span class="pl-s"><span class="pl-pds">"</span>this is the prompt<span class="pl-pds">"</span></span> -m pro</pre></div>
<p>It should exit with an error if <code>GOOGLE_API_KEY</code> is not set</p>
<p>It should use a temporary file for input.json which is deleted on completion</p>
<p>If no file was provided it should skip the inlineData bit</p>
<p>It should use the correct mimeType for PNG or PDF or JPG or JPEG or GIF depending on the file extension</p>
<p>The -m option should follow the following rules: it defaults to 8b, or it can be:</p>
<p>8b =&gt; gemini-1.5-flash-8b-latest (the default)
flash-8b =&gt; gemini-1.5-flash-8b-latest
flash =&gt; gemini-1.5-flash-latest
pro =&gt; gemini-1.5-pro-latest</p>
<p>Any other value should be passed used directly in the <code>gemini-1.5-flash:generateContent</code> portion of the URL</p>
</blockquote>
<p>Here's <a href="https://gist.github.com/simonw/7cc2a9c3e612a8af502d733ff619e066">the full Claude transcript</a>.</p>
<p>Then I added the <code>-r</code> option by pasting in the previous script and prompting:</p>
<blockquote>
<p>Modify this script to add an extra <code>-r</code> option which, if present, causes the final line to pipe through <code>jq</code> like this:</p>
<pre><code>... | jq '.candidates[0].content.parts[0].text' -r
</code></pre>
</blockquote>
<p><a href="https://gist.github.com/simonw/b1bffe54ebdf3583ec4e3639fb535567">Claude transcript here</a>.</p>


guid: tag:til.simonwillison.net,2020-04-30:cloudflare_cache-control-transform-rule.md
title: Setting cache-control: max-age=31536000 with a Cloudflare Transform Rule
author: Simon Willison
url: https://til.simonwillison.net/cloudflare/cache-control-transform-rule
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I ran <a href="https://simonwillison.net/" rel="nofollow">https://simonwillison.net/</a> through <a href="https://pagespeed.web.dev/" rel="nofollow">PageSpeed Insights</a> and it warned me that my static assets were not being served with browser caching headers:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/115514c7-a51d-410b-a4dd-67c34084420d"><img src="https://github.com/user-attachments/assets/115514c7-a51d-410b-a4dd-67c34084420d" alt="A screenshot of a web performance report showing static asset cache policies. Header reads &quot;Serve static assets with an efficient cache policy — 20 resources found&quot; followed by a list of URLs from static.simonwillison.net showing file names, cache TTL status (mostly &quot;None&quot;), and transfer sizes ranging from 96 KiB to 8,871 KiB." style="max-width: 100%;"></a></p>
<p>I serve static assets for my blog (mainly images) from <code>static.simonwillison.net</code>, which is an AWS S3 bucket served via Cloudflare.</p>
<p>I investigated with <code>curl -i</code>:</p>
<div class="highlight highlight-source-shell"><pre>curl -I https://static.simonwillison.net/static/2024/prompt-gemini-extract.gif</pre></div>
<pre><code>HTTP/2 200 
date: Wed, 23 Oct 2024 20:02:52 GMT
content-type: image/gif
content-length: 160263
x-amz-id-2: LpMSRQox/dMz/qS2p2jb6OmP7/YsEa4YZnOfEVgB5biGOPynNxqs8eY5EgWW0xzbclBnOI6LTAQ=
x-amz-request-id: BK8F63EJYFP5K1YB
last-modified: Wed, 23 Oct 2024 18:18:49 GMT
etag: "38a67c2deb66fe4b6f57796c2f2f5c44"
cf-cache-status: HIT
age: 149
accept-ranges: bytes
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=xu4hL1Hr5W2ZDo6VSCd4NBfPp2tmCYiTNbct64mFfU6AAkBofX%2BHSZ6P%2FpIo3QKX5CqweeBhgS%2BYNOuzWW2E4pb4O3V1pISkOHzU0NPM%2B9QgGyuquDc%2FOl2aZM5HL3MjBPm0oKJD2AWWeyY%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8d7453c3481b9e5e-SJC
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&amp;rtt=23918&amp;sent=5&amp;recv=9&amp;lost=0&amp;retrans=0&amp;sent_bytes=2907&amp;recv_bytes=585&amp;delivery_rate=124913&amp;cwnd=177&amp;unsent_bytes=0&amp;cid=ec38180c3ba3b0e7&amp;ts=47&amp;x=0"
</code></pre>
<p>That confirms it's being served by Cloudflare that Cloudflare itself is caching the page (<code>cf-cache-status: HIT</code>) - but there's no <code>cache-control</code> header in there telling my browser to cache the image. This means the image is fetched again from Cloudflare every time the page loads.</p>
<p>I wanted a quick way to add <code>cache-control: max-age=31536000</code> headers to every one of those resources. I already have a policy that I rename a file any time I upload a new version so I may as well cache everything in the browser for 31536000 (24 * 60 * 60 * 365 seconds aka a full year).</p>
<div class="markdown-heading"><h2 class="heading-element">Using Cloudflare Transform Rules</h2><a id="user-content-using-cloudflare-transform-rules" class="anchor" aria-label="Permalink: Using Cloudflare Transform Rules" href="#using-cloudflare-transform-rules"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Cloudflare has a bewildering array of options these days. I asked Claude and it suggested using a "Transform Rule".</p>
<p>Here's the form I found in the Cloudflare dashboard for adding a new rule:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/42c0ab9f-5b78-4dfa-b684-aae2fe156e5f"><img src="https://github.com/user-attachments/assets/42c0ab9f-5b78-4dfa-b684-aae2fe156e5f" alt="A screenshot of a Cloudflare Transform Rules configuration page showing the creation of a new HTTP Response Header Modification Rule. The rule is named &quot;Cache-Control: max-age=31536000&quot; and is set to apply to requests where the hostname equals &quot;static.simonwillison.net&quot;. The rule adds a cache-control header with max-age=31536000." style="max-width: 100%;"></a></p>
<p>Cloudflare is configured for my entire <code>simonwillison.net</code> domain with the S3 bucket configured as a proxied caching CNAME to S3.</p>
<p>As such, I only wanted my rule to apply to <code>static.simonwillison.net</code>. I configured that using a custom filter expression of <code>Hostname = static.simonwillison.net</code>.</p>
<p>Then I told it to add a header name of <code>cache-control</code> and a value of <code>max-age=31536000</code> to all matching responses.</p>
<div class="markdown-heading"><h2 class="heading-element">Confirming it worked</h2><a id="user-content-confirming-it-worked" class="anchor" aria-label="Permalink: Confirming it worked" href="#confirming-it-worked"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I ran this again:</p>
<div class="highlight highlight-source-shell"><pre>curl -I https://static.simonwillison.net/static/2024/prompt-gemini-extract.gif</pre></div>
<p>And got back:</p>
<pre><code>HTTP/2 200 
date: Wed, 23 Oct 2024 20:07:12 GMT
content-type: image/gif
content-length: 160263
x-amz-id-2: LpMSRQox/dMz/qS2p2jb6OmP7/YsEa4YZnOfEVgB5biGOPynNxqs8eY5EgWW0xzbclBnOI6LTAQ=
x-amz-request-id: BK8F63EJYFP5K1YB
last-modified: Wed, 23 Oct 2024 18:18:49 GMT
etag: "38a67c2deb66fe4b6f57796c2f2f5c44"
cf-cache-status: HIT
age: 409
accept-ranges: bytes
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=2HCHW9L52YEhmN%2FbP1ZVxQLL%2FtLrpf5GTN1J%2B8dUnvC53hhSyW2PDiAHv%2B43DvJnsUsnrInVAxdZCgTsIfQ7JCg98jFzHMdu6xb3uOGAHPdC1aE9utyUGvAedNa6RVxO9yx30qkUSuiwR2k%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
cache-control: max-age=31536000
server: cloudflare
cf-ray: 8d745a2099107ac4-SJC
alt-svc: h3=":443"; ma=86400
server-timing: cfL4;desc="?proto=TCP&amp;rtt=23766&amp;sent=5&amp;recv=8&amp;lost=0&amp;retrans=0&amp;sent_bytes=2907&amp;recv_bytes=585&amp;delivery_rate=131337&amp;cwnd=117&amp;unsent_bytes=0&amp;cid=bd0475734101defd&amp;ts=55&amp;x=0"
</code></pre>
<p>Note the new <code>cache-control:</code> header in there.</p>
<p>I then tested it in Firefox. I loaded the page, opened up the network tab in browser DevTools and hit refresh. I got this:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/94b4eae4-c530-4379-b655-bea1f3689092"><img src="https://github.com/user-attachments/assets/94b4eae4-c530-4379-b655-bea1f3689092" alt="A screenshot of a browser network requests panel showing 18 HTTP requests for various resources including images, JavaScript, and icons. The requests are primarily to static.simonwillison.net, with most returning 200 status codes and &quot;cached&quot; transfer status. The page took 786ms to finish loading with DOMContentLoaded at 59ms." style="max-width: 100%;"></a></p>
<p>Note the summary that says "1.38MB / 1.62 kB transferred" - the caching is working extremely well.</p>
<div class="markdown-heading"><h2 class="heading-element">Bonus: access-control-allow-origin</h2><a id="user-content-bonus-access-control-allow-origin" class="anchor" aria-label="Permalink: Bonus: access-control-allow-origin" href="#bonus-access-control-allow-origin"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I have an S3 bucket behind Cloudflare and I wanted a quick and easy way to put files online that are served with <code>access-control-allow-origin: *</code>, initially so I could experiment with things like <a href="https://lite.datasette.io/" rel="nofollow">Datasette Lite</a> and the <a href="https://duckdb.org/docs/api/wasm/overview.html" rel="nofollow">DuckDB WASM Shell</a>.</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/bd476176-a942-47af-9d73-c9cbde476f89"><img src="https://github.com/user-attachments/assets/bd476176-a942-47af-9d73-c9cbde476f89" alt="Screenshot of a CORS configuration interface with fields for rule name &quot;static.simonwillis.net/static/cors-allow/&quot;, custom filter expression selected, URI Full field with wildcard operator and value &quot;https://static.simonwillison.net/static/cors-allow/&quot;, and response header configuration for access-control-allow-origin set to &quot;*&quot;" style="max-width: 100%;"></a></p>


guid: tag:til.simonwillison.net,2020-04-30:python_uv-cli-apps.md
title: Using uv to develop Python command-line applications
author: Simon Willison
url: https://til.simonwillison.net/python/uv-cli-apps
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I finally figured out a process that works for me for hacking on Python CLI utilities using <a href="https://docs.astral.sh/uv/" rel="nofollow">uv</a> to manage my development environment, thanks to a little bit of help from Charlie Marsh.</p>
<div class="markdown-heading"><h2 class="heading-element">Starting a new app with cookiecutter</h2><a id="user-content-starting-a-new-app-with-cookiecutter" class="anchor" aria-label="Permalink: Starting a new app with cookiecutter" href="#starting-a-new-app-with-cookiecutter"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I already have a <a href="https://cookiecutter.readthedocs.io/" rel="nofollow">cookiecutter</a> template I like using for CLI applications: <a href="https://github.com/simonw/click-app">simonw/click-app</a>.</p>
<p>Thanks to <code>uvx</code> I don't even need to install <code>cookiecutter</code> to use it:</p>
<div class="highlight highlight-source-shell"><pre>uvx cookiecutter gh:simonw/click-app</pre></div>
<p>This outputs a set of questions:</p>
<pre><code>  [1/6] app_name (): demo-app
  [2/6] description (): Demo
  [3/6] hyphenated (demo-app): 
  [4/6] underscored (demo_app): 
  [5/6] github_username (): simonw
  [6/6] author_name (): Simon Willison
</code></pre>
<p>Which creates a <code>demo-app</code> directory containing the skeleton of a Python project.</p>
<div class="markdown-heading"><h2 class="heading-element">Setting up the uv virtual environment</h2><a id="user-content-setting-up-the-uv-virtual-environment" class="anchor" aria-label="Permalink: Setting up the uv virtual environment" href="#setting-up-the-uv-virtual-environment"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p><code>uv</code> has a number of different commands that can create and work with a <code>.venv</code> virtual environment directory.</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">cd</span> demo-app</pre></div>
<p>In this case, my <code>pyproject.toml</code> file (created by that cookiecutter template) defines a separate block of test dependencies. Here's that TOML file in full:</p>
<div class="highlight highlight-source-toml"><pre>[<span class="pl-en">project</span>]
<span class="pl-smi">name</span> = <span class="pl-s"><span class="pl-pds">"</span>demo-app<span class="pl-pds">"</span></span>
<span class="pl-smi">version</span> = <span class="pl-s"><span class="pl-pds">"</span>0.1<span class="pl-pds">"</span></span>
<span class="pl-smi">description</span> = <span class="pl-s"><span class="pl-pds">"</span>Demo<span class="pl-pds">"</span></span>
<span class="pl-smi">readme</span> = <span class="pl-s"><span class="pl-pds">"</span>README.md<span class="pl-pds">"</span></span>
<span class="pl-smi">authors</span> = [{<span class="pl-smi">name</span> = <span class="pl-s"><span class="pl-pds">"</span>Simon Willison<span class="pl-pds">"</span></span>}]
<span class="pl-smi">license</span> = {<span class="pl-smi">text</span> = <span class="pl-s"><span class="pl-pds">"</span>Apache-2.0<span class="pl-pds">"</span></span>}
<span class="pl-smi">requires-python</span> = <span class="pl-s"><span class="pl-pds">"</span>&gt;=3.8<span class="pl-pds">"</span></span>
<span class="pl-smi">classifiers</span> = [
    <span class="pl-s"><span class="pl-pds">"</span>License :: OSI Approved :: Apache Software License<span class="pl-pds">"</span></span>
]
<span class="pl-smi">dependencies</span> = [
    <span class="pl-s"><span class="pl-pds">"</span>click<span class="pl-pds">"</span></span>
]

[<span class="pl-en">build-system</span>]
<span class="pl-smi">requires</span> = [<span class="pl-s"><span class="pl-pds">"</span>setuptools<span class="pl-pds">"</span></span>]
<span class="pl-smi">build-backend</span> = <span class="pl-s"><span class="pl-pds">"</span>setuptools.build_meta<span class="pl-pds">"</span></span>

[<span class="pl-en">project</span>.<span class="pl-en">urls</span>]
<span class="pl-smi">Homepage</span> = <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/demo-app<span class="pl-pds">"</span></span>
<span class="pl-smi">Changelog</span> = <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/demo-app/releases<span class="pl-pds">"</span></span>
<span class="pl-smi">Issues</span> = <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/demo-app/issues<span class="pl-pds">"</span></span>
<span class="pl-smi">CI</span> = <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/demo-app/actions<span class="pl-pds">"</span></span>

[<span class="pl-en">project</span>.<span class="pl-en">scripts</span>]
<span class="pl-smi">demo-app</span> = <span class="pl-s"><span class="pl-pds">"</span>demo_app.cli:cli<span class="pl-pds">"</span></span>

[<span class="pl-en">project</span>.<span class="pl-en">optional-dependencies</span>]
<span class="pl-smi">test</span> = [<span class="pl-s"><span class="pl-pds">"</span>pytest<span class="pl-pds">"</span></span>]</pre></div>
<p>The <code>[project.optional-dependencies]</code> section lists that <code>test</code> block. I can create a new virtual environment in <code>.venv/</code> and install both my project dependencies and those test dependencies like this:</p>
<div class="highlight highlight-source-shell"><pre>uv sync --extra <span class="pl-c1">test</span></pre></div>
<p>Here's the output:</p>
<pre><code>Using CPython 3.11.1
Creating virtual environment at: .venv
Resolved 9 packages in 207ms
   Built demo-app @ file:///private/tmp/for-uv/demo-app
Prepared 1 package in 614ms
Installed 6 packages in 8ms
 + click==8.1.7
 + demo-app==0.1 (from file:///private/tmp/for-uv/demo-app)
 + iniconfig==2.0.0
 + packaging==24.1
 + pluggy==1.5.0
 + pytest==8.3.3
</code></pre>
<div class="markdown-heading"><h2 class="heading-element">Running the tests</h2><a id="user-content-running-the-tests" class="anchor" aria-label="Permalink: Running the tests" href="#running-the-tests"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Now I can run <code>pytest</code> using the <code>uv run</code> command:</p>
<div class="highlight highlight-source-shell"><pre>uv run pytest</pre></div>
<pre><code>==================== test session starts ====================
platform darwin -- Python 3.11.1, pytest-8.3.3, pluggy-1.5.0
rootdir: /private/tmp/for-uv/demo-app
configfile: pyproject.toml
collected 1 item                                                                                          

tests/test_demo_app.py .                                                                            [100%]

===================== 1 passed in 0.03s =====================
</code></pre>
<p>This runs the <code>pytest</code> binary in the current <code>.venv/</code> environment. Note that I no longer have to "activate my virtual environment" - using <code>uv run</code> habitually solves that for me.</p>
<div class="markdown-heading"><h2 class="heading-element">Running the CLI tool itself</h2><a id="user-content-running-the-cli-tool-itself" class="anchor" aria-label="Permalink: Running the CLI tool itself" href="#running-the-cli-tool-itself"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>This line in <code>pyproject.toml</code> defines a script entry point for my CLI tool:</p>
<div class="highlight highlight-source-toml"><pre>[<span class="pl-en">project</span>.<span class="pl-en">scripts</span>]
<span class="pl-smi">demo-app</span> = <span class="pl-s"><span class="pl-pds">"</span>demo_app.cli:cli<span class="pl-pds">"</span></span></pre></div>
<p>If the tool is correctly installed, I should be able to run it like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run demo-app</pre></div>
<pre><code>Usage: demo-app [OPTIONS] COMMAND [ARGS]...

  Demo

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  command  Command description goes here
</code></pre>
<p>I can also run it via Python like this (producing the same output):</p>
<div class="highlight highlight-source-shell"><pre>uv run python -m demo_app</pre></div>
<p>Crucially, the only reason this works is that I included <em>this</em> section in <code>pyproject.toml</code>:</p>
<div class="highlight highlight-source-shell"><pre>[build-system]
requires = [<span class="pl-s"><span class="pl-pds">"</span>setuptools<span class="pl-pds">"</span></span>]
build-backend = <span class="pl-s"><span class="pl-pds">"</span>setuptools.build_meta<span class="pl-pds">"</span></span></pre></div>
<p>This may seem unrelated, but it's necessary for the <code>demo-app</code> alias to be correctly installed. As Charlie Marsh explained it:</p>
<blockquote>
<p>We support two kinds of projects: packages and non-packages. You want the former in this case, because you actually want to install the package in the environment. (We used to require this, but a lot of people want to be able to create lightweight projects that are just collections of scripts and don't need to be buildable / installable into an environment. We call those non-package projects.)</p>
<p>We consider a project to be a "package" if <code>[build-system]</code> is defined or you set <code>tool.uv.package = true</code></p>
<p>Otherwise, we don't install the project itself into the environment.</p>
</blockquote>
<div class="markdown-heading"><h2 class="heading-element">Using dev-dependencies instead</h2><a id="user-content-using-dev-dependencies-instead" class="anchor" aria-label="Permalink: Using dev-dependencies instead" href="#using-dev-dependencies-instead"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>The only reason I needed to use <code>uv sync</code> here was to specify that <code>--extra test</code> to get my test dependencies installed as well.</p>
<p>As an aside, the following would have worked instead:</p>
<div class="highlight highlight-source-shell"><pre>uv run --extra <span class="pl-c1">test</span> pytest</pre></div>
<p>I'd only need to pass that <code>--extra test</code> option the first time I ran <code>uv run</code> - on subsequent runs the test dependencies would already be installed.</p>
<p>Another option here would be to use the newer concept of <a href="https://docs.astral.sh/uv/concepts/dependencies/#development-dependencies" rel="nofollow">dev-dependencies</a>. <code>uv</code> supports these right now, and they've just been standardized by <a href="https://peps.python.org/pep-0735/" rel="nofollow">PEP 735</a>. To use those, add this to the <code>pyproject.toml</code> file:</p>
<div class="highlight highlight-source-toml"><pre>[<span class="pl-en">tool</span>.<span class="pl-en">uv</span>]
<span class="pl-smi">dev-dependencies</span> = [<span class="pl-s"><span class="pl-pds">"</span>pytest<span class="pl-pds">"</span></span>]</pre></div>
<p>Then <code>uv run pytest</code> would work without needing to use <code>--extra</code> to ensure the test dependencies are installed, and without needing to use <code>uv sync</code> at all.</p>
<div class="markdown-heading"><h2 class="heading-element">There's no need for uv pip</h2><a id="user-content-theres-no-need-for-uv-pip" class="anchor" aria-label="Permalink: There's no need for uv pip" href="#theres-no-need-for-uv-pip"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I got into a tangle at first trying to figure this out, because I thought I needed to use <code>uv pip</code> to manage my environment... and it turns out <code>uv pip</code> followed <a href="https://docs.astral.sh/uv/pip/environments/#discovery-of-python-environments" rel="nofollow">these rules</a>:</p>
<blockquote>
<p>When running a command that mutates an environment such as <code>uv pip sync</code> or <code>uv pip install</code>, uv will search for a virtual environment in the following order:</p>
<ul>
<li>An activated virtual environment based on the <code>VIRTUAL_ENV</code> environment variable.</li>
<li>An activated Conda environment based on the <code>CONDA_PREFIX</code> environment variable.</li>
<li>A virtual environment at <code>.venv</code> in the current directory, or in the nearest parent directory.</li>
</ul>
</blockquote>
<p><strong>Update:</strong> This changed in <a href="https://github.com/astral-sh/uv/releases/tag/0.5.0">uv 0.5.0</a> so Conda should no longer result in this confusion.</p>
<p>I had Conda installed, which means I had a <code>CONDA_PREFIX</code> environment variable set, which meant <code>uv pip</code> was ignoring my <code>.venv</code> directory entirely and using the Conda environment instead!</p>
<p>This caused all manner of confusion. I <a href="https://gist.github.com/simonw/975dfa41e9b03bca2513a986d9aa3dcf">put together this document</a> and asked Charlie for help, and he graciously unblocked me.</p>


guid: tag:til.simonwillison.net,2020-04-30:python_installing-flash-attention.md
title: Installing flash-attn without compiling it
author: Simon Willison
url: https://til.simonwillison.net/python/installing-flash-attention
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>If you ever run into instructions that tell you to do this:</p>
<div class="highlight highlight-source-shell"><pre>pip install flash-attn --no-build-isolation</pre></div>
<p><strong>Do not try to do this</strong>. It is a trap. For some reason attempting to install this runs a compilation process which can take <em>multiple hours</em>. I tried to run this in Google Colab on an A100 machine that I was paying for and burned through $2 worth of "compute units" and an hour and a half of waiting before I gave up.</p>
<blockquote>
<p><strong>Update</strong>: I may be wrong about this, <a href="https://github.com/Dao-AILab/flash-attention/blob/c1d146cbd5becd9e33634b1310c2d27a49c7e862/setup.py#L54-L56">the setup.py</a> for the project includes code that attempts to install wheels directly from the GitHub releases. That didn't work for me and I don't understand why.</p>
</blockquote>
<p>Thankfully <a href="https://twitter.com/Sampson4242/status/1849666226299281443" rel="nofollow">I learned</a> that there's an alternative: the Flash Attention team provide pre-built wheels for their project exclusively through GitHub releases. You can find them attached to the most recent release on <a href="https://github.com/Dao-AILab/flash-attention/releases">https://github.com/Dao-AILab/flash-attention/releases</a></p>
<p>But which one should you use out of the 83 files listed there?</p>
<p>Google Colab has a "ask Gemini" feature so I tried "Give me as many clues as possible as to what flash attention wheel filename would work on this system" and it suggested I look for a <code>cp310</code> one (for Python 3.10) on <code>linux_x86_64</code> (Colab runs on Linux).</p>
<p>In browsing through the list of 83 options I thought <code>flash_attn-2.6.3+cu123torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl</code> might be the right one (shrug?). So I tried this:</p>
<pre><code>!wget https://github.com/Dao-AILab/flash-attention/releases/download/v2.6.3/flash_attn-2.6.3+cu123torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
!pip install --no-dependencies --upgrade flash_attn-2.6.3+cu123torch2.4cxx11abiFALSE-cp310-cp310-linux_x86_64.whl
</code></pre>
<p>This <em>seemed</em> to work (and installed in just a couple of seconds):</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">flash_attn</span>
<span class="pl-s1">flash_attn</span>.<span class="pl-s1">__version__</span></pre></div>
<pre><code>2.6.3
</code></pre>
<p>But the thing I was trying to run (<a href="https://github.com/deepseek-ai/Janus">deepseek-ai/Janus</a>) failed with an error:</p>
<blockquote>
<p><code>NameError: name '_flash_supports_window_size' is not defined</code></p>
</blockquote>
<p>At this point I gave up. It's possible I picked the wrong wheel, but there may have been something else wrong.</p>


guid: tag:til.simonwillison.net,2020-04-30:llms_docs-from-tests.md
title: Generating documentation from tests using files-to-prompt and LLM
author: Simon Willison
url: https://til.simonwillison.net/llms/docs-from-tests
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I was experimenting with <a href="https://github.com/bytecodealliance/wasmtime-py">wasmtime-py</a> today and found the <a href="https://bytecodealliance.github.io/wasmtime-py/" rel="nofollow">current documentation</a> didn't quite give me the information that I needed.</p>
<p>The package has <a href="https://github.com/bytecodealliance/wasmtime-py/tree/main/tests">a solid looking test suite</a>, so I decided to see if I could generate additional documentation based on that.</p>
<p>I started with a checkout of the repo:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">cd</span> /tmp
git clone https://github.com/bytecodealliance/wasmtime-py</pre></div>
<p>The tests are all in the <code>test/</code> folder, so I used my <a href="https://github.com/simonw/files-to-prompt">files-to-prompt</a> tool to turn every <code>.py</code> file in that folder into a single prompt, using the XML-ish format that Claude likes (the <code>-c</code> option):</p>
<div class="highlight highlight-source-shell"><pre>files-to-prompt wasmtime-py/tests -e py -c</pre></div>
<p>Piping that through <a href="https://github.com/simonw/ttok">ttok</a> shows that it's about 34,780 OpenAI tokens. I pasted the whole thing (<code> | pbcopy</code> to copy to my clipboard) into my <a href="https://tools.simonwillison.net/claude-token-counter" rel="nofollow">Claude token counter</a> tool and got 43,490 - easily enough to fit Claude 3.5 Sonnet's 200,000 limit.</p>
<p>So I ran this:</p>
<div class="highlight highlight-source-shell"><pre>files-to-prompt -e py wasmtime-py/tests -c <span class="pl-k">|</span> \
  llm -m claude-3.5-sonnet -s \
  <span class="pl-s"><span class="pl-pds">'</span>write detailed usage documentation including realistic examples<span class="pl-pds">'</span></span></pre></div>
<p>I'm using <a href="https://github.com/simonw/llm-claude-3">llm-claude-3</a> here and a system prompt of:</p>
<blockquote>
<p><code>write detailed usage documentation including realistic examples</code></p>
</blockquote>
<p>Here's a short excerpt of the result:</p>
<blockquote>
<div class="markdown-heading"><h2 class="heading-element">Basic Usage</h2><a id="user-content-basic-usage" class="anchor" aria-label="Permalink: Basic Usage" href="#basic-usage"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="markdown-heading"><h3 class="heading-element">Engine and Store</h3><a id="user-content-engine-and-store" class="anchor" aria-label="Permalink: Engine and Store" href="#engine-and-store"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">wasmtime</span> <span class="pl-k">import</span> <span class="pl-v">Engine</span>, <span class="pl-v">Store</span>

<span class="pl-c"># Create an engine</span>
<span class="pl-s1">engine</span> <span class="pl-c1">=</span> <span class="pl-v">Engine</span>()

<span class="pl-c"># Create a store </span>
<span class="pl-s1">store</span> <span class="pl-c1">=</span> <span class="pl-v">Store</span>(<span class="pl-s1">engine</span>)</pre></div>
<p>The Engine handles compilation settings and caching. The Store holds runtime state for instances.</p>
<div class="markdown-heading"><h3 class="heading-element">Loading Modules</h3><a id="user-content-loading-modules" class="anchor" aria-label="Permalink: Loading Modules" href="#loading-modules"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">wasmtime</span> <span class="pl-k">import</span> <span class="pl-v">Module</span>

<span class="pl-c"># From WAT text format</span>
<span class="pl-s1">module</span> <span class="pl-c1">=</span> <span class="pl-v">Module</span>(<span class="pl-s1">engine</span>, <span class="pl-s">'(module)'</span>)

<span class="pl-c"># From binary Wasm</span>
<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s">'module.wasm'</span>, <span class="pl-s">'rb'</span>) <span class="pl-k">as</span> <span class="pl-s1">f</span>:
    <span class="pl-s1">module</span> <span class="pl-c1">=</span> <span class="pl-v">Module</span>(<span class="pl-s1">engine</span>, <span class="pl-s1">f</span>.<span class="pl-en">read</span>())</pre></div>
</blockquote>
<p>You can see <a href="https://gist.github.com/simonw/351cffbd254af5cbf329377fb95fcc13">the full transcript here</a>, saved using <code>llm logs -c | pbcopy</code> (I then hand-edited in a <code>&lt;details&gt;&lt;summary&gt;</code> to hide the lengthy piped input). As always I'm sharing the transcript in a private Gist to keep this AI-generated, unverified text from being indexed by search engines.</p>
<p>At the end of the first chunk of output Claude offered the following:</p>
<blockquote>
<p>This documentation covers the core functionality. The bindings also support more advanced features like:</p>
<ul>
<li>Component model and interface types</li>
<li>Resource types and references</li>
<li>Custom linking and importing</li>
<li>Memory management controls</li>
<li>Execution limits and interruption</li>
</ul>
<p>Let me know if you would like me to expand on any of these topics!</p>
</blockquote>
<p>So I followed up with another prompt (using <code>llm -c</code> for "continue current conversation"):</p>
<div class="highlight highlight-source-shell"><pre>llm -c <span class="pl-s"><span class="pl-pds">'</span>write a detailed section about memory management and one about execution limits<span class="pl-pds">'</span></span></pre></div>
<p>This produced <a href="https://gist.github.com/simonw/351cffbd254af5cbf329377fb95fcc13#response-1">a useful continuation of the documentation</a>.</p>
<p>How good is this documentation? It's pretty solid! The <em>only</em> thing it had to go on was the content of those tests, so I can be reasonably confident it didn't make any glaringly terrible mistakes and that the examples it gave me are more likely than not to execute.</p>
<p>Someone with more depth of experience with the project than me could take this as an initial draft and iterate on it to create verified, generally useful documentation.</p>


guid: tag:til.simonwillison.net,2020-04-30:github-actions_cog.md
title: Running cog automatically against GitHub pull requests
author: Simon Willison
url: https://til.simonwillison.net/github-actions/cog
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I really like <a href="https://nedbatchelder.com/code/cog/" rel="nofollow">Cog</a> (<a href="https://til.simonwillison.net/python/cog-to-update-help-in-readme" rel="nofollow">previously</a>) as a tool for automating aspects of my Python project documentation - things like the SQL schemas shown on the <a href="https://llm.datasette.io/en/latest/logging.html#sql-schema" rel="nofollow">LLM logging page</a>.</p>
<p>When using <code>cog</code> in this way it's important to remember to run <code>cog -r</code> to update those generated files before pushing a commit.</p>
<p>I've previously been enforcing this using GitHub Actions - as part of my tests I run <code>cog --check *.md</code> so that my test suite fails if the generated files are out of date.</p>
<p>This morning I switched to a friendlier version. I now run a GitHub Actions workflow against any created or modified pull requests and, if <code>cog</code> needs to be run, the workflow runs it and then helpfully commits the files back to that repo.</p>
<p>I <a href="https://gist.github.com/simonw/28a9e7438f3115c27dc5fbf0d5690ebf">got Claude to write the initial workflow</a>, and then iterated on it to get it working.</p>
<p>This goes in <code>.github/workflows/cog.yml</code>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Run Cog</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">pull_request</span>:
    <span class="pl-ent">types</span>: <span class="pl-s">[opened, synchronize]</span>

<span class="pl-ent">permissions</span>:
  <span class="pl-ent">contents</span>: <span class="pl-s">write</span>
  <span class="pl-ent">pull-requests</span>: <span class="pl-s">write</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">run-cog</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v4</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">ref</span>: <span class="pl-s">${{ github.head_ref }}</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python 3.11</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v5</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">python-version</span>: <span class="pl-s"><span class="pl-pds">'</span>3.11<span class="pl-pds">'</span></span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
        <span class="pl-ent">run</span>: <span class="pl-s">pip install -e '.[test]'</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Run cog</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          cog -r docs/**/*.md docs/*.md</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Check for changes</span>
        <span class="pl-ent">id</span>: <span class="pl-s">check-changes</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          if [ -n "$(git diff)" ]; then</span>
<span class="pl-s">            echo "changes=true" &gt;&gt; $GITHUB_OUTPUT</span>
<span class="pl-s">          else</span>
<span class="pl-s">            echo "changes=false" &gt;&gt; $GITHUB_OUTPUT</span>
<span class="pl-s">          fi</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push if changed</span>
        <span class="pl-ent">if</span>: <span class="pl-s">steps.check-changes.outputs.changes == 'true'</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          git config --local user.email "github-actions[bot]@users.noreply.github.com"</span>
<span class="pl-s">          git config --local user.name "github-actions[bot]"</span>
<span class="pl-s">          git add -A</span>
<span class="pl-s">          git commit -m "Ran cog"</span>
<span class="pl-s">          git push</span></pre></div>
<p>The <code>[opened, synchronize]</code> trigger means that the workflow will run whenever a pull request is either created or updated in some way.</p>
<p>Using that <code>user.email</code> and <code>user.name</code> causes the bot to get a nice icon of its own:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/d4a59c91-8ad4-46a6-9e85-74a53badf766"><img src="https://github.com/user-attachments/assets/d4a59c91-8ad4-46a6-9e85-74a53badf766" alt="List of commits on GitHub - the last one reads Ran cog and shows a GitHub avatar icon" style="max-width: 100%;"></a></p>
<p>Squash-merging the PR causes the bot to be <a href="https://github.com/simonw/llm/commit/febbc04fb6d847bd35ce5ba0156ef010f5894564">credited as a co-author</a>.</p>


guid: tag:til.simonwillison.net,2020-04-30:cloudflare_workers-github-oauth.md
title: GitHub OAuth for a static site using Cloudflare Workers
author: Simon Willison
url: https://til.simonwillison.net/cloudflare/workers-github-oauth
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>My <a href="https://tools.simonwillison.net/" rel="nofollow">tools.simonwillison.net</a> site is a growing collection of small HTML and JavaScript applications hosted as static files on GitHub Pages.</p>
<p>Many of those tools take advantage of external APIs such as those provided by OpenAI and Anthropic and Google Gemini, thanks to the increasingly common <code>access-control-allow-origin: *</code> <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS" rel="nofollow">CORS header</a>.</p>
<p>I want to start building tools that work with <a href="https://docs.github.com/en/rest">the GitHub API</a>, in order to implement things like saving data to a <a href="https://gist.github.com/">Gist</a></p>
<p>To do that, I needed to implement <a href="https://docs.github.com/en/apps/oauth-apps">OAuth</a>: redirecting users to GitHub to request permission to access their data and then storing an access token in their browser's <code>localStorage</code> to be used by JavaScript running on my site.</p>
<p>There is just one catch: it currently isn't possible to implement GitHub OAuth entirely from the client, because that API depends on a secret that must be held server-side and cannot be exposed.</p>
<p>This morning, I had an idea: my tools site is <a href="https://github.com/simonw/tools">hosted by GitHub Pages</a>, but it's served via my <a href="https://www.cloudflare.com/" rel="nofollow">Cloudflare</a> account for the <code>simonwillison.net</code> domain.</p>
<p>Could I spin up a tiny <a href="https://workers.cloudflare.com/" rel="nofollow">Cloudflare Workers</a> server-side script implementing GitHub OAuth and add it to a path on that <code>tools</code> subdomain?</p>
<p>The answer turned out to be yes.</p>
<div class="markdown-heading"><h2 class="heading-element">Getting Claude to write me a Worker</h2><a id="user-content-getting-claude-to-write-me-a-worker" class="anchor" aria-label="Permalink: Getting Claude to write me a Worker" href="#getting-claude-to-write-me-a-worker"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I prompted <a href="https://claude.ai/" rel="nofollow">Claude</a> with the following:</p>
<blockquote>
<p><code>Write a Cloudflare worker that implements an oauth flow with GitHub to get a token scoped for gist read and write only</code></p>
<p><code>Landing on the worker page redirects to GitHub for the oauth - GitHub sends back to the same page, which then outputs a script block that sets the access key in localstorage</code></p>
</blockquote>
<p>This is the simplest possible design for an OAuth flow: send the user straight to GitHub, then exchange the resulting <code>?code=</code> for an access token and write that to <code>localStorage</code>.</p>
<p>Here's <a href="https://gist.github.com/simonw/975b8934066417fe771561a1b672ad4f">the full Claude transcript</a>. Claude gave me almost <em>exactly</em> what I needed - the only missing detail was that it set the <code>redirectUri</code> to <code>url.origin</code> (just the site domain) when it should have been the full URL to the worker page.</p>
<p>I tweaked the code to fix this, and later again to add error handling and then to address a potential security issue. My currently deployed code looks like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">export</span> <span class="pl-k">default</span> <span class="pl-kos">{</span>
  <span class="pl-k">async</span> <span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s1">request</span><span class="pl-kos">,</span> <span class="pl-s1">env</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">const</span> <span class="pl-en">generateHTML</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-kos">{</span> title<span class="pl-kos">,</span> content<span class="pl-kos">,</span> isError <span class="pl-c1">=</span> <span class="pl-c1">false</span><span class="pl-kos">,</span> headers <span class="pl-c1">=</span> <span class="pl-kos">{</span><span class="pl-kos">}</span> <span class="pl-kos">}</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
      <span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span><span class="pl-s">`</span>
<span class="pl-s">        &lt;!DOCTYPE html&gt;</span>
<span class="pl-s">        &lt;html&gt;</span>
<span class="pl-s">          &lt;head&gt;</span>
<span class="pl-s">            &lt;title&gt;<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">title</span><span class="pl-kos">}</span></span>&lt;/title&gt;</span>
<span class="pl-s">            &lt;style&gt;</span>
<span class="pl-s">              body {</span>
<span class="pl-s">                font-family: -apple-system, system-ui, sans-serif;</span>
<span class="pl-s">                padding: 2rem;</span>
<span class="pl-s">                max-width: 600px;</span>
<span class="pl-s">                margin: 0 auto;</span>
<span class="pl-s">                text-align: center;</span>
<span class="pl-s">              }</span>
<span class="pl-s">              .message {</span>
<span class="pl-s">                padding: 1rem;</span>
<span class="pl-s">                margin: 1rem 0;</span>
<span class="pl-s">                border-radius: 4px;</span>
<span class="pl-s">                background-color: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">isError</span> ? <span class="pl-s">'#ffebee'</span> : <span class="pl-s">'#e8f5e9'</span><span class="pl-kos">}</span></span>;</span>
<span class="pl-s">                border: 1px solid <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">isError</span> ? <span class="pl-s">'#ffcdd2'</span> : <span class="pl-s">'#c8e6c9'</span><span class="pl-kos">}</span></span>;</span>
<span class="pl-s">                color: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">isError</span> ? <span class="pl-s">'#b71c1c'</span> : <span class="pl-s">'#2e7d32'</span><span class="pl-kos">}</span></span>;</span>
<span class="pl-s">              }</span>
<span class="pl-s">            &lt;/style&gt;</span>
<span class="pl-s">          &lt;/head&gt;</span>
<span class="pl-s">          &lt;body&gt;</span>
<span class="pl-s">            &lt;div class="message"&gt;</span>
<span class="pl-s">              <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">content</span><span class="pl-kos">}</span></span></span>
<span class="pl-s">            &lt;/div&gt;</span>
<span class="pl-s">            <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">isError</span> ? <span class="pl-s">'&lt;p&gt;Please close this window and try again. If the problem persists, contact support.&lt;/p&gt;'</span> : <span class="pl-s">''</span><span class="pl-kos">}</span></span></span>
<span class="pl-s">          &lt;/body&gt;</span>
<span class="pl-s">        &lt;/html&gt;</span>
<span class="pl-s">      `</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
        <span class="pl-c1">headers</span>: <span class="pl-kos">{</span>
          <span class="pl-s">'Content-Type'</span>: <span class="pl-s">'text/html'</span><span class="pl-kos">,</span>
          ...<span class="pl-s1">headers</span>
        <span class="pl-kos">}</span><span class="pl-kos">,</span>
        <span class="pl-c1">status</span>: <span class="pl-s1">isError</span> ? <span class="pl-c1">400</span> : <span class="pl-c1">200</span>
      <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span><span class="pl-kos">;</span>

    <span class="pl-k">try</span> <span class="pl-kos">{</span>
      <span class="pl-k">const</span> <span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">url</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">clientId</span> <span class="pl-c1">=</span> <span class="pl-s1">env</span><span class="pl-kos">.</span><span class="pl-c1">GITHUB_CLIENT_ID</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">clientSecret</span> <span class="pl-c1">=</span> <span class="pl-s1">env</span><span class="pl-kos">.</span><span class="pl-c1">GITHUB_CLIENT_SECRET</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">redirectUri</span> <span class="pl-c1">=</span> <span class="pl-s1">env</span><span class="pl-kos">.</span><span class="pl-c1">GITHUB_REDIRECT_URI</span><span class="pl-kos">;</span>
      
      <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-c1">!</span><span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">has</span><span class="pl-kos">(</span><span class="pl-s">'code'</span><span class="pl-kos">)</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-c">// Initial authorization request</span>
        <span class="pl-k">const</span> <span class="pl-s1">state</span> <span class="pl-c1">=</span> <span class="pl-s1">crypto</span><span class="pl-kos">.</span><span class="pl-en">randomUUID</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-k">const</span> <span class="pl-s1">githubAuthUrl</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s">'https://github.com/login/oauth/authorize'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-s1">githubAuthUrl</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">set</span><span class="pl-kos">(</span><span class="pl-s">'client_id'</span><span class="pl-kos">,</span> <span class="pl-s1">clientId</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-s1">githubAuthUrl</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">set</span><span class="pl-kos">(</span><span class="pl-s">'redirect_uri'</span><span class="pl-kos">,</span> <span class="pl-s1">redirectUri</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-s1">githubAuthUrl</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">set</span><span class="pl-kos">(</span><span class="pl-s">'scope'</span><span class="pl-kos">,</span> <span class="pl-s">'gist'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-s1">githubAuthUrl</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">set</span><span class="pl-kos">(</span><span class="pl-s">'state'</span><span class="pl-kos">,</span> <span class="pl-s1">state</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

        <span class="pl-c">// Create headers object with the state cookie</span>
        <span class="pl-k">const</span> <span class="pl-s1">headers</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Headers</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
          <span class="pl-s">'Location'</span>: <span class="pl-s1">githubAuthUrl</span><span class="pl-kos">.</span><span class="pl-en">toString</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
          <span class="pl-s">'Set-Cookie'</span>: <span class="pl-s">`github_auth_state=<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">state</span><span class="pl-kos">}</span></span>; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=3600`</span>
        <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

        <span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span><span class="pl-c1">null</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
          <span class="pl-c1">status</span>: <span class="pl-c1">302</span><span class="pl-kos">,</span>
          headers
        <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span>

      <span class="pl-c">// Callback handling</span>
      <span class="pl-k">const</span> <span class="pl-s1">returnedState</span> <span class="pl-c1">=</span> <span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-s">'state'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">cookies</span> <span class="pl-c1">=</span> <span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">headers</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-s">'Cookie'</span><span class="pl-kos">)</span> <span class="pl-c1">||</span> <span class="pl-s">''</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">stateCookie</span> <span class="pl-c1">=</span> <span class="pl-s1">cookies</span><span class="pl-kos">.</span><span class="pl-en">split</span><span class="pl-kos">(</span><span class="pl-s">';'</span><span class="pl-kos">)</span>
        <span class="pl-kos">.</span><span class="pl-en">map</span><span class="pl-kos">(</span><span class="pl-s1">cookie</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">cookie</span><span class="pl-kos">.</span><span class="pl-en">trim</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
        <span class="pl-kos">.</span><span class="pl-en">find</span><span class="pl-kos">(</span><span class="pl-s1">cookie</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">cookie</span><span class="pl-kos">.</span><span class="pl-en">startsWith</span><span class="pl-kos">(</span><span class="pl-s">'github_auth_state='</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-k">const</span> <span class="pl-s1">savedState</span> <span class="pl-c1">=</span> <span class="pl-s1">stateCookie</span> ? <span class="pl-s1">stateCookie</span><span class="pl-kos">.</span><span class="pl-en">split</span><span class="pl-kos">(</span><span class="pl-s">'='</span><span class="pl-kos">)</span><span class="pl-kos">[</span><span class="pl-c1">1</span><span class="pl-kos">]</span> : <span class="pl-c1">null</span><span class="pl-kos">;</span>

      <span class="pl-c">// Cookie cleanup header</span>
      <span class="pl-k">const</span> <span class="pl-s1">clearStateCookie</span> <span class="pl-c1">=</span> <span class="pl-kos">{</span>
        <span class="pl-s">'Set-Cookie'</span>: <span class="pl-s">'github_auth_state=; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=0'</span>
      <span class="pl-kos">}</span><span class="pl-kos">;</span>

      <span class="pl-c">// Validate state parameter</span>
      <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-c1">!</span><span class="pl-s1">savedState</span> <span class="pl-c1">||</span> <span class="pl-s1">savedState</span> <span class="pl-c1">!==</span> <span class="pl-s1">returnedState</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-en">generateHTML</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
          <span class="pl-c1">title</span>: <span class="pl-s">'Invalid State Parameter'</span><span class="pl-kos">,</span>
          <span class="pl-c1">content</span>: <span class="pl-s">`</span>
<span class="pl-s">            &lt;h3&gt;Security Error&lt;/h3&gt;</span>
<span class="pl-s">            &lt;p&gt;Invalid state parameter detected. This could indicate a CSRF attempt.&lt;/p&gt;</span>
<span class="pl-s">          `</span><span class="pl-kos">,</span>
          <span class="pl-c1">isError</span>: <span class="pl-c1">true</span><span class="pl-kos">,</span>
          <span class="pl-c1">headers</span>: <span class="pl-s1">clearStateCookie</span>
        <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span>

      <span class="pl-k">const</span> <span class="pl-s1">tokenResponse</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s">'https://github.com/login/oauth/access_token'</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
        <span class="pl-c1">method</span>: <span class="pl-s">'POST'</span><span class="pl-kos">,</span>
        <span class="pl-c1">headers</span>: <span class="pl-kos">{</span>
          <span class="pl-s">'Content-Type'</span>: <span class="pl-s">'application/json'</span><span class="pl-kos">,</span>
          <span class="pl-s">'Accept'</span>: <span class="pl-s">'application/json'</span>
        <span class="pl-kos">}</span><span class="pl-kos">,</span>
        <span class="pl-c1">body</span>: <span class="pl-c1">JSON</span><span class="pl-kos">.</span><span class="pl-en">stringify</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
          <span class="pl-c1">client_id</span>: <span class="pl-s1">clientId</span><span class="pl-kos">,</span>
          <span class="pl-c1">client_secret</span>: <span class="pl-s1">clientSecret</span><span class="pl-kos">,</span>
          <span class="pl-c1">code</span>: <span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">searchParams</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-s">'code'</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
          <span class="pl-c1">redirect_uri</span>: <span class="pl-s1">redirectUri</span><span class="pl-kos">,</span>
          <span class="pl-c1">state</span>: <span class="pl-s1">returnedState</span>
        <span class="pl-kos">}</span><span class="pl-kos">)</span>
      <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      
      <span class="pl-k">const</span> <span class="pl-s1">tokenData</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">tokenResponse</span><span class="pl-kos">.</span><span class="pl-en">json</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      
      <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">tokenData</span><span class="pl-kos">.</span><span class="pl-c1">error</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-en">generateHTML</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
          <span class="pl-c1">title</span>: <span class="pl-s">'GitHub OAuth Error'</span><span class="pl-kos">,</span>
          <span class="pl-c1">content</span>: <span class="pl-s">`</span>
<span class="pl-s">            &lt;h3&gt;Authentication Error&lt;/h3&gt;</span>
<span class="pl-s">            &lt;p&gt;Error: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">tokenData</span><span class="pl-kos">.</span><span class="pl-c1">error</span><span class="pl-kos">}</span></span>&lt;/p&gt;</span>
<span class="pl-s">            <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">tokenData</span><span class="pl-kos">.</span><span class="pl-c1">error_description</span> ? <span class="pl-s">`&lt;p&gt;Description: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">tokenData</span><span class="pl-kos">.</span><span class="pl-c1">error_description</span><span class="pl-kos">}</span></span>&lt;/p&gt;`</span> : <span class="pl-s">''</span><span class="pl-kos">}</span></span></span>
<span class="pl-s">          `</span><span class="pl-kos">,</span>
          <span class="pl-c1">isError</span>: <span class="pl-c1">true</span><span class="pl-kos">,</span>
          <span class="pl-c1">headers</span>: <span class="pl-s1">clearStateCookie</span>
        <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span>
      
      <span class="pl-c">// Success response with cookie cleanup</span>
      <span class="pl-k">return</span> <span class="pl-en">generateHTML</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
        <span class="pl-c1">title</span>: <span class="pl-s">'GitHub OAuth Success'</span><span class="pl-kos">,</span>
        <span class="pl-c1">content</span>: <span class="pl-s">`</span>
<span class="pl-s">          &lt;h2&gt;Authentication successful!&lt;/h2&gt;</span>
<span class="pl-s">          &lt;p&gt;You can close this window.&lt;/p&gt;</span>
<span class="pl-s">          &lt;script&gt;</span>
<span class="pl-s">            try {</span>
<span class="pl-s">              localStorage.setItem('github_token', '<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">tokenData</span><span class="pl-kos">.</span><span class="pl-c1">access_token</span><span class="pl-kos">}</span></span>');</span>
<span class="pl-s">            } catch (err) {</span>
<span class="pl-s">              document.body.innerHTML += '&lt;p style="color: #c62828;"&gt;Warning: Unable to store token in localStorage&lt;/p&gt;';</span>
<span class="pl-s">            }</span>
<span class="pl-s">          &lt;/script&gt;</span>
<span class="pl-s">        `</span><span class="pl-kos">,</span>
        <span class="pl-c1">headers</span>: <span class="pl-s1">clearStateCookie</span>
      <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

    <span class="pl-kos">}</span> <span class="pl-k">catch</span> <span class="pl-kos">(</span><span class="pl-s1">error</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-c">// Error response with cookie cleanup</span>
      <span class="pl-k">return</span> <span class="pl-en">generateHTML</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
        <span class="pl-c1">title</span>: <span class="pl-s">'Unexpected Error'</span><span class="pl-kos">,</span>
        <span class="pl-c1">content</span>: <span class="pl-s">`</span>
<span class="pl-s">          &lt;h3&gt;Unexpected Error&lt;/h3&gt;</span>
<span class="pl-s">          &lt;p&gt;An unexpected error occurred during authentication.&lt;/p&gt;</span>
<span class="pl-s">          &lt;p&gt;Details: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">error</span><span class="pl-kos">.</span><span class="pl-c1">message</span><span class="pl-kos">}</span></span>&lt;/p&gt;</span>
<span class="pl-s">        `</span><span class="pl-kos">,</span>
        <span class="pl-c1">isError</span>: <span class="pl-c1">true</span><span class="pl-kos">,</span>
        <span class="pl-c1">headers</span>: <span class="pl-kos">{</span>
          <span class="pl-s">'Set-Cookie'</span>: <span class="pl-s">'github_auth_state=; Path=/; HttpOnly; Secure; SameSite=Lax; Max-Age=0'</span>
        <span class="pl-kos">}</span>
      <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>The GitHub API has been around for a very long time, so it shouldn't be surprising that Claude knows exactly how to write the above code. I was still delighted at how much work it had saved me.</p>
<p>(I should mention now that I completed the entire initial project on my phone, before I got up to make the morning coffee.)</p>
<div class="markdown-heading"><h2 class="heading-element">Deploying this to Cloudflare Workers</h2><a id="user-content-deploying-this-to-cloudflare-workers" class="anchor" aria-label="Permalink: Deploying this to Cloudflare Workers" href="#deploying-this-to-cloudflare-workers"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>There are four steps to deploying this:</p>
<ol>
<li>Configuring a GitHub OAuth application to get a client ID and secret</li>
<li>Create a new worker with the code</li>
<li>Set the three variables it needs</li>
<li>Configure the correct URL to serve the worker</li>
</ol>
<p>I created the GitHub OAuth app here: <a href="https://github.com/settings/applications/new">https://github.com/settings/applications/new</a></p>
<p>The most important thing to get right here is the "Authorization callback URL": I set that to <code>https://tools.simonwillison.net/github-auth</code> - a URL that didn't exist yet but would after I deployed the Worker.</p>
<p>Then in my Cloudflare dashboard, I navigated to Workers &amp; Pages and clicked "Create" and then "Create Worker".</p>
<p>I deployed the default "Hello world" example and edited it to paste in the code that Claude had written for me.</p>
<p>The Cloudflare editing UI was <em>not</em> built with mobile phones in mind, but I just managed to paste in my code with it!</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://github.com/user-attachments/assets/25c8bce0-39db-4e0e-93ba-69912a1411d6"><img src="https://github.com/user-attachments/assets/25c8bce0-39db-4e0e-93ba-69912a1411d6" alt="The Cloudflare workers editing UI looks awful in Mobile Safari - a very narrow column with some visible code and half the page taken up with extra settings." style="max-width: 100%;"></a></p>
<p>I used the "settings" page to set the <code>GITHUB_CLIENT_ID</code>, <code>GITHUB_CLIENT_SECRET</code> and <code>GITHUB_REDIRECT_URI</code> environment variables needed by the Worker.</p>
<p>The last step was to configure that URL. I navigated to my <code>simonwillison.net</code> Cloudflare dashboard, hit "Workers Routes" and then added a new route mapping <code>tools.simonwillison.net/github-auth*</code> to the Worker I had just created.</p>
<p>That final <code>*</code> wildcard turned out to be necessary to ensure that <code>?code=</code> querystring URLs would also activate the <code>/github-auth</code> worker - without that I got a 404 served by GitHub Pages for those URLs instead.</p>
<p>Here's the deployed page - visiting it should redirect you straight to GitHub: <a href="https://tools.simonwillison.net/github-auth" rel="nofollow">https://tools.simonwillison.net/github-auth</a></p>
<div class="markdown-heading"><h2 class="heading-element">Using this from an application</h2><a id="user-content-using-this-from-an-application" class="anchor" aria-label="Permalink: Using this from an application" href="#using-this-from-an-application"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>The OAuth flow works by setting a <code>github_token</code> key in <code>localStorage()</code> for the entire <code>tools.simonwillison.net</code> domain - which means JavaScript on any page can check for that key and make API calls to the GitHub Gist API if the key is present.</p>
<p>Here's some more code <a href="https://gist.github.com/simonw/29efb202da39775761ab6ab498d942ca">Claude wrote</a> for integrating the new auth mechanism into an existing tool:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">function</span> <span class="pl-en">checkGithubAuth</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">const</span> <span class="pl-s1">token</span> <span class="pl-c1">=</span> <span class="pl-s1">localStorage</span><span class="pl-kos">.</span><span class="pl-en">getItem</span><span class="pl-kos">(</span><span class="pl-s">'github_token'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">token</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-s1">authLinkContainer</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">display</span> <span class="pl-c1">=</span> <span class="pl-s">'none'</span><span class="pl-kos">;</span>
        <span class="pl-s1">saveGistBtn</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">display</span> <span class="pl-c1">=</span> <span class="pl-s">'inline-block'</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span> <span class="pl-k">else</span> <span class="pl-kos">{</span>
        <span class="pl-s1">authLinkContainer</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">display</span> <span class="pl-c1">=</span> <span class="pl-s">'inline-block'</span><span class="pl-kos">;</span>
        <span class="pl-s1">saveGistBtn</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">display</span> <span class="pl-c1">=</span> <span class="pl-s">'none'</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
<span class="pl-kos">}</span>

<span class="pl-k">function</span> <span class="pl-en">startAuthPoll</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">const</span> <span class="pl-s1">pollInterval</span> <span class="pl-c1">=</span> <span class="pl-en">setInterval</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
        <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">localStorage</span><span class="pl-kos">.</span><span class="pl-en">getItem</span><span class="pl-kos">(</span><span class="pl-s">'github_token'</span><span class="pl-kos">)</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
            <span class="pl-en">checkGithubAuth</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-en">clearInterval</span><span class="pl-kos">(</span><span class="pl-s1">pollInterval</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-kos">}</span>
    <span class="pl-kos">}</span><span class="pl-kos">,</span> <span class="pl-c1">1000</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span>

<span class="pl-s1">authLink</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">'click'</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">open</span><span class="pl-kos">(</span><span class="pl-s">'https://tools.simonwillison.net/github-auth'</span><span class="pl-kos">,</span> <span class="pl-s">'github-auth'</span><span class="pl-kos">,</span> <span class="pl-s">'width=600,height=800'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-en">startAuthPoll</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p><code>authLink</code> here is a reference to a "Authenticate with GitHub" link on that page. Clicking that link uses <code>window.open</code> to open a popup showing my new <code>/github-auth</code> page, which then automatically redirects to GitHub for permission.</p>
<p>Clicking the link also starts a every-second poll to check if <code>github_token</code> has been set in <code>localStorage</code> yet. As soon as that becomes available the polling ends, the "Authenticate with GitHub" UI is hidden and a new <code>saveGistBtn</code> (a button to save a Gist) is made visible.</p>
<p>Here's the page that uses that: <a href="https://tools.simonwillison.net/openai-audio-output" rel="nofollow">https://tools.simonwillison.net/openai-audio-output</a> - you'll need to provide an OpenAI API key and submit a prompt in order to see the option to Authenticate with GitHub.</p>
<div class="markdown-heading"><h2 class="heading-element">Adding error handling</h2><a id="user-content-adding-error-handling" class="anchor" aria-label="Permalink: Adding error handling" href="#adding-error-handling"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>The initial code that Claude wrote was missing an important detail: error handling. If the GitHub API returns an error - e.g. because the <code>?code=</code> is invalid - the page won't reflect that to the user.</p>
<p>I pasted in the code and prompted:</p>
<blockquote>
<p><code>Add error handling to this Cloudflare Workers script - if the GitHub API returns an error it should be shown to the user</code></p>
</blockquote>
<p>Claude <a href="https://gist.github.com/simonw/85debbdf3d981ff7e54f8cdb6be47578#create-github-oauth-cloudflare-worker-with-error-handling">wrote more code</a>, but it was a bit verbose. I prompted:</p>
<blockquote>
<p><code>refactor that code to have less code for the HTML</code></p>
</blockquote>
<p>And <a href="https://gist.github.com/simonw/85debbdf3d981ff7e54f8cdb6be47578#rewrite-untitled">got back this</a>, which is much better.</p>
<p>Here's <a href="https://tools.simonwillison.net/github-auth?code=bad-code" rel="nofollow">an example page</a> showing the new error message.</p>
<div class="markdown-heading"><h2 class="heading-element">Preventing CSRF attacks</h2><a id="user-content-preventing-csrf-attacks" class="anchor" aria-label="Permalink: Preventing CSRF attacks" href="#preventing-csrf-attacks"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>Robert Munteanu <a href="https://fosstodon.org/@rombert/113566295983095957" rel="nofollow">on Mastodon</a>:</p>
<blockquote>
<p>I have looked into OAuth 2.0 and OIDC recently, I wonder what your toughts are about adding CSRF protection? There seems to be no checking of the state parameter in the workers code.</p>
<p><a href="https://www.rfc-editor.org/rfc/rfc6749#section-10.12" rel="nofollow">https://www.rfc-editor.org/rfc/rfc6749#section-10.12</a></p>
</blockquote>
<p>He's absolutely right! The <code>state=</code> parameter was being set but in the first few versions of the code it wasn't being checked later.</p>
<div class="markdown-heading"><h3 class="heading-element">Understanding the attack</h3><a id="user-content-understanding-the-attack" class="anchor" aria-label="Permalink: Understanding the attack" href="#understanding-the-attack"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I started thinking this through with the help of Claude: I <a href="https://gist.github.com/simonw/e87e55dfe13e7201dc0ae5042bc4d4eb">pasted in the code</a> and prompted:</p>
<blockquote>
<p><code>Explain the consequences of this not checking the state parameter</code></p>
</blockquote>
<p>After some back and forth I'm ready to explain this in my own words.</p>
<p>The specific attack to worry about here is one where an attacker tricks/forces a user into signing in to an account that the <em>attacker</em> controls.</p>
<p>For my Gist example here, imagine if I could create a brand new GitHub account and then trick you into signing in to that account using OAuth, while giving you the impression that you had signed into your own account.</p>
<p>If the user saves a Gist containing their private information, you can now access that as the real owner of the account.</p>
<p>With GitHub OAuth here's how that could happen: as an attacker, I could initiate the OAuth flow against my new dedicated malicious account, and then at the end intercept that final redirect URL with the <code>?code=</code> parameter:</p>
<pre><code>https://tools.simonwillison.net/github-auth?code=auth-code-I-generated
</code></pre>
<p>Rather than visit that URL I instead send it to my target and trick them into clicking on it.</p>
<p>When they visit that page the Worker exchanges that <code>?code=</code> for an access token against MY account and stores that in my victim's <code>localStorage</code>.</p>
<p>Now any Gists they save will be visible to me as the account's real owner.</p>
<div class="markdown-heading"><h3 class="heading-element">How to prevent it</h3><a id="user-content-how-to-prevent-it" class="anchor" aria-label="Permalink: How to prevent it" href="#how-to-prevent-it"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>This attack is why the OAuth specification describes the <code>&amp;state=</code> parameter. This is a random value that the client generates and then sends to the server. The server echoes that value back to the client in the final redirect URL as the <code>?state=</code> parameter.</p>
<p>To avoid CSRF attacks, we need to record that initial generated <code>state</code> and then compare it to the <code>state=</code> in the final redirect URL.</p>
<p>I pasted in another copy of my script and prompted Claude:</p>
<blockquote>
<p><code>Modify this to store the state= parameter in an HTTP only session cookie called github_auth_state and then compare that when the user comes back and show an error if they do not match, otherwise unset the cookie and complete the operation</code></p>
</blockquote>
<p>Claude <a href="https://gist.github.com/simonw/ae56f00572dd80f9180687f9532a8226#create-github-oauth-worker-with-state-validation">wrote some code</a>, but when I tried it out on Cloudflare I got this error... which I pasted back into Claude as a follow-up prompt:</p>
<blockquote>
<p><code>Unexpected Error An unexpected error occurred during authentication. Details: Can't modify immutable headers.</code></p>
</blockquote>
<p>This time it <a href="https://gist.github.com/simonw/ae56f00572dd80f9180687f9532a8226#rewrite-untitled">wrote code that worked</a> - it turns out the correct pattern for sending custom HTTP headers in a Cloudflare Worker looks like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span><span class="pl-s">`</span>
<span class="pl-s">  &lt;!DOCTYPE html&gt;</span>
<span class="pl-s">  &lt;html&gt;...&lt;/html&gt;</span>
<span class="pl-s">`</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
  <span class="pl-c1">headers</span>: <span class="pl-kos">{</span>
    <span class="pl-s">'Content-Type'</span>: <span class="pl-s">'text/html'</span><span class="pl-kos">,</span>
    ...<span class="pl-s1">headers</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span><span class="pl-kos"></span>
<span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


guid: tag:til.simonwillison.net,2020-04-30:npm_npm-publish-github-actions.md
title: Publishing a simple client-side JavaScript package to npm with GitHub Actions
author: Simon Willison
url: https://til.simonwillison.net/npm/npm-publish-github-actions
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>Here's what I learned about publishing a single file JavaScript package to NPM for my <a href="https://simonwillison.net/2024/Dec/7/prompts-js/" rel="nofollow">Prompts.js</a> project.</p>
<p>The code is in <a href="https://github.com/simonw/prompts-js">simonw/prompts-js</a> on GitHub. The NPM package is <a href="https://www.npmjs.com/package/prompts-js" rel="nofollow">prompts-js</a>.</p>
<div class="markdown-heading"><h2 class="heading-element">A simple single file client-side package</h2><a id="user-content-a-simple-single-file-client-side-package" class="anchor" aria-label="Permalink: A simple single file client-side package" href="#a-simple-single-file-client-side-package"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>For this project, I wanted to create an old-fashioned JavaScript file that you could include in a web page using a <code>&lt;script&gt;</code> tag. No TypeScript, no React JSK, no additional dependencies, no build step.</p>
<p>I also wanted to ship it to NPM, mainly so it would be magically available from various CDNs.</p>
<p>I think I've boiled that down to about as simple as I can get. Here's the <code>package.json</code> file:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>prompts-js<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.0.4<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"description"</span>: <span class="pl-s"><span class="pl-pds">"</span>async alternatives to browser alert() and prompt() and confirm()<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"main"</span>: <span class="pl-s"><span class="pl-pds">"</span>index.js<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"homepage"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/prompts-js<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"scripts"</span>: {
    <span class="pl-ent">"test"</span>: <span class="pl-s"><span class="pl-pds">"</span>echo <span class="pl-cce">\"</span>Error: no test specified<span class="pl-cce">\"</span> &amp;&amp; exit 1<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"author"</span>: <span class="pl-s"><span class="pl-pds">"</span>Simon Willison<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"license"</span>: <span class="pl-s"><span class="pl-pds">"</span>Apache-2.0<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"repository"</span>: {
    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>git<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>git+https://github.com/simonw/prompts-js.git<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"keywords"</span>: [
    <span class="pl-s"><span class="pl-pds">"</span>alert<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>prompt<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>confirm<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>async<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>promise<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>dialog<span class="pl-pds">"</span></span>
  ],
  <span class="pl-ent">"files"</span>: [
    <span class="pl-s"><span class="pl-pds">"</span>index.js<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>README.md<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>LICENSE<span class="pl-pds">"</span></span>
  ]
}</pre></div>
<p>That "scripts.test" block probably isn't necessary. The <code>keywords</code> are used when you deploy to NPM, and the <code>files</code> block tells NPM which files to include in the package.</p>
<p>The <code>"repository"</code> block is used by NPM's <a href="https://docs.npmjs.com/generating-provenance-statements" rel="nofollow">provenance statements</a>. Don't worry too much about these - they're only needed if you use the <code>npm publish --provenance</code> option later on.</p>
<p>Really the three most important keys here are <code>"name"</code>, which needs to be a unique name on NPM, <code>"version"</code> and that <code>"main"</code> key. I set <code>"main"</code> to <code>index.js</code>.</p>
<p>All that's needed now is that <code>index.js</code> file - and optionally the <code>README.md</code> and <code>LICENSE</code> files if we want to include them in the package. The <code>README.md</code> ends up displayed on the NPM listing page so it's worth including.</p>
<p>Here's my <a href="https://github.com/simonw/prompts-js/blob/main/index.js">index.js</a> file. It starts and ends like this (an <a href="https://developer.mozilla.org/en-US/docs/Glossary/IIFE" rel="nofollow">IFFE</a>):</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-v">Prompts</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-k">function</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-c">// ...</span>
  <span class="pl-k">return</span> <span class="pl-kos">{</span> alert<span class="pl-kos">,</span> confirm<span class="pl-kos">,</span> prompt <span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<div class="markdown-heading"><h2 class="heading-element">Publishing to NPM</h2><a id="user-content-publishing-to-npm" class="anchor" aria-label="Permalink: Publishing to NPM" href="#publishing-to-npm"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>With these pieces in place, running <code>npm publish</code> in the root of the project will publish the package to NPM - after first asking you to sign into your NPM account.</p>
<div class="markdown-heading"><h2 class="heading-element">Automating this with GitHub Actions</h2><a id="user-content-automating-this-with-github-actions" class="anchor" aria-label="Permalink: Automating this with GitHub Actions" href="#automating-this-with-github-actions"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I use GitHub Actions that trigger on any release to publish all of my Python projects to PyPI. I wanted to do the same for this JavaScript project.</p>
<p>I found <a href="https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-nodejs-packages#publishing-packages-to-the-npm-registry">this example</a> in the GitHub documentation which gave me most of what I needed. This is in <a href="https://github.com/simonw/prompts-js/blob/main/.github/workflows/publish.yml">.github/workflows/publish.yml</a>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Publish Package to npmjs</span>
<span class="pl-ent">on</span>:
  <span class="pl-ent">release</span>:
    <span class="pl-ent">types</span>: <span class="pl-s">[published]</span>
<span class="pl-ent">jobs</span>:
  <span class="pl-ent">build</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">permissions</span>:
      <span class="pl-ent">contents</span>: <span class="pl-s">read</span>
      <span class="pl-ent">id-token</span>: <span class="pl-s">write</span>
    <span class="pl-ent">steps</span>:
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v4</span>
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-node@v4</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">node-version</span>: <span class="pl-s"><span class="pl-pds">'</span>20.x<span class="pl-pds">'</span></span>
          <span class="pl-ent">registry-url</span>: <span class="pl-s"><span class="pl-pds">'</span>https://registry.npmjs.org<span class="pl-pds">'</span></span>
      - <span class="pl-ent">run</span>: <span class="pl-s">npm publish --provenance --access public</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">NODE_AUTH_TOKEN</span>: <span class="pl-s">${{ secrets.NPM_TOKEN }}</span></pre></div>
<p>There's that <code>--provenance</code> option which only works if you have the <code>repository</code> block set up in your <code>package.json</code>.</p>
<p>This needs a secret called <code>NPM_TOKEN</code> to be set up in the GitHub repository settings.</p>
<p>It took me a few tries to get this right. It needs to be a token created on the NPM website using the Access Tokens menu item, then Generate New Token -&gt; Classic Token. As far as I can tell the new "Granular Access Token" format doesn't work for this as it won't allow you to create a token that never expires, and I never want to have to remember to update the secret in the future.</p>
<p>An "Automation" token should do the trick here - it bypasses 2-factor authentication when publishing.</p>
<p>Set that in GitHub Actions as a secret called <code>NPM_TOKEN</code> and now you can publish a new version of your package to NPM by doing the following:</p>
<ol>
<li>Update the version number in <code>package.json</code>
</li>
<li>Create a new release on GitHub with a tag that matches the version number</li>
</ol>


guid: tag:til.simonwillison.net,2020-04-30:python_utc-warning-fix.md
title: Fixes for datetime UTC warnings in Python
author: Simon Willison
url: https://til.simonwillison.net/python/utc-warning-fix
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I was getting the following warning for one of my Python test suites:</p>
<blockquote>
<p><code>DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).</code></p>
</blockquote>
<p>I also saw a deprecation warning elsewhere for my usage of <code>datetime.datetime.utcfromtimestamp(iso_timestamp_string)</code>.</p>
<div class="markdown-heading"><h2 class="heading-element">datetime.datetime.now(datetime.timezone.utc)</h2><a id="user-content-datetimedatetimenowdatetimetimezoneutc" class="anchor" aria-label="Permalink: datetime.datetime.now(datetime.timezone.utc)" href="#datetimedatetimenowdatetimetimezoneutc"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I tried switching to <code>datetime.UTC</code> for the first warning, but that caused my tests to fail on versions of Python prior to 3.11. I support all versions of Python that have not yet hit EOL for support, which is currently <a href="https://devguide.python.org/versions/" rel="nofollow">Python 3.9 and higher</a>.</p>
<p>According to <a href="https://docs.python.org/3/library/datetime.html#datetime.UTC" rel="nofollow">the documentation</a> <code>datetime.UTC</code> was added in Python 3.11 as an alias for <code>datetime.timezone.utc</code> - so I switched my usage of <code>datetime.utcnow()</code> to the following instead:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">datetime</span>

<span class="pl-s1">utcnow</span> <span class="pl-c1">=</span> <span class="pl-s1">datetime</span>.<span class="pl-c1">datetime</span>.<span class="pl-c1">now</span>(<span class="pl-s1">datetime</span>.<span class="pl-c1">timezone</span>.<span class="pl-c1">utc</span>)</pre></div>
<p>These tests now passed for Python 3.9+.</p>
<div class="markdown-heading"><h2 class="heading-element">datetime.datetime.fromtimestamp(s, datetime.timezone.utc)</h2><a id="user-content-datetimedatetimefromtimestamps-datetimetimezoneutc" class="anchor" aria-label="Permalink: datetime.datetime.fromtimestamp(s, datetime.timezone.utc)" href="#datetimedatetimefromtimestamps-datetimetimezoneutc"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I found another hint <a href="https://docs.python.org/3/library/datetime.html#datetime.datetime.utcfromtimestamp" rel="nofollow">in the docs</a> for replacing the now deprecated <code>datetime.datetime.utcfromtimestamp</code>:</p>
<blockquote>
<p><strong>Warning</strong>: Because naive <code>datetime</code> objects are treated by many <code>datetime</code> methods as local times, it is preferred to use aware datetimes to represent times in UTC. As such, the recommended way to create an object representing a specific timestamp in UTC is by calling <code>datetime.fromtimestamp(timestamp, tz=timezone.utc)</code>.</p>
</blockquote>
<p>Again, <code>timezone.utc</code> was the right fix here to keep things working in pre-3.11 versions:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">datetime</span>

<span class="pl-s1">utc_datetime</span> <span class="pl-c1">=</span> <span class="pl-s1">datetime</span>.<span class="pl-c1">datetime</span>.<span class="pl-c1">fromtimestamp</span>(<span class="pl-s1">created_string</span>, <span class="pl-s1">datetime</span>.<span class="pl-c1">timezone</span>.<span class="pl-c1">utc</span>)</pre></div>
<p>Here's <a href="https://github.com/simonw/llm/commit/571f4b2a4da52ad127061b7fa953562f6ba6aeb0">the commit</a> where I fixed both of these warnings for my LLM project.</p>


guid: tag:til.simonwillison.net,2020-04-30:llms_bert-ner.md
title: Named Entity Resolution with dslim/distilbert-NER
author: Simon Willison
url: https://til.simonwillison.net/llms/bert-ner
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I was exploring the original BERT model from 2018, which is mainly useful if you fine-tune a model on top of it for a specific task.</p>
<p><a href="https://huggingface.co/dslim/distilbert-NER" rel="nofollow">dslim/distilbert-NER</a> by David S. Lim is a popular implementation of this, with around 20,000 downloads from Hugging Face every month.</p>
<p>I tried the demo from the README but it didn't quite work - it complained about an incompatibility with Numpy 2.0.</p>
<p>So I used <code>uv run --with 'numpy&lt;2.0'</code> to run it in a temporary virtual environment. Here's a Bash one-liner that demonstrated the model:</p>
<div class="highlight highlight-source-shell"><pre>uv run --with <span class="pl-s"><span class="pl-pds">'</span>numpy&lt;2.0<span class="pl-pds">'</span></span> --with transformers python -c <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">from transformers import AutoTokenizer, AutoModelForTokenClassification</span>
<span class="pl-s">from transformers import pipeline</span>
<span class="pl-s">import json</span>
<span class="pl-s">model = AutoModelForTokenClassification.from_pretrained("dslim/distilbert-NER")</span>
<span class="pl-s">tokenizer = AutoTokenizer.from_pretrained("dslim/distilbert-NER")</span>
<span class="pl-s">nlp = pipeline("ner", model=model, tokenizer=tokenizer)</span>
<span class="pl-s">text = "This is an example sentence about Simon Willison who lives in Half Moon Bay"</span>
<span class="pl-s">print(json.dumps(nlp(text), indent=2, default=repr))<span class="pl-pds">'</span></span></pre></div>
<p>The first time you run this it will download 250MB to your <code>~/.cache/huggingface/hub/models--dslim--distilbert-NER</code> folder.</p>
<p>Example output:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>B-PER<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.9982101<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">7</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>Simon<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">34</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">39</span>
  },
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>I-PER<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.99835676<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">8</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>Willis<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">40</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">46</span>
  },
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>I-PER<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.9977602<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">9</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>##on<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">46</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">48</span>
  },
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>B-LOC<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.99432063<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">13</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>Half<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">62</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">66</span>
  },
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>I-LOC<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.99325883<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">14</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>Moon<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">67</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">71</span>
  },
  {
    <span class="pl-ent">"entity"</span>: <span class="pl-s"><span class="pl-pds">"</span>I-LOC<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"score"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.9919292<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"index"</span>: <span class="pl-c1">15</span>,
    <span class="pl-ent">"word"</span>: <span class="pl-s"><span class="pl-pds">"</span>Bay<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"start"</span>: <span class="pl-c1">72</span>,
    <span class="pl-ent">"end"</span>: <span class="pl-c1">75</span>
  }
]</pre></div>


guid: tag:til.simonwillison.net,2020-04-30:git_size-of-lfs-files.md
title: Calculating the size of all LFS files in a repo
author: Simon Willison
url: https://til.simonwillison.net/git/size-of-lfs-files
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>I wanted to know how large the <a href="https://huggingface.co/deepseek-ai/DeepSeek-V3-Base" rel="nofollow">deepseek-ai/DeepSeek-V3-Base</a> repo on Hugging Face was without actually downloading all of the files.</p>
<p>With <a href="https://gist.github.com/simonw/9d2b780a39e58a230a0cee18452ec9d0">some help from Claude</a>, here's the recipe that worked.</p>
<p>First, clone the repo without having Git LFS download the files:</p>
<div class="highlight highlight-source-shell"><pre>GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/deepseek-ai/DeepSeek-V3-Base
<span class="pl-c1">cd</span> DeepSeek-V3-Base</pre></div>
<p>The <code>git lfs ls-files -s</code> command lists the files along with their sizes:</p>
<div class="highlight highlight-source-shell"><pre>git lfs ls-files -s</pre></div>
<pre><code>3f4e5fcec2 - model-00001-of-000163.safetensors (5.2 GB)
4fb0c2abdd - model-00002-of-000163.safetensors (4.3 GB)
...
</code></pre>
<p>Then I used this <code>awk</code> recipe to add up those numbers:</p>
<div class="highlight highlight-source-shell"><pre>git lfs ls-files -s <span class="pl-k">|</span> grep -o <span class="pl-s"><span class="pl-pds">'</span>[0-9.]\+ GB<span class="pl-pds">'</span></span> <span class="pl-k">|</span> awk <span class="pl-s"><span class="pl-pds">'</span>{sum += $1} END {print sum " GB"}<span class="pl-pds">'</span></span></pre></div>
<p>Output:</p>
<pre><code>687.9 GB
</code></pre>
<p>Since this only counts lines with <code>GB</code> in them I asked Claude for a longer one-liner for handling other units as well. This appears to work but I haven't verified it in depth yet, so use with caution:</p>
<div class="highlight highlight-source-shell"><pre>git lfs ls-files -s <span class="pl-k">|</span> grep -o <span class="pl-s"><span class="pl-pds">'</span>[0-9.]\+ [KMGT]B<span class="pl-pds">'</span></span> <span class="pl-k">|</span> awk <span class="pl-s"><span class="pl-pds">'</span>{ </span>
<span class="pl-s">    split($0, a, " "); </span>
<span class="pl-s">    size=a[1]; </span>
<span class="pl-s">    unit=a[2]; </span>
<span class="pl-s">    if(unit=="KB") size*=1024; </span>
<span class="pl-s">    else if(unit=="MB") size*=1024^2; </span>
<span class="pl-s">    else if(unit=="GB") size*=1024^3; </span>
<span class="pl-s">    else if(unit=="TB") size*=1024^4; </span>
<span class="pl-s">    total+=size</span>
<span class="pl-s">} END { </span>
<span class="pl-s">    if(total&lt;1024) print total " B";</span>
<span class="pl-s">    else if(total&lt;1024^2) print total/1024 " KB";</span>
<span class="pl-s">    else if(total&lt;1024^3) print total/1024^2 " MB";</span>
<span class="pl-s">    else if(total&lt;1024^4) print total/1024^3 " GB";</span>
<span class="pl-s">    else print total/1024^4 " TB"</span>
<span class="pl-s">}<span class="pl-pds">'</span></span></pre></div>


guid: tag:til.simonwillison.net,2020-04-30:tiktok_download-all-videos.md
title: Downloading every video for a TikTok account
author: Simon Willison
url: https://til.simonwillison.net/tiktok/download-all-videos
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>TikTok may or may not be banned in the USA within the next 24 hours or so. Here's a pattern you can use to download all of the videos from a specific account.</p>
<div class="markdown-heading"><h2 class="heading-element">Using yt-dlp directly</h2><a id="user-content-using-yt-dlp-directly" class="anchor" aria-label="Permalink: Using yt-dlp directly" href="#using-yt-dlp-directly"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>After first publishing this TIL I found out you can point <code>yt-dlp</code> directly at an account page and it will download all videos for you:</p>
<pre><code>yt-dlp 'https://www.tiktok.com/@username_goes_here' -o "downloads/%(title)s-%(id)s.%(ext)s"
</code></pre>
<p>The rest of this TIL describes a more complex method.</p>
<div class="markdown-heading"><h2 class="heading-element">Scrape the list of video URLs</h2><a id="user-content-scrape-the-list-of-video-urls" class="anchor" aria-label="Permalink: Scrape the list of video URLs" href="#scrape-the-list-of-video-urls"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I used a variant of my <a href="https://til.simonwillison.net/twitter/collecting-replies" rel="nofollow">Twitter scraping trick</a>. Start by loading up a profile page - like <a href="www.tiktok.com/@ilgallinaio_special">https://www.tiktok.com/@ilgallinaio_special</a> - in Firefox or Chrome or Safari.</p>
<p>Open up the DevTools and paste in the following JavaScript:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">videoUrls</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">Set</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">function</span> <span class="pl-en">collect</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelectorAll</span><span class="pl-kos">(</span><span class="pl-s">'a[href*="/video/"]'</span><span class="pl-kos">)</span><span class="pl-kos">,</span> <span class="pl-s1">el</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">href</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span><span class="pl-s1">href</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
        <span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">videoUrls</span><span class="pl-kos">.</span><span class="pl-en">add</span><span class="pl-kos">(</span><span class="pl-s1">href</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span><span class="pl-kos">)</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span>

<span class="pl-en">setInterval</span><span class="pl-kos">(</span><span class="pl-s1">collect</span><span class="pl-kos">,</span> <span class="pl-c1">500</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>This will scan the page every half a second looking for links to TikTok videos - links with <code>/video/</code> in their URL - and add those to a growing set called <code>videoUrls</code>.</p>
<p>Now switch to the "oldest" sort direction and scroll down the page until you reach the bottom. TikTok implements infinite-ish scrolling so this may take a while for an account with a lot of videos.</p>
<p>Once you get to the bottom, copy out the collected list of URLs. In Firefox I used this command for that:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-en">copy</span><span class="pl-kos">(</span><span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">videoUrls</span><span class="pl-kos">)</span><span class="pl-kos">)</span></pre></div>
<p>That copied the array of URLs to my clipboard. I then pasted them into a file and saved it as <code>videos.json</code> - the file contents looked something like this (but a lot longer):</p>
<div class="highlight highlight-source-json"><pre>[
  <span class="pl-s"><span class="pl-pds">"</span>https://www.tiktok.com/@ilgallinaio_special/video/7204803049351695622<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>https://www.tiktok.com/@ilgallinaio_special/video/7204877634189151493<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>https://www.tiktok.com/@ilgallinaio_special/video/7205157890372537606<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>https://www.tiktok.com/@ilgallinaio_special/video/7205189803074211077<span class="pl-pds">"</span></span>
]</pre></div>
<div class="markdown-heading"><h2 class="heading-element">Download them all with yt-dlp</h2><a id="user-content-download-them-all-with-yt-dlp" class="anchor" aria-label="Permalink: Download them all with yt-dlp" href="#download-them-all-with-yt-dlp"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>The <a href="https://github.com/yt-dlp/yt-dlp">yt-dlp</a> Python program can download from TikTok. I ran it against all of the URLs in my <code>videos.json</code> file like this:</p>
<div class="highlight highlight-source-shell"><pre>mkdir -p downloads
jq -r <span class="pl-s"><span class="pl-pds">'</span>.[]<span class="pl-pds">'</span></span> videos.json <span class="pl-k">|</span> <span class="pl-k">while</span> <span class="pl-c1">read</span> url<span class="pl-k">;</span> <span class="pl-k">do</span>
    uvx yt-dlp -o <span class="pl-s"><span class="pl-pds">"</span>downloads/%(title)s-%(id)s.%(ext)s<span class="pl-pds">"</span></span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$url</span><span class="pl-pds">"</span></span>
    <span class="pl-k">if</span> [[ <span class="pl-smi">$?</span> <span class="pl-k">-eq</span> 0 ]]<span class="pl-k">;</span> <span class="pl-k">then</span>
        <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Successfully downloaded: <span class="pl-smi">$url</span><span class="pl-pds">"</span></span>
    <span class="pl-k">else</span>
        <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Failed to download: <span class="pl-smi">$url</span><span class="pl-pds">"</span></span>
    <span class="pl-k">fi</span>
    sleep 1
<span class="pl-k">done</span></pre></div>
<p>This creates a <code>downloads/</code> folder containing files with names like this:</p>
<pre><code>#perte -7204803049351695622.mp4
#perte -7204877634189151493.mp4
#perte -7205189803074211077.mp4
#perte i galli moroseta🐓🐓🌸🍾🍾💪😅-7205157890372537606.mp4
</code></pre>
<div class="markdown-heading"><h2 class="heading-element">Bonus: running Whisper against them</h2><a id="user-content-bonus-running-whisper-against-them" class="anchor" aria-label="Permalink: Bonus: running Whisper against them" href="#bonus-running-whisper-against-them"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>I did this against an account that wasn't just dancing chickens and decided to use Whisper running on macOS via <a href="https://pypi.org/project/mlx-whisper/" rel="nofollow">mlx-whisper</a> to generate text files with transcripts, so I could search that content later on.</p>
<p>Here's the recipe I used for that, powered by <code>uv run</code>:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">for</span> <span class="pl-smi">f</span> <span class="pl-k">in</span> <span class="pl-k">*</span>.mp4<span class="pl-k">;</span> <span class="pl-k">do</span> [[ <span class="pl-k">!</span> <span class="pl-k">-f</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">${f<span class="pl-k">:</span>r}</span>.txt<span class="pl-pds">"</span></span> ]] <span class="pl-k">&amp;&amp;</span> <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Processing <span class="pl-smi">$f</span><span class="pl-pds">"</span></span> <span class="pl-k">&amp;&amp;</span> uv run --with mlx-whisper mlx_whisper <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$f</span><span class="pl-pds">"</span></span><span class="pl-k">;</span> <span class="pl-k">done</span></pre></div>
<p>This can be run multiple times - it checks to see if a <code>.txt</code> file exists already and only executes against <code>.mp4</code> files that have not yet been processed.</p>
<div class="markdown-heading"><h2 class="heading-element">Extra bonus: adding a progress bar</h2><a id="user-content-extra-bonus-adding-a-progress-bar" class="anchor" aria-label="Permalink: Extra bonus: adding a progress bar" href="#extra-bonus-adding-a-progress-bar"><span aria-hidden="true" class="octicon octicon-link"></span></a></div>
<p>After I kicked this off against a larger account I realized a progress bar would be nice. I got ChatGPT o1 to write me this script:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c">#!/usr/bin/env python3</span>

<span class="pl-k">import</span> <span class="pl-s1">sys</span>
<span class="pl-k">import</span> <span class="pl-s1">time</span>
<span class="pl-k">import</span> <span class="pl-s1">subprocess</span>

<span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-k">if</span> <span class="pl-en">len</span>(<span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>) <span class="pl-c1">&lt;</span> <span class="pl-c1">3</span>:
        <span class="pl-en">print</span>(<span class="pl-s">f"Usage: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>[<span class="pl-c1">0</span>]<span class="pl-kos">}</span></span> &lt;total&gt; &lt;shell_command&gt;"</span>)
        <span class="pl-s1">sys</span>.<span class="pl-c1">exit</span>(<span class="pl-c1">1</span>)
    
    <span class="pl-s1">total</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>[<span class="pl-c1">1</span>])
    <span class="pl-c"># If your command may include spaces, you might need to do this:</span>
    <span class="pl-c"># shell_command = ' '.join(sys.argv[2:])</span>
    <span class="pl-c"># but for the simple example provided:</span>
    <span class="pl-s1">shell_command</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-c1">argv</span>[<span class="pl-c1">2</span>]

    <span class="pl-c"># -- Step 1: Get initial progress and record the time --</span>
    <span class="pl-k">try</span>:
        <span class="pl-s1">initial_output</span> <span class="pl-c1">=</span> <span class="pl-s1">subprocess</span>.<span class="pl-c1">check_output</span>(<span class="pl-s1">shell_command</span>, <span class="pl-s1">shell</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
        <span class="pl-s1">done_initial</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">initial_output</span>.<span class="pl-c1">strip</span>())
    <span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
        <span class="pl-en">print</span>(<span class="pl-s">f"Error running initial command: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">shell_command</span><span class="pl-kos">}</span></span><span class="pl-cce">\n</span><span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">e</span><span class="pl-kos">}</span></span>"</span>)
        <span class="pl-s1">sys</span>.<span class="pl-c1">exit</span>(<span class="pl-c1">1</span>)

    <span class="pl-c"># Clamp in case the command returns something above the total or below zero</span>
    <span class="pl-k">if</span> <span class="pl-s1">done_initial</span> <span class="pl-c1">&lt;</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">done_initial</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
    <span class="pl-k">if</span> <span class="pl-s1">done_initial</span> <span class="pl-c1">&gt;</span> <span class="pl-s1">total</span>:
        <span class="pl-s1">done_initial</span> <span class="pl-c1">=</span> <span class="pl-s1">total</span>

    <span class="pl-s1">time_initial</span> <span class="pl-c1">=</span> <span class="pl-s1">time</span>.<span class="pl-c1">time</span>()

    <span class="pl-c"># Print one quick update before we start the loop</span>
    <span class="pl-en">print_progress</span>(<span class="pl-s1">done_initial</span>, <span class="pl-s1">total</span>, <span class="pl-c1">0</span>, <span class="pl-c1">0</span>)

    <span class="pl-c"># If we already reached (or exceeded) the total, exit immediately</span>
    <span class="pl-k">if</span> <span class="pl-s1">done_initial</span> <span class="pl-c1">&gt;=</span> <span class="pl-s1">total</span>:
        <span class="pl-en">print</span>(<span class="pl-s">"<span class="pl-cce">\n</span>Done!"</span>)
        <span class="pl-s1">sys</span>.<span class="pl-c1">exit</span>(<span class="pl-c1">0</span>)

    <span class="pl-c"># -- Step 2: Repeatedly poll the command to update progress --</span>
    <span class="pl-s1">polling_interval</span> <span class="pl-c1">=</span> <span class="pl-c1">1.0</span>  <span class="pl-c"># seconds between checks</span>

    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">time</span>.<span class="pl-c1">sleep</span>(<span class="pl-s1">polling_interval</span>)

        <span class="pl-c"># Fetch current progress</span>
        <span class="pl-k">try</span>:
            <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">subprocess</span>.<span class="pl-c1">check_output</span>(<span class="pl-s1">shell_command</span>, <span class="pl-s1">shell</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
            <span class="pl-s1">done</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">output</span>.<span class="pl-c1">strip</span>())
        <span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
            <span class="pl-en">print</span>(<span class="pl-s">f"<span class="pl-cce">\n</span>Error running command: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">shell_command</span><span class="pl-kos">}</span></span><span class="pl-cce">\n</span><span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">e</span><span class="pl-kos">}</span></span>"</span>)
            <span class="pl-s1">sys</span>.<span class="pl-c1">exit</span>(<span class="pl-c1">1</span>)
        
        <span class="pl-c"># Clamp done to never exceed total or go below 0</span>
        <span class="pl-k">if</span> <span class="pl-s1">done</span> <span class="pl-c1">&lt;</span> <span class="pl-c1">0</span>:
            <span class="pl-s1">done</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
        <span class="pl-k">if</span> <span class="pl-s1">done</span> <span class="pl-c1">&gt;</span> <span class="pl-s1">total</span>:
            <span class="pl-s1">done</span> <span class="pl-c1">=</span> <span class="pl-s1">total</span>

        <span class="pl-c"># How much progress has been made since we started measuring?</span>
        <span class="pl-s1">delta_done</span> <span class="pl-c1">=</span> <span class="pl-s1">done</span> <span class="pl-c1">-</span> <span class="pl-s1">done_initial</span>
        <span class="pl-s1">delta_time</span> <span class="pl-c1">=</span> <span class="pl-s1">time</span>.<span class="pl-c1">time</span>() <span class="pl-c1">-</span> <span class="pl-s1">time_initial</span>

        <span class="pl-c"># Print the progress bar</span>
        <span class="pl-en">print_progress</span>(<span class="pl-s1">done</span>, <span class="pl-s1">total</span>, <span class="pl-s1">delta_done</span>, <span class="pl-s1">delta_time</span>)

        <span class="pl-k">if</span> <span class="pl-s1">done</span> <span class="pl-c1">&gt;=</span> <span class="pl-s1">total</span>:
            <span class="pl-k">break</span>

    <span class="pl-en">print</span>(<span class="pl-s">"<span class="pl-cce">\n</span>Done!"</span>)


<span class="pl-k">def</span> <span class="pl-en">print_progress</span>(<span class="pl-s1">done</span>, <span class="pl-s1">total</span>, <span class="pl-s1">delta_done</span>, <span class="pl-s1">delta_time</span>):
    <span class="pl-s">"""</span>
<span class="pl-s">    Print a single-line progress bar with percentage and ETA (if possible).</span>
<span class="pl-s">    Overwrites the previous line via carriage return.</span>
<span class="pl-s">    """</span>

    <span class="pl-c"># Fraction complete</span>
    <span class="pl-s1">fraction</span> <span class="pl-c1">=</span> <span class="pl-s1">done</span> <span class="pl-c1">/</span> <span class="pl-s1">total</span> <span class="pl-k">if</span> <span class="pl-s1">total</span> <span class="pl-k">else</span> <span class="pl-c1">1.0</span>

    <span class="pl-c"># Build the bar</span>
    <span class="pl-s1">bar_length</span> <span class="pl-c1">=</span> <span class="pl-c1">50</span>
    <span class="pl-s1">filled_length</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">bar_length</span> <span class="pl-c1">*</span> <span class="pl-s1">fraction</span>)
    <span class="pl-s1">bar</span> <span class="pl-c1">=</span> <span class="pl-s">"#"</span> <span class="pl-c1">*</span> <span class="pl-s1">filled_length</span> <span class="pl-c1">+</span> <span class="pl-s">"-"</span> <span class="pl-c1">*</span> (<span class="pl-s1">bar_length</span> <span class="pl-c1">-</span> <span class="pl-s1">filled_length</span>)

    <span class="pl-c"># Compute ETA based only on new progress (delta_done)</span>
    <span class="pl-k">if</span> <span class="pl-s1">delta_done</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">time_per_item</span> <span class="pl-c1">=</span> <span class="pl-s1">delta_time</span> <span class="pl-c1">/</span> <span class="pl-s1">delta_done</span>
        <span class="pl-s1">remaining</span> <span class="pl-c1">=</span> <span class="pl-s1">total</span> <span class="pl-c1">-</span> <span class="pl-s1">done</span>
        <span class="pl-s1">eta_seconds</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">time_per_item</span> <span class="pl-c1">*</span> <span class="pl-s1">remaining</span>)
        <span class="pl-s1">eta_string</span> <span class="pl-c1">=</span> <span class="pl-en">format_eta</span>(<span class="pl-s1">eta_seconds</span>)
    <span class="pl-k">else</span>:
        <span class="pl-c"># If no new items have completed since the script started, can't guess yet</span>
        <span class="pl-s1">eta_string</span> <span class="pl-c1">=</span> <span class="pl-s">"calculating..."</span>

    <span class="pl-s1">progress_line</span> <span class="pl-c1">=</span> (
        <span class="pl-s">f"<span class="pl-cce">\r</span>[<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">bar</span><span class="pl-kos">}</span></span>] <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">done</span><span class="pl-kos">}</span></span>/<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">total</span><span class="pl-kos">}</span></span> (<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">fraction</span><span class="pl-c1">*</span><span class="pl-c1">100</span>:.1f<span class="pl-kos">}</span></span>%) - ETA: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">eta_string</span><span class="pl-kos">}</span></span>"</span>
    )
    <span class="pl-en">print</span>(<span class="pl-s1">progress_line</span>, <span class="pl-s1">end</span><span class="pl-c1">=</span><span class="pl-s">''</span>, <span class="pl-s1">flush</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)


<span class="pl-k">def</span> <span class="pl-en">format_eta</span>(<span class="pl-s1">seconds</span>):
    <span class="pl-s">"""Convert number of seconds into a H:MM:SS or M:SS format string."""</span>
    <span class="pl-s1">h</span> <span class="pl-c1">=</span> <span class="pl-s1">seconds</span> <span class="pl-c1">//</span> <span class="pl-c1">3600</span>
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> (<span class="pl-s1">seconds</span> <span class="pl-c1">%</span> <span class="pl-c1">3600</span>) <span class="pl-c1">//</span> <span class="pl-c1">60</span>
    <span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-s1">seconds</span> <span class="pl-c1">%</span> <span class="pl-c1">60</span>

    <span class="pl-k">if</span> <span class="pl-s1">h</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
        <span class="pl-k">return</span> <span class="pl-s">f"<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">h</span>:d<span class="pl-kos">}</span></span>:<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">m</span>:02d<span class="pl-kos">}</span></span>:<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">s</span>:02d<span class="pl-kos">}</span></span>"</span>
    <span class="pl-k">else</span>:
        <span class="pl-k">return</span> <span class="pl-s">f"<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">m</span>:02d<span class="pl-kos">}</span></span>:<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">s</span>:02d<span class="pl-kos">}</span></span>"</span>


<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-en">main</span>()</pre></div>
<p>Which I can then run like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run progress.py 45 <span class="pl-s"><span class="pl-pds">'</span>ls *.mp4 | wc -l<span class="pl-pds">'</span></span></pre></div>
<p>The <code>45</code> there is the expected number of downloads (found with <code>jq length &lt; videos.json</code>). The <code>ls *.mp4 | wc -l</code> string is a command to run on each iteration to count how many items have been processed.</p>
<p>This command provides both a visible ASCII progress bar and an ETA prediction of when the program will finish, based on how many items have been processed and how quickly they appear to be running.</p>


guid: tag:til.simonwillison.net,2020-04-30:python_pyobjc-framework-corelocation.md
title: Geocoding from Python on macOS using pyobjc-framework-CoreLocation
author: Simon Willison
url: https://til.simonwillison.net/python/pyobjc-framework-corelocation
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>Rhet Turnbull <a href="https://twitter.com/RhetTurnbull/status/1883559820541956605" rel="nofollow">shared</a> this <a href="https://gist.github.com/RhetTbull/db70c113efd03029c6ff619f4699ce34">short script</a> for looking up the named timezone for a given location from Python on macOS using <code>objc</code> and the <code>CoreLocation</code> framework. It uses the <code>objc</code> package and <a href="https://pypi.org/project/pyobjc-framework-CoreLocation/" rel="nofollow">pyobjc-framework-CoreLocation</a>.</p>
<p>This piqued my interest, so I <a href="https://gist.github.com/simonw/fed886265bc32af81efa8e7973fea621">conversed with Claude</a> about other things I could do with that same framework. Here's the script we came up with, for geocoding an address passed to it using Core Location's <code>CLGeocoder.geocodeAddressString()</code> method:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># /// script</span>
<span class="pl-c"># requires-python = "&gt;=3.12"</span>
<span class="pl-c"># dependencies = [</span>
<span class="pl-c">#     "pyobjc-core",</span>
<span class="pl-c">#     "pyobjc-framework-CoreLocation",</span>
<span class="pl-c">#     "click"</span>
<span class="pl-c"># ]</span>
<span class="pl-c"># ///</span>
<span class="pl-s">"""Basic geocoding using CoreLocation on macOS."""</span>

<span class="pl-k">import</span> <span class="pl-s1">click</span>
<span class="pl-k">import</span> <span class="pl-s1">objc</span>
<span class="pl-k">from</span> <span class="pl-v">CoreLocation</span> <span class="pl-k">import</span> <span class="pl-v">CLGeocoder</span>
<span class="pl-k">from</span> <span class="pl-v">Foundation</span> <span class="pl-k">import</span> <span class="pl-v">NSRunLoop</span>, <span class="pl-v">NSDate</span>

<span class="pl-k">def</span> <span class="pl-en">forward_geocode</span>(<span class="pl-s1">address</span>: <span class="pl-smi">str</span>) <span class="pl-c1">-&gt;</span> <span class="pl-s1">list</span>[<span class="pl-smi">dict</span>]:
    <span class="pl-k">with</span> <span class="pl-s1">objc</span>.<span class="pl-c1">autorelease_pool</span>():
        <span class="pl-s1">geocoder</span> <span class="pl-c1">=</span> <span class="pl-v">CLGeocoder</span>.<span class="pl-c1">alloc</span>().<span class="pl-c1">init</span>()
        <span class="pl-s1">results</span> <span class="pl-c1">=</span> {<span class="pl-s">"placemarks"</span>: [], <span class="pl-s">"error"</span>: <span class="pl-c1">None</span>}
        <span class="pl-s1">completed</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
        
        <span class="pl-k">def</span> <span class="pl-en">completion</span>(<span class="pl-s1">placemarks</span>, <span class="pl-s1">error</span>):
            <span class="pl-k">nonlocal</span> <span class="pl-s1">completed</span>
            <span class="pl-k">if</span> <span class="pl-s1">error</span>:
                <span class="pl-s1">results</span>[<span class="pl-s">"error"</span>] <span class="pl-c1">=</span> <span class="pl-s1">error</span>.<span class="pl-c1">localizedDescription</span>()
            <span class="pl-k">elif</span> <span class="pl-s1">placemarks</span>:
                <span class="pl-s1">results</span>[<span class="pl-s">"placemarks"</span>] <span class="pl-c1">=</span> <span class="pl-s1">placemarks</span>
            <span class="pl-s1">completed</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
            
        <span class="pl-s1">geocoder</span>.<span class="pl-c1">geocodeAddressString_completionHandler_</span>(<span class="pl-s1">address</span>, <span class="pl-s1">completion</span>)
        
        <span class="pl-k">while</span> <span class="pl-c1">not</span> <span class="pl-s1">completed</span>:
            <span class="pl-v">NSRunLoop</span>.<span class="pl-c1">currentRunLoop</span>().<span class="pl-c1">runMode_beforeDate_</span>(
                <span class="pl-s">"NSDefaultRunLoopMode"</span>,
                <span class="pl-v">NSDate</span>.<span class="pl-c1">dateWithTimeIntervalSinceNow_</span>(<span class="pl-c1">0.1</span>)
            )
        
        <span class="pl-k">if</span> <span class="pl-s1">results</span>[<span class="pl-s">"error"</span>]:
            <span class="pl-k">raise</span> <span class="pl-en">Exception</span>(<span class="pl-s">f"Geocoding error: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">results</span>[<span class="pl-s">'error'</span>]<span class="pl-kos">}</span></span>"</span>)
            
        <span class="pl-k">return</span> [{
            <span class="pl-s">"latitude"</span>: <span class="pl-s1">pm</span>.<span class="pl-c1">location</span>().<span class="pl-c1">coordinate</span>().<span class="pl-c1">latitude</span>,
            <span class="pl-s">"longitude"</span>: <span class="pl-s1">pm</span>.<span class="pl-c1">location</span>().<span class="pl-c1">coordinate</span>().<span class="pl-c1">longitude</span>,
            <span class="pl-s">"name"</span>: <span class="pl-s1">pm</span>.<span class="pl-c1">name</span>(),
            <span class="pl-s">"locality"</span>: <span class="pl-s1">pm</span>.<span class="pl-c1">locality</span>(),
            <span class="pl-s">"country"</span>: <span class="pl-s1">pm</span>.<span class="pl-c1">country</span>()
        } <span class="pl-k">for</span> <span class="pl-s1">pm</span> <span class="pl-c1">in</span> <span class="pl-s1">results</span>[<span class="pl-s">"placemarks"</span>]]

<span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-c1">command</span>()</span>
<span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-c1">argument</span>(<span class="pl-s">'address'</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">main</span>(<span class="pl-s1">address</span>):
    <span class="pl-k">try</span>:
        <span class="pl-s1">locations</span> <span class="pl-c1">=</span> <span class="pl-en">forward_geocode</span>(<span class="pl-s1">address</span>)
        <span class="pl-k">for</span> <span class="pl-s1">loc</span> <span class="pl-c1">in</span> <span class="pl-s1">locations</span>:
            <span class="pl-s1">click</span>.<span class="pl-c1">echo</span>(<span class="pl-s">"<span class="pl-cce">\n</span>Location found:"</span>)
            <span class="pl-k">for</span> <span class="pl-s1">key</span>, <span class="pl-s1">value</span> <span class="pl-c1">in</span> <span class="pl-s1">loc</span>.<span class="pl-c1">items</span>():
                <span class="pl-k">if</span> <span class="pl-s1">value</span>:
                    <span class="pl-s1">click</span>.<span class="pl-c1">echo</span>(<span class="pl-s">f"<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">key</span><span class="pl-kos">}</span></span>: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">value</span><span class="pl-kos">}</span></span>"</span>)
    <span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
        <span class="pl-s1">click</span>.<span class="pl-c1">echo</span>(<span class="pl-s">f"Error: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">e</span><span class="pl-kos">}</span></span>"</span>, <span class="pl-s1">err</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
        <span class="pl-k">raise</span> <span class="pl-s1">click</span>.<span class="pl-c1">Abort</span>()

<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-en">main</span>()</pre></div>
<p>This can be run using <code>uv run</code> like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run geocode.py <span class="pl-s"><span class="pl-pds">'</span>500 Grove St, San Francisco, CA<span class="pl-pds">'</span></span></pre></div>
<p>Example output:</p>
<pre><code>Location found:
latitude: 37.777717
longitude: -122.42504
name: 500 Grove St
locality: San Francisco
country: United States 
</code></pre>
<p>I tried this without a network connection and it failed, demonstrating that Core Location uses some form of network-based API to geocode addresses.</p>
<p>There are a few new-to-me tricks in this script.</p>
<p><code>with objc.autorelease_pool()</code> is a neat <a href="https://pyobjc.readthedocs.io/en/latest/api/module-objc.html#objc.autorelease_pool" rel="nofollow">memory management pattern</a> provided by PyObjC for establishing an autorelease memory pool for the duration of a Python <code>with</code> block. Everything allocated by Objective C should be automatically cleaned up at the end of that block.</p>
<p>The <code>geocodeAddressString</code> method takes a completion handler. In this code we're setting that to a Python function that communicates state using shared variables:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">results</span> <span class="pl-c1">=</span> {<span class="pl-s">"placemarks"</span>: [], <span class="pl-s">"error"</span>: <span class="pl-c1">None</span>}
<span class="pl-s1">completed</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>

<span class="pl-k">def</span> <span class="pl-en">completion</span>(<span class="pl-s1">placemarks</span>, <span class="pl-s1">error</span>):
    <span class="pl-k">nonlocal</span> <span class="pl-s1">completed</span>
    <span class="pl-k">if</span> <span class="pl-s1">error</span>:
        <span class="pl-s1">results</span>[<span class="pl-s">"error"</span>] <span class="pl-c1">=</span> <span class="pl-s1">error</span>.<span class="pl-c1">localizedDescription</span>()
    <span class="pl-k">elif</span> <span class="pl-s1">placemarks</span>:
        <span class="pl-s1">results</span>[<span class="pl-s">"placemarks"</span>] <span class="pl-c1">=</span> <span class="pl-s1">placemarks</span>
    <span class="pl-s1">completed</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span></pre></div>
<p>We start that running like so:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">geocoder</span> <span class="pl-c1">=</span> <span class="pl-v">CLGeocoder</span>.<span class="pl-c1">alloc</span>().<span class="pl-c1">init</span>()
<span class="pl-s1">geocoder</span>.<span class="pl-c1">geocodeAddressString_completionHandler_</span>(<span class="pl-s1">address</span>, <span class="pl-s1">completion</span>)</pre></div>
<p>Then the clever bit:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">while</span> <span class="pl-c1">not</span> <span class="pl-s1">completed</span>:
    <span class="pl-v">NSRunLoop</span>.<span class="pl-c1">currentRunLoop</span>().<span class="pl-c1">runMode_beforeDate_</span>(
        <span class="pl-s">"NSDefaultRunLoopMode"</span>,
        <span class="pl-v">NSDate</span>.<span class="pl-c1">dateWithTimeIntervalSinceNow_</span>(<span class="pl-c1">0.1</span>)
    )</pre></div>
<p>Where did this code come from? It turns out Claude lifted that from the Rhet Turnbull script I fed into it earlier. Here's <a href="https://gist.github.com/RhetTbull/db70c113efd03029c6ff619f4699ce34#file-tzname-py-L54-L66">that code with Rhet's comments</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c1">WAIT_FOR_COMPLETION</span> <span class="pl-c1">=</span> <span class="pl-c1">0.01</span>  <span class="pl-c"># wait time for async completion in seconds</span>
<span class="pl-c"># ...</span>

<span class="pl-c"># reverseGeocodeLocation_completionHandler_ is async so run the event loop until completion</span>
<span class="pl-c"># I usually use threading.Event for this type of thing in pyobjc but the the thread blocked forever</span>
<span class="pl-s1">waiting</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span>
<span class="pl-k">while</span> <span class="pl-c1">not</span> <span class="pl-s1">completed</span>:
    <span class="pl-v">NSRunLoop</span>.<span class="pl-c1">currentRunLoop</span>().<span class="pl-c1">runMode_beforeDate_</span>(
        <span class="pl-s">"NSDefaultRunLoopMode"</span>,
        <span class="pl-v">NSDate</span>.<span class="pl-c1">dateWithTimeIntervalSinceNow_</span>(<span class="pl-c1">WAIT_FOR_COMPLETION</span>),
    )
    <span class="pl-s1">waiting</span> <span class="pl-c1">+=</span> <span class="pl-c1">WAIT_FOR_COMPLETION</span>
    <span class="pl-k">if</span> <span class="pl-s1">waiting</span> <span class="pl-c1">&gt;=</span> <span class="pl-c1">COMPLETION_TIMEOUT</span>:
        <span class="pl-k">raise</span> <span class="pl-en">TimeoutError</span>(
            <span class="pl-s">f"Timeout waiting for completion of reverseGeocodeLocation_completionHandler_: <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">waiting</span><span class="pl-kos">}</span></span> seconds"</span>
        )</pre></div>
<p>Is this the best pattern for  my own, simpler script? I don't know for sure, but it works. Approach with caution!</p>
<p>Since my script has inline script dependencies and I've <a href="https://gist.github.com/simonw/178ea93ac035293744bde97270d6a7a0">published it to a Gist</a> you can run it directly with <code>uv run</code> without first installing anything else like this:</p>
<div class="highlight highlight-source-shell"><pre>uv run https://gist.githubusercontent.com/simonw/178ea93ac035293744bde97270d6a7a0/raw/88c817e4103034579ec7523d8591bf60aa11fa67/geocode.py \
  <span class="pl-s"><span class="pl-pds">'</span>500 Grove St, San Francisco, CA<span class="pl-pds">'</span></span></pre></div>


guid: tag:til.simonwillison.net,2020-04-30:pytest_pytest-uv.md
title: Running pytest against a specific Python version with uv run
author: Simon Willison
url: https://til.simonwillison.net/pytest/pytest-uv
feedurl: https://til.simonwillison.net/tils/feed.atom
content_mime_type: text/html
content:
<p>While <a href="https://github.com/simonw/datasette/issues/2461">working on this issue</a> I figured out a neat pattern for running the tests for my project locally against a specific Python version using <a href="https://docs.astral.sh/uv/guides/scripts/" rel="nofollow">uv run</a>:</p>
<div class="highlight highlight-source-shell"><pre>uv run --python 3.12 --with <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span> pytest</pre></div>
<p>The new-to-me trick here is that <code>--with '.[test]</code> works for adding the project dependencies <em>and</em> the test dependencies from the <code>setup.py</code> or <code>pyproject.toml</code> file in the current project directory.</p>
<p>This makes it trivial to try running the test suite against different Python versions on demand without needing to worry about manually creating a virtual environment for each one.</p>
<p>This pattern works for more complex scenarios too. My project's GitHub Actions CI runs an additional variant that uses the <code>pytest-cov</code> plugin to generate coverage reports. I could simulate that locally by including that as another <code>--with pytest-cov</code> option - here I'm also adding the <code>-Werror</code> flag so any warnings would be treated as errors:</p>
<div class="highlight highlight-source-shell"><pre>uv run --python 3.12 \
  --with pytest-cov \
  --with <span class="pl-s"><span class="pl-pds">'</span>.[test]<span class="pl-pds">'</span></span> pytest \
    -Werror \
    --cov=datasette \
    --cov-config=.coveragerc \
    --cov-report xml:coverage.xml \
    --cov-report term  </pre></div>


guid: https://lwn.net/Articles/1007887/
title: Firefox 135.0 released
author: corbet
url: https://lwn.net/Articles/1007887/
feedurl: https://lwn.net/headlines/rss
content:
<a href="https://www.mozilla.org/en-US/firefox/135.0/releasenotes/">Version
135.0</a> of the Firefox web browser has been released.  Changes include
more languages for the translations feature, increasing roll-out of the
credit-card autofill and AI chatbot features, and (perhaps most welcome):
<p>
<blockquote class="bq">
	Firefox now includes safeguards to prevent sites from abusing the
	history API by generating excessive history entries, which can make
	navigating with the back and forward buttons difficult by
	cluttering the history. This intervention ensures that such
	entries, unless interacted with by the user, are skipped when using
	the back and forward buttons.
</blockquote>


guid: https://lwn.net/Articles/1006375/
title: [$] An update on sealed system mappings
author: daroc
url: https://lwn.net/Articles/1006375/
feedurl: https://lwn.net/headlines/rss
content:
<p>
Jeff Xu has been working on
<a href="https://lwn.net/ml/all/20241125202021.3684919-1-jeffxu@google.com/">
a patch set</a> that makes certain mappings in a process's address space
impossible to change, sealing them against tampering. This has some potential
security benefits — mainly, making
sure that someone cannot relocate the
<a href="https://lwn.net/Articles/446528/">vsyscall and
vDSO mappings</a> — but some kernel developers haven't
been impressed with the patches.
While the core functionality (sealing the mappings) is sound, some of the
supporting code for enabling and disabling the new feature caused concern by
going against the normal design for such things. Reviewers also questioned
how this feature would interact with checkpointing and with sandboxing.
</p>


guid: https://lwn.net/Articles/1008051/
title: Security updates for Wednesday
author: jzb
url: https://lwn.net/Articles/1008051/
feedurl: https://lwn.net/headlines/rss
content:
Security updates have been issued by <b>Debian</b> (firefox-esr), <b>Fedora</b> (fastd, ovn, and yq), <b>Mageia</b> (libreoffice), <b>Slackware</b> (mozilla), <b>SUSE</b> (google-osconfig-agent, grafana, helm, and rime-schema-all), and <b>Ubuntu</b> (linux-azure, linux-azure-5.4, linux-lowlatency, openjdk-17, openjdk-21, openjdk-23, openjdk-8, and openjdk-lts).


guid: https://lwn.net/Articles/1007689/
title: [$] Exposing concurrency bugs with a custom scheduler
author: daroc
url: https://lwn.net/Articles/1007689/
feedurl: https://lwn.net/headlines/rss
content:
<p>
Jake Hillion gave
<a href="https://fosdem.org/2025/schedule/event/fosdem-2025-4489-concurrency-testing-using-custom-linux-schedulers/">
a presentation</a> at
<a href="https://fosdem.org/2025/">
FOSDEM</a> about using
<a href="https://www.kernel.org/doc/html/next/scheduler/sched-ext.html">
sched_ext</a>, the BPF
scheduling framework that was introduced in kernel version 6.12, to help find
elusive concurrency problems. In collaboration with Johannes Bechberger, he has
built a scheduler that can reveal theoretically possible but unobserved
concurrency bugs in test code in a few minutes. Since their scheduler only
relies on mainline kernel features, it can theoretically be applied to any
application that runs on Linux — although there are a number of caveats since
the project is still in its early days.
</p>


guid: https://lwn.net/Articles/1006001/
title: LWN site tour 2025
author: jzb
url: https://lwn.net/Articles/1006001/
feedurl: https://lwn.net/headlines/rss
content:
<p>Over the past year or so, LWN has added a number of useful new
features for our subscribers to enhance the experience of reading and
commenting on our content. Those features are of little use, however,
to readers who do not know about them. It has been more than a decade
since we last provided <a href="https://lwn.net/Articles/622988/">a
tour of the site</a>&mdash;it seems that another is in
order. Walk this way for a look at the LWN kernel source database (KSDB),
enhanced commenting features, EPUB downloads, and more.</p>


guid: https://lwn.net/Articles/1008137/
title: Servo in 2024: stats, features and donations
author: jzb
url: https://lwn.net/Articles/1008137/
feedurl: https://lwn.net/headlines/rss
content:
<p>The <a href="https://servo.org/">Servo</a> Rust-based rendering
engine project has <a
href="https://servo.org/blog/2025/01/31/servo-in-2024/">published</a>
an article summarizing its progress in&nbsp;2024, and plans for the
future:</p>

<blockquote class="bq">
Servo main dependencies (SpiderMonkey, Stylo and WebRender) have been
upgraded, the new layout engine has kept evolving adding support for
floats, tables, flexbox, fonts, etc. By the end of&nbsp;2024 Servo passes&nbsp;1,515,229 WPT subtests (79%). Many other new features have been under
active development: WebGPU, Shadow DOM, ReadableStream, WebXR, ... Servo
now supports two new platforms: Android and OpenHarmony. And we have
got the first experiments of applications using Servo as a web engine
(like <a
href="https://servo.org/blog/2024/01/19/embedding-update/">Tauri</a>,
<a href="https://github.com/DioxusLabs/blitz">Blitz</a>, <a
href="https://github.com/KDABLabs/cxx-qt-servo-webview">QtWebView</a>,
<a
href="https://github.com/mcclure/cuervo/tree/unstable_traversal_experiment">Cuervo</a>,
<a href="https://github.com/versotile-org/verso/">Verso</a> and <a
href="https://github.com/moto-browser/moto">Moto</a>).
</blockquote>

<p></p>


guid: https://lwn.net/Articles/1007000/
title: [$] LWN.net Weekly Edition for February 6, 2025
author: corbet
url: https://lwn.net/Articles/1007000/
feedurl: https://lwn.net/headlines/rss
content:
Inside this week's LWN.net Weekly Edition:
        <p>
        <ul>
<li> <a href="https://lwn.net/Articles/1007000/">Front</a>: Finding concurrency bugs with sched_ext; Rust abstractions; 6.14 Merge window; Sealed system mappings; OpenSUSE board; Julia; Site tour.
            <li> <a href="https://lwn.net/Articles/1007002/">Briefs</a>: Binutils 2.44; Firefox 135.0; Freedesktop GitLab; GNU C Library 2.41; GTK; Servo; Thunderbird updates; Sanctions; Quotes; ...
            <li> <a href="https://lwn.net/Articles/1007003/">Announcements</a>: Newsletters, conferences, security updates, patches, and more.
            </ul>

        


guid: https://lwn.net/Articles/1008275/
title: Security updates for Thursday
author: jake
url: https://lwn.net/Articles/1008275/
feedurl: https://lwn.net/headlines/rss
content:
Security updates have been issued by <b>Debian</b> (asterisk and chromium), <b>Fedora</b> (FlightGear, java-1.8.0-openjdk, java-11-openjdk, java-17-openjdk, java-latest-openjdk, and SimGear), <b>Mageia</b> (bind, chromium-browser-stable, python-django, and vim), <b>Oracle</b> (buildah, bzip2, firefox, keepalived, mariadb:10.11, and podman), <b>Slackware</b> (curl, mariadb, and mozilla), <b>SUSE</b> (cargo-audit-advisory-db-20250204 and python311-scikit-learn), and <b>Ubuntu</b> (ckeditor, krb5, and ruby2.7).


guid: https://lwn.net/Articles/1007426/
title: [$] The selfish contributor revisited
author: jzb
url: https://lwn.net/Articles/1007426/
feedurl: https://lwn.net/headlines/rss
content:
<p>Open source is often described as a "gift economy"&mdash;an
ecosystem where contributors are motivated by a desire to make the
world a better place. That is, sometimes, true. However, James
Bottomley used his <a
href="https://fosdem.org/2025/schedule/event/fosdem-2025-6540-the-selfish-contributor-revisited/">main
track slot</a> at <a href="https://fosdem.org/2025/">FOSDEM 2025</a>,
on February&nbsp;1, to make the case that it is better to bank on the
selfish motivations of individuals to drive community success than to
rely on their altruism.</p>


guid: https://lwn.net/Articles/1008339/
title: OpenWrt 24.10.0 released
author: corbet
url: https://lwn.net/Articles/1008339/
feedurl: https://lwn.net/headlines/rss
content:
<a href="https://lwn.net/ml/all/607e9b10-1b06-4e68-9be8-1f2d26d5ef24@hauke-m.de">Version
24.10.0</a> of the OpenWrt router-oriented distribution has been released.
Changes include an update to the 6.6 kernel, use of access control lists on
larger systems, multipath TCP support, better WiFi6 support, the
beginning of WiFi7 support, and more.


guid: https://lwn.net/Articles/1008342/
title: LibreOffice 25.2 released
author: corbet
url: https://lwn.net/Articles/1008342/
feedurl: https://lwn.net/headlines/rss
content:
Version 25.2 of the LibreOffice productivity suite is out.  Changes include
the ability to remove all personal information from any document, support
for ODF version&nbsp;1.4, a number of accessibility improvements, and more;
see <a href="https://wiki.documentfoundation.org/ReleaseNotes/25.2">the
release notes</a> for details.


guid: https://lwn.net/Articles/1008386/
title: OpenInfra board calls for input on joining Linux Foundation
author: jzb
url: https://lwn.net/Articles/1008386/
feedurl: https://lwn.net/headlines/rss
content:
<p>Jonathan Bryce has <a href="https://lists.openinfra.org/archives/list/foundation@lists.openinfra.org/thread/3B7OWPRXB4KD2DVX7SYYSHYYRNCKVV46/">announced</a> two open community meetings to hear
input on the topic of the <a href="https://openinfra.org/">OpenInfra
Foundation</a> migrating to the Linux Foundation. Bryce
wrote that the OpenInfra board has carefully evaluated its options,
and sees joining the Linux Foundation as the best way forward.</p>

<p>Like the Linux Foundation, the OpenInfra Foundation is 501(c)(6)
nonprofit. According to the <a
href="https://board.openinfra.org/strategic-consideration/faq">FAQ</a>,
OpenInfra "<q>is in great health, financially and otherwise</q>" with
a growth in membership of about 15% in the last year. However, its
needs in 2025 are different than when it was founded as the OpenStack
Foundation in 2012.</p>

<blockquote class="bq">
While the opportunities ahead for open source to make a positive
impact on the world are greater than they have ever been, the
challenges are more significant as well, particularly with respect to
regulations, licensing and geopolitical tensions that threaten global
collaboration.
</blockquote>

<p>The meetings will be held on February&nbsp;11 and February&nbsp;13 as
Zoom calls. The OpenInfra board will schedule a vote after feedback
has been collected and draft governance documents have been
published.</p>

<p></p>


guid: https://lwn.net/Articles/1008502/
title: Security updates for Friday
author: daroc
url: https://lwn.net/Articles/1008502/
feedurl: https://lwn.net/headlines/rss
content:
Security updates have been issued by <b>Debian</b> (openjdk-17), <b>Fedora</b> (firefox, FlightGear, java-1.8.0-openjdk, java-11-openjdk, java-latest-openjdk, and SimGear), <b>Mageia</b> (gstreamer), <b>Red Hat</b> (firefox, kernel, kernel-rt, libsoup, and python-jinja2), <b>SUSE</b> (bind, curl, dcmtk, etcd, firefox, google-osconfig-agent, krb5, openssl-1_1, podman, python311-cbor2, thunderbird, wget, and xrdp), and <b>Ubuntu</b> (glibc).


guid: https://lwn.net/Articles/1007947/
title: [$] Improved load-time checking for BPF kfuncs
author: corbet
url: https://lwn.net/Articles/1007947/
feedurl: https://lwn.net/headlines/rss
content:
The <a href="https://lwn.net/Articles/982077/">BPF verifier</a> is charged with the
challenging task of ensuring that a BPF program is safe for the kernel to
run before that program is loaded.  Among many other concerns, the verifier
must ensure that any kfuncs (kernel functions that have been exported to
BPF programs) are called with the correct parameters and from the right
context.  The "context" part of that enforcement is showing its age in ways
that are hurting performance; Juntong Deng has been working on
infrastructure to provide finer-grained control over when a kfunc can be
called.


guid: https://lwn.net/Articles/1008641/
title: Three stable kernel updates
author: corbet
url: https://lwn.net/Articles/1008641/
feedurl: https://lwn.net/headlines/rss
content:
The
<a href="https://lwn.net/Articles/1008642/">6.13.2</a>,
<a href="https://lwn.net/Articles/1008643/">6.12.13</a>, and
<a href="https://lwn.net/Articles/1008644/">6.6.76</a>
stable kernels have been released; each contains another set of important
fixes.


guid: https://willhbr.net/2024/07/11/merging-jj-repos/
title: Merging JJ Repos
url: https://willhbr.net/2024/07/11/merging-jj-repos/
feedurl: https://willhbr.net/feed.xml
content:
<p>So here’s a weird thought: can you merge two repositories in <a href="https://github.com/martinvonz/jj">JJ</a>? I have just ended up in a weird state (for unimportant reasons) where I had a remote repo and a local repo that diverged entirely due to me rewriting the commit history. I was going to delete the local copy and re-clone (since the remote was now the source of truth) but I thought “what happens if I add the remote and fetch from it?”, and thought I might as well satisfy my curiosity first.</p>

<p>I added the remote and fetched it, and sure enough it worked without any issues. Since there were no commits in common, I was left with two diverging chains starting at the root commit (an empty commit present at the base of all JJ repos). I could then delete the unwanted chain of commits and get to the state I’d be in if I’d just done a fresh clone.</p>

<p>Or I could do one <code class="language-plaintext highlighter-rouge">jj rebase</code> and move all the commits in one chain atop the commits in the other, effectively rebasing one repo on another. This led me to the cursed realisation that I could take two completely unrelated repos and just splice them together, resulting in a repo with the contents and commit histories of both. Of course this will almost certainly result in horrible merge conflicts, but you can <em>do</em> it.</p>


<a href="https://willhbr.net/2024/07/11/merging-jj-repos/">Permalink</a>
&bull; July 11, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Merging%20JJ%20Repos" target=_blank>email</a>.


guid: https://willhbr.net/2024/07/17/upsetting-the-apricot-cart/
title: Upsetting the Apricot Cart
url: https://willhbr.net/2024/07/17/upsetting-the-apricot-cart/
feedurl: https://willhbr.net/feed.xml
content:
<p>Imagine for a minute that I had a pair of apricot trees in my garden. I don’t actually have a garden, so you’ll have to imagine that I have one of those too. I love a nice fresh ripe apricot. However, I can’t eat every single apricot that comes off the trees—there are just too many! There aren’t enough apricots to start an apricot-selling business, plus I’ve got a job anyway. Instead I setup a stall in the street and give away my apricots to anyone passing by that shares my love of stone fruits. After a while word gets around and people in the neighbourhood know that they can get some apricots from Will’s stall, and they’ll drop by to see if I’ve got any to give away.</p>

<p>Someone puts up a sign at the end of the street: “Tasty Free Apricots from Will’s Tree!”. It’s not what I would’ve done, but to be fair it is a bit difficult to find my stall, so making it easier to find is convenient for me too. I’m happy that more people appreciate the apricots, and I enjoy talking about fruit with the people that stop by. Having a few more people turn up that like apricots but didn’t otherwise know about my stall is nice.</p>

<p>A little later, someone notices my stall and the boxes of fruit that I have sitting behind it. They take a box of apricots and give them away at a local market.</p>

<p>Multiple people have the same idea—it entices more people to their stand if they’ve got apricots to give away. Some of them will tell shoppers where the apricots came from—and where they can find more themselves—but others don’t pass that information on to whoever is attending their stand.</p>

<p>When I find out, I’m a little put out by this. My friends say that the result is the same—the apricots get given away to people that want them, what’s the problem? Wasn’t the whole point of this stall to give away the fruit that I didn’t need?</p>

<p>They don’t understand that while yes, I was giving away the fruit for free so I haven’t lost anything financially. What I have lost is a connection with the people that received the fruit. I don’t get to see that they appreciate it, they don’t come back and tell me about the desert they cooked with the apricots last week. They can’t see the time and effort that I put in to building the stall, or hear about my many apricot recipe recommendations.</p>

<hr />

<p>Completely unrelated to that imaginary scenario, on this website I share my thoughts, opinions, and projects I’ve been working on. I’m happy to give them away because I have such an abundance of opinions I have to put them <em>somewhere</em>, and I’m self-aware enough to know that no one would be bothered paying for this nonsense. However I write with the hope that readers will appreciate not just the ideas, but the way the ideas are presented, both in how they’re written and how they’re presented on my website.</p>


<a href="https://willhbr.net/2024/07/17/upsetting-the-apricot-cart/">Permalink</a>
&bull; July 17, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Upsetting%20the%20Apricot%20Cart" target=_blank>email</a>.


guid: https://willhbr.net/2024/07/18/lazy-jekyll-hacks-for-more-accurate-publication-times/
title: Lazy Jekyll Hacks for More Accurate Publication Times
url: https://willhbr.net/2024/07/18/lazy-jekyll-hacks-for-more-accurate-publication-times/
feedurl: https://willhbr.net/feed.xml
content:
<p>So here’s something terrible that I’ve just done. Enthusiastic readers with an attention to detail (especially those who read the RSS feed) will notice that all my posts are published at midnight. This is actually a lie—I do usually write posts late at night, but I don’t carefully wait until exactly midnight to publish them. The real reason is that I don’t include a published time on my posts—just a date—so if your feed reader wants to show a time, it’ll show <code class="language-plaintext highlighter-rouge">00:00</code>.</p>

<p>Of course I could just add a time to each post, but that requires effort. The time only really matters for the first 24 hours or so after publication to make posts from multiple websites appear in order in a feed reader. After that, only the date matters, since I almost never publish multiple posts in one day.</p>

<p>Instead of adding the time in manually, I’ve made a lower-effort solution to do some terrible things with <a href="http://jekyllrb.com">Jekyll</a> to fake it.</p>

<p>Jekyll includes a <code class="language-plaintext highlighter-rouge">site.time</code> variable which is the current time when the site is built. Anywhere that uses this will be updated every time the site is regenerated. I use this in my feeds for the “last updated” fields, and in the footer to put <code class="language-plaintext highlighter-rouge">$CURRENT_YEAR</code> in the copyright notice.</p>

<p>What I’ve now done is when I need a post time, I check if the post was published on the same day as the site was built. If it was, I use the site build time instead of the post date. So when I add a new post, the site is built and that build time is used in all post metadata. When feed readers come along they will have an accurate publication time to show to users.</p>

<p>Of course if I push another change on the same day that doesn’t update the post, the publication date will jump ahead, and when I push a change the following day, the publication date will be truncated back to just being midnight. I did say it was a hack.</p>

<p>I don’t <em>think</em> this will cause problems for feed readers, as they should be using the <code class="language-plaintext highlighter-rouge">guid</code> (for RSS) or <code class="language-plaintext highlighter-rouge">id</code> (for JSON Feed) fields to identify posts. The most realistic failure most I can see is that the post jumps up or down a few places in a list of posts when the feed is re-fetched, but I don’t think that’s any worse than having the post appear out-of-order for everyone all the time.</p>


<a href="https://willhbr.net/2024/07/18/lazy-jekyll-hacks-for-more-accurate-publication-times/">Permalink</a>
&bull; July 18, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Lazy%20Jekyll%20Hacks%20for%20More%20Accurate%20Publication%20Times" target=_blank>email</a>.


guid: https://willhbr.net/2024/07/20/a-new-home-for-my-photos/
title: A New Home for My Photos
url: https://willhbr.net/2024/07/20/a-new-home-for-my-photos/
feedurl: https://willhbr.net/feed.xml
content:
<p>A few weeks ago Eugen Rochko (creator of Mastodon) published a <a href="https://eugenrochko.com">new photography portfolio website</a>. This quickly made me want to improve my own photography website. Eugen’s site puts more of a focus on small groups of photos, and makes the photo metadata prominent—Eugen shoots on film so this is information about the camera as well as the type of film used and where it was developed.</p>

<p>My photo website was forked from this site a few years ago. The purpose of the design is to make it seem like it’s part of the same site—the colours and header match exactly. It’s built mostly around a photo grid, replicating the view you’d get on Instagram, with square thumbnails for all pictures. There isn’t really anywhere to show additional information or commentary about the photo, and the fact that the style matches this site means I have to keep the two repositories in sync.<sup id="fnref:no-theme" role="doc-noteref"><a href="#fn:no-theme" class="footnote" rel="footnote">1</a></sup></p>

<p loading="lazy"><img src="/images/2024/old-photos-website.webp" alt="screenshot of my old photos website" /></p>

<p class="caption">That’s my old photos website, with an Instagram-style grid layout.</p>

<p>Over the course of about a day I pulled together a new site that is less of a copy of Instagram and more of a photo journal. You can <a href="https://photography.willhbr.net">see it live here</a>.</p>

<p loading="lazy"><img src="/images/2024/new-photos-website.webp" alt="screenshot of my new photos website" /></p>

<p class="caption">The current state of my new photos website.</p>

<p>I also took inspiration from <a href="https://sdw.space/photography/">Sebastiaan de With’s photo blog</a>. I am perpetually jealous of the stunning photos he shoots on his phone.</p>

<p>Probably the biggest pain was my decision to not have titles for posts. They can have a location, but otherwise they are just identified by the date. Whenever I show a post I display a heading built from the location, date, and number of photos. The format of this is changed depending on how many photos there are, and whether there is a location set on the post. The complexity in implementation is worth it for me, as it reduces the overhead to publishing something—photos can be added with no additional commentary.</p>

<p>The photos linked to a single post are defined as a list in the <a href="http://jekyllrb.com">Jekyll</a> frontmatter. This includes information pulled from the EXIF metadata—camera, lens, aperture, shutter speed, and focal length. If any of these are omitted, they will be gracefully omitted on the website. The actual body of the post allows me to write as much or as little as I want.</p>

<p>The layout is fairly simple—it’s mostly just a vertical stack of elements. I’m no CSS wizard. The one trick that I think is worthwhile is setting the <code class="language-plaintext highlighter-rouge">max-height</code> of an image to <code class="language-plaintext highlighter-rouge">80vh</code> so you’ll always be able to see the whole image, no matter the size of your browser window. I decided that any kind of fancy flex-box-y layout wasn’t really worth it since most people would be looking at this on a phone, where you only want to show one image at a time anyway.</p>

<p>I put the same amount of care into the RSS feed as I did into the main website. Since the photos are defined in the frontmatter and not in the body of the post, it’s easy to render them differently for the feed with fewer HTML tags, as well as formatting the pseudo-title (built from the date and location, remember) for the plaintext title of the feed item. My hope here is that if you follow the site via RSS (or JSON Feed!) you’ll get as good an experience as if you were viewing the web page.</p>

<p>There’s also a little Ruby script that reads the EXIF data using <a href="https://imagemagick.org">ImageMagick</a>, creates the Markdown files with frontmatter, and recompresses the images. Who knows how I’ll end up <a href="/2023/05/22/complicated-solutions-to-photo-publishing/">publishing photos in the long term</a>, but for now the script will have to do.</p>

<p>There are 80 posts and 200 photos already on there, a different collection to what I had on my previous site. Perhaps you should go and <a href="https://photography.willhbr.net">have a look</a>.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:no-theme" role="doc-endnote">
      <p>If I did GitHub Pages with a custom build action, perhaps I could turn this into a theme that could be properly shared between the two sites. <a href="#fnref:no-theme" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/07/20/a-new-home-for-my-photos/">Permalink</a>
&bull; July 20, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20A%20New%20Home%20for%20My%20Photos" target=_blank>email</a>.


guid: https://willhbr.net/2024/08/18/understanding-revsets-for-a-better-jj-log-output/
title: Understanding Revsets for a Better JJ Log Output
url: https://willhbr.net/2024/08/18/understanding-revsets-for-a-better-jj-log-output/
feedurl: https://willhbr.net/feed.xml
content:
<p>In git you can do something like <code class="language-plaintext highlighter-rouge">HEAD~</code> to refer to the parent commit of <code class="language-plaintext highlighter-rouge">HEAD</code>. Mercurial has a similar feature <a href="https://repo.mercurial-scm.org/hg/help/revsets">called revsets</a> which <a href="https://martinvonz.github.io/jj/latest/revsets/">JJ took inspiration from</a> (including the name).</p>

<p>The revset language is a declarative query language—not unlike SQL—that lets you specify a set of revisions (a revset) that match certain criteria. It ends up looking more like set operations than SQL, but the idea is similar. In JJ you can use <code class="language-plaintext highlighter-rouge">@</code> to mean “the current commit”, or <code class="language-plaintext highlighter-rouge">mine()</code> to mean “all the commits that I authored”, or <code class="language-plaintext highlighter-rouge">trunk()</code> to mean “the base branch that code will be merged into”.</p>

<p>That’s getting a little bit ahead of ourselves. Why did I go down a rabbit hole of learning about revsets in the first place? Well, one of the <a href="/2024/04/01/its-not-me-its-git/">many nice things</a> about JJ is that the default output of the <code class="language-plaintext highlighter-rouge">log</code> command is to show just the stuff you care about, not the full history. For example here’s the current state of the repo for my website:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>jj log
<span class="go">@  t Will Richardson now e
│  Add post about revsets
◆  k Will Richardson 4 weeks ago main HEAD@git 5
│  photo post
~
</span></code></pre></div></div>

<p>I don’t care about the contents of the other 300+ commits in the repo most of the time. Showing just the stuff that hasn’t been merged into <code class="language-plaintext highlighter-rouge">main</code> is great.</p>

<p>However, if you run <code class="language-plaintext highlighter-rouge">jj log</code> on some other repos, things aren’t quite as neat. Take the <a href="https://github.com/crystal-lang/crystal">Crystal language repo</a> as an example. The output of <code class="language-plaintext highlighter-rouge">jj log</code> is over 2000 lines! That doesn’t include any of the 15,000 commits that have been merged into the main branch, that’s just JJ showing commits that we might want to work on.</p>

<p>What causes this huge output is the fact that the Crystal repo has 93 branches, with commits that haven’t been merged into the main branch, including long-lived branches that have years of work that isn’t in <code class="language-plaintext highlighter-rouge">master</code><sup id="fnref:toilet" role="doc-noteref"><a href="#fn:toilet" class="footnote" rel="footnote">1</a></sup>. The default set of revisions that JJ logs is anything yet to be merged into <code class="language-plaintext highlighter-rouge">trunk()</code>—the main branch. This is a sensible default as it avoids a situation where commits become invisible to the user—they’re always either in the main branch, or they’re shown in <code class="language-plaintext highlighter-rouge">jj log</code>.</p>

<p>However what I want to show is just commits that <em>I’ve</em> written that haven’t been merged, and ignore all these other branches. To do that we need to understand some revsets.</p>

<p>If you want to learn the revset language properly, you should read the <a href="https://martinvonz.github.io/jj/latest/revsets/">revset language documentation</a>, but I’m going to walk through how I settled on my default log output.</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[revsets]</span>
<span class="py">log</span> <span class="p">=</span> <span class="s">'@ | ancestors(trunk()..(visible_heads() &amp; mine()), 2) | trunk()'</span>
</code></pre></div></div>

<p class="caption">That’s the config that I <a href="https://github.com/willhbr/dotfiles/blob/main/git/jjconfig.toml">ended up settling on</a>.</p>

<p>I wanted the log to show all the commit branches (branches as in the sense of a tree structure, not branches as in git branches) that I had authored that weren’t merged into the main branch.</p>

<p><code class="language-plaintext highlighter-rouge">visible_heads()</code> gives me all the leaf nodes in the repository. In a git repo this would basically be all the feature branches that people had pushed to the remote but not yet had their pull request merged. Branches like <code class="language-plaintext highlighter-rouge">main</code> or <code class="language-plaintext highlighter-rouge">master</code> usually wouldn’t be in this list as there will almost certainly be commits somewhere that build on top of it.</p>

<p>In the Crystal repo this gives us a huge output, let’s peek at just the top:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>jj log <span class="nt">-r</span> <span class="s1">'visible_heads()'</span> | <span class="nb">head</span> <span class="nt">-n</span> 20
<span class="go">@  x Will Richardson 14 minutes ago c
│  (no description set)
~

◆  nsxyl Johannes Müller 2 days ago changelog/1.13.2@origin 7214
│  Add changelog for 1.13.2
~

◆  wvqv Johannes Müller 2 days ago revert-14878-docs-generator-dont-mention-nodoc-types@origin a17af
</span><span class="gp">│  Revert "Fix: Don't link to undocumented types in API docs (#</span>14878<span class="o">)</span><span class="s2">"
</span><span class="go">~

◆  xxkqm renovate[bot] 3 days ago renovate/gh-actions@origin d6ce
│  Update actions/checkout action to v4
~

◆  rolu Johannes Müller 3 weeks ago infra/macos-14@origin b479
│  verbose spec output
~
</span></code></pre></div></div>

<p>There’s my working copy commit, and then a bunch of in-flight work like the <a href="https://github.com/crystal-lang/crystal/pull/14914">changelist for version 1.13.2</a>. Each of these heads are a single leaf commit, and JJ dutifully only prints the commit and omits their parent(s).</p>

<p>If we want to show a bit more context, we can use the <code class="language-plaintext highlighter-rouge">ancestors()</code> function (yeah there are functions in revsets) to get the parents of each of the heads:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>jj log <span class="nt">-r</span> <span class="s1">'ancestors(visible_heads(), 1)'</span>
<span class="go">@  x Will Richardson 21 minutes ago c
│  (no description set)
○  p Will Richardson 21 minutes ago HEAD@git 6
│  Allow serving index.html from StaticFileHander
~  (elided revisions)
│ ◆  nsxyl Johannes Müller 2 days ago changelog/1.13.2@origin 7214
├─╯  Add changelog for 1.13.2
◆  okrx Quinton Miller 2 days ago fa02
</span><span class="gp">│  Support LLVM OrcV2 codegen specs (#</span>14886<span class="o">)</span>
<span class="go">~  (elided revisions)
│ ◆  wvqv Johannes Müller 2 days ago revert-14878-docs-generator-dont-mention-nodoc-types@origin a17af
</span><span class="gp">├─╯  Revert "Fix: Don't link to undocumented types in API docs (#</span>14878<span class="o">)</span><span class="s2">"
</span><span class="go">◆  vrny Johannes Müller 3 days ago 93ac1
</span><span class="gp">│  Refactor interpreter stack code to avoid duplicate macro expansion (#</span>14876<span class="o">)</span>
<span class="go">~  (elided revisions)
│ ◆  xxkqm renovate[bot] 3 days ago renovate/gh-actions@origin d6ce
├─╯  Update actions/checkout action to v4
◆  tnzt Johannes Müller 3 days ago f0fe
</span><span class="gp">│  [CI] Update GitHub runner to `macos-14` (#</span>14833<span class="o">)</span>
<span class="go">~  (elided revisions)
</span></code></pre></div></div>

<p>JJ shows the heads, and then the commit immediately before the head. If you look at the full log output you can see that it starts linking up commits because some of them share the same parent—and it won’t print the same commit twice.</p>

<p>The default JJ log output uses the <code class="language-plaintext highlighter-rouge">ancestors()</code> function to show a bit more context in the log, rather than just showing the un-merged commits. This typically results in the tip of the main branch being visible in the log, which is nice as you can see where you’ve synced your repo to.</p>

<p>What we actually want is to show the commits that <em>we’ve</em> made, which we can access with the <code class="language-plaintext highlighter-rouge">mine()</code> function. This uses the configured author email to filter the commits. I can log the commits that I’ve made to Crystal with <code class="language-plaintext highlighter-rouge">jj log -r 'mine()'</code>.</p>

<p>This can be pieced together now using a few of the revset operators. I can find all the commits that I’ve authored that aren’t in the main branch by subtracting all the commits in the main branch from all my commits: <code class="language-plaintext highlighter-rouge">mine() ~ ::trunk()</code>. The <code class="language-plaintext highlighter-rouge">::</code> prefix operator gives all the ancestors up until the revision, so <code class="language-plaintext highlighter-rouge">::trunk()</code> gives me everything until the last commit on the main branch. The <code class="language-plaintext highlighter-rouge">~</code> operator subtracts the right hand side from the left hand side. I basically just think of these as set operations, which I guess they’re not <em>quite</em> but it gets me most of the way there.</p>

<p>That’s already a pretty nice log output, we could wrap that in <code class="language-plaintext highlighter-rouge">ancestors()</code> or just append <code class="language-plaintext highlighter-rouge">| trunk()</code> to show the tip of the main branch to get that extra bit of context.</p>

<p>My revset is slightly different, I use <code class="language-plaintext highlighter-rouge">visible_heads() &amp; mine()</code> to get the leaf commits that I have authored, and then get all the commits between <code class="language-plaintext highlighter-rouge">trunk()</code> and those with the <code class="language-plaintext highlighter-rouge">..</code> operator: <code class="language-plaintext highlighter-rouge">trunk()..(visible_heads() &amp; mine())</code>. I then get the <code class="language-plaintext highlighter-rouge">ancestors()</code> of those to show additional context.</p>

<p>Using the union (<code class="language-plaintext highlighter-rouge">|</code>) operator with both <code class="language-plaintext highlighter-rouge">trunk()</code> and <code class="language-plaintext highlighter-rouge">@</code> ensures that the main branch and current revision are <strong>always</strong> visible in the log output, even if I’ve got myself into a serious pickle—perhaps if I’ve pulled someone else’s un-merged branch and have checked out one of their commits.</p>

<p>I’ve had this in my config for a few months now and it seems to be working well. I have an alias that will log <em>everything</em> <code class="language-plaintext highlighter-rouge">jj log -r 'all()'</code> so if I do mess something up, I can still find my commit there. Or I can always just remove the <code class="language-plaintext highlighter-rouge">revsets.log</code> option from my config file and go back to the standard output. If you want this output, pop this in your <code class="language-plaintext highlighter-rouge">.jjconfig.toml</code>:</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[revsets]</span>
<span class="py">log</span> <span class="p">=</span> <span class="s">'@ | ancestors(trunk()..(visible_heads() &amp; mine()), 2) | trunk()'</span>
</code></pre></div></div>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:toilet" role="doc-endnote">
      <p>There’s a <a href="https://github.com/crystal-lang/crystal/compare/master...toilet">branch named “toilet”</a> with one commit from 4 years ago adding more calls to <code class="language-plaintext highlighter-rouge">IO#flush</code> throughout the standard library. <a href="#fnref:toilet" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/08/18/understanding-revsets-for-a-better-jj-log-output/">Permalink</a>
&bull; August 18, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Understanding%20Revsets%20for%20a%20Better%20JJ%20Log%20Output" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/02/horrifying-volume-control-interface/
title: Horrifying Volume Control Interface
url: https://willhbr.net/2024/09/02/horrifying-volume-control-interface/
feedurl: https://willhbr.net/feed.xml
content:
<p>I just bought a new (second-hand) TV—upgrading from my 42” 1080p second-hand TV to the latest 55” 4K second-hand TV<sup id="fnref:new-tv" role="doc-noteref"><a href="#fn:new-tv" class="footnote" rel="footnote">1</a></sup>—and I can’t get over the volume control. This is what it looks like:</p>

<p loading="lazy"><img src="/images/2024/tv-volume-control.webp" alt="The volume control" /></p>

<p class="caption">The TV was cheap because of the panel defect around the edge.</p>

<p>It appears on the left edge of the screen, and is vertically centred. The green bar shows the fraction of maximum volume, the output device is labelled, and the current volume is shown as a number between 0 and 100.</p>

<p>While writing this I am reminded that Windows 10 (and possibly 11 as well) <a href="https://devblogs.microsoft.com/oldnewthing/20170321-00/?p=95795">increases the volume in multiples of 2</a> when you use the volume keys, which would drive me up the wall because if your scale only needs 50 points of points of precision, why are you putting it on a scale with 100 points? It seems like you <em>can</em> set it to odd numbers if you use a control other than the volume keys. Either way I still prefer the non-numbered volume control on MacOS and iOS.</p>

<p>Anyway, back to my TV. As well as showing the current volume (in the photo above the current volume is <strong>9</strong>), there are numbers above and below the main volume box. These represent the next step up and down on the volume scale.</p>

<p>One might assume that when you pressed the volume up or down the numbers would animate up or down, as though you are rotating a physical wheel with numbers printed around the rim, viewed through a window that shows three digits, similar to the odometer on a car.</p>

<p>To my absolute horror, this is not the case. Upon pressing the volume buttons, the numbers instantly jump either up or down with absolutely no animation. The two additional digits don’t provide any kind of skeuomorphic representation of a volume control, instead they just serve as a reminder of the preceding and succeeding volume levels, in case you forgot that 10 comes after 9.</p>

<p>I understand the desire to have some kind of numbering system so that users can gauge the volume without having to first play audio. You want to be able to set the TV to an appropriate volume level for the surroundings before you play something so you don’t miss something or deafen your neighbours. On this TV I already know that having the volume between 10 and 20 is fine for my quiet apartment, but ultimately it’ll depend on what I’m playing.</p>

<p>If I were to redesign this, I would make the bar indicator much larger—perhaps reaching all the way across the bottom of the screen—and leave the number and output device in the bottom corner. Hopefully with a much wider bar indicator, the filled portion would be obvious enough that the numbers could be removed entirely.</p>

<p>Of course if I could only make one change, I would remove the preceding and succeeding numbers.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:new-tv" role="doc-endnote">
      <p>It’s a Samsung TU-8000, if you’re interested. It replaces some random Panasonic TV that I’ve had for years. <a href="#fnref:new-tv" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/09/02/horrifying-volume-control-interface/">Permalink</a>
&bull; September 2, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Horrifying%20Volume%20Control%20Interface" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/05/implicit-lifetimes-and-undroppable-types/
title: Implicit Lifetimes and Undroppable Types
url: https://willhbr.net/2024/09/05/implicit-lifetimes-and-undroppable-types/
feedurl: https://willhbr.net/feed.xml
content:
<p>Ok I need to put a really heavy caveat at the start here: I am not a Rust expert. I wouldn’t even call myself a Rust novice. I’m more of a Rust admirer. I’ve spent plenty of time trying to learn Rust and then just going back to <a href="/2023/06/24/why-crystal-is-the-best-language-ever/">Crystal</a> whenever I actually want to get something done. I don’t think this is a knock on Rust, it’s just got a very different set of priorities to Crystal. The biggest difference is that the Crystal standard library is batteries-included whereas in Rust you have to find a module for the anode and cathode separately. The furthest I’ve got into a Rust project was rewriting <a href="https://pod.willhbr.net">pod</a>. I got about 60% of the way through and then ran out of free time and enthusiasm.<sup id="fnref:cool-maths" role="doc-noteref"><a href="#fn:cool-maths" class="footnote" rel="footnote">1</a></sup></p>

<p>So if you <em>are</em> a Rust expert, either stop reading now or just read this with the mindset of when a small child is telling you about what they learnt in school today.</p>

<p>With that out of the way, instead of talking about Rust, let’s talk about Java. Without even realising it, you think about the lifetimes of variables implicitly as you’re writing a program. Consider this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">handleRequest</span><span class="o">(</span><span class="nc">Request</span> <span class="n">req</span><span class="o">,</span> <span class="nc">ResponseWriter</span> <span class="n">writer</span><span class="o">)</span> <span class="o">{</span>
  <span class="n">checkValid</span><span class="o">(</span><span class="n">req</span><span class="o">);</span>

  <span class="n">database</span><span class="o">.</span><span class="na">getUser</span><span class="o">(</span><span class="n">req</span><span class="o">.</span><span class="na">id</span><span class="o">);</span>
  <span class="o">...</span>
<span class="o">}</span>
</code></pre></div></div>

<p>Without looking at the implementation of <code class="language-plaintext highlighter-rouge">checkValid</code>, you can be pretty sure from context that it:</p>

<ul>
  <li>Will not hold onto a reference to the request</li>
  <li>Will not mutate the request</li>
  <li>Will throw an exception if the request is invalid</li>
</ul>

<p>None of that is enforced by the type system though, you have to just assume that whoever wrote <code class="language-plaintext highlighter-rouge">checkValid</code> wrote it in good faith and good taste. The difference in Rust is that you can enforce the first two invariants using the type system (Rust doesn’t have exceptions so you can’t do the last one, you’d have to design the API differently).</p>

<p>So in Java we could totally do this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">checkValid</span><span class="o">(</span><span class="nc">Request</span> <span class="n">req</span><span class="o">)</span> <span class="o">{</span>
  <span class="nc">GlobalCheckedRequestStore</span><span class="o">.</span><span class="na">addRequest</span><span class="o">(</span><span class="n">req</span><span class="o">);</span>
  <span class="n">req</span><span class="o">.</span><span class="na">id</span> <span class="o">+=</span> <span class="mi">4</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>

<p>And there’s nothing the type system can do prevent us from doing this. However in Rust, the <code class="language-plaintext highlighter-rouge">Request</code> would be <em>borrowed</em> by the <code class="language-plaintext highlighter-rouge">checkValid</code> function, and so you can’t give it to something else that will hold onto it for longer than the function. That would forbid is from passing it to our sneaky <code class="language-plaintext highlighter-rouge">GlobalCheckedRequestStore.addRequest</code>, since that will store it and keep it around after <code class="language-plaintext highlighter-rouge">checkValid</code> has returned.</p>

<p>As soon as you start looking, you’ll see these kind of things appear constantly. In Python you shouldn’t do this:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_file</span> <span class="o">=</span> <span class="nb">open</span><span class="p">(</span><span class="s">'/tmp/file'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span>
<span class="n">my_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'some text'</span><span class="p">)</span>
<span class="n">my_file</span><span class="p">.</span><span class="n">close</span><span class="p">()</span>
<span class="n">my_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'some more text'</span><span class="p">)</span>
</code></pre></div></div>

<p><em>Obviously</em> that won’t work, and since you’re really clever you can spot it. However when the program gets complicated, you’ll probably start to struggle. Python has a solution to this problem though: <code class="language-plaintext highlighter-rouge">with</code>. Just use <code class="language-plaintext highlighter-rouge">with</code> and you can never accidentally write to a closed file!</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'/tmp/file'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">my_file</span><span class="p">:</span>
  <span class="n">my_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'some text'</span><span class="p">)</span>
  <span class="n">write_more_text</span><span class="p">(</span><span class="n">my_file</span><span class="p">)</span>
</code></pre></div></div>

<p>Problem solved!</p>

<p>Wait but what if <code class="language-plaintext highlighter-rouge">write_more_text</code> does something that holds onto a reference to <code class="language-plaintext highlighter-rouge">my_file</code> after the function returns… <strong>damnit</strong> we’re doing implicit lifetimes again! Or what if we’re really silly and do:</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">my_file</span> <span class="o">=</span> <span class="bp">None</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="s">'/tmp/file'</span><span class="p">,</span> <span class="s">'w'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
  <span class="n">my_file</span> <span class="o">=</span> <span class="n">f</span>
<span class="n">my_file</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'some text'</span><span class="p">)</span>
</code></pre></div></div>

<p>This just appears over and over and over, especially when async work is involved. You’ll see a version of this problem in basically every server API (HTTP, TCP, WebSockets, etc), and File APIs. It’s particularly easy to do this when you’re working asynchronously:</p>

<div class="language-crystal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">require</span> <span class="s2">"http/server"</span>

<span class="n">server</span> <span class="o">=</span> <span class="no">HTTP</span><span class="o">::</span><span class="no">Server</span><span class="p">.</span><span class="nf">new</span> <span class="k">do</span> <span class="o">|</span><span class="n">context</span><span class="o">|</span>
  <span class="n">context</span><span class="p">.</span><span class="nf">response</span><span class="p">.</span><span class="nf">content_type</span> <span class="o">=</span> <span class="s2">"text/plain"</span>
  <span class="n">spawn</span> <span class="k">do</span>
    <span class="nb">sleep</span> <span class="mi">10</span><span class="p">.</span><span class="nf">seconds</span>
    <span class="n">context</span><span class="p">.</span><span class="nf">response</span><span class="p">.</span><span class="nf">print</span> <span class="s2">"I'm a slow response"</span>
  <span class="k">end</span>
  <span class="n">context</span><span class="p">.</span><span class="nf">response</span><span class="p">.</span><span class="nf">print</span> <span class="s2">"Hello world, got </span><span class="si">#{</span><span class="n">context</span><span class="p">.</span><span class="nf">request</span><span class="p">.</span><span class="nf">path</span><span class="si">}</span><span class="s2">!"</span>
<span class="k">end</span>
</code></pre></div></div>

<p>The response will have already been sent to the client and <code class="language-plaintext highlighter-rouge">context.response</code> closed by the time we try to write “I’m a slow response”.</p>

<p>Rust solves this problem with strict lifetime checks. This isn’t a problem in Rust because the compiler will stop you from setting <code class="language-plaintext highlighter-rouge">my_file</code> to <code class="language-plaintext highlighter-rouge">f</code> or sharing access to the <code class="language-plaintext highlighter-rouge">context</code> from two async contexts.</p>

<p>What Rust doesn’t do is prevent you from forgetting to close a file. The <a href="https://doc.rust-lang.org/std/fs/struct.File.html">actual <code class="language-plaintext highlighter-rouge">File</code> implementation in Rust says</a>:</p>

<blockquote>
  <p>Files are automatically closed when they go out of scope. Errors detected on closing are ignored by the implementation of <code class="language-plaintext highlighter-rouge">Drop</code>. Use the method <code class="language-plaintext highlighter-rouge">sync_all</code> if these errors must be manually handled.</p>
</blockquote>

<p>The type system can’t save you from this one. As soon as a file goes out of scope, it is closed and any error is silently ignored.</p>

<p>What actually started this whole train of thoughts was reading <a href="https://without.boats/blog/ownership/">this blog post</a> about ownership in Rust. I’m not going to pretend I understand even some of the type theory, but this one section stood out to me:<sup id="fnref:is-this-the-point" role="doc-noteref"><a href="#fn:is-this-the-point" class="footnote" rel="footnote">2</a></sup></p>

<blockquote>
  <p>Let’s take a very simple example, a transaction which must be committed or aborted:</p>

  <div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span> <span class="n">Transaction</span> <span class="p">{</span>
    <span class="k">pub</span> <span class="k">fn</span> <span class="nf">commit</span><span class="p">(</span><span class="k">self</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="n">TransactionResult</span><span class="p">;</span>

    <span class="k">pub</span> <span class="k">fn</span> <span class="nf">abort</span><span class="p">(</span><span class="k">self</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div>  </div>

  <p>The problem with a session type like this is that the user can just drop the transaction. You might say in this case that you’ll write a destructor and abort the transaction on drop (I’ll return to that idea in the next section), but you can surely imagine more complex examples in which there isn’t an obvious state transition to perform on drop.</p>
</blockquote>

<p>This made me see lifetimes in a new way that I hadn’t considered before: you can use a lifetime to <em>prevent</em> an object from being discarded when it’s in a particular state.</p>

<p>Currently every type in Rust can be implicitly dropped when it reaches the end of the lexical scope it is bound in. The type can define what happens at this point by overriding the destructor, which is how things like <a href="https://doc.rust-lang.org/std/sync/struct.Arc.html"><code class="language-plaintext highlighter-rouge">Arc</code></a> work. What the type <em>can’t</em> do is say “I cannot be discarded”.</p>

<p>If we could do this, it would look something like this:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
  <span class="k">let</span> <span class="n">undroppable</span> <span class="o">=</span> <span class="nn">Undroppable</span><span class="p">::</span><span class="nf">new</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>

<p>That would of course fail to compile:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>cargo run
<span class="go">   Compiling
</span><span class="gp"> --&gt;</span><span class="w"> </span>src/main.rs:2:7
<span class="go">  |
</span><span class="gp">2 |     let undroppable = Undroppable::new();</span><span class="w">
</span><span class="go">  |         ^ help: This type does not implement std::ops::Drop
  |
</span></code></pre></div></div>

<p class="caption">This is not real, it’s just an illustration.</p>

<p>A type that you cannot instantiate because it’ll stop your project from compiling doesn’t seem that useful, but it’s like the <a href="https://doc.rust-lang.org/std/primitive.never.html"><code class="language-plaintext highlighter-rouge">never</code> type</a> (that also appears in <a href="https://developer.apple.com/documentation/swift/never">Swift</a> and <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-nothing.html">Kotlin</a>).</p>

<p>We can apply this to our file API. Instead of just having a <code class="language-plaintext highlighter-rouge">File</code> type, we’ll have a <code class="language-plaintext highlighter-rouge">Path</code> which represents the location of the file before we open it, an <code class="language-plaintext highlighter-rouge">OpenFile</code> which we can read and write from, and a <code class="language-plaintext highlighter-rouge">ClosedFile</code>.</p>

<p>The trick is that <code class="language-plaintext highlighter-rouge">OpenFile</code> cannot be dropped, which means the compiler will force us to call <code class="language-plaintext highlighter-rouge">.close()</code> on the <code class="language-plaintext highlighter-rouge">OpenFile</code>.</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">main</span><span class="p">()</span> <span class="p">{</span>
  <span class="k">let</span> <span class="n">path</span> <span class="o">=</span> <span class="nn">Path</span><span class="p">::</span><span class="nf">new</span><span class="p">(</span><span class="s">"/tmp/file"</span><span class="p">);</span>
  <span class="k">let</span> <span class="n">file</span><span class="p">:</span> <span class="n">OpenFile</span> <span class="o">=</span> <span class="n">path</span><span class="nf">.open</span><span class="p">();</span>
  <span class="nf">write_contents_to_file</span><span class="p">(</span><span class="o">&amp;</span><span class="n">file</span><span class="p">);</span>
  <span class="c1">// This turns the OpenFile into a ClosedFile</span>
  <span class="n">file</span><span class="nf">.close</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span> <span class="c1">// The ClosedFile gets dropped here</span>
</code></pre></div></div>

<p>Of course there has to be an internal way for the <code class="language-plaintext highlighter-rouge">OpenFile</code> to be translated into a <code class="language-plaintext highlighter-rouge">ClosedFile</code> within the <code class="language-plaintext highlighter-rouge">.close()</code> method—for example a private destructor that can only be called from methods in that type. That <code class="language-plaintext highlighter-rouge">.close()</code> method signature would look something like:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span> <span class="n">OpenFile</span> <span class="p">{</span>
  <span class="k">fn</span> <span class="nf">close</span><span class="p">(</span><span class="k">self</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="nb">Result</span><span class="o">&lt;</span><span class="n">ClosedFile</span><span class="p">,</span> <span class="nb">Err</span><span class="o">&gt;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>It takes ownership of the <code class="language-plaintext highlighter-rouge">OpenFile</code>, closes the system resource, uses the internal destructor to drop the <code class="language-plaintext highlighter-rouge">OpenFile</code> and returns a <code class="language-plaintext highlighter-rouge">ClosedFile</code> that represents the state of the file on close. This forces the caller to close the file and handle the errors.</p>

<p>This same pattern could apply in so many situations where some explicit action <strong>must</strong> be done before an object is discarded. Make sure that a <code class="language-plaintext highlighter-rouge">Response</code> object has been written to, require that a <code class="language-plaintext highlighter-rouge">Thread</code> gets <code class="language-plaintext highlighter-rouge">join</code>-ed, make sure a <code class="language-plaintext highlighter-rouge">Channel</code> is closed, that every <code class="language-plaintext highlighter-rouge">async</code> is <code class="language-plaintext highlighter-rouge">await</code>-ed, or make sure a database transaction either committed or rolled back. The list goes on.</p>

<p>This could even work around issues like <a href="https://without.boats/blog/asynchronous-clean-up/">async cleanup</a>. A type that needs to do async work (or error-prone work) to clean itself up just has to make itself un-droppable and move the cleanup into a method that transitions to a type that can be trivially dropped.</p>

<p>I remember in university being <em>explicitly</em> told that objected-oriented “becomes-a” is a terrible anti-pattern, which I suppose is still true if you’re talking about the Java type system, which can’t even express nullability properly.</p>

<p>Last year <a href="/2023/10/31/how-i-learned-to-stop-worrying-and-love-concurrency/">I wrote a long ramble about concurrency</a>, where I mostly glossed over the whole concept of lifetimes, but wrote:</p>

<blockquote>
  <p>What I’d like is—somehow—for concurrent data access to be verified as easily as types are checked in Crystal. I get most of the benefits of static typing and dynamic typing by using Crystal’s type inference, can the lifetimes of variables be inferred in a similar way? I think this would be a very hard problem, and probably only practical if the general population of developers was already used to adding lifetime annotations—like they are with types—so you could just require fewer of them.</p>
</blockquote>

<p>Much like nullability in Java, adding any kind of lifetimes into an existing type system would be incredibly difficult, as you don’t know the lifetime of any unannotated type, so all bets are immediately off. In Java you <code class="language-plaintext highlighter-rouge">@NotNull</code> as much as you want, but someone can totally just throw a <code class="language-plaintext highlighter-rouge">null</code> your way and you have to deal with that at runtime.</p>

<p>You’d maybe end up in a similar situation to C++, where you can use <code class="language-plaintext highlighter-rouge">std::unique_ptr&lt;&gt;</code> and references as much as you want, but eventually a function is going to expect a raw pointer, and you’re going to have to work out whether that pointer will move, share, or borrow ownership.</p>

<p>I think what I’d really like is a language with a batteries-included standard library, an ownership model like Rust’s, but also a shorter syntax for creating reference-counted types.<sup id="fnref:pony" role="doc-noteref"><a href="#fn:pony" class="footnote" rel="footnote">3</a></sup></p>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:cool-maths" role="doc-endnote">
      <p>I did do a cool thing where all the operations that you can do on a container were reversible, so every operation would return the operation that reversed it, which would theoretically make automatic rollbacks really easy to implement. <a href="#fnref:cool-maths" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:is-this-the-point" role="doc-endnote">
      <p>I don’t know if this is really the main point of that post, but this is what I got out of it. <a href="#fnref:is-this-the-point" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:pony" role="doc-endnote">
      <p>Maybe that’s <a href="https://www.ponylang.io">Pony</a>? <a href="#fnref:pony" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/09/05/implicit-lifetimes-and-undroppable-types/">Permalink</a>
&bull; September 5, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Implicit%20Lifetimes%20and%20Undroppable%20Types" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/22/grabbing-github-release-assets-using-jq/
title: Grabbing GitHub Release Assets Using JQ
url: https://willhbr.net/2024/09/22/grabbing-github-release-assets-using-jq/
feedurl: https://willhbr.net/feed.xml
content:
<p><code class="language-plaintext highlighter-rouge">JJ</code> has become enough of a must-have tool that I wanted to have my dotfiles install it automatically. In <a href="https://github.com/willhbr/dotfiles/blob/main/install.sh"><code class="language-plaintext highlighter-rouge">install.sh</code></a> I install a handful of utilities via <code class="language-plaintext highlighter-rouge">apt</code>, but JJ does not have a Debian/Ubuntu package so I have improvised something. It’s a little bit of a hack.</p>

<p>You can download prebuilt JJ binaries for most distributions/architectures from the GitHub releases page, which is how I’d typically install it. I used the GitHub API and <a href="https://github.com/jqlang/jq"><code class="language-plaintext highlighter-rouge">jq</code></a> to parse the list of releases and get the download URL for the right architecture, then download, untar, and install the binary.</p>

<p>The API endpoint is:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>https://api.github.com/repos/martinvonz/jj/releases
</code></pre></div></div>

<p>The important part of the response looks like this:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"assets"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
    </span><span class="p">{</span><span class="w">
      </span><span class="nl">"url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://api.github.com/repos/martinvonz/jj/releases/assets/190411619"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"id"</span><span class="p">:</span><span class="w"> </span><span class="mi">190411619</span><span class="p">,</span><span class="w">
      </span><span class="nl">"node_id"</span><span class="p">:</span><span class="w"> </span><span class="s2">"RA_kwDOEzi53M4LWXNj"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"jj-v0.21.0-aarch64-apple-darwin.tar.gz"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"label"</span><span class="p">:</span><span class="w"> </span><span class="s2">""</span><span class="p">,</span><span class="w">
      </span><span class="nl">"uploader"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="err">...</span><span class="w"> </span><span class="p">},</span><span class="w">
      </span><span class="nl">"content_type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"application/octet-stream"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"state"</span><span class="p">:</span><span class="w"> </span><span class="s2">"uploaded"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"size"</span><span class="p">:</span><span class="w"> </span><span class="mi">8907534</span><span class="p">,</span><span class="w">
      </span><span class="nl">"download_count"</span><span class="p">:</span><span class="w"> </span><span class="mi">42</span><span class="p">,</span><span class="w">
      </span><span class="nl">"created_at"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2024-09-04T17:23:32Z"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"updated_at"</span><span class="p">:</span><span class="w"> </span><span class="s2">"2024-09-04T17:23:32Z"</span><span class="p">,</span><span class="w">
      </span><span class="nl">"browser_download_url"</span><span class="p">:</span><span class="w"> </span><span class="s2">"https://github.com/martinvonz/jj/releases/download/v0.21.0/jj-v0.21.0-aarch64-apple-darwin.tar.gz"</span><span class="w">
    </span><span class="p">},</span><span class="w">
    </span><span class="err">...</span><span class="w">
  </span><span class="p">],</span><span class="w">
  </span><span class="err">...</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<p>I just need to grab the latest release, and from the release get the asset that matches the architecture of the current OS. After some <code class="language-plaintext highlighter-rouge">jq</code> trial and error, I ended up with:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.[0].assets
| .[]
| select(.name | contains("$arch"))
| select(.name | contains("$os"))
| .browser_download_url
</code></pre></div></div>

<p>Just set <code class="language-plaintext highlighter-rouge">$arch</code> and <code class="language-plaintext highlighter-rouge">$os</code> to the right values (“x64_64” and “linux” will probably do the trick) and you’ve got a script that can download the latest release on demand. I <a href="https://github.com/willhbr/dotfiles/blob/8602f53addbb51e77a27897fef1eba074a826f08/install.sh#L15-L33">popped this into my install script</a> so I can keep my install up-to-date and setup new machines quickly.</p>


<a href="https://willhbr.net/2024/09/22/grabbing-github-release-assets-using-jq/">Permalink</a>
&bull; September 22, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Grabbing%20GitHub%20Release%20Assets%20Using%20JQ" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/26/copy-the-output-of-the-last-command-using-tmux/
title: Copy the Output of the Last Command Using tmux
url: https://willhbr.net/2024/09/26/copy-the-output-of-the-last-command-using-tmux/
feedurl: https://willhbr.net/feed.xml
content:
<p>I just came across <a href="https://ianthehenry.com/posts/tmux-copy-last-command/">this post</a> while doing some background reading for another project. I know people that do this kind of thing and it always seemed like a bit of a hack, but putting the non-breaking space in the prompt just pushes this over the line of being probably-robust enough for me to use it.</p>

<p>The basic idea is that you put a non-breaking space (that looks just like a normal space) in your prompt, which makes it easier to automatically search back through your history to find where the previous prompt was. For someone like me with a simple prompt—just <code class="language-plaintext highlighter-rouge">||&gt;</code>—without the non-breaking space it wouldn’t be that uncommon to have those three characters appear in the output of a command.</p>

<p>Check out <a href="https://ianthehenry.com/posts/tmux-copy-last-command/">the post</a> for a full explanation, but the gist is that you use the tmux copy-mode features to search back and create a selection that starts at the previous prompt and ends at the current prompt.</p>

<p>The thing that caught me out that was that to put a unicode escape sequence in your prompt, you need to prefix the string with <code class="language-plaintext highlighter-rouge">$</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">PROMPT</span><span class="o">=</span><span class="s1">$'||&gt;</span><span class="se">\U</span><span class="s1">00A0'</span>
</code></pre></div></div>

<p>Without that you just get the literal string <code class="language-plaintext highlighter-rouge">||&gt;\U00A0</code> as your prompt, which is not what we want.</p>

<p>Hopefully this will be useful in cases where I run a long-running command, and realise I want to do some transformation on the text <em>after</em> seeing the output. Previously I’d either manually copy the output, or re-run the command and pipe the output to a file.</p>

<p>As one final tip, don’t forget to set your <code class="language-plaintext highlighter-rouge">history-limit</code> to some large number so the command output doesn’t fall off the top of the scrollback!</p>


<a href="https://willhbr.net/2024/09/26/copy-the-output-of-the-last-command-using-tmux/">Permalink</a>
&bull; September 26, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Copy%20the%20Output%20of%20the%20Last%20Command%20Using%20tmux" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/27/mockito-type-checking-and-the-perils-of-dsl-design/
title: Mockito, Type Checking, and the Perils of DSL Design
url: https://willhbr.net/2024/09/27/mockito-type-checking-and-the-perils-of-dsl-design/
feedurl: https://willhbr.net/feed.xml
content:
<p>So this one takes a bit of explaining but we’ll get there in the end. The Mockito <code class="language-plaintext highlighter-rouge">when()</code> function has a really subtle trap that can lead to unexpected behaviour. For the longest time I thought it was because the authors wanted a slightly cleaner API, but after some more thought I realised that it’s almost unavoidable. We’ll get to that later, first let’s get into the actual API.</p>

<p>The basic API for Mockito is super simple, you create a mock with <code class="language-plaintext highlighter-rouge">mock(MyClass.class)</code> and then you use a DSL to tell the mock how to act under certain conditions:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThingSizer</span> <span class="n">myMock</span> <span class="o">=</span> <span class="n">mock</span><span class="o">(</span><span class="nc">ThingSizer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="n">when</span><span class="o">(</span><span class="n">myMock</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>

<span class="nc">System</span><span class="o">.</span><span class="na">out</span><span class="o">.</span><span class="na">println</span><span class="o">(</span><span class="n">myMock</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">());</span>
<span class="c1">// =&gt; ThingSize.LARGE</span>
</code></pre></div></div>

<p>It’s that <code class="language-plaintext highlighter-rouge">when()</code> call that throws things off. It might look like you need to put the function call inside the brackets, but that’s not how Java works. The only thing that <code class="language-plaintext highlighter-rouge">when()</code> receives is the result of the function call, allowing you to do this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThingSize</span> <span class="n">size</span> <span class="o">=</span> <span class="n">myMock</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
<span class="n">when</span><span class="o">(</span><span class="n">size</span><span class="o">).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>
</code></pre></div></div>

<p>The way this actually works is that every method call on a mock object is added to a global list, and <code class="language-plaintext highlighter-rouge">when()</code> just grabs the latest mock call (in this case that’s <code class="language-plaintext highlighter-rouge">getThingSize()</code>) and attaches the provided return value for the next time that method gets called.</p>

<p>With that in mind, we don’t even have to pass the result of the mock method call into <code class="language-plaintext highlighter-rouge">when()</code>, we just have to pass an object that matches the return type:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">myMock</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
<span class="n">when</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">TINY</span><span class="o">).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>

<span class="n">myMock</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
<span class="c1">// =&gt; ThingSize.LARGE</span>
</code></pre></div></div>

<p>In this case it’s pretty obvious that you’re doing something wrong, but it’s not hard to run into cases where the failure is much more subtle. Since that list of method invocations is global, it doesn’t matter where it’s called from, just as long as it’s called before <code class="language-plaintext highlighter-rouge">when()</code>.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThingSize</span> <span class="nf">retrieveThingSizeFrom</span><span class="o">(</span><span class="nc">ThingSizer</span> <span class="n">sizer</span><span class="o">)</span> <span class="o">{</span>
  <span class="k">return</span> <span class="n">sizer</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
<span class="o">}</span>

<span class="n">when</span><span class="o">(</span><span class="n">retrieveThingSizeFrom</span><span class="o">(</span><span class="n">myMock</span><span class="o">)).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>
</code></pre></div></div>

<p>To a novice mockito-er, it looks like we’re mocking the <code class="language-plaintext highlighter-rouge">retrieveThingSizeFrom()</code> method, but we’re actually setting the mock behaviour on code that’s <em>inside</em> that method and it just happens to work because we call <code class="language-plaintext highlighter-rouge">getThingSize()</code> within <code class="language-plaintext highlighter-rouge">retrieveThingSizeFrom()</code>. Obviously a terrible thing to do would be:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThingSize</span> <span class="nf">getRandomThingSize</span><span class="o">(</span><span class="nc">ThingSizer</span> <span class="n">a</span><span class="o">,</span> <span class="nc">ThingSizer</span> <span class="n">b</span><span class="o">)</span> <span class="o">{</span>
  <span class="k">if</span> <span class="o">(</span><span class="nc">Random</span><span class="o">.</span><span class="na">rand</span><span class="o">()</span> <span class="o">&lt;</span> <span class="mf">0.5</span><span class="o">)</span> <span class="o">{</span>
    <span class="k">return</span> <span class="n">a</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
  <span class="o">}</span> <span class="k">else</span> <span class="o">{</span>
    <span class="k">return</span> <span class="n">b</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">();</span>
  <span class="o">}</span>
<span class="o">}</span>

<span class="nc">ThingSizer</span> <span class="n">mockA</span> <span class="o">=</span> <span class="n">mock</span><span class="o">(</span><span class="nc">ThingSizer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="nc">ThingSizer</span> <span class="n">mockB</span> <span class="o">=</span> <span class="n">mock</span><span class="o">(</span><span class="nc">ThingSizer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="n">when</span><span class="o">(</span><span class="n">getRandomThingSize</span><span class="o">(</span><span class="n">mockA</span><span class="o">,</span> <span class="n">mockB</span><span class="o">)).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>
</code></pre></div></div>

<p>There’s a 50% chance that <code class="language-plaintext highlighter-rouge">mockA</code> will be configured, and a 50% chance that <code class="language-plaintext highlighter-rouge">mockB</code> will be configured. How do you end up with code like this in reality? It’s easy—you just forget that an object isn’t a mock.</p>

<p>Let’s assume that this is setup in some test helper somewhere. <code class="language-plaintext highlighter-rouge">ProxySizer</code> wraps our <code class="language-plaintext highlighter-rouge">ThingSizer</code> and maybe does some additional checks or whatever.</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">ThingSizer</span> <span class="n">mock</span> <span class="o">=</span> <span class="n">mock</span><span class="o">(</span><span class="nc">ThingSizer</span><span class="o">.</span><span class="na">class</span><span class="o">);</span>
<span class="nc">ProxySizer</span> <span class="n">sizer</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">ProxySizer</span><span class="o">(</span><span class="n">mock</span><span class="o">);</span>
</code></pre></div></div>

<p>Later in our test we want to change what our proxy returns, so we do this:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">sizer</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>
</code></pre></div></div>

<p>And of course it works, despite <code class="language-plaintext highlighter-rouge">sizer</code> not being a mock. Since it proxies to an object that <em>is</em> a mock, the only thing that matters is that the return types match up and <code class="language-plaintext highlighter-rouge">thenReturn</code> is able to grab the mock invocation from the global list. This becomes an even subtler trap to fall into when the return type is a primitive or another commonly-used type.</p>

<p>I’ve known about these failures for ages and I always thought it was just a poor API design choice that Mockito was stuck with. They decided on the <code class="language-plaintext highlighter-rouge">when</code> “DSL” syntax and then later realised that it had some pitfalls that users could fall into. My assumption was that you could just do:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">sizer</span><span class="o">).</span><span class="na">getThingSize</span><span class="o">().</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">MEDIUM</span><span class="o">);</span>
</code></pre></div></div>

<p>The <code class="language-plaintext highlighter-rouge">when()</code> call takes an object, validates that it’s a mock, and returns a special type that lets you configure the mocking without relying on a global list of objects. I then actually sat down and thought about this some more—what would the type of that object be? <a href="https://site.mockito.org/javadoc/current/org/mockito/Mockito.html#when(T)">The signature for <code class="language-plaintext highlighter-rouge">when()</code> is</a>:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="nc">OngoingStubbing</span><span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="nf">when</span><span class="o">(</span><span class="no">T</span> <span class="n">methodCall</span><span class="o">)</span>
</code></pre></div></div>

<p>It takes in the returned value (labelled somewhat dishonestly as <code class="language-plaintext highlighter-rouge">methodCall</code>) and uses generic type inference to return an <code class="language-plaintext highlighter-rouge">OngoingStubbing&lt;T&gt;</code> where <code class="language-plaintext highlighter-rouge">T</code> is the return type. This means that <a href="https://site.mockito.org/javadoc/current/org/mockito/stubbing/OngoingStubbing.html#thenReturn(T)"><code class="language-plaintext highlighter-rouge">.thenReturn</code></a> can have a generic constraint such that any argument to it must be of type <code class="language-plaintext highlighter-rouge">T</code>. This is the only way to get some type safety when setting up the mock. If <code class="language-plaintext highlighter-rouge">when()</code> instead took the mock object as an argument instead of the return value, you wouldn’t be able to return your <code class="language-plaintext highlighter-rouge">OngoingStubbing</code> object to be able to call <code class="language-plaintext highlighter-rouge">thenReturn</code>. The signature of <code class="language-plaintext highlighter-rouge">when()</code> would look like:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="nc">MockConfigurator</span><span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="nf">when</span><span class="o">(</span><span class="no">T</span> <span class="n">mock</span><span class="o">)</span>
</code></pre></div></div>

<p>What methods would we add on that <code class="language-plaintext highlighter-rouge">MockConfigurator</code>? It’s supposed to allow us to set the return values for different method calls, but we don’t know the methods that are on <code class="language-plaintext highlighter-rouge">T</code> to be able to expose a compile-time DSL on the <code class="language-plaintext highlighter-rouge">MockConfigurator</code> object, we’d just get an error:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">sizer</span><span class="o">).</span><span class="na">getThingSize</span><span class="o">().</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">MEDIUM</span><span class="o">);</span>
           <span class="o">^</span> <span class="nc">Unknown</span> <span class="n">method</span> <span class="nf">getThingSize</span><span class="o">()</span> <span class="k">for</span> <span class="nc">MockConfigurator</span><span class="o">&lt;</span><span class="nc">ThingSizer</span><span class="o">&gt;</span>
</code></pre></div></div>

<p>Of course we could just use strings to match the method name at runtime (that’s basically what’s going to happen under the hood anyway):</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">sizer</span><span class="o">).</span><span class="na">onCall</span><span class="o">(</span><span class="s">"getThingSize"</span><span class="o">).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">MEDIUM</span><span class="o">);</span>
</code></pre></div></div>

<p>Although that means we just lost type checking on the method name <em>and</em> the return type. We got rid of the global list of mocked method calls, at the cost of completely losing our type checking within a single method call. There’s no nice way in Java for us to communicate the type information using generics.</p>

<p>You could make some steps forward by passing <em>both</em> the mock and the method in the form of a lambda into the <code class="language-plaintext highlighter-rouge">when</code> call:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="kd">static</span> <span class="o">&lt;</span><span class="no">M</span><span class="o">,</span> <span class="no">T</span><span class="o">&gt;</span> <span class="nc">OngoingStubbing</span><span class="o">&lt;</span><span class="no">T</span><span class="o">&gt;</span> <span class="nf">when</span><span class="o">(</span><span class="no">M</span> <span class="n">mock</span><span class="o">,</span> <span class="nc">Function</span><span class="o">&lt;</span><span class="no">M</span><span class="o">,</span> <span class="no">T</span><span class="o">&gt;</span> <span class="n">method</span><span class="o">)</span>
</code></pre></div></div>

<p>With access to the mock that we’re operating on, the <code class="language-plaintext highlighter-rouge">when()</code> method doesn’t have to consult a global list of mock method invocations, it can interrogate the mock directly. By passing in a lambda that calls the method we want to mock, <code class="language-plaintext highlighter-rouge">when()</code> can interrogate the mock immediately after running the lambda. The downside is that this approach is much more verbose:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">mockSizer</span><span class="o">,</span> <span class="n">m</span> <span class="o">-&gt;</span> <span class="n">m</span><span class="o">.</span><span class="na">getThingSize</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">);</span>
</code></pre></div></div>

<p>Of course there’s nothing stopping you running into all the same problems with proxy objects and suchlike as before, but at least now <code class="language-plaintext highlighter-rouge">when()</code> can check that the first argument passed is a mock, and that the lambda calls exactly one method on that mock.</p>

<p>This is a great example of where allowing succinct syntaxes can allow for better APIs and better code. In Swift with a <a href="https://developer.apple.com/documentation/swift/keypath"><code class="language-plaintext highlighter-rouge">KeyPath</code></a> or an <code class="language-plaintext highlighter-rouge">@autoclosure</code> you could reduce the boilerplate on the <code class="language-plaintext highlighter-rouge">when()</code> call:</p>

<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// With a KeyPath</span>
<span class="nf">when</span><span class="p">(</span><span class="n">mockSizer</span><span class="p">,</span> <span class="p">\</span><span class="o">.</span><span class="n">thingSize</span><span class="p">)</span><span class="o">.</span><span class="nf">thenReturn</span><span class="p">(</span><span class="o">.</span><span class="n">large</span><span class="p">)</span>
<span class="c1">// With an @autoclosure</span>
<span class="nf">when</span><span class="p">(</span><span class="n">mockSizer</span><span class="p">,</span> <span class="n">mockSizer</span><span class="o">.</span><span class="nf">getThingSize</span><span class="p">())</span><span class="o">.</span><span class="nf">thenReturn</span><span class="p">(</span><span class="o">.</span><span class="n">large</span><span class="p">)</span>
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">KeyPath</code> can only reference a property, so you’d have to allow both of these approaches, and there’s always the possibility that someone passes two mocks in, which you’d have to handle:</p>

<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">when</span><span class="p">(</span><span class="n">mockA</span><span class="p">,</span> <span class="n">mockB</span><span class="o">.</span><span class="nf">getThingSize</span><span class="p">())</span><span class="o">.</span><span class="nf">thenReturn</span><span class="p">(</span><span class="o">.</span><span class="n">medium</span><span class="p">)</span>
</code></pre></div></div>

<p>Of course, if you were doing this in Swift you’d want to use a macro that can actually introspect on the method call passed to <code class="language-plaintext highlighter-rouge">when()</code> (I’d give you an example but I don’t know anything about Swift macros).</p>

<p>What have I learnt? DSL design in languages like Java with fewer bells and whistles is really tricky, and there’s probably a good reason that an API is designed a particular way. If I find you setting mock behaviour implicitly via a method call on a real object that proxies to a mock, you can expect a slightly passive-aggressive code review coming your way.</p>


<a href="https://willhbr.net/2024/09/27/mockito-type-checking-and-the-perils-of-dsl-design/">Permalink</a>
&bull; September 27, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Mockito,%20Type%20Checking,%20and%20the%20Perils%20of%20DSL%20Design" target=_blank>email</a>.


guid: https://willhbr.net/2024/09/28/when-does-an-api-become-a-dsl/
title: When Does an API Become a DSL?
url: https://willhbr.net/2024/09/28/when-does-an-api-become-a-dsl/
feedurl: https://willhbr.net/feed.xml
content:
<p>In response to my <a href="/2024/09/27/mockito-type-checking-and-the-perils-of-dsl-design/">previous post</a> I got asked when something is a DSL versus just being an API. Of course if you’ve defined a separate grammar and written a parser for that grammar and use that to solve a problem in your specific domain, then you’ve got a DSL. But a lot of the time, a DSL is directly embedded in another language—like how Mockito is just Java—and so the distinction between when you’re just using the API of a library and when that library becomes a DSL is a little blurry.</p>

<p>My immediate response was along the lines of when the API defines its own set of rules and semantics, then you’ve got a DSL. This doesn’t really hold up as plenty of APIs have rules about what you can do when, <a href="/2024/09/05/implicit-lifetimes-and-undroppable-types/">as I’ve discussed before</a>. It really boils down to <a href="https://en.wikipedia.org/wiki/I_know_it_when_I_see_it">“I know it when I see it”</a>, which isn’t a satisfying answer. Obviously if you’re defining your own grammar, you’ve got a DSL, but when does your API in an existing language become a DSL?</p>

<p>If we take Mockito as an example to start with, the API is clearly designed to emulate how you would write out the behaviour in English:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">when</span><span class="o">(</span><span class="n">thingSizer</span><span class="o">.</span><span class="na">getSize</span><span class="o">()).</span><span class="na">thenReturn</span><span class="o">(</span><span class="nc">ThingSize</span><span class="o">.</span><span class="na">LARGE</span><span class="o">)</span>
</code></pre></div></div>

<p>You would explain this to someone as “when I call <code class="language-plaintext highlighter-rouge">thingSizer.getSize()</code> then it should return <code class="language-plaintext highlighter-rouge">LARGE</code>”. What makes me think of this as a DSL is the fact that it’s adding a “feature” that Java doesn’t have—the ability to pass a method call as an argument to a function. Of course as <a href="/2024/09/27/mockito-type-checking-and-the-perils-of-dsl-design/">I previously mentioned</a> this isn’t actually possible, so it’s doing some behind-the-scenes magic to make it <em>appear</em> that the argument to <code class="language-plaintext highlighter-rouge">when()</code> is the actual invocation on the <code class="language-plaintext highlighter-rouge">thingSizer</code> mock, not the result of that method call.</p>

<p>You can imagine writing a code linter that ensured that the argument passed to <code class="language-plaintext highlighter-rouge">when()</code> is always an invocation on a mock, and not some other value. To me this is what makes it a DSL: it’s got its own semantics that are separate from the typical semantics of Java code, in order to provide a particular way of writing code.</p>

<p>Staying within the JVM, Kotlin has some features that make writing DSLs much easier. <a href="https://kotlinlang.org/docs/type-safe-builders.html">Receiver blocks</a> allow for altering the typical scoping rules that determine how method and property names are resolved. The documentation uses an HTML-generator DSL as an example, which is something that <a href="https://github.com/willhbr/status_page/blob/main/src/status_page/html_builder.cr">I’ve also built in Crystal</a>.</p>

<p>You can see how the change in scoping rules breaks how you might refactor the code, for example if we want to avoid repeating the definition of the button:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">html</span> <span class="p">{</span>
  <span class="nf">body</span> <span class="p">{</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is one paragraph"</span>
      <span class="nf">button</span> <span class="p">{</span> <span class="p">+</span><span class="s">"Copy"</span> <span class="p">}</span>
    <span class="p">}</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is another paragraph"</span>
      <span class="nf">button</span> <span class="p">{</span> <span class="p">+</span><span class="s">"Copy"</span> <span class="p">}</span>
    <span class="p">}</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>One might assume that you could extract the <code class="language-plaintext highlighter-rouge">button { ... }</code> block into a variable to avoid re-defining it:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">copyButton</span> <span class="p">=</span> <span class="nf">button</span> <span class="p">{</span> <span class="p">+</span><span class="s">"Copy"</span> <span class="p">}</span>
<span class="nf">html</span> <span class="p">{</span>
  <span class="nf">body</span> <span class="p">{</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is one paragraph"</span>
      <span class="n">copyButton</span>
    <span class="p">}</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is another paragraph"</span>
      <span class="n">copyButton</span>
    <span class="p">}</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>But of course if you’re familiar with Kotlin’s receiver blocks you know that the <code class="language-plaintext highlighter-rouge">button</code> method is actually probably defined on the receiver type for the <code class="language-plaintext highlighter-rouge">body</code> method, so we can’t call it from outside the block passed to <code class="language-plaintext highlighter-rouge">body</code>. Of course if you know the rules, you can refactor your code:</p>

<div class="language-kotlin highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">val</span> <span class="py">addCopyBody</span> <span class="p">=</span> <span class="p">{</span> <span class="n">body</span> <span class="p">-&gt;</span>
  <span class="n">body</span><span class="p">.</span><span class="nf">button</span> <span class="p">{</span> <span class="p">+</span><span class="s">"Copy"</span> <span class="p">}</span>
<span class="p">}</span>

<span class="nf">html</span> <span class="p">{</span>
  <span class="nf">body</span> <span class="p">{</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is one paragraph"</span>
      <span class="nf">addCopyButton</span><span class="p">(</span><span class="k">this</span><span class="p">)</span>
    <span class="p">}</span>
    <span class="nf">p</span> <span class="p">{</span>
      <span class="p">+</span><span class="s">"This is another paragraph"</span>
      <span class="nf">addCopyButton</span><span class="p">(</span><span class="k">this</span><span class="p">)</span>
    <span class="p">}</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Or if you’re really clever, you know that you can define an extension function on the <code class="language-plaintext highlighter-rouge">body</code> type that defines the copy button.</p>

<p>Contrast this with a more traditional API for building HTML using type-safe objects:</p>

<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">HTMLBuilder</span> <span class="n">html</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">HTMLBuilder</span><span class="o">();</span>

<span class="n">html</span><span class="o">.</span><span class="na">body</span><span class="o">().</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">Paragraph</span><span class="o">(</span><span class="s">"This is one paragraph"</span><span class="o">,</span> <span class="k">new</span> <span class="nc">Button</span><span class="o">(</span><span class="s">"Copy"</span><span class="o">)));</span>
<span class="n">html</span><span class="o">.</span><span class="na">body</span><span class="o">().</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">Paragraph</span><span class="o">(</span><span class="s">"This is another paragraph"</span><span class="o">,</span> <span class="k">new</span> <span class="nc">Button</span><span class="o">(</span><span class="s">"Copy"</span><span class="o">)));</span>

<span class="c1">// Or just:</span>
<span class="nc">Button</span> <span class="n">copyButton</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Button</span><span class="o">(</span><span class="s">"Copy"</span><span class="o">);</span>
<span class="n">html</span><span class="o">.</span><span class="na">body</span><span class="o">().</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">Paragraph</span><span class="o">(</span><span class="s">"This is one paragraph"</span><span class="o">,</span> <span class="n">copyButton</span><span class="o">));</span>
<span class="n">html</span><span class="o">.</span><span class="na">body</span><span class="o">().</span><span class="na">add</span><span class="o">(</span><span class="k">new</span> <span class="nc">Paragraph</span><span class="o">(</span><span class="s">"This is another paragraph"</span><span class="o">,</span> <span class="n">copyButton</span><span class="o">));</span>
</code></pre></div></div>

<p>Of course you could extract that <code class="language-plaintext highlighter-rouge">new Button("Copy")</code> call into a common variable to avoid defining it twice.<sup id="fnref:object-references" role="doc-noteref"><a href="#fn:object-references" class="footnote" rel="footnote">1</a></sup></p>

<p>Another good example is HTTP routing APIs, where some router object has methods for “get”, “post”, etc that accept a callback for when a certain HTTP request with a matching path is called. Taking the API for <a href="https://github.com/julienschmidt/httprouter">this library</a> simply as an example:</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">router</span> <span class="o">:=</span> <span class="n">httprouter</span><span class="o">.</span><span class="n">New</span><span class="p">()</span>
<span class="n">router</span><span class="o">.</span><span class="n">GET</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">,</span> <span class="n">_</span> <span class="n">httprouter</span><span class="o">.</span><span class="n">Params</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">fmt</span><span class="o">.</span><span class="n">Fprint</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s">"Welcome!</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
<span class="p">})</span>
</code></pre></div></div>

<p>What happens if you break out of the little API and do something that’s not really intended?</p>

<div class="language-go highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">router</span><span class="o">.</span><span class="n">GET</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="k">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">,</span> <span class="n">_</span> <span class="n">httprouter</span><span class="o">.</span><span class="n">Params</span><span class="p">)</span> <span class="p">{</span>
  <span class="n">router</span><span class="o">.</span><span class="n">GET</span><span class="p">(</span>
    <span class="n">r</span><span class="o">.</span><span class="n">Params</span><span class="p">[</span><span class="s">"new_method"</span><span class="p">],</span>
    <span class="k">func</span><span class="p">(</span><span class="n">w</span> <span class="n">http</span><span class="o">.</span><span class="n">ResponseWriter</span><span class="p">,</span> <span class="n">r</span> <span class="o">*</span><span class="n">http</span><span class="o">.</span><span class="n">Request</span><span class="p">,</span> <span class="n">_</span> <span class="n">httprouter</span><span class="o">.</span><span class="n">Params</span><span class="p">)</span> <span class="p">{</span>
      <span class="n">fmt</span><span class="o">.</span><span class="n">Fprint</span><span class="p">(</span><span class="n">w</span><span class="p">,</span> <span class="s">"Welcome!</span><span class="se">\n</span><span class="s">"</span><span class="p">)</span>
    <span class="p">})</span>
<span class="p">})</span>
</code></pre></div></div>

<p>I don’t know whether this would work (i.e: whether the <code class="language-plaintext highlighter-rouge">router</code> object gets closed for modification when you start accepting requests) but it’s clearly not the intended use of the API as it doesn’t match the simple language that is constructed by the API.</p>

<p>A great example of something that’s clearly a DSL is <a href="https://hexdocs.pm/ecto/Ecto.html">Ecto</a>, the database wrapper/ORM for Elixir. Ecto has a full-on language defined with macros that gets translated to SQL. Take this example from the documentation:</p>

<div class="language-elixir highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">query</span> <span class="o">=</span> <span class="n">from</span> <span class="n">u</span> <span class="ow">in</span> <span class="s2">"users"</span><span class="p">,</span>
          <span class="ss">where:</span> <span class="n">u</span><span class="o">.</span><span class="n">age</span> <span class="o">&gt;</span> <span class="mi">18</span> <span class="ow">or</span> <span class="n">is_nil</span><span class="p">(</span><span class="n">u</span><span class="o">.</span><span class="n">email</span><span class="p">),</span>
          <span class="ss">select:</span> <span class="p">%{</span><span class="ss">name:</span> <span class="n">u</span><span class="o">.</span><span class="n">name</span><span class="p">,</span> <span class="ss">age:</span> <span class="n">u</span><span class="o">.</span><span class="n">age</span><span class="p">}</span>
<span class="no">Repo</span><span class="o">.</span><span class="n">all</span><span class="p">(</span><span class="n">query</span><span class="p">)</span>
</code></pre></div></div>

<p>It might seem like you can call any function in your query—perhaps replacing that <code class="language-plaintext highlighter-rouge">is_nil(u.email)</code> with something that checks the validity of the email instead—but Ecto completely redefines the rules of what can happen within the <code class="language-plaintext highlighter-rouge">from</code> clause. Since it’s a macro, it’s not limited by typical evaluation order of Elixir function calls, and can translate the syntax tree into anything it wants. However at the end of the day it does have to translate it to an SQL query, and SQL can’t evaluate Elixir code.</p>

<p>Much like the scoping rules in the HTML builder, the code inside the <code class="language-plaintext highlighter-rouge">from</code> call are subject to Ecto’s rules, not the standard rules of Elixir function calls.</p>

<p>One last thing that I think differentiates a DSL is a little unease about the exact execution order of your code within the DSL. Are you defining some behaviour that will be aggregated and evaluated later, or will each part simply be evaluated immediately, top to bottom? The HTML builder might defer the serialisation until it’s requested, or it might serialise while you’re calling the builder methods. Or it might serialise some parts, but leave others un-evaluated. If you put a break point or log statement right in the middle, are you confident you know when that’ll actually trigger?</p>

<p>Something like a <a href="https://bundler.io/v2.5/man/gemfile.5.html">Ruby Bundler gemfile</a> might <em>look</em> like it’s installing each gem when you call the <code class="language-plaintext highlighter-rouge">gem</code> function, but—I’m fairly confident—it’s just defining a list of gems that it needs to install after the file is evaluated. You can’t install a gem at the top of the file and then change the gems you’re installing later based on the version you downloaded.</p>

<p>Of course none of these things are doing magic, they all need to use features of the language they live in, and are subject to that language’s limitations—for better or worse. While that does mean that you can debug a DSL down to the non-DSL code that powers it, the intent of the DSL is that you don’t have to, because if used correctly you don’t think about how they’re implemented, it’s just a more convenient way to express your intent.</p>

<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">inter</span> <span class="o">=</span> <span class="s1">'string interpolation'</span>
<span class="n">question</span> <span class="o">=</span> <span class="s2">"Is </span><span class="si">#{</span><span class="n">inter</span><span class="si">}</span><span class="s2"> a DSL?"</span>
<span class="nb">puts</span> <span class="n">question</span>
</code></pre></div></div>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:object-references" role="doc-endnote">
      <p>In all these examples I’m assuming that the <code class="language-plaintext highlighter-rouge">Button</code> objects are interchangeable, and the only state that one button holds is its label, so passing the same instance to two paragraphs is not a problem. <a href="#fnref:object-references" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/09/28/when-does-an-api-become-a-dsl/">Permalink</a>
&bull; September 28, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20When%20Does%20an%20API%20Become%20a%20DSL?" target=_blank>email</a>.


guid: https://willhbr.net/2024/10/03/web-gardening/
title: Web Gardening
url: https://willhbr.net/2024/10/03/web-gardening/
feedurl: https://willhbr.net/feed.xml
content:
<p>Just over four months ago—in <a href="/2024/05/05/happy-birthday-to-website/">a post celebrating the 10th birthday of my website</a>—I wrote:</p>

<blockquote>
  <p>What will the site look like in another ten years? Have I reached the optimal design, or has drudging through ten years of website screenshots inspired a proper redesign? Wait until 2034 to find out!</p>
</blockquote>

<p>And it turns out you won’t need to wait until 2034, because we’ve got a redesign right now! If you’re reading this in a feed reader, break out of the confines of your client’s default styling and <a href="/2024/10/03/web-gardening/">read this post on the web</a>!</p>

<p><img src="/images/2024/website-2024-05-05.webp" alt="A screenshot of my previous design, from May" /></p>

<p class="caption">In case you forgot, this is what my site used to look like.</p>

<h2 id="new-header">New Header</h2>

<p>The most obvious new feature is the big purple header:</p>

<p><img src="/images/2024/new-website.webp" alt="The new website" /></p>

<p>I’d been thinking about a new design basically since I made that comment at the end of the 10th birthday post. No ideas really stuck. I think played around with the most minor of changes—putting the navigation links to the side of the site title—and then thought that maybe the header could be designed as a block that dropped down from the accent strip at the top. I quickly realised that this would look terrible, but realised that on a sufficiently small screen, it would take up the full width—why not just do that for all screens?<sup id="fnref:especially-mobile" role="doc-noteref"><a href="#fn:especially-mobile" class="footnote" rel="footnote">1</a></sup></p>

<p>That ended up looking pretty good, and made me happy with both the light and dark themes for the site (whereas before the light theme just didn’t really hit the spot). The big colourful header makes the site stand out a little more, and hopefully makes it a bit more memorable than the previous mostly-grey or mostly-white design. I did tweak the dark mode colours to be a little more vibrant (the lavender is a little more saturated, the grey background is much darker) which I think better matches the feel of the light mode colours.</p>

<h2 id="no-javascript">No JavaScript</h2>

<p>Previously the main use of JavaScript was to provide an override to the theme, so that you could have my site in dark mode while your device was in light mode, or vice-versa. The control for this was somewhat unceremoniously plonked in the upper-right corner. With the new header design it stood out much more, and I made the decision that providing the toggle wasn’t worth the effort of finding a home for the button, or having every visitor make an additional HTTP request to fetch the JavaScript file.</p>

<h2 id="web-fonts">Web Fonts</h2>

<p>After shunning the idea of web fonts earlier this year, I experimented with using web fonts and liked the look more than I disliked the amount of extra content that has to be downloaded. I hadn’t realised before this that you needed to have a separate font file for regular, italic, bold, and bold italic, each costing around 20kB and a separate network request. I’m currently committing the <a href="https://ellenlupton.com/Thinking-with-Type">type crime</a> of allowing the browser to create a fake bold font instead of providing one myself, since bold text isn’t crucial to my site.</p>

<p>The fonts I settled on were <a href="https://en.wikipedia.org/wiki/DIN_1451">DIN</a> and <a href="https://www.ibm.com/plex/">IBM Plex Sans</a>. DIN—for titles—is very similar to what I was using before, which was either <em>DIN Alternate</em> or <a href="https://learn.microsoft.com/en-us/typography/font-list/bahnschrift"><em>Bahnschrift</em></a> depending on the platform. DIN is slightly tighter leading than either of those two, which I liked enough that I couldn’t go back. I remember when I first changed to using DIN Alternate and Bahnschrift I was sure that I was only a fan of the fact that I’d changed something, rather than the look of the fonts themselves. Well however much time later, and I think I quite like it.</p>

<p>I took a bunch of inspiration from <a href="http://pxlnv.com"><em>Pixel Envy</em></a> while re-thinking some of the design, the most visible being the use of Plex Sans. It’s just distinct enough that you might notice it, but definitely not out-there enough to make things a pain to read.</p>

<p>A really useful resource for doing web typography is <a href="https://sinja.io/blog/web-typography-quick-guide">this blog post</a> which touches on a much of things to consider. It also introduced me to <a href="https://modernfontstacks.com/">Modern Font Stacks</a>, which includes <code class="language-plaintext highlighter-rouge">font-family</code> declarations to get similar-looking fonts across different platforms for a variety of different font styles. If I hadn’t ended up using web fonts, I would have picked a set of fonts from here.</p>

<p>This is where I learnt about <a href="https://sinja.io/blog/web-typography-quick-guide#text-wrapping"><code class="language-plaintext highlighter-rouge">text-wrap: balance</code></a> which makes your titles look better by soft-wrapping pre-emptively instead of filling the entire width first.</p>

<p>Another interesting post was <a href="https://sia.codes/posts/making-google-fonts-faster/">this one</a> about web font performance, which explains what Google Fonts is <em>actually</em> doing when you use the single-line include, and how that can impact the load time on your website. I’ve tried to make my fonts as fast to load as possible by adding a <code class="language-plaintext highlighter-rouge">preload</code> for each in the HTML, so that by the time the browser has loaded the CSS it should already be well on its way to downloading the fonts.</p>

<h2 id="needless-optimisation">Needless Optimisation</h2>

<p>In order to justify the increased initial download size from adding web fonts, I tried to optimise the rest of the site to be as small as possible. The biggest optimisation was to move the paginated list of posts off the front page. Instead of putting 10 posts on the homepage, there’s only one. This cuts the size of the HTML file from &gt;300kB to ~25kB.</p>

<p>Another optimisation that I did—but didn’t end up mattering—was inlining duplicate SASS styles. I had assumed that this was done automatically, but it isn’t (at least not in the version of Ruby-SASS that Jekyll on GitHub Pages uses). So if you have a SASS file with two rules containing the same styles:</p>

<div class="language-sass highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">h1</span>
  <span class="nl">font-family</span><span class="p">:</span> <span class="n">DIN</span><span class="o">,</span> <span class="s2">"DIN Alternate"</span><span class="o">,</span> <span class="n">Bahnschrift</span><span class="o">,</span> <span class="nb">sans-serif</span>
  <span class="nl">text-align</span><span class="p">:</span> <span class="nb">center</span>
  <span class="nl">color</span><span class="p">:</span> <span class="no">tomato</span>
<span class="nt">h2</span>
  <span class="nl">font-family</span><span class="p">:</span> <span class="n">DIN</span><span class="o">,</span> <span class="s2">"DIN Alternate"</span><span class="o">,</span> <span class="n">Bahnschrift</span><span class="o">,</span> <span class="nb">sans-serif</span>
  <span class="nl">text-align</span><span class="p">:</span> <span class="nb">center</span>
  <span class="nl">color</span><span class="p">:</span> <span class="no">tomato</span>
</code></pre></div></div>

<p>The resulting CSS file is a one-to-one syntactic conversion:</p>

<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">h1</span> <span class="p">{</span>
  <span class="nl">font-family</span><span class="p">:</span> <span class="n">DIN</span><span class="p">,</span> <span class="s1">"DIN Alternate"</span><span class="p">,</span> <span class="n">Bahnschrift</span><span class="p">,</span> <span class="nb">sans-serif</span><span class="p">;</span>
  <span class="nl">text-align</span><span class="p">:</span> <span class="nb">center</span><span class="p">;</span>
  <span class="nl">color</span><span class="p">:</span> <span class="no">tomato</span><span class="p">;</span>
<span class="p">}</span>

<span class="nt">h2</span> <span class="p">{</span>
  <span class="nl">font-family</span><span class="p">:</span> <span class="n">DIN</span><span class="p">,</span> <span class="s1">"DIN Alternate"</span><span class="p">,</span> <span class="n">Bahnschrift</span><span class="p">,</span> <span class="nb">sans-serif</span><span class="p">;</span>
  <span class="nl">text-align</span><span class="p">:</span> <span class="nb">center</span><span class="p">;</span>
  <span class="nl">color</span><span class="p">:</span> <span class="no">tomato</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>So all the styles are pointlessly duplicated—or at least they are in simple cases like this, I don’t know if SASS could guarantee that merging the two rules into one would always result in the same behaviour. In this case though, you can inline them into a single rule, and avoid repeating the bodies twice:</p>

<div class="language-css highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nt">h1</span><span class="o">,</span> <span class="nt">h2</span> <span class="p">{</span>
  <span class="nl">font-family</span><span class="p">:</span> <span class="n">DIN</span><span class="p">,</span> <span class="s1">"DIN Alternate"</span><span class="p">,</span> <span class="n">Bahnschrift</span><span class="p">,</span> <span class="nb">sans-serif</span><span class="p">;</span>
  <span class="nl">text-align</span><span class="p">:</span> <span class="nb">center</span><span class="p">;</span>
  <span class="nl">color</span><span class="p">:</span> <span class="no">tomato</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>

<p>In my case the styles for the syntax highlighting included many duplicates, since there are many different token types, each with their own rule, and only a handful of colours that are applied to them. By replacing rules like this:</p>

<div class="language-sass highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.ne</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#ffb964</span>
<span class="nc">.nf</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#fad07a</span>
<span class="nc">.nl</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#ffb964</span>
<span class="nc">.nn</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#e8e8d3</span>
<span class="nc">.nx</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#e8e8d3</span>
<span class="nc">.py</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#e8e8d3</span>
</code></pre></div></div>

<p>With rules like this:</p>

<div class="language-sass highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nc">.ne</span><span class="o">,</span> <span class="nc">.nl</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#ffb964</span>
<span class="nc">.nf</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#fad07a</span>
<span class="nc">.nn</span><span class="o">,</span> <span class="nc">.nx</span><span class="o">,</span> <span class="nc">.py</span>
  <span class="nl">color</span><span class="p">:</span> <span class="mh">#e8e8d3</span>
</code></pre></div></div>

<p>I saved a whole kilobyte of CSS. Although that is before compression, so the actual gain of minimising data transferred over the network would be much smaller. And then of course I decided to make my own highlighting theme, which made this whole exercise pointless.</p>

<h2 id="syntax-highlighting">Syntax Highlighting</h2>

<p>For the longest time I had just used the <a href="https://pygments.org">Pygments</a> <a href="http://railscasts.com/about"><em>Railscasts</em></a> theme, which didn’t really fit in with the rest of the site, and would always be set on a dark background. I’d thought about finding another theme to use in light-mode, but never got around to it.</p>

<p>Instead, I bit the bullet and committed to making my own (very simple) theme that used the site’s purple highlight colour as a base. I then hue-shifted that same colour to get a green (for strings), blue (for keywords) and orange (for literals). I’ve tried to make syntax highlighting themes before and every time I get a newfound appreciation for anyone that makes a well thought-out theme that works with many different languages (like my long-time fave theme <a href="https://github.com/wadackel/vim-dogrun">dogrun</a>).</p>

<p>Initially my process was to assign colours based on <a href="https://github.com/rouge-ruby/rouge/wiki/List-of-tokens">the token types from Rouge</a> but later ended up just making a markdown file with the languages I might include in a post<sup id="fnref:languages" role="doc-noteref"><a href="#fn:languages" class="footnote" rel="footnote">2</a></sup>, and tweaking the theme until they all looked reasonable.</p>

<h2 id="front-page-design">Front Page Design</h2>

<p>The decision to shrink the size of the front page meant that I needed (or wanted) to have some way of showing some older posts on the homepage. My theory is that someone might come to visit my site, see one post and not be interested, and then leave. If I put a few titles of posts at the bottom, then maybe they’ll find what they’re looking for. Of course this is assuming anyone visits my site in the first place.</p>

<p>I wanted to keep the paginated full-text posts on the off chance that someone wants to skim-peruse the back catalog. If I come across a new blog I like to be able to scroll as much as possible, instead of having to click-click-click between different pages. The difficult decision here is deciding when to link to the archive versus linking to the paginated list of posts. I imagine I’ll end up tweaking this in the future.</p>

<p>Something I did to make this slightly more seamless is to add pagination controls to the front page, so it appears as a very short page with one post on it, followed by many pages with 10 posts each. This created a second problem that the first proper paginated page would include the post from the front page, so you could read the first post, then click to view the next page, and appear as though you’ve not moved. To avoid this, I never show the first post in the pagination—so the first page will actually have 9 posts.</p>

<p>The older Jekyll version used by GitHub Pages uses the also-old <code class="language-plaintext highlighter-rouge">jekyll-paginate</code> plugin (instead of the newer <code class="language-plaintext highlighter-rouge">jekyll-paginate-v2</code>). This is accessed with the <code class="language-plaintext highlighter-rouge">paginator</code> object, but I discovered that this is only available on the page that is going to be paginated, not on any other pages. Since my front page isn’t paginated (there’s a separate page for the paginated content) I couldn’t include the same pagination layout on the front page. Instead, I created a separate layout that does some hacks to approximate the same information. It can be a bit simpler since it’ll always be focussed on the “home” page, and it can just use the total post count to count the number of page links to create.</p>

<h2 id="permalinks-and-url-formats">Permalinks and URL Formats</h2>

<p>Something I considered was changing my URL format. Currently this is <code class="language-plaintext highlighter-rouge">/YYYY/MM/DD/:title</code> (the default Jekyll “pretty” URL style), but since I don’t post multiple times per day (especially with the same title) the format is overly verbose. I could just use <code class="language-plaintext highlighter-rouge">/:title</code> since my titles are (so far) unique, but I like indicating the approximate age of the post in the URL. <code class="language-plaintext highlighter-rouge">/YYYY/:title</code> was tempting, but in the end I decided that changing URLs was too big a change for not that much benefit. Especially since I’d have to ensure that all the existing posts maintained their existing links.</p>

<hr />

<p>I’ve seen a <a href="https://blog.jim-nielsen.com/2024/person-in-personal-website/">bunch</a> <a href="https://www.citationneeded.news/posse/">of</a> <a href="https://manuelmoreale.com/@/page/acx1bK7UldiQW556">posts</a> come across my RSS inbox about making personal websites <em>personal</em> and accepting that they’re not necessarily going to be flawless, and showing some character is part of the point. I started to think of my website (and also my <a href="https://github.com/willhbr/dotfiles">shell config</a>) like gardening. It’ll never be finished, there’s always something to move or tweak or move back, it’s in a state of constant refinement. Unlike real gardening I can leave it untouched and know that it won’t get worse.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:especially-mobile" role="doc-endnote">
      <p>Especially since most visitors are probably on phones anyway. <a href="#fnref:especially-mobile" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:languages" role="doc-endnote">
      <p>So far that’s Bash, Clojure, “conf” (for tmux config files), Console (for shell sessions), C++, Crystal, CSS, Docker, Elixir, ERB, Go, Haskell, HTML, Java, JavaScript, JSON, Kotlin, Markdown, Python, Ruby, Rust, Shell, SQL, Swift, TOML, and YAML. <a href="#fnref:languages" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/10/03/web-gardening/">Permalink</a>
&bull; October 3, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Web%20Gardening" target=_blank>email</a>.


guid: https://willhbr.net/2024/11/23/why-does-delta-take-400ms-to-show-diffs/
title: Why Does Delta Take 400ms to Show Diffs?
url: https://willhbr.net/2024/11/23/why-does-delta-take-400ms-to-show-diffs/
feedurl: https://willhbr.net/feed.xml
content:
<p>The other day I saw a link to <a href="https://github.com/dandavison/delta"><code class="language-plaintext highlighter-rouge">delta</code></a>, which is a tool that integrates with git to add both add better diff highlighting (similar to what you’d see in the GitHub/GitLab web UI) and syntax highlighting. I decided to give it a try. There’s even a note in the <a href="https://martinvonz.github.io/jj/latest/config/#processing-contents-to-be-paged">JJ docs</a> on how to set it up (since obviously I’m <a href="/2024/04/01/its-not-me-its-git/">not going to be using git</a>):</p>

<div class="language-toml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">[ui]</span>
<span class="py">pager</span> <span class="p">=</span> <span class="s">"delta"</span>
<span class="py">diff.format</span> <span class="p">=</span> <span class="s">"git"</span>
</code></pre></div></div>

<p>I tried it and the diff looks nice. However I also noticed that it was <em>really</em> slow. On my <a href="https://github.com/willhbr/dotfiles">dotfiles repo</a> it took just over 400ms to diff a commit. That’s not a big repo in any measure: it doesn’t have many files, the files are small, and there aren’t many commits. Showing the diff without <code class="language-plaintext highlighter-rouge">delta</code> took less than 100ms. Even more puzzling was that setting <code class="language-plaintext highlighter-rouge">ui.pager = "delta"</code> also made <code class="language-plaintext highlighter-rouge">jj log</code> take a similar amount of time—it jumped from ~80ms to ~430ms. The output of <code class="language-plaintext highlighter-rouge">jj log</code> doesn’t even show any diffs (by default), so what’s my computer doing for those 400 milliseconds?</p>

<p>Initially I was a bit puzzled as to why <code class="language-plaintext highlighter-rouge">log</code> was also slowing down, but a look at the code reveals that JJ will <a href="https://github.com/martinvonz/jj/blob/ffe5519fd0f2c63a378239fb81b5cbffab06f3e4/cli/src/commands/log.rs#L175">always call out to the pager program</a>, it’s just that <code class="language-plaintext highlighter-rouge">less</code> will check the screen height and do a transparent pass-through if the output is smaller than the height of the screen.</p>

<p>In order to get a few more data points, I tried <code class="language-plaintext highlighter-rouge">jj log | delta</code>, which should result in the same amount of work and thus take the same amount of time. This was not the case—piping the log output into <code class="language-plaintext highlighter-rouge">delta</code> didn’t slow things down at all, it was about the same speed as using <code class="language-plaintext highlighter-rouge">less</code>.</p>

<p>At this point I was suitably puzzled. I knew there <em>could</em> be some difference in how the data is passed between processes with a shell pipe versus a subprocess, but I didn’t know enough to really dig any further. Also the intended use of <code class="language-plaintext highlighter-rouge">delta</code> is to be called as a pager by a version control system, so it would be very odd if this way of using it was much slower than other non-standard uses. Instead I asked the resident performance expert and flame graph enthusiast <a href="https://www.markhansen.co.nz">Mark</a>, who is always easy to nerd-snipe into solving a problem.</p>

<p>His first suggestion was to see if we could spot something in <code class="language-plaintext highlighter-rouge">strace</code>, but then neither of us wanted to wade through the thousands of lines of output that it produced. The next suggestion was to use <a href="https://github.com/mstange/samply"><code class="language-plaintext highlighter-rouge">samply</code></a>. A quick <code class="language-plaintext highlighter-rouge">cargo install</code> later and I was able to grab a trace just by running:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>$ samply record jj log
@  r Will Richardson 1 hour ago b
│  (no description set)
◆  k Will Richardson 1 month ago main@origin 9
│  Add shortcut to copy last command output
~
Local server listening at http://127.0.0.1:3000
Press Ctrl+C to stop.
</code></pre></div></div>

<p><code class="language-plaintext highlighter-rouge">samply</code> grabs the trace and then starts a local webserver to either download it or open it in the <a href="https://profiler.firefox.com">Firefox profiler</a>. It’s mildly inconvenient that it runs the server on <code class="language-plaintext highlighter-rouge">127.0.0.1</code> instead of listening on all interfaces, so I had to setup a port forward as I develop on a remote machine. Thankfully this <a href="https://github.com/mstange/samply/pull/234">has been fixed</a> with the addition of a <code class="language-plaintext highlighter-rouge">--address</code> flag.</p>

<p>The trace showed the <code class="language-plaintext highlighter-rouge">jj</code> and <code class="language-plaintext highlighter-rouge">delta</code> processes, as well as a thread in the <code class="language-plaintext highlighter-rouge">delta</code> process called “find_calling_pr”—which did about 400ms of work while the main <code class="language-plaintext highlighter-rouge">delta</code> thread did nothing. A quick search of the <code class="language-plaintext highlighter-rouge">delta</code> code <a href="https://github.com/dandavison/delta/blob/959471392d5aa0289f979c8898260e0f133d9ae7/src/utils/process.rs#L50">reveals this</a>:</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">pub</span> <span class="k">fn</span> <span class="nf">start_determining_calling_process_in_thread</span><span class="p">()</span> <span class="p">{</span>
    <span class="c1">// The handle is neither kept nor returned nor joined but dropped, so the main</span>
    <span class="c1">// thread can exit early if it does not need to know its parent process.</span>
    <span class="nn">std</span><span class="p">::</span><span class="nn">thread</span><span class="p">::</span><span class="nn">Builder</span><span class="p">::</span><span class="nf">new</span><span class="p">()</span>
        <span class="nf">.name</span><span class="p">(</span><span class="s">"find_calling_process"</span><span class="nf">.into</span><span class="p">())</span>
        <span class="nf">.spawn</span><span class="p">(</span><span class="k">move</span> <span class="p">||</span> <span class="p">{</span>
            <span class="k">let</span> <span class="n">calling_process</span> <span class="o">=</span> <span class="nf">determine_calling_process</span><span class="p">();</span>

            <span class="k">let</span> <span class="p">(</span><span class="n">caller_mutex</span><span class="p">,</span> <span class="n">determine_done</span><span class="p">)</span> <span class="o">=</span> <span class="o">&amp;**</span><span class="n">CALLER</span><span class="p">;</span>

            <span class="k">let</span> <span class="k">mut</span> <span class="n">caller</span> <span class="o">=</span> <span class="n">caller_mutex</span><span class="nf">.lock</span><span class="p">()</span><span class="nf">.unwrap</span><span class="p">();</span>
            <span class="o">*</span><span class="n">caller</span> <span class="o">=</span> <span class="n">calling_process</span><span class="p">;</span>
            <span class="n">determine_done</span><span class="nf">.notify_all</span><span class="p">();</span>
        <span class="p">})</span>
        <span class="nf">.unwrap</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Ok so we’re trying to find something about the parent process. A look back at the trace shows we’re then calling into <a href="https://docs.rs/sysinfo/0.29.11/src/sysinfo/linux/process.rs.html#363"><code class="language-plaintext highlighter-rouge">retrieve_all_new_process_info</code></a> in the <code class="language-plaintext highlighter-rouge">sysinfo</code> crate. Why are we spending so much time getting process information?</p>

<p>I read a bit of the <code class="language-plaintext highlighter-rouge">determine_calling_process</code> function that is being called, and the gist is that it tries to find the parent process in order to work out if <code class="language-plaintext highlighter-rouge">delta</code> has been invoked by <code class="language-plaintext highlighter-rouge">git</code>, <code class="language-plaintext highlighter-rouge">rg</code>, <code class="language-plaintext highlighter-rouge">ack</code>, <code class="language-plaintext highlighter-rouge">sift</code>, or some other tool. It doesn’t just look at one process though, it uses some heuristics to try and find the calling process. If I’m reading the code right, this can end up fetching information for every running process<sup id="fnref:process-count" role="doc-noteref"><a href="#fn:process-count" class="footnote" rel="footnote">1</a></sup>. Since I was running <code class="language-plaintext highlighter-rouge">delta</code> via JJ it would never find a process name it recognised, so I would hit the worst case every time.</p>

<p>At this point I went to look at what this process information is actually used for. Eventually a <code class="language-plaintext highlighter-rouge">CallingProcess</code> enum is created that contains information about either a git subcommand, or one of the various grep-like tools. The only place I can see this information being used is <a href="https://github.com/dandavison/delta/blob/959471392d5aa0289f979c8898260e0f133d9ae7/src/utils/process.rs#L25"><code class="language-plaintext highlighter-rouge">paths_in_input_are_relative_to_cwd()</code></a>, which does exactly what it says: determines if the paths are relative to the current working directory.</p>

<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">impl</span> <span class="n">CallingProcess</span> <span class="p">{</span>
    <span class="k">pub</span> <span class="k">fn</span> <span class="nf">paths_in_input_are_relative_to_cwd</span><span class="p">(</span><span class="o">&amp;</span><span class="k">self</span><span class="p">)</span> <span class="k">-&gt;</span> <span class="nb">bool</span> <span class="p">{</span>
        <span class="k">match</span> <span class="k">self</span> <span class="p">{</span>
            <span class="nn">CallingProcess</span><span class="p">::</span><span class="nf">GitDiff</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="k">if</span> <span class="n">cmd</span><span class="py">.long_options</span><span class="nf">.contains</span><span class="p">(</span><span class="s">"--relative"</span><span class="p">)</span> <span class="k">=&gt;</span> <span class="k">true</span><span class="p">,</span>
            <span class="nn">CallingProcess</span><span class="p">::</span><span class="nf">GitShow</span><span class="p">(</span><span class="n">cmd</span><span class="p">,</span> <span class="n">_</span><span class="p">)</span> <span class="k">if</span> <span class="n">cmd</span><span class="py">.long_options</span><span class="nf">.contains</span><span class="p">(</span><span class="s">"--relative"</span><span class="p">)</span> <span class="k">=&gt;</span> <span class="k">true</span><span class="p">,</span>
            <span class="nn">CallingProcess</span><span class="p">::</span><span class="nf">GitLog</span><span class="p">(</span><span class="n">cmd</span><span class="p">)</span> <span class="k">if</span> <span class="n">cmd</span><span class="py">.long_options</span><span class="nf">.contains</span><span class="p">(</span><span class="s">"--relative"</span><span class="p">)</span> <span class="k">=&gt;</span> <span class="k">true</span><span class="p">,</span>
            <span class="nn">CallingProcess</span><span class="p">::</span><span class="nf">GitBlame</span><span class="p">(</span><span class="n">_</span><span class="p">)</span>
            <span class="p">|</span> <span class="nn">CallingProcess</span><span class="p">::</span><span class="nf">GitGrep</span><span class="p">(</span><span class="n">_</span><span class="p">)</span>
            <span class="p">|</span> <span class="nn">CallingProcess</span><span class="p">::</span><span class="n">OtherGrep</span> <span class="k">=&gt;</span> <span class="k">true</span><span class="p">,</span>
            <span class="n">_</span> <span class="k">=&gt;</span> <span class="k">false</span><span class="p">,</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>So if it’s called by a grep-like tool, the paths are relative. If the git command received the <code class="language-plaintext highlighter-rouge">--relative</code> flag, then it’s relative. If it’s something else, then it’s absolute.</p>

<p>The solution to my slow <code class="language-plaintext highlighter-rouge">delta</code> invocation seems to be that it needs to know about processes named <code class="language-plaintext highlighter-rouge">jj</code>, so it doesn’t waste time scanning for a <code class="language-plaintext highlighter-rouge">git</code> process that doesn’t exist. I did however try this again on a different computer with much less work going on (~300 processes) and <code class="language-plaintext highlighter-rouge">delta</code> was only ~100ms slower than using <code class="language-plaintext highlighter-rouge">less</code>, which is just fast enough that it’s not too noticeable.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:process-count" role="doc-endnote">
      <p>On my machine while I was testing this, <code class="language-plaintext highlighter-rouge">ps -ef | cl -l</code> showed that I had about 1500 processes running. <a href="#fnref:process-count" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/11/23/why-does-delta-take-400ms-to-show-diffs/">Permalink</a>
&bull; November 23, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Why%20Does%20Delta%20Take%20400ms%20to%20Show%20Diffs?" target=_blank>email</a>.


guid: https://willhbr.net/2024/12/02/appreciating-the-fine-art-of-phone-photography/
title: Appreciating the Fine Art of Phone Photography
url: https://willhbr.net/2024/12/02/appreciating-the-fine-art-of-phone-photography/
feedurl: https://willhbr.net/feed.xml
content:
<p>Last month I spent some time in <a href="https://en.wikipedia.org/wiki/Europe">Europe</a> and discovered a new appreciation for phone photography. My Sony a6500 did make the trip with me (along with standard compliment of lenses, filters, and suchlike) but I ended up taking a lot of photos on my phone, and being really happy with the results.</p>

<p>The phone in question is my iPhone 15 Pro, that I bought at release last year. Like with my previous two iPhones (an 8 and a 12 Pro) I always felt that the photos it took were <em>good</em>, but not something that I would take time to edit significantly to achieve a look much different from how it arrives out of the processing pipeline.</p>

<p>The first day in Paris was grey and drizzling with rain. I carried my a6500 with me, but I’m not super comfortable using it in the rain as while the camera body is weather-sealed, the Sony 18-105mm lens is not. This has never been a problem—I’ve got it damp plenty of times before—but I don’t know what the limit is. The result was that I took a lot more photos on my phone that I might have otherwise. Any modern phone is basically completely waterproof. With no articulating screen, interchangeable lens, knobs or dials, I have no concerns about using my phone even in a complete downpour.</p>

<p>A phone form-factor has a bunch of advantages to even a moderately compact dedicated camera. I think the biggest is the social/psychological advantage: taking photos on a phone is much more normal than using a dedicated camera. Someone taking a photo on their phone basically just blends into the background, whereas someone using a dedicated camera with a big lens is clearly doing <em>photography</em> which is treated differently by other people. I always try to make it clear that I’m taking photos of <em>things</em>, not the people around them, and that I’m happy to wait until there’s a clear moment to get my shot—so please just keep doing what you’re doing and don’t worry about where you’re standing. I’ve thankfully never had anyone confront me while I’m taking photos, but I’m socially awkward enough to try and avoid any confrontation preemptively.</p>

<p loading="lazy"><img src="https://photography.willhbr.net/photos/2024-10-23-b12ca7.jpeg" alt="Paris pic" /></p>

<p>During these rainy days I took lots of pictures of the Parisian architecture and edited them in <a href="http://photomator.com">Photomator</a> right on my phone. I’m not a big user of presets, but because of the overcast skies and uniformity in the colours of the buildings in Paris, after editing a few photos I stopped copy-pasting the same edits and just made a preset called “Paris”. This would accentuate the colour of the limestone buildings, add contrast to the sky, and push the colours of the foliage more towards an autumnal tone.</p>

<p loading="lazy"><img src="/images/2024/paris-comparison.webp" alt="A classic &quot;Paris&quot; photo" /></p>

<p class="caption">A comparison of the original image (left) and the edited image (right). See the full <a href="https://photography.willhbr.net/2024-10-23/">photo here</a>.</p>

<p>While the “camera you have with you” maxim is a bit boring, it is true. Even if I had my a6500 with me—in my bag or on a strap—it is often more convenient to just use my phone. Taking a photo from a bus, train, or cramped cablecar is just easier when you can already have your phone in your hand. This is even easier with the “action button” on the iPhone 15. It means I can be launching the camera <em>while</em> I’m still taking the phone out of my pocket. By the time the phone is pointed where I need it, the camera is already open and ready to capture. Launching the camera from the lock screen software button is incredibly fiddly in comparison.<sup id="fnref:android-lock" role="doc-noteref"><a href="#fn:android-lock" class="footnote" rel="footnote">1</a></sup></p>

<p>iOS 18 added the ability to customise the buttons on the lock screen, so I can swap out the shortcut to the built-in camera app to instead open <a href="https://halide.cam">Halide</a>. Originally I did use both Halide and the built-in camera app, depending on the shot. However after editing photos from both apps, I ended up just using the built-in camera, since it’s more responsive and I didn’t notice a difference in editability.</p>

<p>The most frustrating thing about the action button is the fact that it could be used to do so much more, but it’s currently limited to a single action. Right now it is activated by a semi-long press, but you could easily imagine having different actions for a double press, short press, or even short press followed by a long press. Having two different actions could allow for using the button for quick camera access <em>and</em> doing the various interesting productivity whatsits that people have devised. As it stands, I don’t want to lose quick access to the camera so reassigning the button is a non-starter.</p>

<p loading="lazy"><img src="https://photography.willhbr.net/photos/2024-11-16-50a80f.jpeg" alt="A photo of a dam with a mountain in the background" /></p>

<p class="caption">I was pleasantly surprised that the preset that I had made for buildings in Paris during overcast weather also applied quite well to completely different scenes.</p>

<h1 id="phone-photography-tips">Phone Photography Tips</h1>

<h2 id="ensure-the-sky-is-exposed-correctly">Ensure the sky is exposed correctly</h2>

<p>While HDR can save you a lot of the time, it’s worth checking—even on an overcast day—that the sky isn’t blown out. It’s probably the easiest tell for phone photography to have a corner of pure-white sky. This was especially relevant when taking pictures in small streets with just a little bit of sky visible at the top of frame. The phone will expose for the buildings, and leave the sky blown out. If you just tap on the section of sky, you’ll probably find that you can get both exposed well, and then just bump up the shadows and bring down the highlights later.</p>

<p>Sometimes the dynamic range is just too much for the phone, in which case just take the photo anyway, but then reconsider your framing and take some alternatives that don’t push it as much. Maybe the first one will be recoverable, but maybe you’ll end up with a better exposed shot even if it’s not the composition you wanted originally.</p>

<p loading="lazy"><img src="https://photography.willhbr.net/photos/2024-11-09-49d938.jpeg" alt="a photo of an alleyway in Marseille where the sky could be blown out if you're not careful" /></p>

<p class="caption">For photos like this your phone will often expose for the street, and over-expose the patch of sky.</p>

<h2 id="look-out-for-over-saturated-highlights">Look out for over-saturated highlights</h2>

<p>I’m not sure of the exact cause of this, but this happens with my phone much more than it happens with my a6500 or my drone. If you take a picture of a well-lit subject (like a beautiful pastel-coloured building in Provence) and then go to play with the colours a bit, you can get weird-looking highlights that are almost clipping in a certain colour. Things usually get less saturated as they get brighter, so you probably just need to desaturate the offending colour channel slightly.</p>

<h2 id="dont-shoot-into-the-sun">Don’t shoot into the sun</h2>

<p>Because of the lower dynamic range, I find that it’s just not worth it to compose a shot that includes the sun in the frame. That is unless you’re willing to edit it stylistically so that the sky is completely blown out. With a dedicated camera like my a6500 I can play with the exposure enough to get a somewhat pleasant falloff without banding. The best examples I can find from my a6500 are <a href="https://photography.willhbr.net/2021-12-20/">this shot at Bronte</a>, <a href="https://photography.willhbr.net/2023-02-16/">this shot in Adelaide</a> or <a href="https://photography.willhbr.net/2023-12-05/">this shot in Tokyo</a> that all include a fairly bright sky that I was able to recover.</p>

<h2 id="use-the-volume-buttons-to-capture-photos">Use the volume buttons to capture photos</h2>

<p>If you’re this far into a phone photography blog post I assume you already know that you can use the volume buttons to capture a shot, instead of just using the on-screen button. This is useful if you’ve only got one hand free—the other hand busy with a half-eaten pain au chocolat—and want to take a quick snap. You can definitely contort your hand to press the on-screen button, but that’s a great way to drop your phone.</p>

<p>A side-tip is to use the action button to launch the camera, so you can launch the camera app and capture a photo with one hand.</p>

<h2 id="consider-which-lens-to-use">Consider which lens to use</h2>

<p>This will depend on which model of phone you have, so it’s worth looking up some info on the different lenses your phone has—and possibly the different faux-lenses that it emulates. For my 15 Pro I have:</p>

<h3 id="ultrawide-13mm">Ultrawide (13mm)</h3>

<p>This uses a dedicated 13mm-equivalent lens, which isn’t as high quality as the others. I only use it when I need the wider field-of-view. Motion blur is less noticeable with wide-angle lenses, so you can get some good results in lower light with the ultra wide.</p>

<h3 id="mainwide-lens-24mm-28mm-35mm-48mm">Main/wide lens (24mm, 28mm, 35mm, 48mm)</h3>

<p>The biggest advantage of the main lens is that it’s got a larger sensor, wider aperture, and more megapixels than the other two cameras. It’s a nice default, but I find that too many shots at this “kinda-wide” angle just gets a bit boring. Since the sensor and aperture are the biggest, this is the best to use in low light.</p>

<p>There are the 3 additional fake “lenses” that just crop the 48MP image from the 24mm camera: 28mm, 35mm, and 48mm. The image quality from these goes downhill as the crop increases, but they provide enough of a different perspective that I think they’re worth using. I would find myself using the 35mm and 48mm zooms fairly often, to get closer to that “neutral” 50mm zoom. This is often just enough zoom to hide some pesky tourists, cars, or signage just out of frame.</p>

<p loading="lazy"><img src="https://photography.willhbr.net/photos/2024-11-15-257eb8.jpeg" alt="Montserrat, taken with the main lens" /></p>

<p class="caption">Montserrat, taken with the main lens.</p>

<p>I would recommend taking photos with all four of these zoom levels and paying attention to the difference in the level of detail. In <a href="https://photography.willhbr.net/2024-11-15/">these two photos from Montserrat</a> the first (of the sun on the hills below) is taken at the 48mm zoom, and when you zoom in you can see far less detail than the second shot of the monastery. (I heavily compress the images on that website, so you’ll have to believe me or take some photos yourself).</p>

<h3 id="telephoto-77mm">Telephoto (77mm)</h3>

<p>The 3x “telephoto” zoom is in a bit of an awkward spot. It’s not really telephoto enough to give a substantially different perspective, and when it does give one the optics are lower quality enough that the photos are just <em>fine</em>. A 77mm camera with the sensor size and aperture of the iPhone’s main camera would be great for portraits to get more of a natural bokeh. Perhaps the 120mm lens on the iPhone 15 Pro Max and iPhones 16 Pro will be more exciting for me.</p>

<h2 id="use-the-pre-defined-lenses-and-faux-lenses">Use the pre-defined lenses and faux lenses</h2>

<p>I call out these focal lengths specifically because I only ever swap between these zoom levels. I act like I’m swapping between physical prime lenses, instead of zooming in and out by pinching. This is fuelled by an obsession to avoid the possibility of zooming to 23mm and taking a dreadful cropped image from the 13mm lens instead of just zooming in a little more and using the much better 24mm lens.</p>

<p>If you’re an aspiring photographer it’s also useful to get familiar with what field of view you get at each focal length<sup id="fnref:35mm-equiv" role="doc-noteref"><a href="#fn:35mm-equiv" class="footnote" rel="footnote">2</a></sup> so when you pick up a dedicated camera you already have a feeling of what it’ll see before you turn it on.</p>

<h2 id="dont-use-portrait-mode">Don’t use portrait mode</h2>

<p>I know that everyone loves portrait mode, and it’s got so much better than when it was first introduced, but the cookie-cutter edges around people’s heads or un-blurred sections in the gap of someone’s arm just stick out to me immediately. If you do shoot portrait mode, take the time to just tone down the background blur a <em>touch</em>. If you’re taking a photo of someone with frizzy or wispy hair, it’s probably best to just turn it off.</p>

<h2 id="use-photomators-select-sky">Use Photomator’s “select sky”</h2>

<p>When the sky is overcast and cloudy, I often cheat and use the <a href="https://support.pixelmator.com/photomator-user-guide/working-with-selective-adjustments">“select sky” feature</a> in Photomator to create an adjustment layer only touching the sky. I then lower the exposure and crank up the clarity and texture to accentuate the contrast in the clouds.</p>

<p loading="lazy"><img src="https://photography.willhbr.net/photos/2024-10-26-78d2b2.jpeg" alt="a photo of a building in Paris with a very moody sky" /></p>

<p class="caption">It’s easy to get this over-dramatic crunchy sky look with a little bit of ML and a cranked texture slider.</p>

<hr />

<p>I’m still going through and uploading more onto <a href="https://photography.willhbr.net">my photo blog</a>, and will probably start editing my a6500 photos soon. Smash the <a href="https://photography.willhbr.net/feed.xml">RSS feed</a> into your favourite feed reader and get a handful of photos every month.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:android-lock" role="doc-endnote">
      <p>Android’s “double-click the lock button” to launch the camera is similarly convenient, it’s a shame that was never an option on iOS. <a href="#fnref:android-lock" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:35mm-equiv" role="doc-endnote">
      <p>On a full-frame sensor, that is. <a href="#fnref:35mm-equiv" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/12/02/appreciating-the-fine-art-of-phone-photography/">Permalink</a>
&bull; December 2, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Appreciating%20the%20Fine%20Art%20of%20Phone%20Photography" target=_blank>email</a>.


guid: https://willhbr.net/2024/12/06/http-router-for-crystal/
title: HTTP Router for Crystal
url: https://willhbr.net/2024/12/06/http-router-for-crystal/
feedurl: https://willhbr.net/feed.xml
content:
<p>The most common thing that I do with <a href="https://crystal-lang.org/reference/latest/syntax_and_semantics/macros/index.html">Crystal macros</a> is to write some way of generating a <code class="language-plaintext highlighter-rouge">case</code> statement. If you’re writing something that does networking or deserialisation, translating data into method calls is very common, and automatically generating the code to do it is neat.</p>

<p>I’ve been adding some more HTTP endpoints to my container status board, which has meant that my previously-simple routing logic started to get a bit messy, so it was time to write a macro. I did look at <a href="https://github.com/veelenga/awesome-crystal">awesome-crystal</a> to see if there was an existing library that I could use. There are two—<a href="https://github.com/tbrand/router.cr">router.cr</a> and  <a href="https://github.com/obsidian/orion">orion</a>—but neither are quite what I’m looking for. They more closely match the <a href="https://guides.rubyonrails.org/routing.html">Rails-style routing DSL</a>, whereas I just wanted something that would dispatch between methods on an <code class="language-plaintext highlighter-rouge">HTTP::Handler</code> class.</p>

<p>The resulting macro isn’t very big, and all it does is generate a <code class="language-plaintext highlighter-rouge">call(context)</code> method for you with a single <code class="language-plaintext highlighter-rouge">case</code> statement in it. It means you can do this:</p>

<div class="language-crystal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">MyHander</span>
  <span class="kp">include</span> <span class="no">HTTP</span><span class="o">::</span><span class="no">Handler</span>
  <span class="kp">include</span> <span class="no">HTTP</span><span class="o">::</span><span class="no">Router</span>

  <span class="nd">@[HTTP::Route(path: "/stuff")]</span>
  <span class="nd">@[HTTP::Route(path: "/stuff", method: :HEAD)]</span>
  <span class="k">def</span> <span class="nf">get_stuff</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
    <span class="n">context</span><span class="p">.</span><span class="nf">ok_json</span><span class="p">(</span><span class="ss">result: </span><span class="s2">"success!"</span><span class="p">)</span>
  <span class="k">end</span>

  <span class="nd">@[HTTP::Route(path: "/stuff", method: :POST)]</span>
  <span class="k">def</span> <span class="nf">post_stuff</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
    <span class="n">contents</span> <span class="o">=</span> <span class="n">context</span><span class="p">.</span><span class="nf">request</span><span class="p">.</span><span class="nf">body</span><span class="p">.</span><span class="nf">get_to_end</span>
    <span class="nb">puts</span> <span class="n">contents</span>
    <span class="n">context</span><span class="p">.</span><span class="nf">ok_json</span><span class="p">(</span><span class="ss">result: </span><span class="s2">"success!"</span><span class="p">)</span>
  <span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>

<p>And not have to manually write this out:</p>

<div class="language-crystal highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">call</span><span class="p">(</span><span class="n">context</span> <span class="p">:</span> <span class="no">HTTP</span><span class="o">::</span><span class="no">Server</span><span class="o">::</span><span class="no">Context</span><span class="p">)</span>
  <span class="n">req</span> <span class="o">=</span> <span class="n">context</span><span class="p">.</span><span class="nf">request</span>
  <span class="k">case</span> <span class="p">{</span> <span class="n">req</span><span class="p">.</span><span class="nf">method</span><span class="p">,</span> <span class="n">req</span><span class="p">.</span><span class="nf">path</span> <span class="p">}</span>
  <span class="k">when</span> <span class="p">{</span> <span class="s2">"GET"</span><span class="p">,</span> <span class="s2">"/stuff"</span> <span class="p">}</span>
    <span class="nb">self</span><span class="p">.</span><span class="nf">get_stuff</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
  <span class="k">when</span> <span class="p">{</span> <span class="s2">"HEAD"</span><span class="p">,</span> <span class="s2">"/stuff"</span> <span class="p">}</span>
    <span class="nb">self</span><span class="p">.</span><span class="nf">get_stuff</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
  <span class="k">when</span> <span class="p">{</span> <span class="s2">"POST"</span><span class="p">,</span> <span class="s2">"/stuff"</span> <span class="p">}</span>
    <span class="nb">self</span><span class="p">.</span><span class="nf">post_stuff</span><span class="p">(</span><span class="n">context</span><span class="p">)</span>
  <span class="k">else</span>
    <span class="n">call_next</span> <span class="n">context</span>
  <span class="k">end</span>
<span class="k">end</span>
</code></pre></div></div>

<p>You can <a href="https://github.com/willhbr/http-router">have a look at it on GitHub</a>.</p>


<a href="https://willhbr.net/2024/12/06/http-router-for-crystal/">Permalink</a>
&bull; December 6, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20HTTP%20Router%20for%20Crystal" target=_blank>email</a>.


guid: https://willhbr.net/2024/12/16/going-gigabit/
title: Going Gigabit
url: https://willhbr.net/2024/12/16/going-gigabit/
feedurl: https://willhbr.net/feed.xml
content:
<p>I’ve finally got a network and internet connection fit for the 2020s. My internet connection is gigabit, and my wifi router has been replaced with one that can actually make use of it.</p>

<p>Firstly, the internet. Put simply, in Australia the internet situation is bad. Up until recently the fastest home internet plan was 100Mb/s. When faster plans were introduced they were significantly more expensive. I’m not well versed on the history but <a href="https://en.wikipedia.org/wiki/Internet_in_Australia">Wikipedia sums it up well</a>:</p>

<blockquote>
  <p>The national broadband network aimed to provide up to 1000 Mbit/s speeds […] This has since been revised […] for 25 Mbit/s to 100 Mbit/s</p>
</blockquote>

<p class="caption">Although 25Mb/s <strong>is</strong> “up to 1000Mb/s” if you think about it.</p>

<p>Anyway, for boring reasons I was with <a href="https://en.wikipedia.org/wiki/Internode_(ISP)"><em>Internode</em></a> and had managed to get a secret deal for 250Mb/s at $90 per month after calling them up asking to cancel my service. Last month they sent me an email saying they were increasing the price to $100. I reminded them of their promise that the secret deal was permanent and that they weren’t going to increase the price, but they weren’t interested in negotiating.</p>

<p>Being too stubborn to go back to a 100Mb/s plan, I looked around for other options and found that for $99 I could get a gigabit plan from <a href="https://www.canstarblue.com.au/internet/aussie-broadband-launches-buddy-telco/"><em>Buddy Telco</em></a>—over three times faster than the $100 <em>Internode</em> plan. I cancelled my existing plan (at the end of the billing cycle to not waste any money on internet I’d already paid for) and signed up for a new gigabit connection.</p>

<p><em>Buddy Telco</em> is an offshoot of the well-regarded <em>Aussie Broadband</em> ISP, but saving costs by not having phone support. I was a little concerned that the setup would be painful, as when I’ve moved or swapped routers I’ve had to call tech support and either have them tell me some setting I needed to toggle on the router (it’s always vlan tagging) or find out that everything on my end is correct and they just need to press a button on their end. That’s always the most frustrating end to an evening of debugging. At this point if I can’t find the problem in a few minutes of tinkering I’ll just call up customer support, wait on hold while I tinker, and then follow along with whatever their script wants me to do. Reboot the router? Reboot the fridge? I don’t think that’ll do anything but I’ll do it anyway.</p>

<p>My new connection activated last week, I expected to get sent <a href="https://en.wikipedia.org/wiki/Point-to-Point_Protocol_over_Ethernet">PPPoE</a> credentials and some other instructions, but it just said “Please connect to UNI-D port 2, on your NBN device.” I moved the WAN cable from port 1 to port 2, and it worked. No credentials, no vlan tagging, no calling customer support.</p>

<p>Their online portal <em>does</em> have some self-service debugging tools. You can check your connection status, reset the connection, and things like that. They have some basic descriptions to say “do this if you’ve changed your router” and such, so I’m hopeful that the lack of customer support won’t end up being an issue.</p>

<p>I checked Grafana, and there we are: 911Mb/s. I guess that’s gigabit.</p>

<p><img src="/images/2024/gigabit-graph.webp" alt="Graph from Grafana showing connection speed increase from 269Mb/s to 911Mb/s" /></p>

<p>I’ve been monitoring my internet speed <a href="/2023/07/16/simple-home-server-monitoring-with-prometheus-in-podman/">since I setup Prometheus</a>. After a few days I noticed my new connection will periodically drop from ~900Mb/s to 400Mb/s. I wasn’t sure if this is an issue with the connection, or my server that runs the speed test. It’s running on my less powerful “production” server, a <a href="https://blog.codinghorror.com/the-scooter-computer/">“scooter computer”</a> that I bought in 2017 with cheap 2015-ish era hardware. To narrow it down, I ran a parallel speed test on my (much newer) NUC. This seems to have fewer slowdowns, but it’s hard to draw a definitive conclusion with only a few days of data, and a lot of the other changes to my network would confuse this result.</p>

<p>Speaking of hardware, my quest for gigabit also led me to finally take my home networking hardware seriously.</p>

<p>Up until now, I’ve been using the free router that my ISP gave me almost 7 years ago. It’s a <a href="https://www.tp-link.com/au/service-provider/xdsl/archer-vr1600v/">TP-Link Archer VR1600v</a>, one of the handful of routers that Australian ISPs would give out<sup id="fnref:no-routers" role="doc-noteref"><a href="#fn:no-routers" class="footnote" rel="footnote">1</a></sup> and it’s somewhat <a href="https://www.markhansen.co.nz/doubled-ping/">notoriously bad</a>. It’s never seen a software update; which is not what you want from the device between you and the rest of the internet.</p>

<p>Without a fast internet connection, buying a “real” router just seemed like a waste. As I accumulated more home servers (like <a href="/2023/07/03/picking-a-synology/">my Synology</a>), instead of being an adult and buying a switch to chain multiple devices off the poorly-placed Ethernet ports in my apartment, I used another free ISP router in bridge mode to do the job.<sup id="fnref:positioning" role="doc-noteref"><a href="#fn:positioning" class="footnote" rel="footnote">2</a></sup></p>

<p>My home network looked like this:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> o TP-Link Archer VR1600v Router
 ├─ Wifi devices
 ├─ Steve (Scooter computer)
 ╰─o Technicolor TG789vac v2 Router (Bridge mode)
   ├─ Synology DS420j
   ├─ Brett (Intel NUC)
   ╰─o─ Thinkpad thunderbolt dock
     ╰─ MacBook Air M1 (sometimes)
</code></pre></div></div>

<p>It worked and didn’t require buying any new hardware. I was optimistic about the capability of my Wifi 5 (ie 802.11ac) router, since the maximum speed of an 802.11/ac connection is a bit over a gigabit. Sadly this was not the case, it capped out at about 310Mb/s.</p>

<p>Knowing that my wifi router was likely to be the biggest bottleneck in my home network, I wanted to check that there weren’t any other bottlenecks lurking. It would be especially bad if the wiring in my apartment was a limiting factor, since that would be expensive to replace.</p>

<p>I started by generating a 1 gigabyte file using <code class="language-plaintext highlighter-rouge">head -c 1G /dev/random &gt; testfile</code>, and then starting a webserver with <code class="language-plaintext highlighter-rouge">python -m http.server</code> to see how fast I could download it—and what the download speed the browser reported. I ran this on my dev machine—Brett—and that capped out at 80Mb/s. Shouldn’t this be at least as fast as accessing the internet?</p>

<p>Maybe the Python webserver isn’t able to push data out fast enough? Ideally we’d want a server that is designed to push out data as fast as possible. This is where <a href="https://openspeedtest.com"><em>OpenSpeedTest</em></a> comes in. You can run it in a container and run a speed test across your local network:</p>

<div class="language-console highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="gp">$</span><span class="w"> </span>podman run <span class="nt">-d</span> <span class="nt">-p</span> 3000:3000 <span class="nt">-p</span> 3001:3001 docker.io/openspeedtest/latest
</code></pre></div></div>

<p>You just click the button in the web UI, and you get a speed test.</p>

<p>I tested from my laptop (over wifi) to both Steve (scooter) and Brett (NUC). Steve got the speeds I would expect: 298 down, 345 up. Brett gets 92 down and 364 up. It seems like Python wasn’t the problem after all. The only difference in the two servers is Brett is going through the bridging router and Steve is directly connected to my wifi router. I plugged Brett directly into the main router and got 315Mb/s in both directions.</p>

<p>It’s not clear why exactly the bridge-mode router can accept traffic at over 300Mb/s<sup id="fnref:faster-not-wifi" role="doc-noteref"><a href="#fn:faster-not-wifi" class="footnote" rel="footnote">3</a></sup> but can’t sent traffic that fast back in the other direction. Either way, it was a bottleneck and had to go. A quick trip to the shop and it was replaced with a TP-Link 5-port gigabit switch. Now both Steve and Brett have equally fast connections.</p>

<p>I feel a bit silly not having tested this earlier, because my Synology was also sitting behind the bridge-mode router, and so this would have also been slowing down transfers there. Lesson learnt, I guess. Now the limiting factor will be the wifi, rather than me being too lazy to spend $34 on a switch.</p>

<p>The final puzzle piece is wifi. So far all my tests from my laptop have been wireless, and so the limiting factor has been the crappy old VR1600v.</p>

<p>I consulted with a colleague who helped me navigate the complex world of wifi routers. I realised that I’d never actually bought a router before, there was always just one available somewhere. The two important things I learnt is that you can get a wifi 6 router for about $100, but you might not get your full gigabit out of that. For an extra $50 you can push the maximum speed up an extra gigabit.</p>

<p>The router I ended up getting was the <a href="https://www.tp-link.com/au/home-networking/wifi-router/archer-ax53/">TP-Link A53</a>, largely dictated by what was in stock in shops a walking distance from home. It’s exactly the boring appliance I was hoping for; it’s router shaped, doesn’t require you to download an app, has a handful of ports, and gives my laptop fast internet.</p>

<p>It was tempting to get something super fast, but since none of my devices have faster than gigabit ethernet, having wifi that’s substantially faster than my wired networking seemed like a waste. If I make the jump to 2.5 or 10Gb/s devices, I’ll replace the router with something more capable, but I don’t think that’s likely anytime soon.</p>

<p>Is it worth upgrading your home network? If you’re in Australia and have fibre coming into your house or apartment, go get a gigabit plan and a router to match. If you’ve got a slower connection but have some kind of networked storage? Get a better router and switch, I should have done this years ago. Will this make a meaningful difference in my real-world internet speed? Probably not, everything is hosted in the US or Europe anyway and the speed of light isn’t getting any faster.</p>
<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:no-routers" role="doc-endnote">
      <p>This seems to be much less common now. <a href="#fnref:no-routers" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:positioning" role="doc-endnote">
      <p>To clarify, the router and scooter computer are in the cupboard where the fibre connection terminates, which also has a small ethernet patch panel for the other ports in the apartment. The NUC and Synology sit next to my desk, since they’ve got fans and appreciate having a bit more airflow. There’s only one ethernet port there<sup id="fnref:port-placement" role="doc-noteref"><a href="#fn:port-placement" class="footnote" rel="footnote">4</a></sup> so it has to get split between multiple devices. <a href="#fnref:positioning" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:faster-not-wifi" role="doc-endnote">
      <p>Presumably it would be able to go even faster if I wasn’t limited by the speed of my wifi. <a href="#fnref:faster-not-wifi" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
    <li id="fn:port-placement" role="doc-endnote">
      <p>I want to have a serious word with whoever decided the ethernet port placement in this apartment. There are more ports by my bed than there are at my desk. Are they expecting me to use an alarm clock with wired networking? <a href="#fnref:port-placement" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>


<a href="https://willhbr.net/2024/12/16/going-gigabit/">Permalink</a>
&bull; December 16, 2024
&bull; Send feedback via
  <a href="https://ruby.social/@willhbr" target=_blank>mastodon</a> or
  
  <a href="mailto:feedback@willhbr.net?subject=Feedback:%20Going%20Gigabit" target=_blank>email</a>.


