# rss_item

guid: 604e24039a28aa03ac1f8144
title: MakAir Series: Engineering Focus on Ventilator Pneumatics
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-pneumatics/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics.jpg" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics"><p><em>This is the fourth article <a href="https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/">of a series</a> on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the COVID-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article explains how the MakAir pneumatic circuit works. Without even going into complex electronics and software, pneumatics &#x2014; <em>the air circuit</em> &#x2014; is the first stone to lay in order to build a mechanical ventilator. The pneumatic circuit contains all pipes, actuators and sensors to control the amount of air going into a patient lungs, and is entirely controlled by <a href="https://github.com/makers-for-life/makair-firmware">software</a> running on the MakAir <a href="https://github.com/makers-for-life/makair-motherboard">motherboard</a>.</p><hr><h2 id="assembly-overview">Assembly Overview</h2><p>If you are reading this article to get a quick overview of the MakAir pneumatics, <strong>you may first watch the following video</strong>, with which you will understand the fundamentals of the MakAir pneumatic circuit, and its components.</p><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-overview-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-overview-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-overview-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-overview-video-track.webm">Play the overview video.</a>
</video><!--kg-card-end: html--><hr><h2 id="overall-circuit">Overall Circuit</h2><p>As explained in the article of this series on <a href="https://valeriansaliou.name/blog/makair-series-how-to-ventilate-a-human-patient-101/">how to ventilate a human patient</a>, a mechanical ventilator may, in the case of invasive ventilation, fully take over the functionality of the lungs. Unless the patient gets ventilated under non-invasive ventilation, the ventilator would then rather assist the patient breathing.</p><p>The sheer amount of ventilation modes to support requires <strong>the pneumatic circuit to be as versatile as possible</strong> (meaning: generic), fully operated by software.</p><p>Knowing the human respiratory cycle &#x2014; where at each phase the lungs <strong>inflate with fresh air</strong> (inhale phase), then <strong>retain that air for a few moments</strong> (hold phase), and finally the spoiled <strong>air is discharged from the lungs</strong> (exhale phase) &#x2014; a ventilator pneumatic circuit should have the capability to produce air, push it in the lungs (at a certain pressure), hold it inside the lungs, and finally allow the lungs to deflate.</p><p>Given what is said above, a ventilator pneumatic circuit would need components to <strong>produce air</strong> (a turbine), <strong>measure pressure &amp; volume</strong> (sensors), as well as to <strong>cut off, reduce or increase the air flow</strong> by controlling the pipe aperture (actuators), meaning no air can pass at all or a certain air flow is allowed.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-air-circuit.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The MakAir air circuit, mounted inside a MakAir Model 2. The inspiratory branch is visible at the back, while the expiratory branch is visible at the front.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-pipe.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The standard <a href="https://uk.rs-online.com/web/p/flexible-tubes/4190056/">silicone flexible pipe</a> used in the air circuit (10 mm external diameter). Biocompatible &amp; approved for medical use.</figcaption></figure><p>As explained above, each respiratory cycle is made of three phases: inhale, hold, exhale &#x2014; <em>as described in a previous article of this series on <a href="https://valeriansaliou.name/blog/makair-series-how-to-ventilate-a-human-patient-101/">how to ventilate a human patient</a></em>.</p><p><strong>During each of these three phases, actuators within the air circuit have a specific, changing state, namely:</strong></p><p><strong>1. Inhale</strong>: air must be pushed in the lungs to reach a <code>P(peak)</code> pressure.</p><ul><li><strong>Inspiratory branch</strong>: opened, then closes when <code>P(peak)</code> is reached (<em>positive pressure</em>);</li><li><strong>Expiratory branch</strong>: closed;</li></ul><p><strong>2. Hold</strong>: air must stay in the lungs at a certain pressure, namely <code>P(plateau)</code>, down from <code>P(peak)</code>.</p><ul><li><strong>Inspiratory branch</strong>: closed;</li><li><strong>Expiratory branch</strong>: lets a bit of air out and then closes when <code>P(plateau)</code> is reached (<em>negative pressure</em>);</li></ul><p><strong>3. Exhale</strong>: air is vented out of the lungs to reach a <code>P(expiratory)</code> or <code>PEEP</code> pressure.</p><ul><li><strong>Inspiratory branch</strong>: closed;</li><li><strong>Expiratory branch</strong>: lets a almost all air out and then closes when <code>P(expiratory)</code> is reached (<em>negative pressure</em>);</li></ul><p>Each branch &#x2014; whether inspiratory or expiratory &#x2014; is independent. It contains a set of actuators and sensors. <em>Those components are described in the next sections.</em></p><p>If you need to refer to a schematic of the full pneumatic system, one is provided at the beginning of an article from this series on <a href="https://valeriansaliou.name/blog/makair-series-the-building-blocks-of-a-mechanical-ventilator/">the building blocks of a medical ventilator</a>, although not up-to-date on certain details (eg. the expiratory flow sensor is missing).</p><hr><h2 id="inspiratory-branch">Inspiratory Branch</h2><p>The inspiratory branch is made of the components responsible for inflating the lungs, taking in air from the hospital room, filtering it, pressurizing it, and finally directing it in the patient lungs.</p><p><em>The full list of inspiratory branch components is provided right below, in order.</em></p><h3 id="1-producing-air-with-a-blower-turbine-filter-">1. Producing air with a blower (turbine &amp; filter)</h3><p>First off, a ventilator needs to <strong>gather air from its environment</strong>, <strong>and pressurize it</strong> in order to create a given air flow. A blower turbine is the most convenient component to do that, amongst all available solutions, as it is very compact. One major downside, is that it is noisy, though it can be made silent by caging it in special sound absorbing foam, and noise-insulating its air inlet.</p><p>As the air that gets in comes from the external environment (eg. the hospital room), and gets right in the lungs, <strong>it may contain pollutants</strong>. A filter box must be placed right on the blower inlet. The filter box fits an <a href="https://www.qleanair.com/en/getqlean/hepa-14-filters-for-viruses-and-bacteria">HEPA 14</a> filter, capable of filtering germs, viruses, <a href="https://en.wikipedia.org/wiki/Particulates">particulate matter</a>, and more.</p><p>While designing the MakAir ventilator, we tried building our own blower out of DC motors scavenged from a previous drone project. While successful at making it work and producing the required minute volume and pressure, we were not comfortable with the required biocompatibility certification process, as well as complexity of assembly, noise coming from the motor and long-term reliability. If the turbine fails, the ventilator shuts down, potentially harming or killing the patient. Hence, <strong>we resorted to existing blowers from the market, matching our specifications</strong>.</p><p>Still, we designed the filter box part ourselves, for which CAD models are available. This filter box would ideally be manufactured using SLS 3D printing, as this would make it biocompatible (if properly post-processed, ie. cleaned).</p><p><strong>Filter box:</strong></p><ul><li>&#x27A1;&#xFE0F; <strong>Final inspiratory filter box CAD model</strong>: <a href="https://github.com/makers-for-life/makair-parts/blob/master/src/machine-filter-box">CAD files</a>, <a href="https://a360.co/2x1mhIx">3D preview</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-filter-box-inspiratory.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The inspiratory filter box is made of two 3D printed parts, which are screwed together. This filter box is pressed against the blower inlet with an o-ring gasket.</figcaption></figure><p><strong>Blower:</strong></p><ul><li>&#x27A1;&#xFE0F; <strong>Experimental blower CAD model</strong>: <a href="https://github.com/makers-for-life/makair-parts/tree/master/src/blower">CAD files</a>, <a href="https://a360.co/2JIBr8d">3D preview</a> (<em>reminder: we use an on the shelf blower in our production ventilator</em>).</li><li>&#x27A1;&#xFE0F; <strong>Connector part</strong>: <a href="https://us.intersurgical.com/products/critical-care/elastomeric-soft-connectors#1704000">Intersurgical straight elastomeric connector 22mm ID</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-blower.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The on the shelf blower we picked for the production MakAir. Unfortunately, this reference is private as this blower is not available on the public market. You can easily find alternatives, though.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-blower-3d-printed.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The 3D printed blower we designed, and experimented with. While it fits the minute volume and pressure requirements to ventilate humans, we do not advise using this in any production ventilator due to the noise and (predicted) poor reliability.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-blower-sla-resin.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The 3D printed blower parts, coming out of a <a href="https://formlabs.com/">Formlabs</a> Form 2 printer (before washing &amp; post-processing). The fan part is visible on the bottom left. The casing parts are visible on the top left and top right.</figcaption></figure><hr><h3 id="2-mixing-air-with-oxygen-o2-mixer-">2. Mixing air with oxygen (O2 mixer)</h3><p>Once the ventilator generates the required air flow from its blower, <strong>pure oxygen might need to be mixed</strong> (or might not, depending on the patient condition).</p><p>Pure oxygen comes from the hospital oxygen system. The oxygen concentration (<a href="https://en.wikipedia.org/wiki/Fraction_of_inspired_oxygen">FiO2</a>) can be adjusted with an <a href="https://www.precisionmedical.com/hospital/flowmeters/s-93">oxygen flowmeters</a>. The doctors can then connect the oxygen pipe to the oxygen inlet of the ventilator, which is connected to the internal O2 mixer.</p><ul><li>&#x27A1;&#xFE0F; <strong>O2 mixer part</strong>: <a href="https://us.intersurgical.com/products/critical-care/22mm-straight-connectors-and-mouthpieces#1964001">Intersurgical straight connector 22mm OD - 22mm ID</a>.</li><li>&#x27A1;&#xFE0F; <strong>Connector part</strong>: <a href="https://us.intersurgical.com/products/critical-care/15mm-straight-connectors#1962000">Intersurgical straight connector 15mm OD - 22mm ID</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-o2-mixer.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>An O2 mixer, made of a biocompatible Intersurgical straight connector. Oxygen comes in from the top section, and gets mixed with the unidirectional air flow from the blower.</figcaption></figure><hr><h3 id="3-moderating-egress-air-flow-pinch-valve-">3. Moderating egress air flow (pinch valve)</h3><p>Once the blown air is enriched in oxygen, we need to be able to control the internal diameter of the flexible pipe, in order to <strong>let more or less (or none) air flow pass</strong>. A reduced pipe internal diameter means that the air flow is reduced as well. The pinch valve is responsible for that.</p><p>While typical commercial ventilator use more complex components to achieve the same task, we have chosen to keep things simple: the pinch valve is wrapping around the biocompatible flexible pipe in which the air is passing, and applies a variable force on it in order to reduce or increase its aperture.</p><p>This method is interesting because it does not introduce pneumatic connections in the system, as we did in earlier prototypes of valves through the project (unsuccessfully). This helps ensure that no leak is introduced, and prevents the air from passing through non-biocompatible materials. All we need here, is a way to press against a pipe, and a flexible silicon hose. Simple.</p><p>Here is how it works: <strong>the pinch valves is controlled by software</strong>, and <strong>takes a target angle as its sole input</strong>. An angle relates to how much pressure is applied on the pipe. A Faulhaber DC servo motor rotates to the target angle, which moves two ball bearings and <strong>presses the pipe more or less</strong>.</p><p>We recommend using <strong>industrial-grade high-quality servo motors</strong>, and ditch cheaper hobbyist servo motors, as they would not last the required half a million cycles to ventilate a COVID-19 patient over the course of 2 weeks.</p><p>Ball bearings have been chosen in order to limit friction between materials to a minimum, as the bearings are &#x201C;rolling&#x201D; on the pipe, instead of &#x201C;rubbing&#x201D; against it. <strong>This reduces strain on the DC motor, saves battery and minimizes wear</strong>. In order to apply pressure given an angle, the ball bearings are held on an axis which is not centered, that way rotating the axis across Y (horizontal) would create a variable pressure across Y (vertical).</p><p>You may wonder as well why is the chassis made of stainless steel. We could have chosen traditional plastic instead, though note that in this case the plastic would have to be high-density and rigid. Indeed, the pressure applied is such that we have observed 3D printed plastic parts to change shape and ultimately break. Going to metal was an easy fix for us, but you may as well find a strong plastic as an alternative.</p><ul><li>&#x27A1;&#xFE0F; <strong>Pinch valve DC motor</strong>: <a href="https://www.faulhaber.com/en/products/series/2232bx4/">Faulhaber 2232 S 024 BX4 CSD</a>.</li><li>&#x27A1;&#xFE0F; <strong>Pinch valve bearings (large)</strong>: <a href="https://www.skf.com/us/products/rolling-bearings/ball-bearings/deep-groove-ball-bearings/productid-W%206200-2RS1">SKF 30 mm W 6200-2RS1</a>.</li><li>&#x27A1;&#xFE0F; <strong>Pinch valve bearings (small)</strong>: <a href="https://www.skf.com/us/products/rolling-bearings/ball-bearings/deep-groove-ball-bearings/productid-W%20625-2RS1">SKF 16 mm W 625-2RS1</a>.</li><li>&#x27A1;&#xFE0F; <strong>Pinch valve chassis CAD model</strong>: <a href="https://github.com/makers-for-life/makair-parts/tree/master/src/pressure-valve">CAD files</a>, <a href="https://a360.co/2KZLXeW">3D preview</a>.</li><li>&#x27A1;&#xFE0F; <strong>Pinch valve board PCB model</strong>: <a href="https://github.com/makers-for-life/makair-boards/tree/849aa0baf45ca9caebf3e4f976a62c3f877d9b93/src/pinch-valve-controller">PCB model files</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-pinch-valve.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The MakAir pinch valve. Assembled on a custom-built metal chassis, are a Faulhaber DC motor and a custom PCB, as well as two ball bearings fit on a custom axis.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-pinch-valve-schematic.png" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>A schematic of the MakAir pinch valve, seen from the right side. This shows the offset of the bearings away from the X axis, which enables the bearings to press on the pipe along the Y axis when rotating.</figcaption></figure><hr><h3 id="4-sensing-egress-air-flow-flow-sensor-">4. Sensing egress air flow (flow sensor)</h3><p>In order for the algorithm to to control the air flow and pressure via the pinch valves, it needs to be fed <strong>real-time measurements on the air flow in the circuit</strong>.</p><p>The MakAir includes a Sensirion SFM 3019 mass flow meter sensor near the end of its inspiratory branch, right after the pinch valve. <strong>This sensor provides measurements in liters per minute (of air)</strong>.</p><ul><li>&#x27A1;&#xFE0F; <strong>Mass flow meter sensor</strong>: <a href="https://www.sensirion.com/en/flow-sensors/mass-flow-meters-for-high-precise-measurement-of-gases/high-volume-digital-mass-flow-meter/">Sensirion SFM 3019</a>.</li><li>&#x27A1;&#xFE0F; <strong>Connector</strong>: <a href="https://us.intersurgical.com/products/critical-care/15mm-straight-connectors#1979000">Intersurgical straight connector 15mm ID - 22mm ID</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-flow-sensor-inspiratory.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The Sensirion SFM 3019 mass flow meter (black piece), with a set of Intersurgical connectors fit on it (green pieces).</figcaption></figure><hr><h3 id="5-preventing-spoiled-air-reflux-non-return-valve-">5. Preventing spoiled air reflux (non-return valve)</h3><p>When a sick patient breathes on a mechanical ventilator, <strong>the exhaled air from their lungs could contain viruses or other germs</strong>, that we do not want to spread everywhere in the pneumatic circuit. Especially, if the ventilator is to be re-used on another patient later.</p><p>The <strong>non-return valve lets air pass in a direction</strong>, but not the other. This component, <em>here from Intersurgical</em>, eases the sterilization work of maintenance personnel after use, as it protects the inspiratory branch from contamination. That is, up to the pinch valve, as the blower-to-pinch valve section is protected anyway due to constant positive pressure.</p><p>The expiratory branch, as we will see later, is designed to be contaminated anyway, as there is no other way around it. Though, as the branch is simple and made of a single pipe and a single flow sensor, it is deemed acceptable to throw everything away and replace the whole branch with new parts.</p><p><em>Note that the non-return valve was not shown in the overview video above, as the ventilator functions normally without it. Though, it is very desirable, and could be mandatory for medical certification purposes.</em></p><ul><li>&#x27A1;&#xFE0F; <strong>Non-return valve</strong>: <a href="https://www.intersurgical.com/products/critical-care/directional-valves#1921000">Intersurgical one-way directional valve 22M flow in - 22F flow out</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-valve-non-return.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The Intersurgical non-return valve &#x2014; also called <a href="https://en.wikipedia.org/wiki/Check_valve">check valve</a> &#x2014; used on the MakAir. It is placed at end of the inspiratory branch, and protects all previous components from air reflux.</figcaption></figure><hr><h3 id="6-sensing-egress-air-pressure-pressure-sensor-">6. Sensing egress air pressure (pressure sensor)</h3><p>Just as with the air flow sensor, <strong>algorithms need to know the current pressure in the lungs</strong>, using a pressure sensor. This sensor is mounted on the motherboard, and a small pipe connects it to the air circuit.</p><p>As this sensor does not measure the pressure in the inspiratory branch, but rather in the lungs, it needs to be located after the non-return valve. That is, <strong>as near as possible to the patient lungs</strong>, if we were to look at a schematic of the pneumatic circuit.</p><p>An Intersurgical-made biocompatible straight connector is used to connect the pressure sensor component on the motherboard, to the air circuit. It is very similar to the Intersurgical part used for the O2 mixer.</p><ul><li>&#x27A1;&#xFE0F; <strong>Pressure extractor</strong>: <a href="https://us.intersurgical.com/products/critical-care/22mm-straight-connectors-and-mouthpieces#1966001">Intersurgical straight connector 22mm OD - 22mm OD</a>.</li><li>&#x27A1;&#xFE0F; <strong>Exit fit</strong>: <a href="https://us.intersurgical.com/products/critical-care/22mm-straight-connectors-and-mouthpieces#1961000">Intersurgical straight connector 22mm OD - 22mm ID</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-pressure-sensor.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The pressure sensor, mounted on the MakAir motherboard. A small grey pipe is connected on it, and then fitted to a larger silicon pipe that is connected on the Intersurgical part.</figcaption></figure><hr><h2 id="expiratory-branch">Expiratory Branch</h2><p>The expiratory branch is made of the components responsible for deflating the lungs, taking in air from the lungs, filtering it, and exhausting it to the hospital room.</p><p><em>The full list of expiratory branch components is provided right below, in order.</em></p><h3 id="1-sensing-ingress-air-flow-flow-sensor-">1. Sensing ingress air flow (flow sensor)</h3><p>As with the egress air flow on the inspiratory branch, <strong>the ingress air flow must be measured on the expiratory branch</strong>, and fed to the algorithm. There, a Sensirion SFM 3300D digital flow meter is used.</p><p>As air coming from the patient lungs circulate within the expiratory branch, all components on its path could be contaminated with viruses and germs. This means that <strong>this branch would need to be sterilized after each use on a patient</strong>, or more simply put, <strong>all the components on that branch could be thrown away and replaced with brand new components</strong>.</p><p>This sensor is fortunately much less expensive than the inspiratory branch sensor, costing about 40 EUR, and thus we recommend that it is replaced after each use.</p><p>To ease with replacement and maintenance, a 3D printed connector hat was designed, which snaps onto the sensor and can be easily removed. The connector hat contains a PCB which presses electrical connectors on the sensor PCB, helping connecting the sensor to the motherboard.</p><p><strong>Sensor:</strong></p><ul><li>&#x27A1;&#xFE0F; <strong>Digital flow meter sensor</strong>: <a href="https://www.sensirion.com/en/flow-sensors/mass-flow-meters-for-high-precise-measurement-of-gases/proximal-flow-sensors-sfm3300-autoclavable-washable-or-single-use/">Sensirion SFM 3300D</a>.</li><li>&#x27A1;&#xFE0F; <strong>Entry fit</strong>: <a href="https://us.intersurgical.com/products/critical-care/22mm-straight-connectors-and-mouthpieces#1961000">Intersurgical straight connector 22mm OD - 22mm ID</a>.</li><li>&#x27A1;&#xFE0F; <strong>Connector</strong>: <a href="https://us.intersurgical.com/products/critical-care/15mm-straight-connectors#1962000">Intersurgical straight connector 15mm OD - 22mm ID</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-flow-sensor-expiratory.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The Sensirion SFM 3300 digital flow meter, without its hat. You can see the Sensirion PCB with the electrical fits.</figcaption></figure><p><strong>Connector hat:</strong></p><ul><li>&#x27A1;&#xFE0F; <strong>Connector hat CAD model</strong>: <a href="https://github.com/makers-for-life/makair-parts/blob/master/src/mass-flow-meter-cap">CAD files</a>, <a href="https://a360.co/39bc62M">3D preview</a>.</li><li>&#x27A1;&#xFE0F; <strong>Connector hat PCB model</strong>: <a href="https://github.com/makers-for-life/makair-boards/tree/849aa0baf45ca9caebf3e4f976a62c3f877d9b93/src/sfm-3300-adapter/boards/V1">Eagle files</a>.</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-flow-sensor-hat.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>The MakAir SFM hat, with its PCB. Notice that the electrical pins match the fits on the Sensirion PCB.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-flow-sensor-hat-3dp.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"><figcaption>Eight 3D printed MakAir SFM hats, ready for use, in a Form Cure from <a href="https://formlabs.com/">Formlabs</a>.</figcaption></figure><hr><h3 id="2-moderating-ingress-air-flow-pinch-valve-">2. Moderating ingress air flow (pinch valve)</h3><p>This section works exactly as the one with the pinch valve on the inspiratory branch. The ingress air flow needs to be moderated using a pinch valve as well.</p><p><em>As the pinch valve is exactly the same as on the inspiratory branch, please refer to explanations and pictures from the inspiratory pinch valve section above.</em></p><hr><h3 id="3-venting-out-ingress-air-flow-filter-">3. Venting out ingress air flow (filter)</h3><p>During the exhale phase, air is vented outside of the patient lungs into the expiratory circuit, and ultimately out of the ventilator.</p><p>Given that <strong>the air might be contaminated</strong>, <strong>it needs to be filtered</strong> using the same HEPA 14 filter as used on the inspiratory filter.</p><p>Note that this expiratory filter box is almost the same as the one from the inspiratory branch. The connector is different, as the expiratory filter box connects to the flexible pipe.</p><ul><li>&#x27A1;&#xFE0F; <strong>Final expiratory filter box CAD model</strong>: <a href="https://github.com/makers-for-life/makair-parts/blob/master/src/machine-filter-box">CAD files</a>, <a href="https://a360.co/2XeSVAP">3D preview</a>.</li></ul><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-pneumatics-filter-box-expiratory.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Pneumatics" loading="lazy"></figure><hr><h2 id="closing-notes">Closing Notes</h2><p>This article was the fourth of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>Coming up with the air circuit described in this article could only be achieved by taking a trial and error approach. The MakAir engineering team started experimenting with 3D printed parts (eg. the O2 mixer) but ultimately settled on <strong>using on the shelf biocompatible parts</strong> manufactured by <a href="https://www.intersurgical.com/">Intersurgical</a>, except for some parts which are not in contact with air (eg. the 3D printed expiratory flow sensor hat, or the CNC-machined pinch valve metal chassis).</p><p><strong><em>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</em></strong></p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 60583e91596c5b03eededcc1
title: MakAir Series: Engineering Focus on Ventilator Electronics
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-electronics/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics.jpg" alt="MakAir Series: Engineering Focus on Ventilator Electronics"><p><em>This is the fifth article <a href="https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/">of a series</a> on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the COVID-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article explains how the MakAir electronics work. Electronics are the center piece of the MakAir ventilator, as they interconnect software &amp; algorithms with pneumatics &amp; actuators. As well, they handle all user controls, eg. changing an option on the touchscreen. While early MakAir prototypes ran on development boards, we quickly moved to custom-built production-ready boards, allowing us to fit all required components on a single PCB.</p><hr><h2 id="assembly-overview">Assembly Overview</h2><p>If you are reading this article to get a quick overview of the MakAir electronics, <strong>you may first watch the following video</strong>, with which you will understand what makes the MakAir electronics, and how they work together.</p><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-overview-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-overview-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-overview-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-overview-video-track.webm">Play the overview video.</a>
</video><!--kg-card-end: html--><hr><h2 id="the-big-picture">The Big Picture</h2><p>We have described the pneumatics system in the <a href="https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-pneumatics/">previous article</a>, which is made of <strong>passive components</strong> (eg. pipes), as well as <strong>active components</strong> (sensors and actuators) that need to be controlled by some central brain. This is where electronics come into play.</p><p>The MakAir ventilation software &#x2014; <em>we call it the firmware</em> &#x2014; runs on a custom motherboard, that houses a STMicroelectronics <a href="https://www.st.com/en/microcontrollers-microprocessors/stm32f411.html">STM32F411</a> microcontroller (or <a href="https://en.wikipedia.org/wiki/Microcontroller">MCU</a>), as well as input/output ports for sensors and actuators. This piece of electronics is the sole responsible for running the ventilation software, which acts &#x2014; <em>in worst cases</em> &#x2014; as a replacement diaphragm for the patient lungs for approximately 2 weeks. It must not fail, or the patient will be in serious trouble: life-threatening style, until medical personnel intervenes.</p><p>We made sure to choose electronic components that were known for their reliability and stability, as well as simplicity. For instance, <strong>the MakAir motherboard does not run any operating system</strong>; the <a href="https://github.com/makers-for-life/makair-firmware">MakAir Firmware</a> <strong>runs directly</strong> on the CPU with no abstraction layer (<em>our firmware boots at memory address 0</em>).</p><p>On the other hand, as doctors are used to modern commercial ventilators that <strong>can be monitored and configured over a large touch screen</strong>, we decided to add one to the MakAir as well. This large touch screen requires a complex desktop-class CPU, GPU, video drivers, as well as an operating system, which make this device less reliable, by design &#x2014; <em>unless we were to write image frames in the <a href="https://en.wikipedia.org/wiki/Framebuffer">framebuffer</a>, skipping abstraction layers</em>. We chose to <strong>use commodity electronics</strong> to build this part of the ventilator, as the market is already full of highly-efficient ARM-based boards. In this specific case we chose the Raspberry Pi 4.</p><p>This touchscreen computer runs the <a href="https://github.com/makers-for-life/makair-control-ui">MakAir Control UI</a>, which talks to the firmware via the <a href="https://github.com/makers-for-life/makair-telemetry">MakAir Telemetry Protocol</a>, over a bidirectional serial connection. <strong>The MakAir Telemetry Protocol acts as the interface between the MakAir Firmware and the MakAir Control UI</strong>, and lets the touch UI commit ventilation settings changes, while the firmware feeds the UI with real-time ventilation metrics, that are used to show live graphs of air pressure and air flow.</p><p>It is <strong>particularly important that those two computing units are independent</strong>, as one is clearly more critical than the other, pretty much like how the drive computer and the multimedia computer are split in a Tesla. When a failure on the touch UI computer would be inconvenient at worst, <strong>a failure on the ventilator motherboard is definitely a life-threatening situation</strong>.</p><p>All those power-hungry electronics could not work without proper DC current. Given how unstable the AC power grid can be in certain areas of the world, and that a patient may need to be moved rooms or transported on the road, any ventilator must provide backup power in case AC is unplugged or fails. The MakAir comes equipped with a 60W AC-DC 24V power supply, capable of powering the ventilator and charge the two onboard 12V 7Ah lead-acid batteries <em>at the same time</em>. Those <strong>batteries can provide backup power to the ventilator for up to 4 hours</strong>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-all.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>All MakAir electronics, laying on a table. Put them all together, and you get a working ventilator. All components are commodity electronics that can be found online and in stores, except from the motherboard and the RTC board which we designed.</figcaption></figure><hr><h2 id="motherboard">Motherboard</h2><p>The MakAir motherboard is a custom-built <a href="https://en.wikipedia.org/wiki/Printed_circuit_board">printed circuit board</a>, <strong>dedicated to running the firmware &amp; ventilation algorithms</strong>, and allowing software to <strong>gather realtime data from sensors and control pneumatic actuators</strong>. It also serves other functions, such as handling user input through hardware buttons, or emitting high-pitched noises through a buzzer whenever an alarm is triggered.</p><p>All credits for the MakAir motherboard design go to <a href="https://fr.linkedin.com/in/vincent-le-cunff-343327a5">Vincent Le Cunff</a> from <a href="https://www.tronico-alcen.com">Tronico</a>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-motherboard.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>Revision 3 of the MakAir motherboard, on which all major I/O are labelled. This motherboard is used on the MakAir Model 2 ventilator. Note the black &amp; white display, which is legacy and should be removed in next revisions, as the touchscreen supersedes it.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-motherboard-render.png" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>A 3D render of the MakAir motherboard, which can be viewed <a href="https://a360.co/3eaMoN0">in your browser here</a>.</figcaption></figure><hr><h3 id="motherboard-revisions">Motherboard revisions</h3><p>While initially the MakAir team has been working on development boards &#x2014; an Arduino Uno, then a <a href="https://www.st.com/en/evaluation-tools/nucleo-f411re.html">Nucleo F411RE</a> &#x2014; it was soon time to design our own motherboard PCB, integrating a <a href="https://www.st.com/en/microcontrollers-microprocessors/stm32f411re.html">STM32F411RE</a> MCU as on the Nucleo. That motherboard went into 3 design revisions.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-motherboard-revisions.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>Going from revision 1 (left) to revision 3 (right) of the MakAir motherboard design. The first revision did not implement an onboard MCU, but instead relied on a Nucleo for computing power. The last revision integrated the STM32F411RE MCU, hence it did not require the Nucleo board anymore.</figcaption></figure><p>In addition to running the firmware, the motherboard provides <strong>user interaction via hardware buttons</strong>, though we are slowly deprecating those in favor of the touch UI. As well, an alarm buzzer and LEDs are mounted on the board to provide audible and visible feedback whenever an alarm is raised, eg. for instance a target pressure is not reached, which is abnormal and thus for which human intervention might be required. Here as well, we may deprecate the LEDs in favor of the touch UI, though the buzzer is staying.</p><p>In the latest revision, we added a <a href="https://en.wikipedia.org/wiki/LoRa">LoRa</a> wireless module, allowing the firmware to broadcast telemetry data to a receiver device. Though unused at the moment, we got the idea that building a centralized monitoring dashboard could be much appreciated in pandemic situations, where too many patients would be ventilated under the oversight of too few doctors. <strong>Hundreds of MakAir ventilators could be used at the same time, with their alarms and metrics being transmitted realtime to a &#x201C;control center&#x201D; room in the hospital</strong>. LoRa (for Long Range), is capable of transmitting information with a range of 10 kilometers, at a low data transfer rate (from 0.3 kbit/s to 27 kbit/s). It doesn&#x2019;t require relaying infrastructure like WiFi, as in our use case it is simply made of N transmitters (MakAir ventilators) sending to 1 receiver (MakAir monitoring dashboard).</p><p>Note that a black &amp; white display is mounted on the motherboard. While today&#x2019;s MakAir ventilators use a touchscreen, the monochromatic screen is still used during the production process to perform end-of-line tests &amp; calibrations. This display will go away in next motherboard revisions, in a similar fashion to the hardware buttons.</p><hr><h3 id="motherboard-schematics">Motherboard schematics</h3><p>The electrical schematics for the motherboard <a href="https://github.com/makers-for-life/makair-motherboard/blob/5f22867dbabc98d7a29f48b42c1a5f4ddd8c4fb7/src/schematics/V3/Electrical%20Schematics.pdf">are open-source</a>, which let anyone make a list of required components, or reproduce and improve the existing motherboard with specialized electronic design software.</p><p>As well, the motherboard <a href="https://en.wikipedia.org/wiki/Gerber_format">Gerber</a> files <a href="https://github.com/makers-for-life/makair-motherboard/tree/7c8bef7288f532eec5beec81997939fd6b86adde/src/gerber/V3">available in our repository</a>. They can be used to send our current design to PCB factories for production.</p><p>The list of components to solder on the PCB is available through the schematics. They can be applied on the PCB by a <a href="https://en.wikipedia.org/wiki/Pick-and-place_machine">pick-and-place machine</a>. You can also use the <a href="https://github.com/makers-for-life/makair-motherboard/blob/c2fc20c9bbde774d00185fa5e59260fa42d64d74/src/bom/V3/Bill%20of%20Materials.pdf">Bill of Materials</a> that we published to proceed with bulk-ordering all required components.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-motherboard-gerber.png" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>An imprint of a <a href="https://en.wikipedia.org/wiki/Gerber_format">Gerber</a> file for the MakAir motherboard PCB. Routes can be easily spotted. The isolated area on the bottom right of the board is the USB input/output interface.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-motherboard-schematics-stm32.png" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>An extract of the MakAir motherboard <a href="https://github.com/makers-for-life/makair-motherboard/blob/5f22867dbabc98d7a29f48b42c1a5f4ddd8c4fb7/src/schematics/V3/Electrical%20Schematics.pdf">electrical schematics</a> showing the MCU wiring diagram. You can see that the MCU has direct control on the LEDs, buzzer, and pinch valve motors via its outputs, and gets sensor measurements via its inputs.</figcaption></figure><hr><h3 id="programming-interface">Programming interface</h3><p>In order to flash a firmware to the motherboard, a programming interface is required. <strong>The MakAir motherboard offers two ways to flash a firmware</strong>: either use a STM32 programmer on <a href="https://en.wikipedia.org/wiki/JTAG">JTAG</a> ports, or commit it over USB. We tend to prefer using the programmer from the Nucleo development board, which we broke apart from a Nucleo board, then we connect it to the motherboard JTAG ports, and finally we plug it on a computer over USB and upload the firmware with <a href="https://www.st.com/en/development-tools/stm32cubeide.html">STM32Cube</a>.</p><hr><h3 id="wiring-to-sensors-actuators">Wiring to sensors &amp; actuators</h3><p>In order to connect all sensors (eg. flow sensors) and actuators (eg. pinch valves, blower), a <a href="https://en.wikipedia.org/wiki/Cable_harness">cable harness</a> should be built, and would connect to the motherboard via a single connector. Note tough that the two flow sensors connect to an individual extension port each.</p><p>The blower, while controlled by the motherboard, is not directly connected to it. As the motherboard would be sending a <a href="https://en.wikipedia.org/wiki/Pulse-width_modulation">PWM</a> signal with the target blower speed, this target value would need to be converted to a 3-phase signal that would make the blower DC motor spin. That is the job of the ESC board, which takes in a speed target via a PWM signal, as well as a power source (high amperage line).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-flow-sensor.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The egress and ingress flow sensors are connected to the two extension ports on the motherboard. Note that the ingress flow sensor (SFM 3300D) is optional, while the egress flow sensor (SFM 3019) is required for the MakAir firmware to boot up, as algorithms rely on its measurements.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-pinch-valve.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The two pinch valve PCBs are connected to the motherboard over the cable harness. They get power and target angle commands from the firmware over those connectors.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-esc.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The blower <a href="https://en.wikipedia.org/wiki/Electronic_speed_control">ESC</a> is connected to the motherboard over the cable harness. It takes a target speed input from the motherboard and creates a 3 phase signal output to the blower. It draws its current from the power line directly (ie. not from the motherboard given the high amperage required). Please ignore the cooling fan in the foreground, as it is not connected to the ESC and thus is unrelated.</figcaption></figure><hr><h2 id="control-ui">Control UI</h2><p>The MakAir Control UI runs on <strong>a second computer board, which is connected to the motherboard</strong> over a GPIO ribbon cable. As it might be less reliable than the motherboard due to its advanced hardware and software complexity, the probability of failures is much higher, and thus in the event it fails, ventilation algorithms must still go on.</p><h3 id="system-design">System design</h3><p>We chose a <a href="https://www.raspberrypi.org/products/raspberry-pi-4-model-b/">Raspberry Pi 4</a> to run the Control UI software, which is plugged to a <a href="https://www.raspberrypi.org/products/raspberry-pi-touch-display/">7&#x201D; Raspberry Pi Touch Display</a>.</p><p>The Raspberry is powered by the motherboard itself. In order to address possible failures, it sends periodic &#x201C;heartbeats&#x201D; to the firmware, which is able to run a power cycle sequence if it misses an heartbeat announcement, effectively rebooting the Control UI computer electrically.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-rpi.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The <a href="https://www.raspberrypi.org/products/raspberry-pi-4-model-b/">Raspberry Pi 4</a>, which runs the MakAir Control UI, is connected to a <a href="https://www.raspberrypi.org/products/raspberry-pi-touch-display/">7&#x201D; Raspberry Pi Touch Display</a>. It is then connected to the motherboard with a GPIO ribbon cable, which feeds it with power and serial data.</figcaption></figure><hr><h3 id="ventilator-black-box">Ventilator black box</h3><p>When testing new ventilation algorithms on a live patient, or during clinical trials, anomaly events might be reported by medical personnel and may require further investigation by software or hardware engineers.</p><p>The Control UI computer allows for an USB key to be plugged on one of the four USB ports of the Raspberry Pi. <strong>Whenever an USB key gets plugged in, the system detects it and starts recording telemetry logs</strong>. This USB key effectively acts as a &#x201C;black box&#x201D;.</p><p>Those telemetry logs can be retrieved later on, and replayed on a simulated Control UI on a computer. It also indicates the exact date and time at which an hardware button was pressed to adjust a setting, eg. increase a target pressure value.</p><p><em>Note that telemetry recordings are actually a raw dump of the very same telemetry protocol used over serial between the motherboard and the touch UI.</em></p><hr><h3 id="keeping-track-of-time">Keeping track of time</h3><p>As the Linux system running on the Raspberry is not connected to the Internet, and thus cannot reach a <a href="https://en.wikipedia.org/wiki/Network_Time_Protocol">NTP</a> server whenever it boots up, its internal clock will lose track of time whenever the MakAir is powered off. Recovering the actual date and time can be important when recording ventilation logs to the ventilator black box (ie. the USB key).</p><p>To address this issue, we designed a custom Raspberry Pi RTC module that plugs to the Raspberry Pi GPIO port (RTC comes for <a href="https://en.wikipedia.org/wiki/Real-time_clock">Real-Time Clock</a>). It can be polled by the Linux system during boot to <strong>retrieve the actual date and time from the RTC hardware clock</strong>. A CR2032 battery powers the RTC chip, which keeps track of time when electronics are powered off.</p><p>The <a href="https://github.com/makers-for-life/makair-boards/tree/849aa0baf45ca9caebf3e4f976a62c3f877d9b93/src/raspberry-pi-rtc">source files</a> for this board are open-source. Credits for the RTC PCB design go to <a href="https://github.com/pi-r-p">Pierre Papin</a>, from the MakAir team.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-rtc.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The Raspberry Pi RTC hat which we designed. It keeps track of time whenever the MakAir is powered off, and provides the Raspberry Pi with the current date and time whenever its Linux system boots up. It is sandwiched between the Raspberry Pi GPIO socket, and the GPIO ribbon cable that comes from the motherboard.</figcaption></figure><hr><h2 id="power-battery">Power &amp; Battery</h2><p>Mechanical ventilators, while mostly used stationary and thus plugged to AC power, must allow medical personnel to continue ventilating the patient as it is being moved from place to place, and thus should be able to fallback to battery DC power.</p><p>The MakAir ventilator is powered by a medical-grade <a href="https://medical.meanwell.com/webapp/product/search.aspx?prod=RPS-60">Mean Well RPS-60 24V</a> power supply. It feeds the electronics with a stable 24V DC voltage at 2.5A maximum, while at the same time charging the two 12V 7Ah lead-acid batteries.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-electronics-power.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Electronics" loading="lazy"><figcaption>The MakAir power components. The <a href="https://www.meanwell.com/">Mean Well</a> power supply can be seen on the left, while the two lead-acid batteries can be seen on the right. An ON/OFF kill switch is mounted at the rear of the ventilator. Spot the AC expander, which lets the firmware know if the ventilator is AC-powered or battery-powered (via a boolean high/low state).</figcaption></figure><hr><h3 id="addressing-current-peaks">Addressing current peaks</h3><p>When running, all ventilator components put together draw a mean current of 1.2A (~30W at 24V), which the power supply can happily deliver.</p><p>Whenever algorithms decide to increase the blower speed &#x2014; either due to a change in lung compliance or in the target tidal volume value &#x2014; the blower may draw too high of a current than the power supply is able to deliver, <em>for a few milliseconds</em>. This event can be quite catastrophic if the batteries are not available to deliver the spare current, as the voltage would drastically drop within the system, <strong>which would trigger a general system reboot</strong>.</p><p>During clinical trials, unfortunately such an event occurred due to medical personnel starting ventilating a patient without connecting the batteries inside the MakAir. After some hours of successfully ventilating the patient, a sudden blower speed increase event occurred, to maximum throttle &#x2014; <em>due to the patient coughing</em>. As you&#x2019;d expect, the ventilator power cycled itself, which hopefully the medical team handled quickly enough.</p><p>In order to address this &#x2014; <em>let&#x2019;s admit it, rare case</em> &#x2014; we implemented a progressive blower speed ramp up algorithm, <strong>spreading the speed increase over a timeframe of 2-3 seconds</strong>. This has been tested to limit the current draw of the blower to an acceptable level, effectively addressing the issue.</p><hr><h3 id="choosing-lead-acid-over-lithium-ion">Choosing lead-acid over Lithium-ion</h3><p>Given the superior energy density of Lithium-ion batteries over lead-acid, one might be tempted to choose them over the latter.</p><p>While more modern and more compact than their older counterparts, <strong>Lithium-ion batteries require more electronics</strong> to operate in a safe way; they require a <a href="https://en.wikipedia.org/wiki/Battery_management_system">Battery Management System</a> (BMS), increasing complexity and cost.</p><p>In comparison, <strong>lead-acid batteries are happy being wired directly</strong> to their power source (the AC-DC transformer) and their load (the motherboard, blower and others).</p><p>As well, lead-acid batteries are <strong>very common and affordable</strong>. The 12V batteries used in the MakAir ventilator can be found in any battery shop, and can even be scavenged on scooters. In comparison, Lithium-ion batteries are harder to source. This comes especially true as you go to remote area throughout the world.</p><p>Adding to that, airline safety regulations regarding the transportation of Lithium-ion batteries is quite stringent. Given that in pandemic situations MakAir ventilators could be shipped by thousands in cargo planes, Lithium-ion seemed like an obvious no-go.</p><hr><h3 id="estimating-the-state-of-charge-soc">Estimating the state of charge (SoC)</h3><p>Being a medical ventilator, the MakAir must be capable of functioning on battery while ventilating a patient <strong>for at least 30 minutes</strong>. Our measurements indicated that the MakAir ventilator could go well above that, <strong>reaching the 4 hours autonomy mark</strong>.</p><p>In order to provide the end-user with a quick snapshot of the battery <a href="https://en.wikipedia.org/wiki/State_of_charge">state of charge</a> (SoC) through the UI, expressed in remaining battery percentage (eg. 93% battery left), we needed to characterize the battery discharge rate on a known load (eg. 1A).</p><p>The only reading letting us approximate the state of charge of the battery is a live voltage measurement. This voltage measurement, while indicative, can be very unreliable, as slight voltage drops can occur eg. if the blower ramps up and pulls more current, which could result in an approximated SoC that would be much lower than it really is. A more reliable voltage measurement would be an open-circuit one, performed after letting the batteries cool down a bit and giving time for their chemistry to stabilize. Note that in practice, <em>we cannot do so</em>, as the ventilator is running non-stop.</p><p>After trial and error, as well as performing a <a href="https://github.com/makers-for-life/makair-motherboard/blob/2045e8a381580d1ccf87689391af9ae7623d6b63/doc/research/PbAc%202S-12V-7Ah%20Kt/pbac-kt_v-cmv.csv">full discharge characterization</a> of the batteries of a MakAir under a typical load, <strong>we came up with a software-based &#x201C;BMS&#x201D; capable of estimating the SoC</strong> at any time, based on voltage readings and approximated current draw of the system given the blower speed. You can <a href="https://github.com/makers-for-life/makair-control-ui/blob/22d4035746ba18a90e6da3891ff57b20e6b6b3dd/src/utilities/battery.rs#L11">read its source code here</a>.</p><hr><h3 id="detecting-when-ac-power-is-unplugged">Detecting when AC power is unplugged</h3><p>The MakAir UI should <strong>show an alarm whenever the ventilator is unplugged from AC power</strong>, and is thus on backup power.</p><p>As the system voltage is a bit higher when AC power is connected than when on batteries, we traditionally used live voltage measurements to speculate as to which power source was in use.</p><p>This method was not very reliable; it sometimes reported that the ventilator was still on AC, although it was running on battery power (SoC at 100%).</p><p>In order to address this (<em>slight</em>) issue, we drew a cable from the power supply to the motherboard, which we call the expander. This cable reports the power supply output DC voltage, independent from the batteries. That DC output would be 0V if the power supply was not powered over AC. The firmware can then read this voltage value, and use it as a high/low boolean state to show which power source is being used.</p><hr><h2 id="closing-notes">Closing Notes</h2><p>This article was the fifth of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>The whole process of designing the electronics of the MakAir was an iterative one. While the team started building on an Arduino Uno board, it then moved to a <em>more reliable</em> Nucleo board, and finally <strong>it released 3 revisions of the custom-built MakAir motherboard PCB</strong>. That, only to end up adding one more board with computing power, &#x2014; <em>the control UI</em>. This final design is still the one we have today, though <strong>we may have some plans to integrate the Raspberry Pi board on the motherboard itself</strong>, in a dual CPU architecture (one for the firmware, one for the UI).</p><p><strong><em>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</em></strong></p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 6063913eab07df03b5260dc5
title: MakAir Series: Engineering Focus on Ventilator Software
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-software/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/03/makair-software-banner.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-banner.jpg" alt="MakAir Series: Engineering Focus on Ventilator Software"><p><em>This is the sixth (and last!) article <a href="https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/">of a series</a> on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the COVID-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article explains how the MakAir software works. While a MakAir ventilator is rather simple in its <a href="https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-pneumatics/">pneumatics</a> &amp; <a href="https://valeriansaliou.name/blog/makair-series-engineering-focus-on-ventilator-electronics/">electronics</a>, its complexity lies in software. Pretty much like a Tesla, there exists tens of different MakAir hardware revisions and builds, though they all still receive software updates <em>as of today</em>. As the marginal cost of software tends to zero, keeping MakAir ventilators up-to-date with latest algorithms and safety improvements is basically free, even years after a ventilator unit was built.</p><p>The MakAir software has been split into two subsystems with different reliability guarantees: the <strong>ventilation firmware</strong> (<em>it keeps the patient alive</em>), and the <strong>control UI</strong> (<em>it allows ventilation to be monitored, and settings to be adjusted</em>). Let&#x2019;s find out how they function and interconnect!</p><hr><h2 id="setup-overview">Setup Overview</h2><p>If you are reading this article to get a quick overview of the MakAir software, <strong>you may first watch the following video</strong>, with which you will understand how to build the MakAir firmware, how to flash it, as well as which components run which part of the software.</p><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-overview-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-overview-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-overview-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-overview-video-track.webm">Play the overview video.</a>
</video><!--kg-card-end: html--><hr><h2 id="firmware-ventilation-algorithms">Firmware &amp; Ventilation Algorithms</h2><p>The <a href="https://github.com/makers-for-life/makair-firmware">firmware</a> is the brain of the MakAir: it has the responsibility of <strong>keeping the patient alive</strong>, by providing enough air to their lungs (<em>and not too much</em>). The firmware centralizes input from all sensors (pressure &amp; flow), and outputs commands to actuators (eg. <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/pressure_valve.cpp">pinch valves</a>, <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/blower.cpp">blower</a>, <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/alarm_controller.cpp">alarms</a>). It implements <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/main_state_machine.cpp">ventilation algorithms</a>, and ensures <strong>patient ventilation stays stable</strong> under the configured minimum targets and safety limits. Things are kept simple, meaning that the firmware has a single task, and does it in the most reliable way possible. If it fails at its mission, the patient dies, <em>period</em>.</p><p><strong>Reliability comes from simplicity</strong>, as we can analyze all layers of complexity that are necessary for the system to function. It is the reason why the firmware does not run on any operating system. It is an embedded system built in C++, executed with a real-time computing constraint. The MCU it runs on is an ARM-based <a href="https://www.st.com/en/microcontrollers-microprocessors/stm32f411.html">STM32F411</a> microcontroller, clocked at 100 MHz. <em>Code ought to be as efficient as possible, in order not to overload this tiny processor</em>.</p><p>The ventilation algorithms make an intensive use of <a href="https://en.wikipedia.org/wiki/PID_controller">Proportional&#x2013;Integral&#x2013;Derivative controllers</a> (PID) when eg. calculating the angle of the pinch valves, resulting in more or less air passing in either the inspiratory or expiratory branch, until the sensor measurements stabilize to the desired target value.</p><h3 id="building-the-firmware">Building the firmware</h3><p>The firmware can be compiled from <code>arduino-cli</code> (though it does not target an Arduino board), in a single command:</p><pre><code>arduino-cli compile --fqbn STM32:stm32:Nucleo_64:opt=o3std,pnum=NUCLEO_F411RE --verbose srcs/respirator.cpp --output output/respirator-production
</code></pre><p>This command produces a binary file, which can then be flashed onto the target board.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-motherboard.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Software" loading="lazy"><figcaption>The MakAir motherboard V3, integrating the STM32F411 MCU on which the firmware runs, as well as all the I/O available from the firmware.</figcaption></figure><hr><h3 id="flashing-a-firmware-build">Flashing a firmware build</h3><p>As explained in the video, flashing a MakAir motherboard (V3 revision), can be done by <strong>plugging a Nucleo F411RE programmer to the MCU JTAG port</strong> on the motherboard.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-nucleo.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Software" loading="lazy"><figcaption>The Nucleo F411RE programmer can be broken apart from the Nucleo. You may discard the rest of the board.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-programmer.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Software" loading="lazy"><figcaption>The Nucleo STM32 programmer is plugged to our computer over USB, as well as the motherboard MCU over JTAG. The command <code>arduino-cli board list</code> should show the programmer serial identifier to be used when flashing.</figcaption></figure><p>Flashing can be done in a single command &#x2014; once firmware binaries have been compiled &#x2014; by passing the programmer serial port identifier for <code>{SERIAL_PORT}</code>:</p><pre><code>arduino-cli upload --port {SERIAL_PORT} --fqbn STM32:stm32:Nucleo_64:pnum=NUCLEO_F411RE,upload_method=swdMethod --input output/respirator-production
</code></pre><p>Note that <a href="https://www.st.com/en/ecosystems/stm32cube.html">STM32Cube drivers</a> are required.</p><p>Flashing the firmware this way is basically how we iterated on developing the firmware source code. We could simply leave our computer connected to the programmer, re-executing the compile &amp; flash commands when needed. A flash operation only takes 1-2 seconds.</p><hr><h3 id="hardware-revision-compatibility">Hardware revision compatibility</h3><p>During the first months of the project, the hardware team iterated on 3 motherboard revisions, tens of assemblies, packing different flow sensors, adding new components along the way. We introduced <strong>conditional compilation variables</strong>, in order, say, to be able to build the same firmware version for a Honeywell flow sensor or a Sensirion flow sensor.</p><p>This means that the current MakAir firmware supports a wide range of sensors and actuators alternatives, that perform the same function within the system (eg. <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/mass_flow_meter.cpp">measure air flow</a>, or <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/pressure_valve.cpp">apply pressure</a> on the air pipes). As well, some sensors such as the expiratory air flow sensor might be missing on certain hardwares, in which case the firmware will use an estimated value instead (based on measured pressure and measured inspiratory air flow).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-team-and-hardware-revisions.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Software" loading="lazy"><figcaption>The MakAir team on 24th April 2020, with all hardware revisions produced so far. The firmware was compatible with (almost) all visible MakAir versions, though note we did not have a touchscreen hardware version yet. Photo credits: <a href="https://abacapress.com">ABACAPRESS</a>.</figcaption></figure><hr><h3 id="regulatory-safety">Regulatory &amp; safety</h3><p>As use of firmware improvements on human patients is subject to regulatory approval, proper firmware versioning is an important matter. Simply put, a MakAir should only be used on a patient if the firmware version that was flashed onto its MCU has been certified.</p><p>Given the relative complexity of the control UI interface, and the potential safety issues that could arise in the event the control UI system was to freeze while the firmware is ventilating a patient &#x2014; thereby preventing doctors from monitoring and adjusting settings &#x2014; we implemented a <strong>control UI watchdog in the firmware</strong>.</p><p>As <strong>the firmware decides if the control UI board gets 5V power</strong> over the GPIO ribbon cable, in the event that the firmware has missed a heartbeat announcement from the control UI, <strong>it can decide to power cycle it</strong>, effectively rebooting the control UI systems and hopefully restoring normal function. This is an automated process, which the user would barely notice, as in such cases the UI would appear frozen for ~30 seconds. The Raspberry Pi watchdog code can be <a href="https://github.com/makers-for-life/makair-firmware/blob/7a16c22936bff896683d042ef8d842e378bbe041/srcs/rpi_watchdog.cpp">read from the firmware</a>.</p><p>Note that we implemented this control UI watchdog preventively, ie. not as a patchy way to fix obscure freeze bugs we saw. With that said, you should understand that <strong>we never saw the control UI freeze yet</strong>. <em>Better safe than sorry, though.</em></p><hr><h2 id="control-ui-settings">Control UI &amp; Settings</h2><p>The <a href="https://github.com/makers-for-life/makair-control-ui">control UI</a> is the presentation layer of the MakAir: it provides medical personnel with <strong>live measurements of the ventilation process</strong>, with a graph of pressure and flow metrics. As well, it lets the <strong>ventilator be configured from a touch panel</strong>, and shows alarms whenever an anomaly occurs.</p><p>The control UI is a graphical interface built in <a href="https://www.rust-lang.org/">Rust</a>, that runs on any UNIX operating system, with a stripped-down Linux target on MakAir ventilators.</p><p>The control UI does not run any ventilation algorithm, nor does it have direct control over ventilator actuators. The firmware handles the ventilation process, while <strong>the control UI communicates with the firmware over a telemetry protocol</strong>, and handles metrics and ventilation settings.</p><h3 id="building-running-the-control-ui">Building &amp; running the control UI</h3><p>Building the control UI is as simple as:</p><pre><code>cargo build
</code></pre><p>It can then be ran using a sample telemetry recording with:</p><pre><code>./makair-control --input=/path/to/makair-telemetry/records/v2/short.record
</code></pre><hr><h3 id="video-recording">Video recording</h3><p>A video of the control UI has been shot. This control UI was ran on a computer, and <strong>replays a telemetry dump</strong> that has previously been recorded with a test balloon.</p><p><em>Please note that the pressure and flow graphs do not reflect the ones you would see on a proper test lung, or a real human lung.</em></p><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-control-ui-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-control-ui-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-control-ui-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-control-ui-video-track.webm">Play the overview video.</a>
</video><!--kg-card-end: html--><hr><h3 id="project-architecture-performance">Project architecture &amp; performance</h3><p>The control UI is written in Rust, and runs on an OpenGL graphical backend provided by <code>glium</code>, with the help of <code>conrod</code> as to draw texts, images and shapes. It does not pack any heavy runtime; all potentially heavy components have been stripped down.</p><p>The control UI imports code from the <code>makair-telemetry</code> library, which exposes a <strong>simple programmatic interface</strong> which helps connect to the <strong>serial telemetry channel</strong>, and read payloads coming to this channel, as well as transmit commands (eg. change a setting).</p><p>The rest of the control UI is just <strong>reacting to <a href="https://github.com/makers-for-life/makair-control-ui/blob/f25f037ef7265ac72a1f2ac38b57220dd4bb2fcf/src/serial/poller.rs">telemetry events</a></strong>, maintaining a local store reflecting the state of the firmware and the underlying ventilation process, as well as <strong>handling <a href="https://github.com/makers-for-life/makair-control-ui/blob/f25f037ef7265ac72a1f2ac38b57220dd4bb2fcf/src/display/events.rs#L83">user inputs</a></strong> (eg. screen taps). Different views are served to the user <a href="https://github.com/makers-for-life/makair-control-ui/blob/f25f037ef7265ac72a1f2ac38b57220dd4bb2fcf/src/display/renderer.rs#L86">based on the state</a> of the ventilator unit (eg. stopped, running, in error, in a setting modal, etc.).</p><p>As the UI would be running all the time over the course of 2 weeks when ventilating a patient, we took special care to <strong>prevent memory leaks</strong> (the RAM usage of the UI is stable at 36 MB on a 32-bit SoC), as well as <strong>reduce CPU and GPU usage</strong> to a bare minimum.</p><p>Indeed, as the interface needs to show graphs in real-time by refreshing the screen at 30 frames per second, non-optimized code could result in elevated CPU usage, which in turn would cause the CPU to heat up. As we opted for the passive cooling of the control UI board (a Raspberry Pi 4), we made sure to keep CPU usage low to a mean ~30% of 1 ARM CPU core out of the 4 available. Our measurements indicate that in a 20&#xB0;C room, <strong>the CPU did not get hotter than 45&#xB0;C</strong> (<em>that is quite satisfying!</em>).</p><hr><h3 id="on-the-importance-of-ux">On the importance of UX</h3><p>Starting with MakAir hardware V1, ventilation settings could be changed with physical buttons, mounted on the motherboard. As the project grew, we added so many settings that they could not fit on hardware buttons anymore, <em>unless we transformed the ventilator to look like a plane cockpit</em>. Thus, we made the choice to move <em>all</em> settings to the touchscreen UI, effectively deprecating hardware buttons.</p><p>In practice, using a touchscreen for controls is much more versatile, as the ventilator user interface becomes a single clean flat surface, that <strong>can be easily disinfected</strong> after use; contrasting with the bumpy button-based facade. As well, before the touch UI was mounted, we had to design a front panel printout for each targeted user language, whereas the UI can be translated virtually in any language, and <strong>makes MakAir hardware country independent</strong>.</p><p>In real world situations, a MakAir ventilator might be placed meters away from where medical personnel stand. Whenever a quick ventilation measurement reading is needed, people should be able to see the text shown on screen from meters away. This <strong>forced us to use quite a large font size</strong>. Similarly, we had to adapt button sizes and implement <strong>double confirmation prompts everywhere</strong>, as doctors may interact with the ventilator while wearing gloves, sometimes in emergency situations.</p><hr><h3 id="efficient-real-time-graph-rendering">Efficient real-time graph rendering</h3><p>The control UI renders <a href="https://github.com/makers-for-life/makair-control-ui/blob/f25f037ef7265ac72a1f2ac38b57220dd4bb2fcf/src/widget/graph.rs">two independent graphs</a> at the same time: pressure and flow. Each graph draws <strong>data sampled at a rate of 100 points per second</strong>, and is <strong>refreshed at 30 frames per second</strong>.</p><p>To render graphs, the control UI is using the <code>plotters</code> library, written in Rust. <code>plotters</code> renders its plots through a rendering backend, which can be customized. When we started up the control UI project, the only plotter that could be interfaced with our <code>conrod</code> drawing backend was the Bitmap backend, meaning <code>plotters</code> had to use a CPU-based rasterizer do draw each graph pixel-by-pixel, and feed it as an image to <code>conrod</code> for drawing.</p><p>As expected, this CPU-based rendering method <strong>did not allow us to draw two graphs simultaneously on the same CPU core</strong>, the control UI rendering loop being mono-threaded. A single graph consumed nearly 70% of one CPU core at 30 FPS, thus dramatically reducing the FPS if we were to draw a second graph (140% CPU).</p><p>This is where we decided to build a native <code>plotters</code> backend for <code>conrod</code>, named <a href="https://github.com/valeriansaliou/plotters-conrod">plotters-conrod</a>. Such a backend would draw using <code>conrod</code> shape primitives instead, which in turn would use OpenGL as <strong>to render them from the GPU</strong>.</p><p>This rendering method <strong>has proven to be much more efficient</strong>, as it freed CPU resources, delegated all plotting work to the GPU, allowing two plots to be rendered from a single thread at 30 FPS. As a bonus, the GPU performs vector graphics in contrast to the CPU raster graphics, meaning that antialiasing could be enabled at no cost, <strong>producing much better looking graphs</strong>.</p><hr><h3 id="hardware-running-the-ui">Hardware running the UI</h3><p>The control UI runs on a <a href="https://www.raspberrypi.org/products/raspberry-pi-4-model-b/">Raspberry Pi 4</a>, equipped with 1 GB of RAM (though 128 MB RAM only would do as well). The Raspberry Pi is connected to a <a href="https://www.raspberrypi.org/products/raspberry-pi-touch-display/">7&#x201D; Raspberry Pi Touch Display</a>.</p><p>The control UI <strong>requires a single ARM core</strong>, running in a low-frequency mode through the <code>powersave</code> CPU governor, ensuring the CPU stays cool (600 MHz).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-software-rpi.jpg" class="kg-image" alt="MakAir Series: Engineering Focus on Ventilator Software" loading="lazy"><figcaption>The Raspberry Pi 4 on which the control UI runs, plugged to a 7&#x201D; touch display. The GPIO pins are used to connect the control UI to the firmware, and carry the telemetry protocol serial channel.</figcaption></figure><p>As we were limited in time, note that we made the choice to lock the control UI drawing area to the 7&#x201D; touch display native resolution, <strong>that is 800 x 480 pixels</strong>. If a larger or higher density screen was to be used, the control UI source code would need to be adjusted to upsize the interface, or make all interface elements upsize dynamically based on their parent container size.</p><p>Note that <strong>we share prepared system images</strong> that can be readily flashed to a 8GB microSD card. This will get you a working Raspberry Pi system and control UI in a matter of minutes. The system image is available in a <a href="https://github.com/makers-for-life/makair-files">special large files repository</a>.</p><hr><h3 id="on-a-deeper-integration-of-systems">On a deeper integration of systems</h3><p>For now, any firmware update must be applied by plugging in a STM32 programmer to JTAG pins on the motherboard. While keeping a MakAir firmware up-to-date is important, this maintenance operation can be quite tedious and complex for some MakAir end-users.</p><p>The last motherboard revision exposes an USB port and packs an integrated programmer on that port, which can be used by the control UI running on the Raspberry Pi <strong>to initiate a firmware update sequence</strong>. The MakAir team is therefore thinking about <strong>integrating a signed archive of the firmware within the control UI binary</strong>. This would allow the UI to show a new maintenance tab, letting the user flash new firmware code on the motherboard.</p><p>On its end, the UI could show a &#x201C;check for updates&#x201D; button in the maintenance tab, and perform an OTA update whenever the MakAir ventilator gets connected to the Internet over the Ethernet port on its back side. Updating the UI from the Internet would result in the control UI restarting to its new version in a matter of seconds. Upon booting up, this new UI runtime could <strong>prompt to flash the new firmware build it contains</strong> on the MCU, if it detects that an outdated firmware version is running on the motherboard.</p><p>Note that the current Linux image that the control UI runs on is based on Arch Linux. Running on a desktop/server-grade Linux distribution is sub-efficient, as it packs a lot of useless binaries and kernel modules. We have a project to build a smaller, minimal Linux image. This would help strip down control UI boot time from ~20 seconds to ~5 seconds.</p><hr><h2 id="telemetry-protocol">Telemetry Protocol</h2><p>The telemetry protocol is the glue binding the firmware and the control UI together. We designed this protocol to be as <strong>lightweight</strong> and <strong>parser-efficient</strong> as possible, as the firmware, that sends the most protocol frames, runs on a rather slow ARM MCU.</p><p>Telemetry protocol <a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/proto/telemetry.proto">frames</a> are <strong>transmitted through a RX/TX serial channel</strong>, meaning it is bidirectional. The control UI is able to transmit protocol frames to the firmware to change ventilation settings, while the firmware sends over a stream of telemetry frames as to let the control UI show beautiful real-time pressure &amp; flow graphs.</p><p>Telemetry frames get serialized/deserialized by a dedicated <a href="https://github.com/makers-for-life/makair-telemetry">telemetry library</a>, which abstracts the wire protocol into more expressive Rust code.</p><h3 id="telemetry-frame-types">Telemetry frame types</h3><p>The telemetry protocol is made of predefined frame types, that can either flow from the firmware to the control UI, or from the control UI to the firmware.</p><p><strong>Frames sent from the firmware, as of protocol V2:</strong></p><ul><li><code>BootMessage</code>: the firmware announces that it has finished its boot sequence, sensor self tests and is now ready for use (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L365">code</a>);</li><li><code>StoppedMessage</code>: ventilation is stopped, though all settings get broadcast at a periodic interval (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L388">code</a>);</li><li><code>DataSnapshot</code>: ventilation is active, instant sensor measurements are reported every 10 ms (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L471">code</a>);</li><li><code>MachineStateSnapshot</code>: ventilation is active, all current settings are reported at the end of each respiratory cycle, which should be every 2-3 seconds (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L510">code</a>);</li><li><code>AlarmTrap</code>: an alarm has been triggered, and should be displayed on screen, eg. ventilator is on battery (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L607">code</a>);</li><li><code>ControlAck</code>: the firmware acknowledges that it updated a setting that was committed by the control UI via a <code>ControlSetting</code> frame (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L648">code</a>);</li><li><code>FatalError</code>: the ventilator raised an error and is thus not usable in this state, eg. sensor self tests failed on boot (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L669">code</a>);</li><li><code>EolTestSnapshot</code>: sent when the manufacturer performs end of line tests, ie. before a MakAir is validated and packaged (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/structures.rs#L688">code</a>);</li></ul><p><strong>Frames sent from the control UI, as of protocol V2:</strong></p><ul><li><code>ControlSetting</code>: the control UI commits an updated setting to the firmware, and waits for an acknowledgement via a <code>ControlAck</code> frame (<a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/control.rs#L15">code</a>);</li></ul><hr><h3 id="protocol-versions-backward-compatibility">Protocol versions &amp; backward compatibility</h3><p>As the firmware and control UI might be updated separately, where the control UI could be updated more often than the firmware &#x2014; <em>due to new firmware versions being subject to regulatory approvals</em> &#x2014; it is critical that <strong>a firmware running an older telemetry protocol version</strong> is still able to feed its data to <strong>an up-to-date control UI</strong>.</p><p>The telemetry protocol has this covered, as all telemetry frames put on the wire <strong>get tagged with a version number</strong>. Upon receival, a protocol V1 frame can thus be directed to a <a href="https://github.com/makers-for-life/makair-telemetry/tree/4f424c64671bcd8a2c3e024e28352f68702bd542/src/parsers">different parser</a> than a protocol V2 frame, albeit ultimately producing a consistent, common, data structure that is usable by the control UI. In fact, the control UI does not even care which telemetry protocol the firmware is speaking, as this is abstracted by the <a href="https://github.com/makers-for-life/makair-telemetry">telemetry library</a>. This means that the control UI can use the same data structures for different protocol versions.</p><p>Still, support for older telemetry protocol versions might be dropped when they become too legacy. For any telemetry version support pulled from the telemetry library, the control UI handles this gracefully. In the event that an unsupported telemetry frame gets received, the UI will show an <a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/src/error.rs#L10">error</a> message prompting the user to update the ventilator firmware to a newer version.</p><hr><h3 id="protocol-flow-example">Protocol flow example</h3><p>As an example of a normal protocol flow, when booting up a MakAir ventilator running telemetry protocol V2, <strong>this is what you would see</strong> if you were to tap the serial wire:</p><ol><li>The firmware announces it has booted and that it is now ready with a <code>BootMessage</code> frame;</li><li>As the ventilator unit is not running yet, the firmware sends <code>StoppedMessage</code> frames at periodic intervals;</li><li>The user starts tapping on the touchscreen and adjusting ventilation settings: what follows is a request-response exchange of <code>ControlSetting</code> and <code>ControlAck</code>;</li><li>The user starts the ventilation unit, another <code>ControlSetting</code> and <code>ControlAck</code> exchange occur, with <code>ControlSetting::RespirationEnabled</code> set to value <code>1</code>;</li><li>The ventilator is now running, it starts broadcasting a <code>DataSnapshot</code> frame every 10 ms, with measurements from its sensors;</li><li>At the end of each ventilation cycle (that is, at 15 cycles per minute, every 4 second), a <code>MachineStateSnapshot</code> is announced, as to re-synchronize any setting that may have gone out of sync on the control UI (eg. if the UI was rebooted when the ventilator was already running);</li><li>The user unplugs the ventilator to move the patient, an <code>AlarmTrap</code> event is raised, with alarm code <code>31</code> (for <code>AlarmCodeDescription::PowerCableUnplugged</code>), telling the UI to show an alert that the ventilator is now on battery;</li><li>(&#x2026;)</li><li>The user unplugs the patient, and stops the ventilator, a <code>ControlSetting</code> and <code>ControlAck</code> exchange occur, with <code>ControlSetting::RespirationEnabled</code> set to value <code>0</code>;</li><li>The ventilator is now stopped, the firmware sends <code>StoppedMessage</code> frames at periodic intervals (<code>DataSnapshot</code> and <code>MachineStateSnapshot</code> are not sent anymore);</li></ol><hr><h3 id="telemetry-recordings-replays">Telemetry recordings &amp; replays</h3><p>In order to help debug telemetry frames, as well as provide replays for library and control UI development purposes, telemetry frames can be dumped in a record file, in which binary telemetry messages are encoded in <a href="https://en.wikipedia.org/wiki/Base64">Base64</a>. Such a record file for the telemetry protocol V2 can be <a href="https://github.com/makers-for-life/makair-telemetry/blob/4f424c64671bcd8a2c3e024e28352f68702bd542/records/v2/short.record">found there</a>.</p><p>Telemetry records can be easily <strong>replayed on a computer</strong>, with the help of the control UI, in a single command:</p><pre><code>makair-control --input=/path/to/makair-telemetry/records/v2/short.record
</code></pre><p>The ability of replaying records <strong>lets MakAir developers debug records</strong> from eg. clinical trials, where ventilation anomalies or particular events would be reported by medical personnel, and developers would then try to understand what happened, eg. &#x201C;<em>is this strange ventilation behavior related to a doctor changing a setting and making a mistake in the decision?</em>&#x201D;.</p><p>If <strong>an USB key is plugged</strong> on any USB port on the board that runs the control UI, telemetry frames will be dumped on a record file on that USB key. The USB key can then be pulled from the ventilator and its contents analyzed and replayed, much like a <a href="https://en.wikipedia.org/wiki/Flight_recorder">flight recorder</a> on a plane (ie. black box).</p><hr><h2 id="firmware-simulator">Firmware Simulator</h2><p>The <a href="https://github.com/makers-for-life/makair-simulator">firmware simulator</a> is one of the latest experiment of the MakAir project. It basically integrates the firmware as a library, and simulates hardware components, namely sensors (pressure &amp; flow sensors) and actuators (pinch valves). <strong>It models the effect of the patient lung on air pressure and air flow</strong>.</p><p><strong>The firmware simulator was built with three goals in mind:</strong></p><ul><li>Let <strong>developers &amp; UI designers contribute</strong> to ventilation algorithms and the control UI, where they have no access to a MakAir ventilator hardware;</li><li>Let <strong>theoretical researchers</strong> in ventilation algorithms implement their algorithms, and test them out on an actual ventilator &amp; lung model;</li><li>Package the simulator, firmware and control UI in an app that could be used by <strong>trainees in medical schools</strong> to learn how to use a ventilator and ventilate a (simulated) patient, where the lung model could be chosen from a list, in order to simulate a wide range of pathologies (from a healthy adult to an <a href="https://en.wikipedia.org/wiki/Acute_respiratory_distress_syndrome">ARDS</a> patient);</li></ul><p>We have also managed to run the firmware simulator with its attached control UI on a cloud server, running a VNC server. We exposed a VNC client on a Web page, which we shared to doctors as to demo the MakAir user interface and firmware. Through this demo server, <strong>doctors could interact with the simulated MakAir ventilator</strong>, right from their Web browser. They could change ventilation settings and evaluate the quality of our ventilation algorithms in comparison to commercial ventilators.</p><p><strong>This experiment was a huge success</strong>, which suggests that we should scale this &#x201C;marketing&#x201D; method as a way to showcase the robustness of our ventilation algorithms, in turn to build trust in the MakAir software, which could ultimately lead to more doctors using MakAir ventilators on patients.</p><hr><h2 id="closing-notes">Closing Notes</h2><p>This article was the sixth (and last!) of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>While our first software revision was only made of a firmware running on an Arduino board, trying to reproduce the <a href="https://valeriansaliou.name/blog/makair-series-how-to-ventilate-a-human-patient-101/">pressure curve</a> as close as possible, it was a matter of days before we moved over to a <strong>more robust <a href="https://en.wikipedia.org/wiki/PID_controller">PID</a>-controlled algorithm</strong>, running on a <a href="https://www.st.com/en/microcontrollers-microprocessors/stm32-32-bit-arm-cortex-mcus.html">STM32 MCU</a>. We added a touchscreen interface to the system thereafter, improving user experience and bringing ventilation monitoring capabilities.</p><p><strong>We are now iterating on the firmware and control UI</strong> to improve safety and UX, as well as adding easy maintenance features such as the ability to perform OTA updates of the firmware and the control UI, right from the touchscreen. We are trying to improve the developer experience as well, with the inception of a firmware simulator, that mimics a MakAir hardware running with simulated physics, which lets developers contribute from their laptop, <em>anywhere in the world</em>.</p><p><strong><em>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</em></strong></p><hr><p>&#x1F1F5;&#x1F1F9; <em>Written from Lisbon, Portugal.</em></p>


guid: 6135df393daca803cc39bddc
title: Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS
author: Valerian Saliou
url: https://valeriansaliou.name/blog/announcing-prose-decentralized-team-messaging/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/09/prose-banner.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/09/prose-banner.jpg" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS"><p>Nowadays, almost every company is using messaging tools within their teams. Such tools help businesses boost productivity. They (almost) entirely replaced email. But there&#x2019;s a hitch: those new platforms are all centralized. From Slack, Skype for Business to Microsoft Teams, or the now-defunct HipChat, all company&#x2019;s industrial secrets &amp; small talk get relayed and stored on centralized cloud servers (located mostly in the US). The exact same is happening to personal communications, with Facebook Messenger, Telegram, WhatsApp and iMessage.</p><p>While the Internet makes decentralization technically possible and desirable, it is not a de-facto requirement for service operators. Instead, all VC-funded companies have a strong economic incentive to centralize, as a means to concentrate more value in their walled gardens. While this centralized model is efficient at building $1B+ public companies, it goes against what the Internet was built for.</p><p>On the side, it has been more than 20 years that a modern open messaging standard was invented: <a href="https://xmpp.org/">XMPP</a> (<em>I am not mentioning <a href="https://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a> and the likes here, as those cannot be considered &#x201C;modern&#x201D; anymore</em>). Those open protocols, while mature and supported in a wide range of servers, lack the applicative ecosystem they deserve: existing apps are mostly made for techies (<em>want to chat in the terminal?</em>). This, in part, explains why Slack and its competitors won the war so far.</p><p><strong>It&#x2019;s time to change the status quo with Prose</strong>, a native cross-platform team messaging app, built on solid foundations, thanks to the XMPP standard. This article announces the <a href="https://prose.org/">Prose project</a>, which I am starting aside from <a href="https://crisp.chat/">Crisp</a>, my main business.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/09/prose-macos-login.jpg" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"><figcaption><em>Hello World. This is Prose.</em></figcaption></figure><h1 id="a-short-history-of-team-messaging">A Short History of Team Messaging</h1><p>Instant Messaging first penetrated our personal lives in late 1996, with the advent of ICQ. This redefined the way we communicate with friends and loved ones, at a time businesses were still massively using email (or old-fashioned telephone &amp; fax).</p><p>IM quickly became ubiquitous with WhatsApp, iMessage and Facebook Messenger in the early 2010s. At the same time, online communities were using IRC or XMPP to gather hundred of people in group chats, while businesses were still&#x2026; using email, which enabled G Suite and Outlook to thrive.</p><p>The <a href="https://en.wikipedia.org/wiki/XMPP">XMPP standard</a>, introduced as &#x201C;<a href="https://en.wikipedia.org/wiki/Jabber">Jabber</a>&#x201D; in 1999, set out to build a modern open and extensible instant messaging standard. Servers and clients (ie. applications) were to implement the protocol and interconnect, forming a decentralized network. Anyone willing to join the network as a user can choose a public XMPP server, and register for an email-looking address like: <code>john.doe@acme.com</code> (called <a href="https://en.wikipedia.org/wiki/JID_(Jabber)">Jabber ID</a>). In that regard, XMPP can be compared to the way email operates. XMPP quickly became popular in online and open-source software communities, alongside IRC.</p><p>Slack, launched in 2013, went on a mission to redefine how people communicate within the workspace. It aimed at making team messaging more efficient &amp; instant, and replace old-fashioned mailing lists with Slack channels. While in the first years, early adopters were mostly tech-savvy companies, startups and communities, Slack then gained traction in more traditional businesses, schools, universities and large corporations.</p><p>More recently, team messaging services have seen their user base explode. During the COVID-19 pandemic, a lot of businesses were forced to use Slack or competitors. This pushed businesses away from email &#x2014; <em>which is a standard protocol forming a decentralized network</em> &#x2014; to proprietary centralized walled gardens, whether it be Slack or Microsoft Teams. On the side, <a href="https://mattermost.com/">Mattermost</a>, an open-source self-hosted Slack alternative, is also used by a smaller number of organizations, though it was not built on any standard. This sadly defeats interoperability and inter-team connectivity.</p><p>Today, <a href="https://techcrunch.com/2020/12/01/salesforce-buys-slack/">Slack is owned by Salesforce</a>, which operates as a huge conglomerate of SaaS services, while Microsoft Teams and Skype for Business are ran by Microsoft. This means that two large US corporations dominate the market of workspace messaging, <em>worldwide</em>. To me, this looks like an unhealthy duopoly. 43% of Fortune 100 businesses pay to use Slack, but note that Microsoft <a href="https://dispatch.m.io/enterprise-messaging-wars-slack-microsoft-teams/">dominates almost all</a> of the market with Skype and Teams.</p><h1 id="team-messaging-in-an-ideal-world">Team Messaging in an Ideal World</h1><p><strong>Person-to-person interactions should be kept private in all situations</strong>, regardless of the importance of what is being discussed. From small talk to handing over the designs of a revolutionary nuclear fusion reactor that works, messages should not get exchanged via a single middleman. Rather, each company should have the choice of which cloud server relays their message. If they are not happy with the available cloud offerings, they should be able to deploy the system on their own premises. They would have <em>choice</em> as to where their data lives.</p><p>Neither should users not trust their own server. In fact, messages may sometimes unintentionally get read by engineers while debugging something on the server. Logging could also sometimes be intentional, eg. for espionage or law enforcement purposes. Therefore, <strong>the applications that people use must enforce end-to-end encryption</strong>. A message that gets end-to-end encrypted can only be opened by its recipient, because only the sender and the recipient are aware of the cryptographic key that lets them decrypt it. <em>Note that this fantastic privacy feature already exists under the <a href="https://en.wikipedia.org/wiki/OMEMO">OMEMO</a> standard, which is available within the XMPP protocol.</em></p><p>We usually do not tolerate that other people listen to our real-life conversations, say in a public caf&#xE9;. On the contrary, we do not seem to be <em>that</em> concerned by the privacy of the conversations we have on the Internet via private messaging. This is exactly what is happening today. We relay our conversations through huge centralized services, that ingest and forward messages for 100s of millions of people daily. Not only do they relay messages, but they also store them in the clear for history purposes. This means that someone in the future could go back in time through anyone&#x2019;s conversations.</p><p>Moving back communications to a decentralized model also makes the whole system resilient against potential worldwide network fragmentation in the future. What would happen in the event of a new World War? Some countries, like China and Russia, already implement some level of &quot;Internet within the Internet&quot; with eg. the <a href="https://en.wikipedia.org/wiki/Great_Firewall">Great Firewall</a>, and therefore can block access to a messaging platform (as it once was with <a href="https://en.wikipedia.org/wiki/Blocking_Telegram_in_Russia">Telegram in Russia</a>). In the event of a worldwide conflict between two blocs, the Internet could be split into two independant networks. For instance, certain IP addresses could be made un-routable in certain countries. What happens, then, if all your centralized messaging services are hosted on the other side of the virtual &quot;wall&quot;? Decentralization guarantees that systems stay resilient against network fragmentation, as only a subset of your teammates or friends would become unreachable, while you could still communicate with people who are within your side of the Internet. <em>That is worst-case scenario, not to say that this could actually happen &#x2014; but hey, who knows what the future is made of</em>.</p><h1 id="the-choice-of-the-xmpp-standard">The Choice of the XMPP Standard</h1><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2021/09/xmpp-logo-wide.png" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"></figure><p><strong>XMPP is a 20+ years old messaging protocol</strong>. Its acronym stands for Extensible Messaging and Presence Protocol. The extensibility of the protocol is especially important, as it lets engineers add new features over time, in order to keep the protocol in line with modern uses (eg. chat stickers were not a thing twenty years ago, though now they are &#x2014; this has been covered under <a href="https://xmpp.org/extensions/xep-0449.html">this protocol extension</a>, aka <em>XEP</em>). Since those features are being built as protocol extensions, the protocol foundations are left clean &amp; untouched, which is a desirable trait of a durable protocol.</p><p>Being <strong>stable and mature</strong>, XMPP has all the required properties for a project with a long-term vision. Furthermore, all features Prose aims to develop are already available within the XMPP standard (eg. end-to-end encryption is covered under the <a href="https://xmpp.org/extensions/xep-0384.html">OMEMO XEP</a>, calls are covered under the <a href="https://xmpp.org/extensions/xep-0166.html">Jingle XEP</a>, while user profiles are handled by the <a href="https://xmpp.org/extensions/xep-0292.html">vCard4 over XMPP XEP</a>).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/09/xmpp-network.png" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"><figcaption>A simplified scheme of how users and servers for different teams interact on the XMPP network. Users are hosted on a server. Servers connect to each other. Note that while some servers are self-hosted, I expect a majority of organizations to use third-party cloud hosting, for the convenience it provides. This cloud service could be provided by Prose itself.</figcaption></figure><p>Multiple good-quality open-source XMPP server implementations are available, such as <a href="https://www.ejabberd.im/">ejabberd</a>, <a href="https://prosody.im/">Prosody</a> or <a href="https://www.igniterealtime.org/projects/openfire/">OpenFire</a>. Those servers can be connected-to by any client application over the XMPP protocol. Servers can open connections to each other, so that users on different servers can communicate (exchange messages, call each other, view profiles, etc.). Instead of reinventing the wheel, those servers can be used as a foundation to build great user-facing applications.</p><p>As well, XMPP client libraries are available, such as <a href="https://strophe.im/libstrophe/">libstrophe</a>. A client library manages the connection to an XMPP server and handles packet routing and parsing with a clean abstraction layer. That way, application developers can focus on implementing UI and application logic, leaving the connection management and obscure protocol work to the underlying XMPP client library.</p><p>Note that I am not by any means new to the world of XMPP. I taught myself how to code by starting <a href="https://github.com/jappix/jappix">Jappix</a>, <strong>the first ever Web-based XMPP client</strong>, at a time where Web apps were not (yet) a thing and Google Chrome just came out. Jappix was used by communities, businesses and organizations such as <a href="https://home.cern/">CERN</a> (the European Council for Nuclear Research; <a href="http://indico.eblida.org/ihelp/html/Chat/XMPPClients.html">proof</a>), at a time Slack and similar did not yet exist.</p><p><strong>Prose is the culmination of what I started with Jappix 10 years ago</strong>. It will be built on my past experience of working with the XMPP protocol.</p><p><em>By the way, I made a public presentation of XMPP 5 years ago, <a href="https://www.slideshare.net/ValrianSaliou/xmpp-technical-overview-jingle-protocol-study">available in those slides</a>. They can help you get a quick technical overview of how it works (audio comments not available).</em></p><h1 id="prose-project-vision-timeline">Prose Project Vision &amp; Timeline</h1><p>The Prose project has been started to provide organizations of all sizes with a first-class alternative to Slack and Microsoft Teams. Think of the Prose project mission as one of liberating person-to-person workspace communication, from messaging to audio &amp; video calls.</p><h2 id="what-will-be-built">What will be built</h2><p>Prose will start small. <strong>A native app for macOS will be released in 2022</strong>, then its mobile version for iOS later the same year. Finally, support on other platforms (Windows, Android, Linux) will be progressively added.</p><p>Prose will not take the path of a single Web app bundled in Electron, targeting all platforms at once. Web apps for the Desktop, while convenient for projects with limited means, do not feel that <em>that great</em> for the end user. Laptop batteries drain faster and user experience is suboptimal. People deserve native apps, built with the system&#x2019;s native GUI frameworks.</p><p>Once all the apps are out, we will be releasing a way for organizations to easily deploy their XMPP server on their premises (called a &#x201C;pod&#x201D;). Aside from that, we will also offer a cloud hosting service, as we know that running a server can be a no-go for most users.</p><p>To make this vision a reality, <strong>I will assemble a remote team of developers &amp; designers</strong>. I will fund it all from my own pocket at the start, until a sustainable business model has been built. Note that all Prose apps will be open-source, therefore revenue would have to flow from cloud server hosting or other commercial services.</p><p><strong>If you are a talented developer or designer, interested in joining Prose, please <a href="mailto:valerian@valeriansaliou.name">contact me</a></strong>. Wherever you are in the world, this is a remote team!</p><h3 id="an-unified-messaging-experience">An unified messaging experience</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/09/prose-macos-messaging.jpg" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"><figcaption>Share messages and files with your colleagues, with a best-in-class native messaging experience. Prose is designed for remote teams, split across different timezones.</figcaption></figure><h3 id="call-anyone-from-a-single-app">Call anyone from a single app</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/09/prose-macos-call-pstn.jpg" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"><figcaption>Start audio and video calls with your team members, or even call external phone numbers. Prose makes it easy to organize conference calls inside of outside of your workspace.</figcaption></figure><h3 id="ubiquitous-end-to-end-encryption-made-easy">Ubiquitous end-to-end encryption, made easy</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/09/prose-macos-profile-omemo.jpg" class="kg-image" alt="Announcing Prose: Decentralized Team Messaging in an Era of Centralized SaaS" loading="lazy"><figcaption>All messages sent and received are end-to-end encrypted. This means that even the server cannot read messages. Easily revoke a connected device access to all of your future encrypted messages.</figcaption></figure><h2 id="on-the-business-model-of-an-open-source-project">On the business model of an open-source project</h2><p><strong>Making money with an open-source business is incredibly hard</strong>. A lot of open-source businesses fail, while others need to continuously raise VC funds in order to sustain their team.</p><p>Prose will be separating the user-facing apps &#x2014; which will be open source &#x2014; from the server-side, for which a cloud service will be offered. I am a big fan of the <a href="https://ghost.org/">Ghost Foundation</a>, that succeeded at making money as an open-source blogging software (their current revenue is tracked <a href="https://ghost.org/about/">on this page</a>).</p><p>Ghost paved the way for profitable foundation-ran software project. Prose will follow suit. In order to protect the project on the long run, <strong>Prose will be incorporated under a not-for-profit foundation</strong>. This means that the Prose Foundation cannot be split into share and therefore it cannot be sold to private investors. Selling a project can affect the direction it takes and alter its founding values, which is why Prose needs to hedge itself against this risk.</p><h1 id="closing-notes">Closing Notes</h1><p>With the centralization of workspace messaging towards two huge conglomerates, it is time to provide businesses with a first-class decentralized and open alternative.</p><p>Prose will offer a set of native open-source apps, from Desktop to Mobile, all built on solid foundations: the XMPP protocol.</p><p><strong>I am looking to assemble a world-class remote team of developers &amp; designers, working towards this vision with me. If you are interested, please <a href="mailto:valerian@valeriansaliou.name">drop me a line</a></strong>. I am looking for macOS &amp; iOS developers (Swift/SwiftUI), as well as Windows and Android developers. Adding to that, UI designers and illustrators to design &amp; maintain <a href="https://prose.org/">Prose&#x2019;s website</a> (<code>prose.org</code>), identity and app designs.</p><p>Valerian.</p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 61ce27bfbefb1e03ba43d433
title: Estimating My Carbon Footprint for The Year
author: Valerian Saliou
url: https://valeriansaliou.name/blog/estimating-my-carbon-footprint-for-the-year/
feedurl: https://journal.valeriansaliou.name/rss/
content_mime_type: text/html
content:
<p><em>I usually say that one cannot understand what one does not measure. Our modern daily lives revolve around energy. We power our lights &amp; electronics, we heat our home, we commute, we travel, we buy imported goods, and we rely on social services from our states. And as a result, we emit carbon, a lot.</em></p><p>The human body, runs on an average of 100W of power at rest. That said, each one of us <strong>uses much more power than what our whole bodies consume</strong>. For instance, the MacBook Pro and external display from which I am writing this article, consume north of 150W, which is 50% more than my body sitting at my desk.</p><h1 id="primer">Primer</h1><p>When we reflect about the energy we consume, it&#x2019;s important to think about the equivalent Joules, regardless on how it gets generated (coal, wind, nuclear, organic, etc.). 1 Joule, as per Wikipedia, is:</p><blockquote>Equal to the energy transferred to an object when a force of one newton acts on that object in the direction of the force&apos;s motion through a distance of one meter.</blockquote><p>1 Joule equals 1 Watt-second. We should first strive to reduce to number of Joules powering our daily lives in order to reduce the amount of carbon that we put in the air.</p><p>Obviously, each Joule of energy that we consume was not generated in the same way. Depending on the source of energy, <strong>the carbon intensity of each Joule varies dramatically</strong>. For instance &#x2014; <em>using the Watt-hour unit, where 1 kWh = 3.6 Megajoules</em> &#x2014; 1 kWh of electricity generated through coal emits about 1kg of CO2; natural gas 410g, while nuclear would only be about 10-20g (fission reactors only &#x201C;emit&#x201D; water vapor, though the end-to-end production life-cycle still emits small amounts of CO2).</p><p>All that said, we do not yet fathom how much tons of CO2 a year we emit through seemingly harmless consumption habits. Here are some recent <a href="https://en.wikipedia.org/wiki/Global_warming_potential">CO2eq (CO2-equivalent)</a> estimates of average per-capita yearly carbon footprint (also accounting for all per-capita emissions off-country eg. you buy goods manufactured in China):</p><ul><li>&#x1F1FA;&#x1F1F8; <strong>United States</strong>: 17.10 tons of CO2eq / capita</li><li>&#x1F1E8;&#x1F1F3; <strong>China</strong>: 10.10 tons of CO2eq / capita</li><li>&#x1F1EB;&#x1F1F7; <strong>France</strong>: 11.0 tons of CO2eq / capita (<em>note that local emissions are only 4.60 CO2eq / capita, as most goods consumed are produced off country</em>)</li></ul><p>Note that as the middle class rises in developing countries, <strong>a lot more countries will ramp up to this ~10.00 tCO2eq per capita number</strong>, which seems to represent the environmental cost of enjoying a modern way of life based off of fossil fuels. For instance, the tCO2eq per capita in Peru was only 1.87 tons per person in 2016 (not sure if this accounts for off-country emissions of imported goods, though).</p><h1 id="popular-misconceptions">Popular Misconceptions</h1><p>Some of us would feel like saving the planet by not buying this new shiny iPhone. <strong>Numbers prove that this is, relatively, a pointless effort</strong>. Indeed, a lot of people are still living in poorly isolated homes heated with natural gas. To provide some numbers, as per <a href="https://www.apple.com/environment/pdf/products/iphone/iPhone_13_PER_Sept2021.pdf">Apple&#x2019;s environmental report</a>, the iPhone 13 has a lifecycle CO2eq of (only) 64kg, while <strong>heating a 100 sqm home with natural gas emits in the range of 1 to 4 tons of CO2eq (a year!)</strong>, depending on various factors such as insulation, efficiency and type of gas used.</p><p>Given all those popular misconceptions about &#x201C;what&#x2019;s green&#x201D; and &#x201C;what&#x2019;s not green&#x201D;, the best thing to do is to estimate one&#x2019;s carbon footprint for the year! You&#x2019;d be surprised how poorly certain things we take for granted perform in terms of tCO2eq (such as heating).</p><h1 id="estimating-one%E2%80%99s-footprint">Estimating One&#x2019;s Footprint</h1><p>Quite a few carbon footprint estimation tools are available online. But after trying out 3 of them, inputing the same data, they all gave me results that were off of each other by 30%. There is a huge uncertainty factor on the numbers I got, which lead me not to trust them.</p><p>This brought me to the realization that <strong>I had to come up with my own model to estimate my carbon footprint</strong>, all in a spreadsheet. This forced me to research each area that should be considered in the model, and well as seek to justify where each number comes from. The results I got are super interesting. My model seems to be reliable, as my footprint is in par with the average per-capita footprint for France!</p><h1 id="the-estimation-model">The Estimation Model</h1><p>The estimation model I came up with can be downloaded and filled with your data, using the Numbers app on macOS. <em>It contains my carbon footprint data for 2021 as an example.</em></p><p>&#x2B07;&#xFE0F; <strong><a href="https://valeriansaliou.name/blog/content/images/2021/12/carbon-footprint-model.numbers">Download the model here</a></strong></p><p>The model is divided into the following categories, that all add up to a grand total:</p><ul><li><strong>Trips</strong> (plane, car, bus and train);</li><li><strong>Manufactured goods</strong> (eg. the new computer you bought);</li><li><strong>Leisure</strong> (eg. hotels);</li><li><strong>Living</strong> (eg. food + trash);</li><li><strong>Energy</strong> (eg. electricity + heating);</li><li><strong>Indirect</strong> (eg. state services);</li></ul><p>While obtaining a total tCO2eq for certain categories <strong>is fairly easy and reliable</strong> (such as everything that falls into trips, given that it&#x2019;s a number of km multiplied by the grams of CO2eq / km, which is always provided by car manufacturers or online estimations), other categories such as leisure (think: hotels) <strong>are a bit more tricky</strong> to get right. In the latter case, rough estimations are made, based on online searches, adjusted with a pessimistic corrective tCO2eq number. If we cannot reliably estimate a number, let&#x2019;s at least be pessimists to get things right. Your estimations are as good as your sources are.</p><p>For example, the category for car trips gives the following:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/12/carbon-footprint-example-category-car.png" class="kg-image" alt loading="lazy"><figcaption>Note here how comparatively &#x201C;clean&#x201D; an EV is when charged over renewables/nuclear.</figcaption></figure><p>Input your trip routes, the number of km, the typical gCO2/km for the car used, as given by the manufacturer, and let the spreadsheet do the rest!</p><h1 id="my-own-results-for-2021">My Own Results For 2021</h1><p><strong>Those are the totals I came up with for my 2021 carbon footprint</strong>:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/12/carbon-footprint-overview.png" class="kg-image" alt loading="lazy"><figcaption>Plane is responsible for more than half of my emissions for 2021.</figcaption></figure><p><strong>You can also view my full <a href="https://valeriansaliou.name/blog/content/images/2021/12/carbon-footprint-valerian-2021.pdf">carbon footprint estimation report</a> for 2021</strong>, containing all details and calculation methods. This document was exported from my model spreadsheet.</p><p><strong>My grand total for 2021</strong>, including off-country emissions due to eg. buying electronics manufactured in China, is: <strong>12.24 tons of CO2eq</strong>. Which is about 11% above France per-capita average (11.00 tCO2eq), but way below the US per-capita average (17.10 tCO2eq, for reference). This is by no means a good number, which can be easily improved.</p><p>Here&#x2019;s an overview of my situation regarding habits and energy use (which are probably not typical, but it significantly affects the results I got):</p><ul><li>Living in Nantes, France; where <strong>typical winters temperatures are cold</strong>: fluctuating from -5C to 10C. Summers are typically 25C;</li><li>My apartment is brand new, 70 sqm, built with the <strong>latest thermal insulation norms</strong>;</li><li><strong>Heating through natural gas</strong>, though I opted for <a href="https://en.wikipedia.org/wiki/Biogas">biogas</a> which is about 2x less carbon intense per kWh (through my energy provider). I heat to 18C during the day in winter, 16C during the night;</li><li>Electricity in France is typically nuclear, though <strong>I opted for hydro power</strong> (through my energy provider). My electricity usage is quite low as most of my appliances are energy efficient;</li><li><strong>My car is an EV</strong> (Tesla Model S 2014), charged in its parking via nuclear;</li><li><strong>I eat a vegetarian diet</strong>, which is less carbon intensive than eating meat. Note that vegan diets are the most efficient ones regarding carbon, but, hey;</li><li><strong>I do not produce a lot of trash</strong> that does not get recycled. It is important to note that the process of incinerating trash produces a lot of CO2, about a bit more than the total weight of trash that gets burned (in the 0.7x-1.7x range, meaning burning 1 ton of trash results in 0.7 tCO2, up to 1.7 tCO2);</li><li><strong>I travel by plane. Super often.</strong> Much more than per-capita average. I&#x2019;ve worked remotely for half of 2021, which led me to take a lot of pan-European and long-distance flights;</li><li>Lastly, I need to account for the <em>incompressible</em> per-capita carbon footprint linked to <strong>services catered by the state</strong> (health, education, infrastructure, defense, etc.). As a French citizen, this is estimated to be 1.5 tCO2eq per capita per year.</li></ul><h1 id="outro">Outro</h1><p>That&#x2019;s all! I hope that sharing my carbon footprint model can save you some time in doing your own. Leading you to ultimately act on reducing your emissions.</p><p><em>To give an example of how I leveraged this model to produce actual results</em>: we used the model to estimate Crisp offices carbon footprint, to discover that our 160 sqm office space heated with natural gas, <strong>emitted a total of 8.0 tCO2eq over the year of 2021</strong> (!!!). That&#x2019;s about 19MWh of energy consumed for heating. We took corrective action to switch to a biogas provider (2x less carbon intensive), modernize the heating system and add a <a href="https://www.netatmo.com/en-us/energy/thermostat">Netatmo thermostat</a> and smart valves on the radiators (2x kWh reduction). We are now <strong>aiming for 2.0 tCO2eq in 2022</strong>, a <strong>4x total reduction</strong> in emissions.</p><p><em>Measuring is the first step of any optimization process. After all, it&#x2019;s not so much different than optimizing software as a software developer. And it can be a lot of fun, too!</em></p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 627839cfd915235741dc579d
title: Horizontal Scaling of Socket.IO Microservices with RabbitMQ
author: Valerian Saliou
url: https://valeriansaliou.name/blog/horizontal-scaling-of-socket-io-microservices-with-rabbitmq/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-socketio-microservices-scaling.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-socketio-microservices-scaling.jpg" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ"><p>At Crisp, we are building a <a href="https://crisp.chat/en/">customer service software</a> that provides our users with a <a href="https://crisp.chat/en/livechat/">chatbox service</a> that they can setup on their website. The Crisp chatbox provides features that require users to stay connected to Crisp servers over WebSocket, via <a href="https://socket.io/">Socket.IO</a>, even if they do not opt to use the chatbox and start a chat.</p><p>This design choice in our product makes for a challenging technical task: <strong>handing millions of simultaneous WebSocket connections</strong>, inside of which real-time events such as messages could be streamed anytime.</p><p>As the Crisp user base grows, we get to face more issues related to the scalability of our systems past a certain point. In our initial model, users get load balanced to the server with the least active connections, such that the distribution of connections stays even over all of our WebSocket workers. Unfortunately, this also means that <strong>events that must be served to connected clients cannot be routed selectively: they must be broadcasted to all our workers</strong> (<em>understand: we do not know which upstream server a chatbox is connected to</em>), which then individually decide which connection to stream events to. In Socket.IO jargon, events get emitted to Socket.IO sockets, which are part of the room that the event targets.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/diagram-amqp-topology-default.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>An overview of our AMQP-based microservice topology. The RabbitMQ server sits at the center of our network of microservices, passing messages along nodes. In this diagram, microservices emit payloads that are destined to all <code>socket-client</code> nodes, which ultimately get emitted to connected chatbox users over WebSocket.</figcaption></figure><p>This dumb and simple topology has served us well for 6+ years, and proved to be stable and fault-tolerant. However, it comes with a huge tax on resources, not only for Socket.IO servers (which receive all events happening on the platform and need to sort through them), but also for our <a href="https://en.wikipedia.org/wiki/Message_broker">message broker</a>, <a href="https://www.rabbitmq.com/">RabbitMQ</a>. Our current topology involves RabbitMQ as the <a href="https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol">AMQP</a> server at the center of our microservices network, passing messages along from exchanges (<em>where they get published</em>), to <a href="https://www.rabbitmq.com/queues.html">queues</a> (<em>where they get consumed</em>).</p><p>As Socket.IO workers expect to receive a broadcast of all events happening, <strong>RabbitMQ has to pick every incoming event on the exchange, and clone them as many times as there are Socket.IO workers</strong> to each individual queue. Each queue is bound to a Socket.IO worker, from which the worker consumes incoming events. While this topology works great, it proved to be very costly to RabbitMQ in terms of CPU resources, especially as the number of platform events per second (EPS) grows, and as we add more Socket.IO worker replicas. As of mid-2022, we run 16 WebSocket servers for the chatbox, which means that the events amplification factor is 16. We could roughly say that we are dealing with a complexity of <code>O(W x R)</code>, where <code>W</code> is the number of workers, and <code>R</code> is the peak EPS.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/diagram-scenario-eps.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>A simulation of total EPS output off of the exchange, as a function of total EPS input to the exchange. The runaway EPS out problem we want to solve here is very visible as we add more Socket.IO nodes, even if the input EPS growth stays under relative control, and is rather low.</figcaption></figure><p>Given the inability of our system to scale further, it was just about time for a change. <a href="https://jam.in/">Baptiste</a> and <a href="https://valeriansaliou.name/">myself</a> pondered on an elegant solution &#x2014; <em>that I describe in this article</em> &#x2014; which builds on the top of our existing RabbitMQ-based topology.</p><hr><h1 id="prelude-what-are-rabbitmq-and-socketio">Prelude: What are RabbitMQ and Socket.IO?</h1><p><strong><a href="https://www.rabbitmq.com/">RabbitMQ</a> is a message broker</strong>. It is at the core of microservice infrastructures by acting as the central server to which nodes publish messages. It then serves messages to recipient consumer nodes in the pool, making sure all your nodes receive their fair share of work. RabbitMQ is very powerful, and can be configured to be fault-tolerant and resilient, as part of a cluster. You can define advanced routing policies based on your needs, which is what we will do in this article.</p><p><strong><a href="https://socket.io/">Socket.IO</a> is an abstraction over WebSocket</strong> that helps implement bidirectional real-time communication channels in applications. A Socket.IO server handles connections from Socket.IO clients, from which it receives input events, and to which it can also deliver output events. Socket.IO clients are part of rooms, which means that they subscribe to certain events only. Socket.IO servers can be replicated and scaled horizontally, to provide fault-tolerance (ie. if one server fails, your service would still work) and serve more users (ie. by adding more server capacity).</p><hr><h1 id="fast-forwards-post-optimization-savings">Fast-Forwards: Post-Optimization Savings</h1><p><em>I know that most of you would like to see the results of the optimizations we made before diving into the details, so here they are!</em></p><p>After we completed deploying the optimizations described later in this article, a stark drop in events per seconds was observed on our <a href="https://www.rabbitmq.com/clustering.html">RabbitMQ cluster</a>. <strong>We have reduced approximately by 40% the CPU load</strong> induced by RabbitMQ on the host server, and <strong>divided by ~8 the total outgoing EPS</strong> on the <code>client</code> chatbox exchange.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-cpu-before-after.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Typical CPU usage of a RabbitMQ node within our cluster, before and after deploying our optimizations, showing the ~40% savings per core.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-exchange-client-rates.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>The routed events rate on the chatbox socket exchange of our RabbitMQ cluster, before and after the enforcement of Socket.IO events routing around 15:35.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-global-rates.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>The total events rate on our RabbitMQ cluster, including all exchanges for all our different kinds of Socket.IO services. The yellow graph shows the events in, while the purple one shows the events out. Amplification due to the need to clone and broadcast events to each Socket.IO worker replica caused the purple and yellow graphs to diverge before applying our routing changes. After deploying routing around 15:35, both graphs converge, which means in/out amplification is now minimal.</figcaption></figure><p>On the Socket.IO servers end, the CPU usage has been completely slashed. This can be explained by the fact that each worker receives much less payloads per second from RabbitMQ (in our case, this hovers from 8 to 16 times less!), as the load is now distributed on 16 Socket.IO servers from the pool, thanks to the efficient routing of payloads. Note that there might still be some double routing due to failover overlaps &#x2014; <em>more on that later in the article</em>. The CPU cost of running each Socket.IO server has been divided by two, as workers do not need to parse every incoming event anymore to take the routing decision.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-broadcast-queues.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>An overview of our 16 chatbox Socket.IO worker queues EPS in broadcast mode and routed mode, for which the ~16-fold saving can be observed. Note that we kept 2 non-routed workers, that still receive all events &#x2014; also more on that later.</figcaption></figure><hr><h1 id="rabbitmq-routing-keys-to-the-rescue">RabbitMQ Routing Keys to the Rescue</h1><p><em>&#x26A0;&#xFE0F; Not so fast&#x2026; Production crashes ensued!</em></p><p>In our previous topology, other nodes in our infrastructure would publish events that should be routed to Socket.IO clients via an exchange named <code>client</code> (for the chatbox). This exchange is bound to multiple queues (one individual queue for each Socket.IO worker). The queue auto-destroys itself whenever its consumer (the Socket.IO worker) comes offline, which means that they are configured as non-durable, <a href="https://www.rabbitmq.com/queues.html#properties">auto-delete queues</a>. We have also made sure to setup an HA Policy (for <a href="https://www.rabbitmq.com/ha.html">High-Availability Policy</a>) of exactly 1 mirror. This means that the queue will not get mirrored over multiple RabbitMQ cluster nodes, as we do not care about data loss since real-time events that get streamed over Socket.IO are transient &#x2014; this configuration greatly helps reduce induced RabbitMQ CPU usage.</p><p>Whenever publishing an event to a target chatbox (designated by its Crisp session identifier, which is set per-user), the publisher node in our infrastructure would publish the payload on RabbitMQ to the <code>client</code> exchange, with an empty routing key. Since all Socket.IO queues are bound to the <code>client</code> exchange with an empty routing key, the published event would get pushed to each individual queue.</p><hr><h2 id="enforcing-routing-keys-usage">Enforcing routing keys usage</h2><p>Our initial topology, that uses empty routing keys, made the upgrade extremely easy: we would only need to consider that whenever a user connects to their session identifier over the Crisp chatbox Socket.IO server, <strong>the server would bind its queue to the <code>client</code> exchange, with a routing key for the session</strong>, eg. <code>session_3de9c68f</code> (<a href="https://www.rabbitmq.com/tutorials/tutorial-four-python.html">read this tutorial</a> from the RabbitMQ website for more on routing keys). Whenever the Socket.IO room for the session is left, the server would unbind <code>session_3de9c68f</code> from the exchange, and voil&#xE0;!</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/crisp-socketio-join-room.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Hooking onto the Socket.IO join room event, by extending default Socket.IO adapter&#x2019;s <code>addAll()</code> method. This is fired every time a user joins a room. Note that we could as well listen on the <code>create-room</code> event from the Socket.IO server.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/crisp-amqp-bind-route.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Binding the AMQP exchange to the Socket.IO worker queue, with the routing key corresponding to the user session identifier. This is called whenever a Socket.IO room is joined for the first time on this server (ie. it has no member yet).</figcaption></figure><p>On their end, <strong>event publishers would attach a routing key to each payload sent to the <code>client</code> exchange</strong> (here: <code>session_3de9c68f</code>), so that RabbitMQ would only forward the event to the queue bound with the same routing key (here: <code>session_3de9c68f</code>).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/crisp-amqp-publish-routing.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Publishing an AMQP event with a routing key. The routing key contains the target user session identifier, which maps to a chatbox user. This makes sure that the event is only served to the Socket.IO worker that needs it (and not all of them as done before).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/diagram-amqp-topology-routing-full.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>AMQP messages now get passed along with a routing key set, which allows RabbitMQ to route them to the <code>socket-client</code> nodes which are bound to the same routing key. If no node is bound to the message routing key, then the message is dropped, as it routes nowhere.</figcaption></figure><p><strong>This elegant method required minimal change to our topology</strong>, and was built on the top of the AMQP protocol, which had been designed for that. <em>There was, however, a hitch&#x2026;</em></p><hr><h2 id="total-production-collapse-ensued">Total production collapse ensued</h2><p><em>Not all things in this world go as planned every time.</em></p><p>In production, we expected to bind A LOT of Socket.IO rooms to RabbitMQ queues, via routing keys. When I say a lot, I mean 50k+ bindings per queue.</p><p>Knowing this constraint, we made hardcore simulations in our development environment, reproducing 10k+ bindings, with a high bind/unbind rate (that is, adding new routing keys to the exchange-queue association, and removing them very fast). RabbitMQ behaved very well on our MacBooks, so we were very confident that it would work nicely in production. Plus, routing is RabbitMQ&#x2019;s job after all, even if it lacks documentation and benchmarks on the matter.</p><p>It was about time to deploy, so we pushed the new routing system to production. It seemed to be a success: routing behaved perfectly as planned and RabbitMQ bindings were populated with thousands of routing keys, with no hitch.</p><p>We however, started to notice that <strong>RabbitMQ memory usage was growing to ~80% of its <a href="https://www.rabbitmq.com/memory.html">high watermark</a> limit</strong>, then the GC would pass and sweep that memory back to ~20%. This memory allocation pattern would repeat itself on and on every ~10 minutes or so, but the overall cluster was stable, therefore this did not raise any concern on our end.</p><p>A few hours after the deployment, we decided to restart a Socket.IO worker node (1 out of the 16) as part of an unrelated server maintenance. This server was shutdown in a rather routine way. We quickly found out however that something was going wrong on our infrastructure as a whole: Crisp messages were not going through anymore, user availabilities were not being processed anymore, etc. We connected to the RabbitMQ Web Management dashboard, to discover with horror that it was unavailable, throwing <code>HTTP 500</code> errors. <strong>All AMQP queues were stuck, and did not accept any publish order anymore.</strong></p><p>Following the meltdown of our entire RabbitMQ cluster, we connected over SSH to the servers hosting RabbitMQ instances, to find out that all CPUs were going through the roof and memory was swapping like crazy. We came to conclude that tearing down a queue with a lot of routing keys (40k+ in this case) was something RabbitMQ could not handle, at least in production. <strong>It was time to cycle the cluster and bring it back up.</strong></p><p>Later that day, we upscaled the VMs hosting our RabbitMQ cluster to increase their CPU and RAM capacity 4x, and enlarge RabbitMQ memory high watermark limit to 6GB of RAM, and tried again running our new routing system. To no luck, the same cluster meltdown was observed following a single Socket.IO worker shutdown. Only this time, our server had plenty of CPU and RAM capacity left, but all <strong>RabbitMQ nodes still froze upon stopping a Socket.IO worker</strong>!</p><p>After a quick rollback to the previous non-routed topology, we came to the conclusion that <strong>binding that many routing keys was just not going to work</strong>, but another way could be possible.</p><p>After delving in a <a href="https://mikehadlow.com/posts/large-numbers-of-bindings-with-rabbitmq/">blog article</a>, we learnt that RabbitMQ behaved very differently when running as part of a cluster (as in production) from running as a single instance (as in development). It is especially inefficient and slow at synchronizing the exchange-queue routing table across all other RabbitMQ instances in the cluster, which (<em>might?</em>) involve distributed locks and latency.</p><hr><h1 id="finding-the-best-balance-with-prefix-routing">Finding the Best Balance with Prefix Routing</h1><p>Based on our production meltdown observations described above, we came to the conclusion that <strong>we had to reduce the <a href="https://en.wikipedia.org/wiki/Cardinality">cardinality</a> of routing keys on exchange-queue bindings</strong> (meaning, limiting the number of possible routing keys to a predictable number). Not only that, but <strong>we also had to minimize the churn rate on routing key bindings</strong> (that is, the rate at which routing keys get bound and unbound), in order to minimize overhead on routing table synchronization across the RabbitMQ cluster.</p><h2 id="moving-to-prefix-routing-keys">Moving to prefix routing keys</h2><p>The trick in our new solution, is that we should hash the original identifier to be used as a routing key (eg. session or website identifier, where an <a href="https://en.wikipedia.org/wiki/Universally_unique_identifier">UUID</a> such as <code>website_ff6b409b-ef69-4ddb-9cf1-ff80e302361a</code> would become <code>30ea6586fe179c00</code> with the <a href="https://github.com/google/farmhash">Farmhash</a> hashing function), and then prepend its type (eg. website would be <code>w:</code>, so here we obtain: <code>w:30ea6586fe179c00</code>).</p><p><strong>Hashing the raw identifier provides the guarantee it is encoded in a compact hexadecimal form</strong>, for which <strong>each character has only 16 different possible values</strong> (<a href="https://en.wikipedia.org/wiki/Hexadecimal">base 16 encoding</a>). The type we just prepended helps debug which kind of routing key it is, either website (<code>w:</code>), session (<code>s:</code>), or something else. We could have chosen to skip this part though, as the prefix is not required <em>per se</em>.</p><p>Once our full routing key is generated, such as: <code>w:30ea6586fe179c00</code>, <strong>we truncate it to the number of hash characters we want our route prefix to be limited to</strong>. The less characters the prefix holds, the smaller the cardinality is, and therefore <strong>the less bindings will be declared in our RabbitMQ <code>client</code> exchange</strong> (<em>which is a desirable property</em>). We opted for 2 characters.</p><p>Crisp chatbox sessions are bound to two Socket.IO rooms: a session identifier and a website identifier. A website on Crisp corresponds to a messaging inbox, where messages get sent. A website holds many sessions. For load-balancing stickiness reasons that I&#x2019;ll come over on later, in the part on NGINX, we could not use sessions as our routing keys, therefore we opted for websites.</p><p>All that said above, in our Crisp <code>client</code> routing keys use case, for the full routing key: <code>w:30ea6586fe179c00</code>, we would obtain the prefix routing key: <code>w:30</code>.</p><p>Let&#x2019;s now check how many different routing keys could be bound to our exchange (<em>the cardinality</em>) in a worst-case scenario. Knowing that hexadecimal-encoded characters can hold 16 different values, we obtain: <code>Card(s) = 16^s</code>, where <code>s</code> is the truncation size and <code>Card(s)</code> the worst-case cardinality given <code>s</code>. Our truncation size is 2, therefore we obtain: <code>Card(2) = 16 x 16 = 256</code>. <strong>It looks like RabbitMQ could happily handle this number of bindings!</strong> What about a truncation size of 3? That would give us <code>Card(3) = 4096</code>, which could start slowing down our RabbitMQ cluster based on our earlier observations in production, so let&#x2019;s stick on a truncation size of 2.</p><hr><h2 id="deciding-when-to-bindunbind-the-routing-key-on-the-amqp-queue">Deciding when to bind/unbind the routing key on the AMQP queue</h2><p>Your AMQP client code will need to decide when to bind a given routing key prefix to the exchange, for its queue. As well, it should decide when to unbind the routing key when it is not needed anymore, because there are no users connected to the associated Socket.IO room anymore.</p><p><strong>We recommend that you opt for a dynamic routing topology</strong>, where you would hook onto your Socket.IO server events, through the <code>create-room</code> and <code>delete-room</code> events (see <a href="https://socket.io/docs/v3/rooms/#room-events">documentation here</a>), and automatically bind or unbind the RabbitMQ queue to its exchange, for the routing key corresponding to the Socket.IO room.</p><p>It means that when the first users joins a room for a website prefix, then <strong>your Socket.IO worker declares to RabbitMQ that it would like to receive incoming events for that prefix</strong>. Subsequent users to join the same room will not result in further AMQP bindings. When users leave, it will wait for the last user to leave the room to unbind itself from that routing prefix on RabbitMQ.</p><p>Given what we described above about the cost of updating the AMQP routing table across a RabbitMQ cluster (binding and unbinding), and the behavior of certain bogous clients that tend to to open WebSocket connections, close them, open them again, etc., <strong>we advise that you introduce a delay in committing the unbind request to RabbitMQ</strong> (could be 20s). You would cancel this delay if a client was to connect to the same room before the delay fires. This technique also helps ease with massive reconnections (and massive churn rates on bindings) that follow a restart of your NGINX load balancers.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/rabbitmq-queue-bindings.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Prefix routing bindings at play in 1 of our 16 Socket.IO worker AMQP queue.</figcaption></figure><p>We can now enforce routing again in our production infrastructure! But wait&#x2026; aren&#x2019;t so few routing keys possibilities (here, 256) equivalent to our previous system (no routing keys at all) when the number of connected users is high? You are just right! Users will get evenly load-balanced to all of our 16 Socket.IO workers, regardless of the resulting routing prefix. It means that each worker queue will be bound to ~256 RabbitMQ routing keys on the exchange, as the number of WebSocket connections tends to infinity.</p><p><strong>Obviously, we do not want that</strong>. It means that our solution to the routing problem is incomplete there. We lack the NGINX part, described right below.</p><hr><h2 id="nginx-affinity-routing-to-the-rescue">NGINX affinity routing to the rescue</h2><p>While we have put routing into place on our AMQP pipeline, we still need to make sure that <strong>the 256 possible routing prefixes are evenly distributed over our 16 Socket.IO workers</strong>. Best-case, it means that each worker would be bound to around 16 routing prefixes, effectively receiving <code>1/16</code> of total events (ie. <code>[256 routing prefixes] divided by [16 total Socket.IO workers] = 16 prefixes per worker</code>).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/diagram-routing-ring.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Routing ring for the 256 possible prefixes. In reality, NGINX uses a <a href="https://www.metabrew.com/article/libketama-consistent-hashing-algo-memcached-clients">Ketama</a> ring, with <code>2^32</code> possible slots, which is simplified to 256 slots here for legibility. Each slot in the ring maps to an upstream.</figcaption></figure><p>We use <a href="https://nginx.org/">NGINX</a> as the load balancer for all our incoming WebSocket connections. When a user connects the Crisp chatbox, a WebSocket connection to <code>wss://client.relay.crisp.chat/</code> is opened.</p><p>The second trick, here, is to <strong>provide the website identifier in its truncated hash form (therefore, as a prefix), in the connection URL</strong>. We would now want to connect to eg. <code>wss://client.relay.crisp.chat/w/30/</code>. Since the chatbox knows its website identifier before opening the WebSocket connection, but does not necessarily know its session identifier, we opted to route based on the website identifier.</p><p>Now that <strong>the routing prefix is given in the URL, NGINX can use this value to route the connection to a preferred Socket.IO worker</strong>. More specifically, this Socket.IO worker will be responsible for all <code>/w/30/</code> connections, and therefore it will be the only one binding itself to the <code>w:30</code> AMQP routing key.</p><p>We have essentially created an affinity routing system. We let the client decide of the selected Socket.IO worker upstream by including the routing key in the URL, which should in 99.99% cases map to the same AMQP routing key. <strong>This effectively groups Crisp chatboxes with the same routing key on the same Socket.IO worker upstream.</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/diagram-amqp-topology-routing-prefix.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>AMQP messages now get passed to the only <code>socket-client</code> node responsible for the routing prefix set in the message routing key. Connected clients are routed to the correct <code>socket-client</code> node thanks to the affinity key passed in the WebSocket URL.</figcaption></figure><p>How did we adjust our NGINX configuration files to enable affinity routing? Fairly easy; we split our upstreams into: <code>default</code> and <code>affinity</code> pools:</p><ul><li>The <code>default</code> pool is used for legacy clients (those not passing the prefix route in the URL);</li><li>The <code>affinity</code> pool is used for newer clients (those passing the prefix route in the URL).</li></ul><p>The <code>default</code> pool would obviously be the slowest one, bound to more AMQP routing prefixes as no affinity hints are provided by the older clients, and therefore exhibiting a much higher CPU usage. Based on your use-case, you could also choose to completely deprecate support for older clients and omit the <code>default</code> pool. <strong>The <code>default</code> pool uses the traditional round-robin balancing method</strong>, which would distribute connections evenly across the pool.</p><p><strong>The <code>affinity</code> upstreams would use the <a href="https://en.wikipedia.org/wiki/Consistent_hashing">consistent hash</a> balancing method</strong>, based on the routing prefix extracted from the request URL (as a variable). This guarantees that <strong>for a given routing prefix, the same upstream will get selected again and again</strong>. As we add more upstreams in the future, this also ensures that most connections would still be routed to the same upstream as before (<em>giving it a nice stickiness property</em>).</p><hr><p><strong>1. Split your NGINX <a href="https://nginx.org/en/docs/http/ngx_http_upstream_module.html">upstreams</a> into default and affinity:</strong></p><pre><code># Default upstream
upstream default.chatbox-socket.local {
    server worker-1.legacy.chatbox-socket.local:3000  max_fails=0;
    server worker-2.legacy.chatbox-socket.local:3000  max_fails=0;
}

# Affinity upstream (website)
upstream website.chatbox-socket.local {
    hash $affinity_key consistent;

    server worker-1.chatbox-socket.local:3000  max_fails=0;
    server worker-2.chatbox-socket.local:3000  max_fails=0;
    server worker-3.chatbox-socket.local:3000  max_fails=0;
    server worker-4.chatbox-socket.local:3000  max_fails=0;
    
    # &#x2014; (10 values eluded for readability) &#x2014;
    
    server worker-15.chatbox-socket.local:3000  max_fails=0;
    server worker-16.chatbox-socket.local:3000  max_fails=0;
}
</code></pre><hr><p><strong>2. Split your NGINX location block into default and affinity:</strong></p><pre><code># Default pool (round-robin routing)
location / {
    proxy_pass http://default.chatbox-socket.local;

    add_header &apos;X-Crisp-Ray&apos; &apos;default $upstream_addr&apos; always;

    # &#x2014; (add your proxy rules here) &#x2014;
}

# Website pool (affinity hash-based routing)
location ~ &quot;^/(w)/([^/]+)&quot; {
    # Extract routing prefix from the URL (gives eg. &apos;w:00&apos;)
    set $affinity_key $1:$2;

    proxy_pass http://website.chatbox-socket.local;

    add_header &apos;X-Crisp-Ray&apos; &apos;website $affinity_key $upstream_addr&apos; always;
    
    # &#x2014; (add your proxy rules here) &#x2014;

    break;
}
</code></pre><p>Restart your NGINX load balancers, and try connecting to the <code>/</code> path, and eg. the <code>/w/00/</code> path over WebSocket. The <code>X-Crisp-Ray</code> should indicate that the <code>default</code> and <code>website</code> pool are respectively being used, and also output the address of the upstream that was picked up.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2022/05/nginx-affinity-url.png" class="kg-image" alt="Horizontal Scaling of Socket.IO Microservices with RabbitMQ" loading="lazy"><figcaption>Headers served by our NGINX load balancers after we deployed affinity keys in the Socket.IO URLs. Opening multiple WebSocket connections for the same affinity key &#x2014; here, <code>w:87</code> &#x2014; shows that the same upstream is used, which is responsible for handling AMQP events with routing key <code>w:87</code> (see the <code>X-Crisp-Ray</code> header).</figcaption></figure><hr><h2 id="maintaining-redundancy-and-failover-capabilities">Maintaining redundancy and failover capabilities</h2><p>Now that we have affinity sat up, you would want to ask: what happens if the Socket.IO worker responsible for my Crisp website identifier is down? Would my chatbox service be 100% unavailable for the duration of the outage? Well, yes, since that&#x2019;s how affinity works: it maps a route prefix to a specific worker in the pool. Unless&#x2026; we allow NGINX to pick the next available Socket.IO worker upstream in the affinity ring.</p><p>All that we had to do to ensure that NGINX would re-route connections to the next Socket.IO worker in the ring, is to add the following rules in our NGINX <code>location</code> group:</p><pre><code>proxy_next_upstream error;
proxy_next_upstream_tries 2;
</code></pre><p>Those proxy rules are described in details <a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_next_upstream">in the NGINX documentation</a>.</p><p>This limits the next upstream tries to 2 attempts in total, so if the second upstream is down as well, the WebSocket connection would generate an <code>HTTP 502</code> error. <strong>The total number of attempts should be limited</strong>, as you&#x2019;d want to fail early in a situation where a large part of your fleet of Socket.IO workers is down, in order to avoid overloading the healthy side of your fleet with a ton of connections.</p><p>In our case, if a second upstream connection attempt was made, a trace would be left in the <code>X-Crisp-Ray</code> HTTP header, which would hold the comma-separated values of the 2 attempted upstreams (in order).</p><hr><h1 id="closing-notes">Closing Notes</h1><p>The optimization techniques that I described in this article should not be considered too early in a distributed Socket.IO environment that uses RabbitMQ to route events between nodes.</p><p><em>As the <a href="https://web.archive.org/web/20130731202547/http://pplab.snu.ac.kr/courses/adv_pl05/papers/p261-knuth.pdf">saying goes</a>:</em></p><blockquote>Premature optimization is the root of all evil.</blockquote><p>Looking back to our early days at Crisp, it would have been overkill to add this kind of complexity to our infrastructure topology earlier than today. However, with the growth of our user base came scaling issues with our initial (simple) broadcast routing system, that made changes necessary to support our growth in the future.</p><p>I know this article was highly technical, and that very few projects run an architecture similar to ours. If you&#x2019;re interested in distributed microservice architectures, we might have an <a href="https://crisp.chat/en/careers/">engineering position</a> open at Crisp for you!</p><p><strong>I would be happy helping you! If you are running into the same problems as we did, and you still have questions, please leave a comment below. If you are just curious and would like to learn more about the microservice architecture that powers Crisp at scale, please also let me know in the comments section.</strong></p><p><em>&#x1F449; Note that this article has been </em><a href="https://crisp.chat/blog/horizontal-scaling-of-socket-io-microservices-with-rabbitmq/"><em>re-published on Crisp blog</em></a><em>; the original version lives here, on my personal journal.</em></p><hr><p>&#x1F1F5;&#x1F1F9; <em>Written from Lisbon, Portugal.</em></p>


guid: 64fdea757cca3e044663c697
title: Deploy a Nomad Cluster on Alpine Linux with Vultr
author: Valerian Saliou
url: https://valeriansaliou.name/blog/deploy-a-nomad-cluster-on-alpine-linux-with-vultr/
feedurl: https://journal.valeriansaliou.name/rss/
content_mime_type: text/html
content:
<p>After spending countless hours trying to understand how to deploy my apps on Kubernetes for the first time to host <a href="https://mirage-ai.com/">Mirage</a>, an AI API service that I run, I ended up making myself a promise that the next app I work on would be using a more productive &amp; simpler orchestration platform. This is where I stumbled upon <a href="https://www.nomadproject.io">Nomad from HashiCorp</a>.</p><p>Nomad runs your apps from definitions files, as code. It&#x2019;s essentially a small binary that you install on your servers, plus a single configuration file. Nomad embraces the same minimalistic approach to software of UNIX. This makes it incredibly easy for an operator to deploy and manage a Nomad cluster. This is the complete opposite of Kubernetes, where the same task is so complicated that users often resort to using managed Kubernetes clusters.</p><p>This article explains how to deploy a Nomad cluster on cheap <a href="https://www.vultr.com/?ref=9538750-8H">Vultr servers</a>, step-by-step, using a minimal Alpine Linux distribution. The Nomad cluster will be made of 3 Nomad servers and as many Nomad clients as you&#x2019;d like. We will be automating the process of adding more Nomad clients as you grow. Note that Nomad servers are the Nomad agents responsible for receiving jobs (ie. apps you wish to deploy), and managing their allocation on Nomad clients. Nomad clients are the Nomad agents which actually run your apps. While a Nomad agent can be both a server and a client, it&#x2019;s usually a good practice to separate them.</p><p>Once our Nomad cluster is running, we will then proceed to host a <a href="https://github.com/valeriansaliou/vigil">Vigil</a> instance, which is an open-source project of mine that can be used to monitor microservices.</p><hr><h2 id="1-setup-vultr-account-for-nomad">1. Setup Vultr Account for Nomad</h2><p>Vultr is a cloud provider which offers cheap $5/month VPS servers, as well as a wide range of useful cloud services such as virtual private LAN networks (VPCs), cloud firewall, and more. You might already know about DigitalOcean &#x2014; <em>Vultr is quite similar in features and server speed</em>. Vultr runs a datacenter on every continent on the planet, so that you can create a server nearby from wherever you live: US, Mexico, Chile, France, Japan, India, Australia, South Africa, and 24 other locations.</p><p>&#x1F449; <strong>If you don&#x2019;t have an account with Vultr, I&#x2019;d suggest you </strong><a href="https://www.vultr.com/?ref=9538750-8H"><strong>create one using this link</strong></a><strong>, which gives you $100 of free credits, enough to run a single server for more than a year at no cost.</strong></p><p>Once your Vultr account is created, and before we can create our first server, we&#x2019;ll start with setting up:</p><ol><li><strong>Our VPC</strong>: the private local network which will allow our servers to talk to each other privately;</li><li><strong>Our Firewall</strong> secures our server by restricting who can access what from the open Internet;</li><li><strong>A Boot Script</strong>: installs all required dependencies to run Nomad when the servers you create boot for the first time, and also auto-configures your Nomad cluster;</li></ol><p>Let&#x2019;s start!</p><h3 id="11-create-a-vpc-network">1.1. Create a VPC network</h3><p>Nomad agents within our cluster need a secure private network link over which to communicate. We will create a VPC network, which is essentially a private LAN between our Vultr servers. Servers connected to the VPC will expose a secondary network interface named <code>eth1</code>. Nomad will bind itself to the VPC interface.</p><p>In the Vultr dashboard top bar, hover the blue Deploy button, then click &#x201C;Create VPC 2.0&#x201D;. Then, pick your target datacenter region, in which you will also create your actual servers. In my case, I went with Paris.</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-vpc-create.png" class="kg-image" alt loading="lazy" width="2000" height="974" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-vpc-create.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-vpc-create.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-vpc-create.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2023/09/nomad-vultr-vpc-create.png 2400w" sizes="(min-width: 720px) 720px"></figure><p>Then, scroll down, and hit &#x201C;Configure IPv4 Range&#x201D; and enter <code>10.24.96.0/20</code> (<code>&#x200C;10.24.96.0</code> is the network address and <code>20</code> the network prefix).</p><p>We&#x2019;re forcing the VPC address range, as our Boot Script will use this hardcoded IP range. If you opt to use an auto-assigned range, you will later need to edit the Boot Script with the IP range that was assigned to your account.</p><p>Finish by giving the network a name, eg. <code>nomad-cluster</code>. It can also be any value you&#x2019;d like.</p><h3 id="12-create-a-firewall">1.2. Create a Firewall</h3><p>Next, we need to write firewall rules so that access to our server TCP/UDP ports is restricted to certain users only. This is especially important, as we won&#x2019;t be enabling ACLs in this Nomad setup, therefore there won&#x2019;t be any sort of authentication guarding our cluster public HTTP API and Web UI against other users. In a real production setup, you&#x2019;d enable ACLs (more on that at the end of the article). Let&#x2019;s create our firewall!</p><p>In the Vultr dashboard top bar, hover the blue Deploy button, then click &#x201C;Add Firewall Group&#x201D; and create a firewall group named eg. <code>nomad-cluster</code> (again, this can be any name you&#x2019;d like).</p><p>Then, in the firewall, make sure to proceed as such for both:</p><ul><li><strong>IPv4 Rules</strong></li><li><strong>IPv6 Rules</strong></li></ul><p>With the following rules:</p><ul><li><em>Accept</em>: <strong>ICMP</strong> from <strong>Anywhere</strong></li><li><em>Accept</em>: <strong>SSH</strong> port <strong>22</strong> from <strong>My IP</strong></li><li><em>Accept</em>: <strong>TCP</strong> port <strong>4646</strong> from <strong>My IP</strong></li><li><em>Accept</em>: <strong>TCP</strong> port range <strong>8000-8999</strong> from <strong>Anywhere</strong></li><li><em>Drop</em>: <strong>Any</strong> from <strong>Anywhere</strong></li></ul><p>Some explanations on certain ports:</p><ul><li>TCP 4646 is used for the Nomad HTTP API, <strong>it is especially important that access is restricted to your IP for now, as we won&#x2019;t be enabling authentication</strong>. Failing to restrict this port will let any public Internet users deploy apps to your cluster, and do more things;</li><li>TCP 8000 to 8999 will be used for your test apps, we want to share them with the world, so we will be accepting connections from anyone to those ports;</li></ul><h3 id="13-create-a-boot-script">1.3. Create a Boot Script</h3><p>Last step is to create a Boot Script, that will be ran exactly once every time a server gets created on Vultr. This script will setup and configure Nomad on the created server. Extra benefit of this script: whenever Vultr tells you that the server was activated, you won&#x2019;t even have to SSH to the machine, Nomad will be 100% ready to go. Let&#x2019;s create your Boot Script now.</p><p>In the Vultr dashboard top bar, hover the blue Deploy button, then click &#x201C;View More Options&#x201D;, then &#x201C;Add Startup Script&#x201D;.</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-boot-script-create.png" class="kg-image" alt loading="lazy" width="2000" height="731" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-boot-script-create.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-boot-script-create.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-boot-script-create.png 1600w, https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-boot-script-create.png 2273w" sizes="(min-width: 720px) 720px"></figure><p>Call your script <code>setup-nomad-node</code>, mark is as <code>Boot</code> type, and enter the following script (<a href="https://gist.github.com/valeriansaliou/1591f7d326a6a8567c3499a8429434a5">or use this Gist</a>):</p><pre><code class="language-bash">#!/bin/bash

# 1. Setup base dependencies

apk update
apk del ufw
apk add htop busybox-extras

# 2. Disable Swap

swapoff -a
sed -i &quot;/swapfile/d&quot; /etc/fstab
rm /swapfile

# 3. Install Nomad + Docker + Consul

apk add nomad
rc-update add nomad default

apk add consul
rc-update add consul default

if [[ $(hostname) = nomad-cluster-client-* ]]; then
  apk add docker
  rc-update add docker default
fi

# 4. Export environment variables

export NOMAD_CLUSTER_REGION=&quot;global&quot;
export NOMAD_CLUSTER_DATACENTER=&quot;vultr-par&quot;

export NOMAD_CLUSTER_NODE_SELF=&quot;{{ GetInterfaceIP \\\&quot;eth1\\\&quot; }}&quot;
export NOMAD_CLUSTER_SERVER_1=&quot;10.24.96.3&quot;
export NOMAD_CLUSTER_SERVER_2=&quot;10.24.96.4&quot;
export NOMAD_CLUSTER_SERVER_3=&quot;10.24.96.5&quot;

export CONSUL_CLUSTER_DATACENTER=${NOMAD_CLUSTER_DATACENTER}

export CONSUL_CLUSTER_NODE_SELF=${NOMAD_CLUSTER_NODE_SELF}
export CONSUL_CLUSTER_SERVER_1=${NOMAD_CLUSTER_SERVER_1}
export CONSUL_CLUSTER_SERVER_2=${NOMAD_CLUSTER_SERVER_2}
export CONSUL_CLUSTER_SERVER_3=${NOMAD_CLUSTER_SERVER_3}

export CONSUL_CLUSTER_CLIENT_LOCAL=&quot;127.0.0.1&quot;

# 5. Configure Consul

rm /etc/consul/*
touch /etc/consul/server.hcl
chmod 640 /etc/consul/server.hcl
chown consul:consul -R /etc/consul/

if [[ $(hostname) = nomad-cluster-server-* ]]; then
  cat &lt;&lt; EOF &gt; /etc/consul/server.hcl
datacenter           = &quot;${CONSUL_CLUSTER_DATACENTER}&quot;

data_dir             = &quot;/var/consul&quot;

disable_update_check = true
enable_syslog        = false

bind_addr            = &quot;${CONSUL_CLUSTER_NODE_SELF}&quot;
advertise_addr       = &quot;${CONSUL_CLUSTER_NODE_SELF}&quot;
client_addr          = &quot;${CONSUL_CLUSTER_CLIENT_LOCAL}&quot;

server               = true
bootstrap_expect     = 3

retry_join = [
  &quot;${CONSUL_CLUSTER_SERVER_1}&quot;,
  &quot;${CONSUL_CLUSTER_SERVER_2}&quot;,
  &quot;${CONSUL_CLUSTER_SERVER_3}&quot;
]

ui_config {
  enabled = false
}
EOF
else
  cat &lt;&lt; EOF &gt; /etc/consul/server.hcl
datacenter           = &quot;${CONSUL_CLUSTER_DATACENTER}&quot;

data_dir             = &quot;/var/consul&quot;

disable_update_check = true
enable_syslog        = false

bind_addr            = &quot;${CONSUL_CLUSTER_NODE_SELF}&quot;
advertise_addr       = &quot;${CONSUL_CLUSTER_NODE_SELF}&quot;
client_addr          = &quot;${CONSUL_CLUSTER_CLIENT_LOCAL}&quot;

retry_join = [
  &quot;${CONSUL_CLUSTER_SERVER_1}&quot;,
  &quot;${CONSUL_CLUSTER_SERVER_2}&quot;,
  &quot;${CONSUL_CLUSTER_SERVER_3}&quot;
]
EOF
fi

# 6. Configure Nomad

if [[ $(hostname) = nomad-cluster-server-* ]]; then
  cat &lt;&lt; EOF &gt; /etc/nomad.d/server.hcl
region               = &quot;${NOMAD_CLUSTER_REGION}&quot;
datacenter           = &quot;${NOMAD_CLUSTER_DATACENTER}&quot;

data_dir             = &quot;/var/lib/nomad&quot;
plugin_dir           = &quot;/usr/lib/nomad/plugins&quot;

disable_update_check = true
enable_syslog        = false

addresses {
  http = &quot;0.0.0.0&quot;
  rpc  = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  serf = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
}

advertise {
  http = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  rpc  = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  serf = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
}

server {
  enabled          = true
  bootstrap_expect = 3

  server_join {
    retry_join = [
      &quot;${NOMAD_CLUSTER_SERVER_1}&quot;,
      &quot;${NOMAD_CLUSTER_SERVER_2}&quot;,
      &quot;${NOMAD_CLUSTER_SERVER_3}&quot;
    ]
    retry_max      = 0
    retry_interval = &quot;5s&quot;
  }
}

consul {
  address = &quot;${CONSUL_CLUSTER_CLIENT_LOCAL}:8500&quot;
}

ui {
  enabled = true
}
EOF
else
  cat &lt;&lt; EOF &gt; /etc/nomad.d/server.hcl
region               = &quot;${NOMAD_CLUSTER_REGION}&quot;
datacenter           = &quot;${NOMAD_CLUSTER_DATACENTER}&quot;

data_dir             = &quot;/var/lib/nomad&quot;
plugin_dir           = &quot;/usr/lib/nomad/plugins&quot;

disable_update_check = true
enable_syslog        = false

addresses {
  http = &quot;0.0.0.0&quot;
  rpc  = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  serf = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
}

advertise {
  http = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  rpc  = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
  serf = &quot;${NOMAD_CLUSTER_NODE_SELF}&quot;
}

server {
  enabled = false
}

client {
  enabled = true

  servers = [
    &quot;${NOMAD_CLUSTER_SERVER_1}&quot;,
    &quot;${NOMAD_CLUSTER_SERVER_2}&quot;,
    &quot;${NOMAD_CLUSTER_SERVER_3}&quot;
  ]

  options = {
    &quot;driver.allowlist&quot; = &quot;docker&quot;
  }
}

consul {
  address = &quot;${CONSUL_CLUSTER_CLIENT_LOCAL}:8500&quot;
}

plugin &quot;docker&quot; {
  config {
    volumes {
      enabled = true
    }
  }
}
EOF
fi

# 7. Reboot

reboot now
</code></pre><p>&#x1F449; <em>If your VPC IP range is different from <code>10.24.96.0/20</code>, you should perform a search &amp; replace on <code>10.24.96.</code> and replace it with your own VPC IP prefix.</em></p><hr><h2 id="2-create-nomad-servers">2. Create Nomad Servers</h2><p>It&#x2019;s time to deploy your first Vultr servers, running Nomad server agents! We will be deploying 3x Nomad server agents, as this is the sweet spot between safety and cost-optimization in a distributed cluster. Indeed, if a leader Nomad server agent fails, then new elections will need to take place to elect a new leader. As you need majority in an election, a minimum of 3 participants are required.</p><p>Those Nomad server nodes will not host our apps directly, therefore we will be creating cheap tiny Vultr servers.</p><h3 id="21-create-the-nomad-servers">2.1. Create the Nomad servers</h3><p>In the Vultr dashboard top bar, hover the blue Deploy button, then click &#x201C;Deploy New Server&#x201D;.</p><p><strong>Follow those steps then:</strong></p><ol><li>Pick &#x201C;Cloud Compute&#x201D;;</li><li>Pick &#x201C;Intel Regular Performance&#x201D;;</li><li>Select your datacenter region (which must be the same than the location you have created your VPC in), eg. mine is Paris;</li><li>Select the &#x201C;Alpine Linux&#x201D; server image;</li><li>Pick the smallest server size, which should be $5/mth;</li><li>In &#x201C;Additional Features&#x201D;, tick &#x201C;Enable IPv6&#x201D; and &#x201C;Enable VPC&#x201D; 2.0, leave the rest unticked;</li><li>Select the VPC that you previously created;</li><li>Select your Boot Script in Startup Script;</li><li>Pick your SSH keys, or add a new one;</li><li>Select your Firewall Group in the dropdown;</li><li>Adjust &#x201C;Server Quantity&#x201D; so that you create a total of 3 servers;</li><li>Finally, name each of your server as such, in chronological order:</li></ol><ul><li><code>nomad-cluster-server-1</code></li><li><code>nomad-cluster-server-2</code></li><li><code>nomad-cluster-server-3</code></li></ul><p><em>&#x26A0;&#xFE0F; The name of your servers is important, as the Boot Script will use the server name to either bootstrap a Nomad server agent, or a Nomad client agent.</em></p><p>When this is all done, hit &#x201C;Deploy Now&#x201D; and wait for all of your servers to come online.</p><h3 id="22-confirm-that-the-nomad-cluster-is-healthy">2.2. Confirm that the Nomad cluster is healthy</h3><p>When servers are ready, SSH to any one of them, and run the following command:</p><pre><code class="language-sh">nomad server members
</code></pre><p>The command should output the list of your 3 Nomad server agents running in the cluster, with one of them marked as <code>leader</code>. If this works, then it means that your cluster is healthy!</p><p>You may also already access your cluster Web UI from your Web browser, by entering the IP of any server in the 3 that you created, followed by port <code>4646</code>.</p><hr><h2 id="3-create-nomad-clients">3. Create Nomad Clients</h2><p>Before we can deploy our first app, we need to deploy at least one Nomad client agent, which will run our apps on Docker.</p><p>Doing so is as simple as repeating the steps we followed to create our Nomad servers, but this time we will be creating larger, beefier, servers:</p><ol><li>&#x201C;Cloud Compute&#x201D;;</li><li>&#x201C;Intel High Performance&#x201D;;</li><li>Eg. 2vCPU and 2GB RAM, or more;</li></ol><p>You may disable &#x201C;Auto Backups&#x201D; on those ones to save some costs, as those client servers can be considered disposable, because they do not hold any state or sensitive data.</p><p>You may create 1 or more client servers, I heavily recommend at least 2 so that you have failover capabilities on your apps in the event that one client server crashes.</p><p>Finally, name each of your client servers as such, in chronological order:</p><ul><li><code>nomad-cluster-client-1</code></li><li><code>nomad-cluster-client-2</code></li><li>(&#x2026;)</li></ul><p><em>&#x26A0;&#xFE0F; The name of your servers is also important there, as the Boot Script will use them to create a Nomad client agent there.</em></p><p>When this is all done, hit &#x201C;Deploy Now&#x201D; and wait for all of your client servers to come online.</p><hr><h2 id="4-deploy-your-first-app">4. Deploy your First App</h2><p>It is now time to deploy our first app on our Nomad cluster.</p><h3 id="41-deploying-vigil">4.1. Deploying Vigil</h3><p>The first step is to install the Nomad CLI on your local machine. If you are using macOS and Homebrew, it is as easy as running: <code>brew install nomad</code></p><p>Now, we need to write a Nomad definition file that specifies how Nomad should run Vigil, we named this file <code>vigil.hcl</code>:</p><pre><code class="language-hcl">job &quot;vigil&quot; {
  region = &quot;global&quot;
  datacenters = [&quot;vultr-par&quot;]
  type = &quot;service&quot;

  group &quot;vigil&quot; {
    count = 1

    network {
      port &quot;http&quot; {
        static = 8003
      }
    }

    task &quot;vigil&quot; {
      driver = &quot;docker&quot;

      config {
        image = &quot;valeriansaliou/vigil&quot;
        ports = [&quot;http&quot;]

        volumes = [
          &quot;local/vigil.cfg:/etc/vigil.cfg&quot;
        ]
      }

      env {
        SERVER_LOG_LEVEL      = &quot;info&quot;
        SERVER_INET           = &quot;0.0.0.0:8003&quot;
        SERVER_MANAGER_TOKEN  = &quot;0394e257-0a33-4209-b26a-9d1eb73ad320&quot;
        SERVER_REPORTER_TOKEN = &quot;f005fa9f-7a6e-4e16-bfc7-9398575b62c2&quot;
      }

      resources {
        cpu    = 100
        memory = 10
      }

      template {
        destination = &quot;local/vigil.cfg&quot;
        data        = file(&quot;vigil.cfg&quot;)
      }

      service {
        name = &quot;vigil-http&quot;
        port = &quot;http&quot;

        check {
          type     = &quot;http&quot;
          path     = &quot;/&quot;
          interval = &quot;5s&quot;
          timeout  = &quot;2s&quot;
        }
      }
    }
  }
}
</code></pre><p>Next, in the same folder, create Vigil&#x2019;s configuration file named <code>vigil.cfg</code>:</p><pre><code class="language-toml">[server]

log_level = &quot;${SERVER_LOG_LEVEL}&quot;
inet = &quot;${SERVER_INET}&quot;

manager_token = &quot;${SERVER_MANAGER_TOKEN}&quot;
reporter_token = &quot;${SERVER_REPORTER_TOKEN}&quot;

[assets]

path = &quot;./res/assets/&quot;

[branding]

page_title = &quot;Crisp Status&quot;
page_url = &quot;https://status.crisp.chat/&quot;
company_name = &quot;Crisp IM SAS&quot;
icon_color = &quot;#1972F5&quot;
icon_url = &quot;https://valeriansaliou.github.io/vigil/images/crisp-icon.png&quot;
logo_color = &quot;#1972F5&quot;
logo_url = &quot;https://valeriansaliou.github.io/vigil/images/crisp-logo.svg&quot;
website_url = &quot;https://crisp.chat/&quot;
support_url = &quot;mailto:support@crisp.chat&quot;
custom_html = &quot;&quot;

[metrics]

[notify]

reminder_interval = 300

[probe]

[[probe.service]]

id = &quot;web&quot;
label = &quot;Web nodes&quot;

[[probe.service.node]]

id = &quot;status&quot;
label = &quot;Access to status page&quot;
mode = &quot;poll&quot;
replicas = [&quot;https://status.crisp.chat/robots.txt&quot;]
http_body_healthy_match = &quot;User-agent:.*&quot;
</code></pre><p>Finally, submit the job to your Nomad cluster (make sure to insert the IP address of either one of your 3 Nomad servers):</p><pre><code class="language-sh">export NOMAD_ADDR=http://[IP_ADDRESS_OF_NOMAD_SERVER]:4646
nomad job run vigil.hcl
</code></pre><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-app-deploy.png" class="kg-image" alt loading="lazy" width="1989" height="681" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-app-deploy.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-app-deploy.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-app-deploy.png 1600w, https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-app-deploy.png 1989w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Vigil is being deployed on our Nomad cluster, from our local terminal using Nomad CLI.</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-app-vigil.png" class="kg-image" alt loading="lazy" width="2000" height="1276" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-app-vigil.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-app-vigil.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-app-vigil.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2023/09/nomad-vultr-app-vigil.png 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Vigil, running on our Nomad cluster!</span></figcaption></figure><h3 id="42-exploring-nomad%E2%80%99s-web-ui">4.2. Exploring Nomad&#x2019;s Web UI</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-ui-topology.png" class="kg-image" alt loading="lazy" width="2000" height="1276" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-ui-topology.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-ui-topology.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-ui-topology.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2023/09/nomad-vultr-ui-topology.png 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the topology of our Nomad cluster, with all running client agents. Note that here I made a mistake and had to destroy 2 client agents, which are marked as ineligible to run apps.</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-ui-client.png" class="kg-image" alt loading="lazy" width="2000" height="1276" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-ui-client.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-ui-client.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-ui-client.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2023/09/nomad-vultr-ui-client.png 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Detailed view of the Nomad client agent number 2, showing jobs (ie. apps) that are running there.</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2023/09/nomad-vultr-ui-job.png" class="kg-image" alt loading="lazy" width="2000" height="1276" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2023/09/nomad-vultr-ui-job.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2023/09/nomad-vultr-ui-job.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2023/09/nomad-vultr-ui-job.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2023/09/nomad-vultr-ui-job.png 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the job that runs Vigil. This is where you can get the IP and port of the service that lets you access Vigil from the outside.</span></figcaption></figure><hr><h2 id="5-important-security-considerations">5. Important Security Considerations</h2><p>The nomad cluster that we deployed herein is by no means secure. For simplicity&#x2019;s sake, I haven&#x2019;t covered here the topics of ACLs, mTLS, and symmetric encryption of the gossip protocol between servers. You may read HashiCorp&#x2019;s documentation about <a href="https://developer.hashicorp.com/nomad/docs/concepts/security">Nomad&#x2019;s security model</a>.</p><p>If you would like to make your Nomad cluster production-ready, you should explore those topics and apply the necessary configuration changes:</p><ul><li>Enable <a href="https://developer.hashicorp.com/nomad/tutorials/transport-security/security-gossip-encryption">gossip encryption</a>;</li><li>Deploy <a href="https://developer.hashicorp.com/nomad/tutorials/access-control">ACLs</a>;</li><li>Setup <a href="https://developer.hashicorp.com/nomad/tutorials/transport-security/security-enable-tls">mTLS</a> (mutual TLS);</li></ul><p>Please share what kind of workloads you are running in the comments below! I&#x2019;d love to hear about your experience setting up Nomad, running apps, as well as any question you&#x2019;d have.</p><p><em>On my side, Nomad will come handy at helping me simplify and automate the complex 100+ servers setup that runs </em><a href="https://crisp.chat/en/"><em>Crisp</em></a><em>, removing the need for frequent human intervention. Automation is key to building a profitable business!</em></p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Lyon, France.</em></p>


guid: 67479bdc89d0950001193cfd
title: Retrying failed requests to an HTTP API with Axios
author: Valerian Saliou
url: https://valeriansaliou.name/blog/retrying-failed-requests-http-api-axios/
feedurl: https://journal.valeriansaliou.name/rss/
content_mime_type: text/html
content:
<p>During the day, I&#x2019;m running company called <a href="https://crisp.chat/en/">Crisp</a>. We&#x2019;re building a customer support software used by 500k companies in the world. We distribute a large VueJS Web application that interacts with an HTTP REST API to load conversations, send messages, and much more.</p><p>Given the highly distributed nature of our user base, we use Cloudflare Tunnel to serve our HTTP API at low latencies, connecting to our systems running in a single datacenter in Amsterdam. This architecture works great 99.9% of the time. However, traffic between Cloudflare PoPs and our servers transit over the public Internet, which might be subject to packet loss or traffic interruption, or even human error (by Cloudflare or ourselves).</p><hr><h2 id="some-context">Some context</h2><p><em>You may skip this part if you came for the JavaScript code!</em></p><p>The current worldwide geopolitical situation has caused many issues for us in 2024, the worst being around May 2024 when a <a href="https://edition.cnn.com/2024/03/04/business/red-sea-cables-cut-internet/index.html">subsea cable cut in the Red Sea</a>, in addition to an earlier <a href="https://www.datacenterdynamics.com/en/news/saboteurs-cut-fiber-cables-in-france-in-second-incident-this-year/">fiber sabotage in a landing point in Marseille, France</a> in 2022, caused the re-routing of a large portion of the Internet traffic between Europe and Asia. Due to the spikes in Internet traffic between those 2 regions at certain times of the day, links became congested, which resulted in very high packet loss between our EU servers and most locations in Asia.</p><p>Yes, I was as surprised as you right now, when I learned that a large portion of European Internet traffic with Asia takes the same route, with no proper geographical redundancy.</p><p>During hours where EU-Asia paths were congested, Crisp customers in Vietnam to Singapore were unable to reliably access their Crisp apps because:</p><ul><li>Some HTTP requests to Crisp API got back to them with 529 and other 5xx HTTP errors</li><li>Some HTTP requests took 10s and more to complete (although they succeeded!), sometimes hitting a client timeout</li></ul><p>Much more recently (last week), an <a href="https://www.cloudflarestatus.com/incidents/f2c54nzb5dbr">over-night maintenance in the Cloudflare AMS PoP</a> to which our Cloudflare Tunnel agents are connected resulted in a total worldwide outage of Crisp applications, lasting 2 hours (!!), with no immediate remedy. Cloudflare&#x2019;s control plane stubbornly served our Clouflare Tunnel agents with AMS PoP during the maintenance (!!!), while we expected them to direct our tunnels to the nearby Frankfurt PoP, which was not under maintenance.</p><p>This was too much. Time to work on an emergency Cloudflare bypass for our apps.</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2024/11/axios-retries-api-downtime.png" class="kg-image" alt loading="lazy" width="1702" height="912" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/axios-retries-api-downtime.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/axios-retries-api-downtime.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/11/axios-retries-api-downtime.png 1600w, https://valeriansaliou.name/blog/content/images/2024/11/axios-retries-api-downtime.png 1702w" sizes="(min-width: 720px) 720px"></figure><p><em>Cloudflare Tunnel, I still love you</em>! Despite all the issues described above, Cloudflare Tunnel is still an amazing product that serves well our needs and just works fine most of the time. However, it seems to be very sensitive to packet loss. The entire tunnel can suffer performance degradation if the link is not stable, resulting in 5xx errors for HTTP clients.</p><hr><h2 id="axios-retry-coming-to-the-rescue">Axios Retry coming to the rescue!</h2><p>Back in May 2024, in order to solve the situation with customers not being able to access Crisp from Asia and resume service for them, a solution we came up with was to set up a secondary domain to our HTTP API, on a direct NGINX server with a LetsEncrypt SSL certificate. This failover API host would act as a Cloudflare Tunnel bypass.</p><p><strong>This would leave us with 2 API hosts:</strong></p><ul><li><code>api.crisp.chat</code> which would go through Cloudflare Tunnel</li><li><code>rescue.api.crisp.chat</code> which would hit a direct NGINX server</li></ul><p>Our application would never need to use <code>rescue.api.crisp.chat</code> in normal times, and would always make HTTP requests to <code>api.crisp.chat</code> since it&#x2019;s lower latency and more secure. In the event of a downtime of <code>api.crisp.chat</code>, our application would however retry any failed HTTP request to <code>rescue.api.crisp.chat</code>.</p><p>Since we use Axios to make HTTP calls to our REST API, we used the amazing <a href="https://www.npmjs.com/package/axios-retry">axios-retry</a> plugin. It acts as a response interceptor on the top of your Axios client, retrying any failed request, for the desired number of times. This means that you do not need to make any change to your existing source code. You can also define on which conditions to retry a request, for instance on which HTTP status code.</p><p>In our case, we wanted to retry any <a href="https://www.theserverside.com/tip/Idempotent-HTTP-methods-and-REST">idempotent</a> HTTP request once, and at the same time opportunistically change the API host from the main host to the failover host.</p><p><strong>We came up with this simple code:</strong></p><pre><code class="language-js">const HTTP_RETRY_ATTEMPTS = 1; // Retry failures once
const HTTP_RETRY_DELAY = 500; // Wait 1/2 second between retries

const HTTP_RETRY_STATUS_FROM = 502; // Retry HTTP 502 to...
const HTTP_RETRY_STATUS_TO = 599; // HTTP 599

// Retry idempotent methods, DO NOT retry POSTs since they could result in \
//   same resources being created multiple times in the database, or a message \
//   being sent twice or more.
const HTTP_RETRY_METHODS = [&quot;get&quot;, &quot;head&quot;, &quot;options&quot;, &quot;put&quot;, &quot;delete&quot;];

axiosRetry(this._client, {
  retries: HTTP_RETRY_ATTEMPTS,

  retryDelay: () =&gt; {
    return HTTP_RETRY_DELAY;
  },

  retryCondition: (error) =&gt; {
    let _status = error?.response?.status || -1;

    // Important: we want to retry requests when we get a status code \
    //   that indicates the error occurred at the edge (ie. Cloudflare) \
    //   rather than at the origin. We DO NOT want ot spam our origins \
    //   with retries here (origin 5xx are transformed to 408). This \
    //   mechanism helps &apos;fixing&apos; most temporary communication errors \
    //   between Cloudflare edges and Crisp origins due to temporary \
    //   cross-region traffic flow interruptions (eg. backbone \
    //   congestion). Also, we want to retry requests for which we got no \
    //   response at all, because of a possible network timeout.
    if (
      _status === -1 ||
      (_status &gt;= HTTP_RETRY_STATUS_FROM &amp;&amp;
      _status &lt;= HTTP_RETRY_STATUS_TO)
    ) {
      let _method = error.config?.method || null;

      // Ensure we know the method (since we only want to retry idempotent \
      //   requests)
      // Notice: some non-idempotent requests may also be marked as \
      //   retriable, meaning they can safely be retried.
      if (_method !== null &amp;&amp; HTTP_RETRY_METHODS.includes(_method)) {
        // Do retry this request
        return true;
      }
    }

    // Do not retry by default
    return false;
  },

  onRetry(retryCount, _, config) {
    // Use different API host? (for next and last retry request)
    if (retryCount === HTTP_RETRY_ATTEMPTS) {
      // Change next base URL to use (roll in circle)
      config.baseURL =
        (config.baseURL !== CONFIG.CRISP_URL_API_FAILOVER)
          ? CONFIG.CRISP_URL_API_FAILOVER
          : CONFIG.CRISP_URL_API;
    }
  },

  // Important: reset timeout between retries, since we may want to retry \
  //   after a timeout occurred on the main API endpoint, although the \
  //   failover would answer to the same request with no issue. This fixes \
  //   retries not firing if the first request timed out at the network \
  //   level.
  shouldResetTimeout: true
});
</code></pre><p>Cool! Testing retries locally involved adding some <code>Math.random()</code> to our backend code and sending random simulated HTTP 529 responses. Our application performed well and retried failed requests transparently, hitting the failover API as expected.</p><p>Note that if an intercepted HTTP response has no status code (we set it to a value of <code>-1</code> in this case), it means that the request has possibly timed out, that is, no response was produced by the server. This may indicate that the user computer is not connected to a network, but it could also be the result of the Cloudflare PoP being too slow to transmit the response from our servers to the end-user, triggering an HTTP client timeout. It is therefore important to retry such failures, due to what I said in the latter point.</p><p>It is however super important NOT to retry requests for which we received a response code that indicates a server application-level error. That is, if the HTTP API responds with HTTP 500, it indicates a server error that&#x2019;s most likely caused by a bug. Retrying the request over a failover network path will produce the same HTTP 500 response, since it is caused by an issue in our API.</p><hr><h2 id="improving-user-experience-during-downtimes">Improving user experience during downtimes</h2><p>While the retry system described above is a very easy and efficient fix to ensure the Crisp application is still usable during a period of downtime of our main HTTP API, in situations where the downtime is prolonged or where the main HTTP endpoint is 100% unreachable, it implies the following:</p><ul><li>The application will work in degraded mode, since non-idempotent HTTP requests will not be retried (POST and PATCH methods), therefore some features might be unusable.</li><li>The application might run considerably slower, if requests timeout after the configured HTTP client timeout of 5s, since <code>axios-retry</code> will kick in for a given request after the first request timed out, thus the application UI might take ages to load data. Eg. if you open a conversation or your CRM during a downtime event, some parts of the app might take 10 seconds to load!</li></ul><p>Well... the fix might be easy here too&#x2026; Why not forcing all future requests to the failover HTTP API instead of attempting every single request to the main HTTP API, if we know the main HTTP API is unreachable? And then, maybe later, try restoring the regular HTTP API when we know it&#x2019;s back up again?</p><p>During the 2h total worldwide downtime of our Cloudflare Tunnels in November 2024, we discovered that our simple <code>axios-retry</code> hack from May 2024 was far from perfect. It sometimes worked, it sometimes did not and resulted in an unusable app during the downtime, even if the failover HTTP API was perfectly reachable. While it helped with our packet loss issue between Asia and Europe, in total downtime scenarios it just did not work for us.</p><p>We therefore came up with a system that we called &#x201C;Failover HTTP API lock mode&#x201D;, consisting of temporarily changing the main HTTP API host to only use the failover host.</p><p><strong>Its principles of operation are the following:</strong></p><ol><li>After 3 failed requests (retries triggered) in the last 1 minute then enable full failover mode for the next 10 minutes for all future requests.</li><li>If failure occurs in the same way as explained in point 1 while in failover lock mode, then disable failover mode and use principal URL back again.</li><li>Per-request retries should be performed on principal URL if first attempt uses failover URL (circle back between URLs).</li></ol><p>Building up on our existing <code>axios-retry</code> code, we added the following:</p><p>We added a global variable to account for retries:</p><pre><code class="language-js">retryCounter = {
  startDate: 0, // Timestamp
  rollbackDate: 0, // Timestamp
  retries: 0 // Counter
};
</code></pre><p>We modified our <code>onRetry()</code> handler so that it now looks like:</p><pre><code class="language-js">onRetry(retryCount, _, config) {
  if (retryCount === HTTP_RETRY_ATTEMPTS) {
    // (...existing code here...)
 
    // Bump retry count (this allows entering or exiting failover lock mode)
    incrementRetryCounter();
    assertRetryCounter();
  }
}
</code></pre><p>Now, we defined some constants based on the principles described above &#x2014; <em>3 retries over the last 1 minute, enabling lock mode for the next 10 minutes</em>:</p><pre><code class="language-js">const HTTP_RETRY_LOCK_MODE_ATTEMPTS = 3; // 3 total retries
const HTTP_RETRY_LOCK_MODE_COUNT_TIMEFRAME = 60000; // 1 minute
const HTTP_RETRY_LOCK_MODE_SWITCH_TIMEFRAME = 600000; // 10 minutes
</code></pre><p>And then we defined the function that increments the global retry counter. This method manages the retry counter and starts the counter again if we passed the 1 minute counting timeframe:</p><pre><code class="language-js">function incrementRetryCounter() {
  // Start a new retry counter? (too long since start of counter)
  let _dateNow = Date.now();

  if (
    _dateNow - retryCounter.startDate &gt;=
    HTTP_RETRY_LOCK_MODE_COUNT_TIMEFRAME
  ) {
    retryCounter.startDate = _dateNow;
    retryCounter.retries = 0;
  }

  // Increment retry count
  retryCounter.retries++;
}
</code></pre><p>Finally, we defined the method that checks the global retry counter, and switches the base URL used by <code>axios</code> for all HTTP requests, if we need to enter failover lock mode:</p><pre><code class="language-js">function assertRetryCounter() {
  // Change main API host? (enter failover lock mode)
  if (retryCounter.retries === HTTP_RETRY_LOCK_MODE_ATTEMPTS) {
    // Toggle base URL
    let _newBaseUrl = toggleBaseUrl();

    // Schedule rollback date (we are now in failover lock mode)
    retryCounter.rollbackDate =
      Date.now() + HTTP_RETRY_LOCK_MODE_SWITCH_TIMEFRAME;
  }
}
</code></pre><p>We also needed a function to toggle the base URL in our <code>axios</code> client:</p><pre><code class="language-js">function toggleBaseUrl(forceDefault = false) {
  // This function helps toggle on or off the failover lock mode
  let _baseUrl = this._client.defaults.baseURL;

  // Acquire new main base URL
  let _newBaseUrl =
    (_baseUrl !== CONFIG.CRISP_URL_API_FAILOVER &amp;&amp; forceDefault !== true)
      ? CONFIG.CRISP_URL_API_FAILOVER
      : CONFIG.CRISP_URL_API;

  // Change main URL to failover URL (or back to main URL)
  this._client.defaults.baseURL = _newBaseUrl;

  // Reset retries counter
  // Notice: this is important, otherwise next asserts will never fire a \
  //   toggle when they should have done so.
  retryCounter.startDate = 0;
  retryCounter.retries = 0;

  return _newBaseUrl;
}
</code></pre><p>Great! Now, simulating some HTTP 529 failures again from the backend, it was fairly easy to confirm that we entered failover lock mode after 3 failed attempts to the main HTTP API host in the same 1 minute.</p><p>But&#x2026; how do we exit the failover lock mode once the main HTTP API is back online?</p><p>The solution we came up with here is to decide whether to exit an active failover lock mode or not whenever making an HTTP request to our API. If we are in failover lock mode and it&#x2019;s been more than 10 minutes that we entered failover lock mode, we would roll back the base URL used by <code>axios</code> to its default value: our main HTTP API host. If we rolled back to the main HTTP API and all next 3 HTTP requests failed, we would then enter failover lock mode back again.</p><p><code>axios</code> comes up with request interceptors, that let us hook onto any HTTP request that is going to be sent, and run some code before the actual request gets executed. It is the perfect place to restore the regular HTTP API URL once failover lock mode expires after 10 minutes:</p><pre><code class="language-js">// Add request interceptor (automatically gets out of an active failover \
//   lock mode)
this._client.interceptors.request.use((request) =&gt; {
  // Check if should rollback an active failover lock mode
  assertRollbackRetryCounter();

  // Important: return un-transformed request.
  return request;
});
</code></pre><p>And finally we defined our URL rollback checker function:</p><pre><code class="language-js">function assertRollbackRetryCounter() {
  // Should the base URL be toggled from failover back to the main URL?
  if (
    retryCounter.rollbackDate &gt; 0 &amp;&amp;
    Date.now() &gt;= retryCounter.rollbackDate
  ) {
    if (this._client.defaults.baseURL !== CONFIG.CRISP_URL_API) {
      // Toggle base URL (force usage of default URL)
      toggleBaseUrl(true);
    }

    // Reset rollback date (not in failover lock mode anymore)
    retryCounter.rollbackDate = 0;
  }
}
</code></pre><p>That&#x2019;s it!</p><p>We&#x2019;ve just improved the resiliency of our application to failed network paths, temporary proxying and tunneling errors and HTTP timeouts, with absolutely no server-side changes.</p><p>I&#x2019;m now waiting for the next downtime to properly test this hardening IRL! &#x1F603;</p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 674b38d0f04ce70001cafb95
title: Designing a Tray for my Vintage Apple Keyboard & Trackpad
author: Valerian Saliou
url: https://valeriansaliou.name/blog/tray-for-my-vintage-apple-keyboard-trackpad/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2024/11/tray-banner.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2024/11/tray-banner.jpg" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad"><p>I really like the ergonomics of my MacBook Pro 16&#x201D; M1 Pro. The location of the trackpad, beneath the keyboard, makes it very comfortable for my hands to switch between writing and moving the mouse. I also enjoy working on my large 27&#x201D; LG UltraFine display, which is basically a plastic-made Apple Studio Display &#x2014; the display panel is very similar, almost identical.</p><p>At the same time, I do not like working on 2 displays simultaneously, this destroys my attention which in turn badly affects my productivity. What this means, is that I work on an external keyboard, mouse and trackpad.</p><p>I tried placing the trackpad on the right of my keyboard, but it really hurts my arm since I need to constantly perform lateral moves. Kind of like a cashier arm movement. This lateral move is also a slow one, which increases my latency when switching from typing to moving my cursor.</p><p>After looking online for well-made trays &#x2014; basically, a plastic or wooden part that would lock the keyboard and the trackpad together into a single solid piece, the trackpad sitting right below the keyboard, I only found them made for the last-generation Apple Magic Keyboard and Magic Trackpad.</p><p>I own the previous generation keyboard and trackpad, what a bummer &#x2014; I&#x2019;ll need to design my own tray from scratch!</p><p>After 1 hour of tinkering with Fusion 360, I got a prototype, which I sliced into 4 parts that could be printed on my FDM 3D printer, since the whole part did not fit the build tray. I then proceeded to assemble the 4 parts together with Neoprene glue.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/tray-fusion360.png" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="2000" height="951" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/tray-fusion360.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/tray-fusion360.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/11/tray-fusion360.png 1600w, https://valeriansaliou.name/blog/content/images/size/w2400/2024/11/tray-fusion360.png 2400w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The tray body in Fusion 360</span></figcaption></figure><p>If you are interested of getting the 3D file to reproduce the tray on your end, I <a href="https://www.printables.com/model/1039974-apple-magic-keyboard-and-trackpad-tray-2012-models">shared all the design sources online on Printables</a>. You can also download the Fusion 360 source file if you need to modify the design, for instance reducing the padding on the sides if you want to match the ergonomics of the smaller MacBook Pro 14&#x201D;.</p><p>The final design validated and fine-tuned (it was almost a first-time perfect match), I sent it over to <a href="https://www.hubs.com/">Hubs</a> for final printing on a large FDM printer, at a total cost of around 100&#x20AC;. The printed part got shipped to my address in Europe about 4 days after ordering it online.</p><p><strong>Here&#x2019;s the final result:</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-structure.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-structure.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-structure.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-structure.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the naked tray from above</span></figcaption></figure><p><strong>How it looks like on my desk:</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-general.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-general.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-general.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-general.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">The tray sitting on my desk</span></figcaption></figure><p><strong>More views of the part:</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-close.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-close.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-close.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-close.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the tray from an angle</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-behind.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-behind.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-behind.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-behind.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the tray from behind</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-above.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-above.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-above.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-above.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the tray from above</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-trackpad-only.jpg" class="kg-image" alt="Designing a Tray for my Vintage Apple Keyboard &amp; Trackpad" loading="lazy" width="1600" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/11/keyboard-tray-trackpad-only.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/11/keyboard-tray-trackpad-only.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/11/keyboard-tray-trackpad-only.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">View of the tray with keyboard removed</span></figcaption></figure><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 674f09bc7125c50001304f01
title: Automating Nomad deployments from GitHub Actions
author: Valerian Saliou
url: https://valeriansaliou.name/blog/automating-nomad-deployments-github-actions/
feedurl: https://journal.valeriansaliou.name/rss/
content_mime_type: text/html
content:
<p>I&#x2019;ve recently migrated both Prose server and my personal server over to Nomad. Services such as websites are now running in the Nomad cluster as independent Docker containers, which are proxied to by a NGINX load balancer also running in the cluster.</p><p>Once the migration to Nomad was completed, I was looking for an easy way to trigger automatic updates of my running services, since the recommended way to update a Nomad job is to modify its HCL job file and update the Docker image tag.</p><p>The image update process is quite tedious in practice, since it implies releasing the Docker image to your container registry (probably automated using a CI/CD pipeline), and then asking your Nomad cluster to pull that image you just released (this step is to be done <strong>by hand</strong> by editing your Nomad job file and submitting the job update to your Nomad cluster). <em>Wouldn&#x2019;t there be a better way?</em></p><p>After searching Nomad&#x2019;s documentation and communities on the matter, and being familiar with Kubernetes, I was looking for a Nomad CLI command similar to Kube&#x2019;s:</p><pre><code class="language-sh">kubectl set image deployment/my-deployment mycontainer=myimage
</code></pre><p>Alas, I realized there was no other way to request an image update with Nomad, than to manually edit the job HCL file. We&#x2019;ll therefore be looking to hack around this manual job edit process to automate image update rollouts from Nomad&#x2019;s CLI.</p><hr><h2 id="1-nomad-variables-to-the-rescue">1. Nomad variables to the rescue</h2><p>Nomad provides a way to store variables, and use them in various places in job files. Variables can be read and modified. If modified, any job using the modified variable can be re-evaluated by the control plane and trigger updates of running containers.</p><p>Let&#x2019;s say we have the following Nomad job file for <code>prose-web</code>, which runs <a href="https://prose.org/">Prose&#x2019;s website</a> (I reduced it to its essential parts for this example):</p><pre><code class="language-hcl">job &quot;prose-web&quot; {
  type = &quot;service&quot;

  group &quot;prose-web&quot; {
    count = 1

    task &quot;prose-web&quot; {
      driver = &quot;docker&quot;

      config {
        image = &quot;ghcr.io/prose-im/prose-web:v1.0.0&quot;
      }
    }
  }
}
</code></pre><p>Here, we&#x2019;re looking at replacing <code>v1.0.0</code> in the Docker image tag to a variable. Let&#x2019;s call this variable <code>IMAGE_TAG</code>.</p><p>First, we&#x2019;re modifying the Docker <code>config</code> so that it becomes:</p><pre><code class="language-hcl">config {
  image = &quot;ghcr.io/prose-im/prose-web:${IMAGE_TAG}&quot;
}
</code></pre><p>Note that <code>IMAGE_TAG</code> is an environment variable in this context; it does not refer to a Nomad variable. We will need to define this environment variable somewhere, and tell Nomad to assign the environment variable value from an actual Nomad variable.</p><p>To achieve that, let&#x2019;s add a <code>template</code> block and tell Nomad to source this file to our environment variables with <code>env = true</code>. This assigns the value of Nomad&#x2019;s variable <code>IMAGE_TAG</code> at path <code>nomad/jobs/prose-web</code> to an environment variable of the same name:</p><pre><code class="language-hcl"># Those variables allow updating the deployed image from CI/CD
template {
  data = &lt;&lt;EOH
IMAGE_TAG=&quot;{{ with nomadVar &quot;nomad/jobs/prose-web&quot; }}{{ .IMAGE_TAG }}{{ end }}&quot;
  EOH

  destination = &quot;local/run.env&quot;
  env         = true
}
</code></pre><p>Perfect! To summarize, our job file now looks like:</p><pre><code class="language-hcl">job &quot;prose-web&quot; {
  type = &quot;service&quot;

  group &quot;prose-web&quot; {
    count = 1

    task &quot;prose-web&quot; {
      driver = &quot;docker&quot;

      # Those variables allow updating the deployed image from CI/CD
      template {
        data = &lt;&lt;EOH
IMAGE_TAG=&quot;{{ with nomadVar &quot;nomad/jobs/prose-web&quot; }}{{ .IMAGE_TAG }}{{ end }}&quot;
        EOH

        destination = &quot;local/run.env&quot;
        env         = true
      }

      config {
        image = &quot;ghcr.io/prose-im/prose-web:${IMAGE_TAG}&quot;
      }
    }
  }
}
</code></pre><p>Now, create a Nomad environment variable named <code>IMAGE_TAG</code>, at path <code>nomad/jobs/prose-web</code>, with a value of <code>latest</code> (this will be updated later on to a version value when deploying, eg. <code>v1.0.0</code>).</p><p>Finally, submit your updated job definition to Nomad with:</p><pre><code class="language-sh">nomad job run your-job.hcl
</code></pre><p>You should now observe your job update being rolled out, with Docker containers at tag <code>latest</code> being pulled on your Nomad client nodes.</p><p>If that worked, we can now proceed to the next step: automatically updating our newly-created <code>IMAGE_TAG</code> variable from our CI/CD pipeline (here: GitHub Actions). But first we have some preparation work to do.</p><hr><h2 id="2-securely-exposing-nomad-api-publicly">2. Securely exposing Nomad API publicly</h2><p>Before we proceed with GitHub Actions, we need to expose our Nomad API in a secure way, so that a Nomad CLI can administrate your Nomad control plane from the public Internet. This is required so that GitHub Action runners can hit your Nomad&#x2019;s API.</p><p>Since I&#x2019;m also running a NGINX load balancer on my Nomad cluster, I&#x2019;ve simply added a <code>admin.prose.org</code> virtual host that proxies to my Nomad API:</p><pre><code class="language-nginx"># --- [admin.prose.org] ---

upstream host-nomad {
    ip_hash;

    # We are proxying to the host Nomad here (running on the same server, \
    #   since Nomad runs on the host system). The target port is local and is \
    #   not accessible by public users, for security reasons. This upstream is \
    #   therefore marked as &apos;host&apos;.
    server [::1]:4646 max_fails=0;
}

server {
    listen [::]:443 ssl;
    server_name admin.prose.org;

    root /dev/null;

    location / {
        # Authenticate users, adding an extra layer of security on the top of \
        #   Nomad API, which may expose some public routes with no ACL.
        auth_basic            &quot;Authenticate with your Proxy Auth keypair&quot;;
        auth_basic_user_file  htpasswd/admin;

        proxy_pass http://host-nomad;

        proxy_connect_timeout 5s;
        proxy_read_timeout    600s;
        proxy_send_timeout    10s;

        proxy_http_version 1.1;

        proxy_set_header Connection &quot;upgrade&quot;;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header Early-Data $ssl_early_data;

        # Do not buffer responses (for Nomad log streaming)
        proxy_buffering off;
    }
}
</code></pre><p>It is to be noted here that I&#x2019;m running my load balancer job in Docker host network mode, so that it has access to my local loopback and can connect to Nomad&#x2019;s API, running on the same server. If you&#x2019;re running a multi-node Nomad cluster, with Nomad clients set apart from Nomad servers, you would have to connect over the LAN instead (modify the <code>upstream</code> block accordingly).</p><p>In my case, I needed to change the <code>network_mode</code> in my <code>task/config</code> block:</p><pre><code class="language-hcl"># Required to be able to see real servers/clients IPs
network_mode = &quot;host&quot;
</code></pre><p>Also, make sure to create a file named <code>htpasswd/admin</code> containing your HTTP Basic authentication username and password. I&#x2019;d suggest reading NGINX&#x2019;s guide on <a href="https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/">how to setup HTTP Basic authentication</a>.</p><hr><h2 id="3-create-nomad-deployment-user-token-and-policy">3. Create Nomad deployment user token and policy</h2><p>We&#x2019;re certainly not going to use our Nomad super-admin bootstrap token with GitHub Actions. We will need to setup a restricted Nomad policy and token.</p><p>Start by creating a new policy, that we&#x2019;ll call <code>deployer</code>, in a file named <code>deployer.policy.hcl</code>:</p><pre><code class="language-hcl">namespace &quot;default&quot; {
  policy = &quot;deny&quot;

  variables {
    path &quot;nomad/jobs/prose-web&quot; {
      capabilities = [&quot;write&quot;]
    }
  }
}
</code></pre><p>This policy gives restricted write access to certain variable paths which you use for jobs that you want to deploy automatically. In our case, the <code>nomad/jobs/prose-web</code> path.</p><p>Then, import that policy to Nomad:</p><pre><code class="language-sh">nomad acl policy apply deployer deployer.policy.hcl
</code></pre><p>Now, create a new token for your <code>deployer</code> policy:</p><pre><code class="language-sh">nomad acl token create -name &quot;GitHub Actions&quot; -policy deployer
</code></pre><p>Take note of the token secret that was issued, we will set it up later on as a GitHub secret.</p><hr><h2 id="4-updating-a-nomad-variable-from-github-actions">4. Updating a Nomad variable from GitHub Actions</h2><p>Great! Now that our job file uses a Nomad variable to specify the Docker image tag, and that our Nomad API is publicly accessible, we can automate the image tag update from our GitHub Actions pipeline. That&#x2019;s the easy step.</p><h3 id="configuring-github-secret-variables">Configuring GitHub secret variables</h3><p>First thing is to setup secrets containing our Nomad API token and our HTTP Basic authentication username/password. We also need to create a regular variable containing our Nomad API URL.</p><p>For improved security, I&#x2019;ve chosen to setup a GitHub Environment, so that only GitHub Action jobs triggered on the <code>production</code> Git branch have access to my secret variables.</p><p>Supposing you&#x2019;re also using the <code>production</code> branch for your deployments, an environment can be easily created in your GitHub project settings:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-environment-list.png" class="kg-image" alt loading="lazy" width="1586" height="416" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-ci-github-environment-list.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-ci-github-environment-list.png 1000w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-environment-list.png 1586w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Create an environment in your GitHub project settings</span></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-environment-settings.png" class="kg-image" alt loading="lazy" width="1634" height="1512" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-ci-github-environment-settings.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-ci-github-environment-settings.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/12/nomad-ci-github-environment-settings.png 1600w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-environment-settings.png 1634w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Restrict your environment to the production branch, and add your secrets and variables.</span></figcaption></figure><h3 id="setting-up-a-deployment-github-actions-step">Setting up a deployment GitHub Actions step</h3><p>The final step, is to update our existing GitHub Actions workflow and add a new <code>deploy</code> step, following an existing <code>build</code> step that would publish our Docker image to GitHub&#x2019;s container registry:</p><pre><code class="language-yaml">on:
  push:
    branches:
      - production

name: Build and Deploy

jobs:
  build:
    # (previous job step here, to release Docker image)

  deploy:
    needs: [build]
    environment: prose.org
    runs-on: ubuntu-latest

    steps:
      - name: Install Nomad
        uses: hashicorp/setup-nomad@v1.0.0

      - name: Request deployment to Nomad
        env:
          NOMAD_ADDR: ${{ vars.NOMAD_ADDR }}
          NOMAD_TOKEN: ${{ secrets.NOMAD_TOKEN }}
          NOMAD_HTTP_AUTH: ${{ secrets.NOMAD_HTTP_AUTH }}
        run: |
          nomad var put -force nomad/jobs/prose-web IMAGE_TAG=${{ needs.build.outputs.revision }}
</code></pre><p>The following environment variables should be sourced from your GitHub secrets and variables: <code>NOMAD_ADDR</code>, <code>NOMAD_TOKEN</code> and <code>NOMAD_HTTP_AUTH</code>. They will be used directly by Nomad CLI&#x2019;s <code>nomad</code> command.</p><p>We&#x2019;re using the following Nomad CLI command to force-update our <code>IMAGE_TAG</code> variable with our new Docker image tag coming out of our <code>build</code> step:</p><pre><code class="language-sh">nomad var put -force nomad/jobs/prose-web IMAGE_TAG=&quot;$version_number&quot;
</code></pre><p>Good! Now we can commit our workflow update to our <code>master</code> branch.</p><hr><h2 id="5-test-a-deployment-from-github-actions">5. Test a deployment from GitHub Actions</h2><p>Let&#x2019;s merge our work from <code>master</code> to <code>production</code>, and watch the magic happen:</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-actions-stages.png" class="kg-image" alt loading="lazy" width="1800" height="494" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-ci-github-actions-stages.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-ci-github-actions-stages.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/12/nomad-ci-github-actions-stages.png 1600w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-actions-stages.png 1800w" sizes="(min-width: 720px) 720px"></figure><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-actions-deploy.png" class="kg-image" alt loading="lazy" width="1800" height="651" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-ci-github-actions-deploy.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-ci-github-actions-deploy.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/12/nomad-ci-github-actions-deploy.png 1600w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-github-actions-deploy.png 1800w" sizes="(min-width: 720px) 720px"></figure><p>If you look at your Nomad Web UI, you should be seeing a job deployment getting triggered a few seconds after your <code>deploy</code> GitHub Actions completed:</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-nomad-deployment.png" class="kg-image" alt loading="lazy" width="1800" height="939" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-ci-nomad-deployment.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-ci-nomad-deployment.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/12/nomad-ci-nomad-deployment.png 1600w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-ci-nomad-deployment.png 1800w" sizes="(min-width: 720px) 720px"></figure><p>Job done! &#x1F603;</p><hr><h2 id="6-summary-of-steps">6. Summary of steps</h2><p><strong>For more clarity, I&#x2019;ve summarized all the steps we&#x2019;ve taken in this article:</strong></p><ol><li>Update your Nomad job definition so that the <strong>Docker image tag is sourced from an environment variable</strong>, itself sourced from a Nomad variable. Also, create this variable in Nomad and initialize it to a value of <code>latest</code>.</li><li><strong>Setup a HTTPS proxy to Nomad</strong> on a public host, it is also recommended to protect this with a HTTP Basic Auth password, in addition to your Nomad token.</li><li><strong>Create a Nomad token solely used by GitHub Actions</strong>, with a restricted policy to give it access to certain actions only (here, write Nomad variables for specified variable paths).</li><li>Configure the Nomad token we&#x2019;ve created, as well as your HTTP Basic authentication password, in your <strong>GitHub project secrets</strong>.</li><li><strong>Build your Docker image</strong> from a GitHub Actions pipeline, and release it to eg. GitHub packages repository.</li><li><strong>Add a deployment step</strong> that uses Nomad&#x2019;s client to hit your Nomad API and update the image tag variable. This will trigger a job re-evaluation and result in Nomad updating your running containers.</li></ol><hr><p>&#x1F1E6;&#x1F1F7; <em>Written from Buenos Aires, Argentina.</em></p>


guid: 676d5e2e3c022300014d95e4
title: Generating SSL certificates in Nomad with Certbot
author: Valerian Saliou
url: https://valeriansaliou.name/blog/ssl-certificates-nomad-certbot/
feedurl: https://journal.valeriansaliou.name/rss/
content_mime_type: text/html
content:
<p>Hosting applications in Nomad is easy, all it takes is writing a small HCL file and all your apps are now magically running on your Nomad cluster with Infrastructure as Code. Hosting your HTTP load balancer on Nomad, acting as a reverse proxy to your HTTP services, is another story: if I want to run my server over HTTPS, how do I generate SSL certificates within Nomad?</p><p>To address this need, you could take the easy path: routing ingress traffic to your HTTP load balancer running in Nomad with Cloudflare Tunnels. Cloudflare would handle SSL termination for you, so that you wouldn&#x2019;t have to deal with SSL certificates.</p><p>However, not all people want to rely on Cloudflare to handle this task. The alternative would be to handle SSL termination on your own NGINX server, and acquire the SSL certificates with Certbot (via Let&#x2019;s Encrypt). In a traditional Linux system setup, whenever your SSL certificate would get renewed, Certbot would signal NGINX to reload itself. In a Nomad cluster, that&#x2019;d play out differently.</p><p>I have came up with an elegant method to manage SSL certificates in a Nomad-native way with Certbot, leveraging Nomad templates, which I&#x2019;ll explain in details in this article. This method makes it possible to provide SSL certificates to any service running in Nomad, from a NGINX load balancer to a Postfix mail server. It also lets you reload Nomad jobs using your SSL certificates whenever they get renewed.</p><hr><h2 id="1-the-big-picture">1. The big picture</h2><p>On a traditional Linux system, you&#x2019;d issue certificates using the <code>certbot</code> command, and then use a cron job in order to trigger renewal checks. In a Nomad environment, we will still use the same <code>certbot</code> command, although it will be packaged in a Docker image. Renewals will be handled by a Nomad <code>batch</code> job &#x2014; <em>an alias name for a Cronjob</em>, which will be scheduled to run periodically (for example, every week). Nomad will handle the responsibility of firing the job whenever necessary.</p><p>Your SSL key and certificate PEMs will be stored in Nomad variables, which can then be used in whichever job uses SSL certificates, such as your NGINX load balancer job. Nomad variables come with the benefit of being able to trigger templates re-rendering whenever a variable change, meaning that Nomad will know when to restart your NGINX server so that it loads your new SSL certificate. Instead of restarting NGINX, we might rather want to reload it with a <code>SIGHUP</code> signal, which will make certificate updates a zero-downtime operation.</p><p>Finally, Nomad variables can be updated via Nomad&#x2019;s HTTP API, which is authenticated with a private token that we will generate. We are looking to use Nomad&#x2019;s HTTP API to submit the new SSL key and certificate whenever it changes, that is, whenever it gets renewed by our <code>certbot</code> job. We will be using Certbot renewal hooks to execute a script whenever our Nomad variables need to be updated with the new SSL key and certificates.</p><hr><h2 id="2-defining-the-certbot-cron-job">2. Defining the Certbot cron job</h2><p>Let&#x2019;s start by creating our <code>certbot</code> periodic job!</p><p><strong>First, we need to create 2 persistent volumes:</strong></p><ul><li>One where Certbot will store its database (we will call it <code>data-certs</code>)</li><li>Another where Certbot will store the temporary domain verification files served over HTTP, in the <code>.well-known/</code> directory (we will call it <code>data-web</code>).</li></ul><p>Note that since my Nomad setup is 1 server + 1 client, I am using host volumes. If your Nomad setup is distributed in a cluster, you would rather be looking to create volumes using CSI, which will be network volumes such as a Block Storage provided by your cloud provider. This aspect is not covered in this article, you may adapt the configurations provided in this article to match your Nomad topology.</p><p>If you need to create those two volumes on your Nomad host, edit your Nomad client agent <code>nomad.hcl</code> configuration and add eg. the following:</p><pre><code class="language-hcl">client {
  host_volume &quot;data-web&quot; {
    path = &quot;/srv/nomad/volumes/web&quot;
  }

  host_volume &quot;data-certs&quot; {
    path = &quot;/srv/nomad/volumes/certs&quot;
  }
}
</code></pre><p>Before we continue, we will also need to create a policy and token to be able to hit the Nomad API when our SSL key and certificate gets renewed.</p><p><strong>Create a policy file named <code>certs-updater.policy.hcl</code>:</strong></p><pre><code class="language-hcl">namespace &quot;default&quot; {
  variables {
    path &quot;certs/*&quot; {
      capabilities = [&quot;write&quot;]
    }
  }
}
</code></pre><p><strong>Now, import this policy:</strong></p><pre><code class="language-sh">nomad acl policy apply certs-updater ./certs-updater.policy.hcl
</code></pre><p><strong>And create a Nomad token:</strong></p><pre><code class="language-sh">nomad acl token create -name &quot;LetsEncrypt (job-certbot)&quot; -policy certs-updater
</code></pre><p><strong>Take note of the generated token, and insert it in a file named <code>job-certbot.nv.hcl</code>:</strong></p><pre><code class="language-hcl">path = &quot;nomad/jobs/job-certbot&quot;

items {
  NOMAD_TOKEN = &quot;YOUR_CERTS_UPDATER_TOKEN_HERE&quot;
}</code></pre><p><strong>And import it to Nomad:</strong></p><pre><code class="language-sh">nomad var put -in hcl @job-certbot.nv.hcl</code></pre><p>Good! Now we can define the actual <code>certbot</code> job.</p><p><strong>Create a job file named <code>job-certbot.hcl</code>:</strong></p><pre><code class="language-hcl">job &quot;job-certbot&quot; {
  type = &quot;batch&quot;

  periodic {
    crons     = [&quot;0 12 */3 * *&quot;]
    time_zone = &quot;Europe/Paris&quot;

    prohibit_overlap = true
  }

  reschedule {
    attempts = 0
  }

  group &quot;job-certbot&quot; {
    restart {
      attempts = 0
      mode     = &quot;fail&quot;
    }

    volume &quot;certbot-data&quot; {
      type   = &quot;host&quot;
      source = &quot;data-certs&quot;
    }

    volume &quot;web-data&quot; {
      type   = &quot;host&quot;
      source = &quot;data-web&quot;
    }

    task &quot;job-certbot&quot; {
      driver = &quot;docker&quot;

      config {
        image = &quot;valeriansaliou/certbot-with-nomad:v2.11.0-v1.5.8&quot;

        entrypoint = [
          &quot;sh&quot;,
          &quot;/usr/local/bin/renew.sh&quot;
        ]

        extra_hosts = [
          &quot;host.docker.internal:host-gateway&quot;
        ]

        mount {
          type     = &quot;bind&quot;
          readonly = true

          source = &quot;local/renew.sh&quot;
          target = &quot;/usr/local/bin/renew.sh&quot;
        }

        mount {
          type = &quot;bind&quot;

          source = &quot;local/renewal-hooks/&quot;
          target = &quot;/etc/letsencrypt/renewal-hooks/&quot;
        }
      }

      template {
        data = &lt;&lt;EOH
#!/bin/sh

set -e

LIVE_PATH=/etc/letsencrypt/live
DEPLOY_HOOK_PATH=/etc/letsencrypt/renewal-hooks/deploy/update-nomad-var-certs

# Mark renewal hooks as executable
chmod +x &quot;$DEPLOY_HOOK_PATH&quot;

# Renew or issue certificate?
if [ ! -d &quot;$LIVE_PATH/domain.tld&quot; ]; then
  echo &quot;Issuing new certificate...&quot;

  certbot certonly \
    --non-interactive \
    --agree-tos \
    --email=contact@domain.tld \
    --preferred-challenges=http \
    --webroot -w /var/www/default/ \
    -d domain.tld \
    -d www.domain.tld

  # Manually trigger deploy hook, since it does not get triggered on \
  #   certificate creation
  echo &quot;Manually triggering deploy hook...&quot;

  sh &quot;$DEPLOY_HOOK_PATH&quot;
else
  echo &quot;Checking if should renew existing certificate...&quot;

  certbot renew \
    --no-random-sleep-on-renew \
    --deploy-hook=&quot;$DEPLOY_HOOK_PATH&quot;
fi

echo &quot;&#x2705; Done.&quot;
        EOH

        destination = &quot;local/renew.sh&quot;
      }

      template {
        data = &lt;&lt;EOH
#!/bin/sh

set -e

echo &quot;Updating certificates in Nomad variables...&quot;

LIVE_PATH=/etc/letsencrypt/live

# Configure Nomad CLI
export NOMAD_ADDR=&quot;http://host.docker.internal:4646&quot;
export NOMAD_TOKEN=&quot;{{ with nomadVar &quot;nomad/jobs/job-certbot&quot; }}{{ .NOMAD_TOKEN }}{{ end }}&quot;

# Read certificates from disk
DOMAIN_TLD_KEY=$(cat &quot;$LIVE_PATH/domain.tld/privkey.pem&quot;)
DOMAIN_TLD_CERTIFICATE=$(cat &quot;$LIVE_PATH/domain.tld/fullchain.pem&quot;)

# Commit new certificates in Nomad
nomad var put -force certs/domain-tld \
  CERT_KEY=&quot;$DOMAIN_TLD_NAME_KEY&quot; \
  CERT_CERTIFICATE=&quot;$DOMAIN_TLD_CERTIFICATE&quot;

echo &quot;Updated.&quot;
        EOH

        destination = &quot;local/renewal-hooks/deploy/update-nomad-var-certs&quot;
      }

      volume_mount {
        volume      = &quot;certbot-data&quot;
        destination = &quot;/etc/letsencrypt/&quot;
      }

      volume_mount {
        volume      = &quot;web-data&quot;
        destination = &quot;/var/www/&quot;
      }

      resources {
        cpu    = 100
        memory = 64
      }
    }
  }
}
</code></pre><p><strong>In the job file, make sure to replace and adapt the following values:</strong></p><ul><li><code>domain.tld</code> and <code>DOMAIN_TLD</code> should be renamed to match your domain name (eg. mine would be <code>valeriansaliou.name</code> and <code>VALERIANSALIOU_NAME</code>)</li><li>Add all your sub-domains in the <code>certbot certonly</code> command arguments, with the <code>-d</code> option (one per sub-domain)</li><li>Replace <code>contact@domain.tld</code> with your email address &#x2014; it is important that you are able to receive email on this address since Let&#x2019;s Encrypt might send you security-related notifications</li></ul><p><strong>Once done, import the job:</strong></p><pre><code class="language-sh">nomad job run job-certbot.hcl
</code></pre><p>The job has been imported, but we are not yet ready to issue our first certificate. We will need to make some changes to our NGINX configuration to be able to pass the Let&#x2019;s Encrypt HTTP verification challenge. However, NGINX will not start if we do not have yet any SSL key and certificate defined in our Nomad variables. It&#x2019;s a chicken and egg problem&#x2026;</p><p>To fix this issue, we will need to populate those Nomad variables with a self-signed certificate to start with, so that NGINX can start, and <code>certbot</code> can issue our real certificate (Let&#x2019;s Encrypt needs to send some HTTP requests to our NGINX during the issuance process). Our users will never see this self-signed certificate in practice, we solely need it to start NGINX before the first issuance.</p><p><strong>Generate a new SSL key and its self-signed certificate:</strong></p><pre><code class="language-sh">openssl ecparam -out key.pem -name secp256r1 -genkey
openssl req -new -key key.pem -x509 -nodes -days 3650 -out cert.pem
</code></pre><p><strong>Now, create a file named <code>certs-domain-tld</code>:</strong></p><pre><code class="language-hcl">path = &quot;certs/domain-tld&quot;

items {
  CERT_KEY = &quot;-----BEGIN EC PRIVATE KEY-----\n[YOUR_PEM_HERE]\n-----END EC PRIVATE KEY-----&quot;
  CERT_CERTIFICATE = &quot;-----BEGIN CERTIFICATE-----\n[YOUR_PEM_HERE]\n-----END CERTIFICATE-----&quot;
}
</code></pre><p>Make sure to replace <code>domain-tld</code> with your domain name, and insert the content of your key and self-signed certificate in <code>[YOUR_PEM_HERE]</code> and <code>[YOUR_PEM_HERE]</code>. Replace all new lines with <code>\n</code>.</p><p><strong>Finally, import it to Nomad with:</strong></p><pre><code class="language-sh">nomad var put -in hcl @certs-domain-tld.nv.hcl
</code></pre><p>Certbot is now ready!</p><p>&#x26A0;&#xFE0F; <strong>Please do not attempt to manually start the <code>certbot</code> job just yet, we need to make some adjustments to our <code>nginx</code> job so that we can pass the Let&#x2019;s Encrypt certificate issuance process.</strong></p><hr><h2 id="3-configuring-nginx-to-use-your-ssl-certificate">3. Configuring NGINX to use your SSL certificate</h2><p>Let&#x2019;s load our SSL key and certificate from our NGINX job. We will be doing this by rendering our <code>nginx.conf</code> configuration using a Nomad template in our existing <code>nginx</code> job:</p><pre><code class="language-hcl">job &quot;nginx&quot; {
  type = &quot;service&quot;

  group &quot;nginx&quot; {
    task &quot;nginx&quot; {
      # (...) The rest of your job config goes here

      template {
        data = &lt;&lt;EOH
http {
    # (...) Your existing nginx.conf configuration goes here
    
    ssl_certificate      certs/domain.tld.crt;
    ssl_certificate_key  certs/domain.tld.key;        
}
                EOH

        destination = &quot;local/nginx/nginx.conf&quot;

        change_mode   = &quot;signal&quot;
        change_signal = &quot;SIGHUP&quot;
      }

      template {
        data = &lt;&lt;EOH
{{ with nomadVar &quot;certs/domain-tld&quot; }}{{ .CERT_KEY }}{{ end }}
        EOH

        destination = &quot;local/nginx/certs/domain.tld.key&quot;

        change_mode   = &quot;signal&quot;
        change_signal = &quot;SIGHUP&quot;
      }

      template {
        data = &lt;&lt;EOH
{{ with nomadVar &quot;certs/domain-tld&quot; }}{{ .CERT_CERTIFICATE }}{{ end }}
        EOH

        destination = &quot;local/nginx/certs/domain.tld.crt&quot;

        change_mode   = &quot;signal&quot;
        change_signal = &quot;SIGHUP&quot;
      }

      # (...) The rest of your job config goes here
    }
  }
}
</code></pre><p>Make sure to replace <code>domain.tld</code> and <code>domain-tld</code> with your domain name, so that it fits the name of the Nomad variable that we defined earlier.</p><p>We will also need to allow our <code>nginx</code> job to read the variables under the <code>certs/</code> path, by deploying a Nomad policy.</p><p><strong>Create a policy file named <code>certs-user.policy.hcl</code>:</strong></p><pre><code class="language-hcl">namespace &quot;default&quot; {
  variables {
    path &quot;certs/*&quot; {
      capabilities = [&quot;read&quot;]
    }
  }
}
</code></pre><p><strong>And then bind this policy to your <code>nginx</code> job:</strong></p><pre><code class="language-sh">nomad acl policy apply -namespace default -job nginx certs-user-nginx ./certs-user.policy.hcl
</code></pre><p>Finally, we need to serve the ACME challenge files that the <code>certbot</code> job will create on our <code>data-web</code> volume, right from our <code>nginx</code> job.</p><p><strong>Add the following to your <code>nginx</code> job configuration:</strong></p><pre><code class="language-hcl">job &quot;nginx&quot; {
  type = &quot;service&quot;

  group &quot;nginx&quot; {
    volume &quot;nginx-web-data&quot; {
      type   = &quot;host&quot;
      source = &quot;data-web&quot;

      read_only = true
    }
    
    task &quot;nginx&quot; {
      # (...) The rest of your job config goes here

      template {
        data = &lt;&lt;EOH
location = /.well-known/acme-challenge/ {
    return 404;
}

location ^~ /.well-known/acme-challenge/ {
    default_type &quot;text/plain&quot;;

    root /var/www/default;

    break;
}
        EOH

        destination = &quot;local/nginx/acme_challenge.conf&quot;
      }

      volume_mount {
        volume      = &quot;nginx-web-data&quot;
        destination = &quot;/var/www/&quot;

        read_only = true
      }

      # (...) The rest of your job config goes here
    }
  }
}
</code></pre><p>Now, you can simply include the <code>acme_challenge.conf</code> file in each <code>server</code> block (ie. each Virtual Host) that should receive a SSL certificate:</p><pre><code class="language-nginx">server {
    listen [::]:443 ssl;
    server_name www.domain.tld;

    # (...) Your server block configuration goes here

    include acme_challenge.conf;
}
</code></pre><p>Don&#x2019;t forget to submit your updated <code>nginx</code> job HCL definition to Nomad before we continue.</p><hr><h2 id="4-issuing-your-first-ssl-certificate">4. Issuing your first SSL certificate</h2><p>We&#x2019;re now ready to issue our first SSL certificate. We will be triggering a manual run of the <code>certbot</code> job, which will then run the <code>certbot certonly</code> command on its first run (since its database is empty, it will issue the first certificate). Later runs of the job will use the <code>certbot renew</code> command (since the database will have pre-existing certificates by then).</p><p>Go to your Nomad Web UI using an administrator token, and click on the &#x201C;Force Launch&#x201D; button:</p><figure class="kg-card kg-image-card"><img src="https://valeriansaliou.name/blog/content/images/2024/12/nomad-certbot-force-launch.png" class="kg-image" alt loading="lazy" width="2000" height="788" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/12/nomad-certbot-force-launch.png 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/12/nomad-certbot-force-launch.png 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2024/12/nomad-certbot-force-launch.png 1600w, https://valeriansaliou.name/blog/content/images/2024/12/nomad-certbot-force-launch.png 2396w" sizes="(min-width: 720px) 720px"></figure><p>Hold on for a few seconds and you should see an allocation for your <code>certbot</code> job appear. Click on the logs viewer, and monitor the progress of your Let&#x2019;s Encrypt certificate issuance.</p><p>When the <code>certbot</code> job run is complete, navigate to your <code>nginx</code> job in Nomad&#x2019;s Web UI, and confirm that a template re-rendering event gets triggered. Once the template re-renders, your NGINX server should get either restarted or reloaded, depending on the reload signal you opted for.</p><p>You may now open your Web browser and navigate to your website URL. You should see a valid SSL certificate now.</p><hr><h2 id="5-testing-auto-renewal-of-your-ssl-certificate">5. Testing auto-renewal of your SSL certificate</h2><p>You may force launch your <code>certbot</code> job once again to test the renewal process. Certificates that are not yet due for renewal will not be renewed, so no attempt will be made at this point.</p><p>To make sure that renewal works for the first time, I recommend that you check the expiration date of the first SSL certificate that was issued to you, take note of it, and come back in the future a few days before its expiration to check it again. The certificate should by then have a new expiration date, at a minimum of 1 month in the future.</p><p>If the expiration date is still the one from your first certificate, then something is wrong. If this happens, I recommend triggering a manual run of your <code>certbot</code> job and observing the logs. Do you see any error during the renewal?</p><p>If nothing goes wrong with the Certbot renewal, I would look for the <code>nginx</code> job to double-check if it&#x2019;s not a simple reloading issue. To confirm that, I&#x2019;d simply manually stop and then start the <code>nginx</code> job and check the certificate that&#x2019;s being used again. If the stop and start operation fixed your issue, then it means that something is wrong with the template re-rendering process in your <code>nginx</code> job, and not the certificate renewal process in the <code>certbot</code> job.</p><p><em>We are all done at this point! Enjoy your Nomad-native Certbot setup</em> &#x1F603;</p><hr><h2 id="6-additional-notes">6. Additional notes</h2><h3 id="accessing-nomad-api-from-a-job">Accessing Nomad API from a job</h3><p>You may have noticed that the <code>certbot</code> job uses the <code>host.docker.internal</code> host to access Nomad&#x2019;s HTTP API. This is the cleanest way to expose Nomad&#x2019;s HTTP API to jobs running in network-isolated Docker containers, which fits my single Nomad client/server setup.</p><p>Since your Nomad deployment might be different than mine, make sure to adjust the <code>host.docker.internal</code> hostname to anything that fits your setup, which could either be:</p><ul><li>Multiple Nomad servers/clients: an IP address pointing to one of your Nomad servers on your VLAN</li><li>Single Nomad server/client + Docker host networking: an IP address such as <code>::1</code> pointing to your Nomad server over <code>localhost</code></li><li>Single Nomad server/client + Docker bridge networking: the <code>host.docker.internal</code> hostname pointing to your Nomad server over the Docker bridge network (in that case, make sure that your Nomad HTTP server listens on <code>172.17.0.1</code>)</li></ul><h3 id="certbot-image-dockerfile">Certbot image Dockerfile</h3><p>If you&#x2019;re looking to build the Certbot Docker image yourself (<code>valeriansaliou/certbot-with-nomad</code> on Docker Hub), here&#x2019;s the source Dockerfile:</p><pre><code class="language-dockerfile"># ------------------ #
# CERTBOT WITH NOMAD #
# ------------------ #

ARG CERTBOT_VERSION=2.11.0

FROM certbot/certbot:v${CERTBOT_VERSION}

ARG NOMAD_VERSION=1.5.8-r4

RUN apk add --no-cache nomad=${NOMAD_VERSION}

ENTRYPOINT [&quot;certbot&quot;]
</code></pre><h3 id="keep-the-certbot-docker-image-cached-locally">Keep the Certbot Docker image cached locally</h3><p>After running the <code>certbot</code> job for the first time, Nomad&#x2019;s garbage collector will run and evict your local <code>valeriansaliou/certbot-with-nomad</code> Docker image after a few minutes/hours.</p><p>Since we will be running the same <code>certbot</code> batch job over and over again every 3 days, it is pointless to let Nomad&#x2019;s GC remove your local <code>valeriansaliou/certbot-with-nomad</code>.</p><p>Let&#x2019;s save some network gigabytes every month and a lot of SSD writes by adjusting Nomad <code>docker</code> driver configuration in your <code>nomad.hcl</code> to increase the GC cleanup delay:</p><pre><code class="language-hcl">plugin &quot;docker&quot; {
  config {
    # Increase Docker images cleanup timer to 10 days, so that weekly periodic \
    #   jobs get a better chance of re-using a cached image upon its next run
    gc {
      image       = true
      image_delay = &quot;240h&quot;
    }
  }
}
</code></pre><blockquote>I unfortunately did not find any no other way to avoid the GC evicting our Certbot Docker image than changing the global GC settings. There&#x2019;s apparently no way to adjust its settings for a particular Nomad job or Docker image.</blockquote><hr><p>&#x1F1E6;&#x1F1F7; <em>Written from El Chalten, Argentina.</em></p>


guid: https://blog.regehr.org/?p=1832
title: The Gods Pocket Peak Trail
author: regehr
url: https://blog.regehr.org/archives/1832
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>Years ago my friend Derek heard that the Jarbidge Wilderness &#8212; a remote, mountainous area along the Idaho-Nevada border &#8212; is one of the least-visited wilderness areas in the USA, and we&#8217;ve wanted to visit since then. There&#8217;s little good information about this area online, but a book I had sitting around listed The Gods Pocket Peak Trail, and we hoped to connect up to some other routes in the area to make a loop. The trailhead for this route is above 9000 feet, but then the trail immediately drops a thousand feet, climbs a thousand feet, and sort of keeps on doing this. The views and microclimates are amazing but it&#8217;s tiring walking. </p>



<p>We left Salt Lake City in the morning, and about four hours later turned off of US 93 in Rogerson, NV. Our trailhead was near the (abandoned) Pole Creek Ranger Station:<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02174.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02174.jpg"></a> We started walking by mid-afternoon with good weather and good views:
<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01966.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01966.jpg"></a>
<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01967.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01967.jpg"></a>
<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01973.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01973.jpg"></a>
</p>



<p> We camped about five miles down the trail on a ridge that had a few trees for shelter and some moderately flat spots to camp. Here&#8217;s Derek&#8217;s tent as we&#8217;re getting ready to sleep: <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01990.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01990.jpg"></a> Gods Pocket Peak in the morning light: <a href="https://blog.regehr.org/extra_files/jarbidge-
2020/DSC01991.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC01991.jpg"></a> After coffee we started a long traverse of the eastern slopes of Gods Pocket Peak and Divide Peak, and then crossed the upper parts of several forks of Camp Creek. Views were outstanding but we kept losing and finding the trail, and progress was overall not super fast.  <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02003.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02003.jpg"></a> <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02005.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02005.jpg"></a> <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02006.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02006.jpg"></a> We ended up beneath the imposing Marys River Peak: <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02019.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02019.jpg"></a> at a 9300&#8242; pass overlooking the East Fork of the Jarbidge River, roughly 12 miles from the trailhead. <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02023.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02023.jpg"></a> <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02030.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02030.jpg"></a> From here we wanted to follow a trail I&#8217;d seen in some old online imagery, but the descent on the south side of the pass was very steep and loose, and we could see no signs at all of a trail that went in the direction we wanted to go. Since it was getting late and we didn&#8217;t feel like bushwhacking, we turned around and went back to an excellent campsite we had passed earlier in the afternoon. At this point we knew the rest of the trip was straightforward, so we put our whisky flasks into a convenient snowbank and relaxed.</p>



<p><p>Next day, we returned to the trailhead, another tiring day of uphill and downhill hiking. And, of course, when walking a faded trail in reverse, you lose the trail exactly the same number of times, but in totally different locations, so we had a decent amount of hunting around for where the trail picked up again. <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02041-Pano.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02041-Pano.jpg"></a> The final uphill slog to the trailhead hurt.<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02051.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02051.jpg"></a> </p> <p>Obligatory selfie on reaching the truck: <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02070.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02070.jpg"></a> We could have just headed out right then, but we had beer, a great car camp site, and we didn&#8217;t feel like a long drive in the dark so we spent the night. </p> <p>Setting up to eat dinner and watch the sunset:<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02078.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02078.jpg"></a></p> <p><a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02081.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02081.jpg"></a></p> <p>It got cold pretty fast:<a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02086.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02086.jpg"></a></p> <p><a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02087.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02087.jpg"></a></p> <p>This is why we leave our houses. <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02096.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02096.jpg"></a></p> <p><a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02119.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02119.jpg"></a></p> <p><a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02132.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02132.jpg"></a></p> <p> Morning light on the main ridge of the Jarbidge Range was good. <a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02157.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02157.jpg"></a></p> <p><a href="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02166.jpg"><img decoding="async" src="https://blog.regehr.org/extra_files/jarbidge-2020/DSC02166.jpg"></a></p></p>



<p>Mid-July was a good time to be in Jarbidge: there was plenty of water around, but not so much that stream crossings were difficult. The first day and night were a bit warm, but after that temperatures were great. We saw oddly little wildlife: no snakes, no frogs, not many birds, and no mammals larger than a marmot. There was plenty of deer and elk scat around, however. Our first campsite had mosquitoes, though they were not particularly bothersome, and we didn&#8217;t have any bugs after that, except that I managed to find a patch of ticks while hiking off trail through some dense vegetation. Who expects ticks in Nevada?</p>



<p>The Gods Pocket Peak Trail gave the impression of having been well maintained up until a few decades ago. Major trail junctions were marked with signs, but most of the signs were faded to near illegibility and many had fallen over. The trail is excellent for long stretches, but then parts have eroded completely away or have been covered by plants to the point that it&#8217;s basically gone. We were happy to have a map I had traced from Google Earth imagery: each time we lost the trail, the GPS made it pretty easy to find again. We didn&#8217;t see any evidence of recent camping activity, such as the fire rings or areas cleared of fallen wood. One of the reasons I&#8217;m posting this trip report is that I think this area could use some more foot traffic: this might keep the trail usable for a few more years.</p>


guid: https://blog.regehr.org/?p=1837
title: Alive2 Part 3: Things You Can and Can’t Do with Undef in LLVM
author: regehr
url: https://blog.regehr.org/archives/1837
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<style type="text/css">
html { font-variant-numeric: lining-nums; }<br />
</style>


<p>[Also see <a href="https://blog.regehr.org/archives/1722">Part 1</a> and <a href="https://blog.regehr.org/archives/1737">Part 2</a> in this series.]</p>



<p>Let&#8217;s talk about these functions: </p>



<pre class="wp-block-preformatted"><strong>unsigned add(unsigned x) { return x + x; }
unsigned shift(unsigned x) { return x &lt;&lt; 1; }</strong></pre>



<p>From the point of view of the C and C++ abstract machines, their behavior is equivalent: in a program you&#8217;re writing, you can always safely substitute add for shift, or shift for add.</p>



<p>If we compile them to LLVM IR with optimizations enabled, <a href="https://gcc.godbolt.org/z/bGurwr">LLVM will canonicalize both functions to the version containing the shift operator</a>. Although it might seem that this choice is an arbitrary one, it turns out there is a very real difference between these two functions:</p>



<pre class="wp-block-preformatted"><code>define i32 @add(i32 %0) { 
  %2 = add i32 %0, %0 
  ret i32 %2
}
define i32 @shift(i32 %0) {
  %2 = shl i32 %0, 1
  ret i32 %2
}</code></pre>



<p>At the LLVM level, <a href="https://alive2.llvm.org/ce/z/pje5Q7">add can always be replaced by shift</a>, but <a href="https://alive2.llvm.org/ce/z/iR2mfT">it is not generally safe to replace shift with add</a>. (In these linked pages, I&#8217;ve renamed the functions to src and tgt so that Alive2 knows which way the transformation is supposed to go.) </p>



<p>To understand why add can be replaced by shift, but shift cannot be replaced by add, we need to know that these functions don&#8217;t have 2<sup>32</sup> possible input values, but rather 2<sup>32</sup>+2. Analogously, an LLVM function taking an i1 (Boolean) argument has four possible input values, not the two you were probably expecting. These extra two values, undef and poison, cause a disproportionate amount of trouble because they follow different rules than regular values follow. This post won&#8217;t discuss <a href="https://llvm.org/docs/LangRef.html#poison-values">poison</a> further. <a href="https://llvm.org/docs/LangRef.html#undefined-values">Undef</a> sort of models uninitialized storage: it can resolve to any value of its type. Unless we can specifically prove that an LLVM value, such as the input to a function, cannot be undef, we must assume that it might be undef. Many compiler bugs have been introduced because compiler developers either forgot about this possibility, or else failed to reason about it correctly.</p>



<p>Let&#8217;s look at some things you&#8217;re allowed to do with undef. You can <a href="https://alive2.llvm.org/ce/z/psNicd">leave it alone</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="764" height="498" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.14.44-AM.png" alt="" class="wp-image-1911" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.14.44-AM.png 764w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.14.44-AM-450x293.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.14.44-AM-150x98.png 150w" sizes="auto, (max-width: 764px) 100vw, 764px" /></figure>



<p>You can <a href="https://alive2.llvm.org/ce/z/DhPanX">replace it with zero</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="652" height="484" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.13-AM-1.png" alt="" class="wp-image-1914" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.13-AM-1.png 652w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.13-AM-1-450x334.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.13-AM-1-150x111.png 150w" sizes="auto, (max-width: 652px) 100vw, 652px" /></figure>



<p>You can <a href="https://alive2.llvm.org/ce/z/ECRrAS">replace it with any number you choose</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="658" height="494" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.22-AM.png" alt="" class="wp-image-1915" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.22-AM.png 658w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.22-AM-450x338.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.22-AM-150x113.png 150w" sizes="auto, (max-width: 658px) 100vw, 658px" /></figure>



<p>You can <a href="https://alive2.llvm.org/ce/z/3h6bFj">replace it with the set of odd numbers</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="644" height="576" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.30-AM.png" alt="" class="wp-image-1916" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.30-AM.png 644w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.30-AM-450x402.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.30-AM-150x134.png 150w" sizes="auto, (max-width: 644px) 100vw, 644px" /></figure>



<p>You can <a href="https://alive2.llvm.org/ce/z/BK6JPt">take a function that returns [0..15] and replace it by a function that returns [3..10]</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="654" height="610" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.38-AM.png" alt="" class="wp-image-1917" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.38-AM.png 654w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.38-AM-450x420.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.20.38-AM-150x140.png 150w" sizes="auto, (max-width: 654px) 100vw, 654px" /></figure>



<p>The common pattern here is that the new function returns a subset of the values returned by the old function. This is the essence of <em>refinement</em>, which is what every step of a correct compilation pipeline must do. (Though keep in mind that right now we&#8217;re looking at a particularly simple special case: a pure function that takes no arguments.) So what can&#8217;t you do with undef? This is easy: you can&#8217;t violate refinement. In other words, you can&#8217;t replace a (pure, zero-argument) function with a function that fails to return a subset of the values that is returned by the original one.</p>



<p>So, for example, you can&#8217;t <a href="https://alive2.llvm.org/ce/z/8VRq8E">take a function that returns [0..15] and replace it by a function that returns [9..16]</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="842" height="1110" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.24.04-AM.png" alt="" class="wp-image-1920" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.24.04-AM.png 842w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.24.04-AM-341x450.png 341w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.24.04-AM-114x150.png 114w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.24.04-AM-768x1012.png 768w" sizes="auto, (max-width: 842px) 100vw, 842px" /></figure>



<p>As we can see in the counterexample, Alive2 has correctly identified 16 as the problematic output. Returning [9..15] would have been fine, as would any other subset of [0..15].</p>



<p>When a function takes inputs, the refinement rule is: for every valuation of the inputs, the new function must return a subset of the values returned by the old function. Refinement gets more complicated for functions that touch memory; my colleagues Juneyoung and Nuno will cover that in a different post.</p>



<p>Now let&#8217;s return to the question from the top of this piece: why can we turn add into shift, but not shift into add? This is because every time an undef is used, the use can result in a different value (<a href="https://en.wikipedia.org/wiki/Static_single_assignment_form">here &#8220;use&#8221; has a precise meaning from SSA</a>). If undef is passed to the shift function, an arbitrary value is left-shifted by one position, resulting in an arbitrary even value being returned. On the other hand, the add function uses the undef value twice. When two different arbitrary values are added together, the result is an arbitrary value. This is a clear failure of refinement because the set of all 32-bit values fails to be a subset of the set of the even numbers in 0..2<sup>32</sup>-1. Why was undef designed this way? It is to give the compiler maximum freedom when translating undef. If LLVM mandated that every use of an undef returned the same value, then it would sometimes be forced to remember that value, in order to follow this rule consistently. The resulting value would sometimes occupy a register, potentially preventing something more useful from being done with that register. As of LLVM 10, this &#8220;remembering&#8221; behavior can be achieved using the freeze instruction. A frozen undef (or poison) value is arbitrary, but consistent. So if we want to transform the shift function into the add function in a way that respects refinement, <a href="https://alive2.llvm.org/ce/z/jGoHFy">we can do it like this</a>:</p>



<figure class="wp-block-image size-large"><img loading="lazy" decoding="async" width="660" height="608" src="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.29.44-AM.png" alt="" class="wp-image-1922" srcset="https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.29.44-AM.png 660w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.29.44-AM-450x415.png 450w, https://blog.regehr.org/wp-content/uploads/2020/07/Screen-Shot-2020-07-20-at-10.29.44-AM-150x138.png 150w" sizes="auto, (max-width: 660px) 100vw, 660px" /></figure>



<p>A general guideline is that an LLVM transformation can decrease the number of uses of a possibly-undef value, or leave the number of uses the same, but it cannot increase the number of uses unless the value is frozen first.</p>



<p>Instead of adding freeze instructions everywhere, one might ask whether it is possible to avoid undef-related problems by proving that a value cannot possibly be undef. This can be done, and as of recently <a href="https://github.com/llvm/llvm-project/blob/65c63eb69cc157b5bf257d6b91ecd758446ee5a1/llvm/lib/Analysis/ValueTracking.cpp#L4775-L4871">Juneyoung Lee has added a static analysis for this</a>. Alas, it is necessarily pretty conservative, it only reaches the desired conclusion under restricted circumstances.</p>



<p>Before concluding, it&#8217;s worth recalling that undefined behavior in an IR is not necessarily closely related to the source-level UB. In other words, we can compile a safe language to LLVM IR, and we can also compile C and C++ to IRs lacking undefined behavior. In light of this, we might ask whether undef, and its friend, poison, are worthwhile concepts in a compiler IR at all, given the headaches they cause for compiler developers. On one hand, we can find IR-like languages such as Java bytecode that lack analogous abstractions. On the other hand, as far as actual IRs for heavily optimizing compilers go, I believe all of the major ones have something more or less resembling poison and/or undef. The fact that a number of different compiler teams invented similar concepts suggests that there might be something fundamental going on here. An open question is to what extent these abstractions are worthwhile in an IR for a heavily-optimizing compiler middle-end that is only targeted by safe languages.</p>



<p>My colleagues and I believe that having both undef and poison in LLVM is overkill, and that undef can and should be removed at some point. This was not true in the past, but we believe that the new freeze instruction, in combination with poison values, provides adequate support for undef&#8217;s use cases.</p>


guid: https://blog.regehr.org/?p=2037
title: Responsible and Effective Bugfinding
author: regehr
url: https://blog.regehr.org/archives/2037
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p><strong>NB: This piece is not about responsible disclosure of security issues.</strong></p>



<p>For almost as long as people have written code, we have also worked to create methods for finding software defects. Much more recently, it has become common to treat “external bug finding” &#8212; looking for defects in other people’s software &#8212; as an activity worth pursuing on its own. Several factors have contributed:</p>



<ul class="wp-block-list"><li>The amount of software in use has grown massively, and presumably the number of latent bugs has grown with it.</li><li>Since software is increasingly used to monitor and control aspects of our lives, and because systems are often networked, bugs can have more impact than they previously did.</li><li>Cheap computer power for brute-force bug finding is ubiquitously available.</li></ul>



<p>The <a href="https://github.com/google/oss-fuzz">OSS-Fuzz</a> project is a good example of a successful external bug finding effort; its web page mentions that it “has found over 20,000 bugs in 300 open source projects.”</p>



<p>This article is about how to conduct an external bug finding effort in a way that is likely to maximize the overall benefit of the project. The high level summary is that this kind of work has to be done carefully, thoughtfully, and in collaboration with the software developers. In contrast, simply throwing a huge pile of bugs over the wall to some project and walking away, is unlikely to do much good. I&#8217;ve not seen a lot written up about this, but&nbsp; “<a href="https://web.stanford.edu/~engler/BLOC-coverity.pdf">A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World</a>” is very good and touches on some of the same issues.</p>



<p>The problem is that there are usually mismatches in costs and incentives between bug finders and project developers. External bug finders would like to find and report as many bugs as possible, and if they have the right technology (like a shiny new fuzzer) it can end up being cheap to find hundreds or thousands of bugs in a large, soft target. Bug finders are motivated altruistically (they want to make the targeted software more reliable by getting lots of bugs fixed) but also selfishly (a bug finding tool or technique is demonstrably powerful if it can find a lot of previously unknown defects). On the other side, developers of the targeted project typically appreciate bugfinding efforts (of course, they too want a reliable system) but for them a huge bug count is undesirable. First, every bug report requires significant attention and effort, usually far more effort than was required to simply find the bug. Second, a large bug count potentially reflects negatively on the project and consequently on its developers.</p>



<p>In the rest of this post I’ll present some questions that people doing external bug finding work should ask themselves about each bug that they find. The answers, and the process of answering, will help them do their work more responsibly and effectively.</p>



<h2 class="wp-block-heading">Is the system being tested an appropriate target for external bug finding?</h2>



<p>This is obvious, but it still needs mentioning: not every code base wants or needs bug reports. There are plenty of throw-aways, research projects, and other code bases out there that have a very small number of users and at most a couple of developers. It is often inappropriate to use this sort of code as a target for external bugfinding work. In contrast, the more users a system has, and the more important its use cases are, the more appropriate a target it becomes. A reasonable bar for external bug finding might be: Does the package manager for your operating system include, by default, a recipe for installing the software you want to test? If not, then perhaps it has not risen to the level where it is a suitable target for external bugfinding.</p>



<h2 class="wp-block-heading">Is the bug a security vulnerability?</h2>



<p>This is a completely separate can of worms; if you have found a potential vulnerability, <a href="https://en.wikipedia.org/wiki/Responsible_disclosure">please seek advice elsewhere</a>.</p>



<h2 class="wp-block-heading">Is the bug already known?</h2>



<p>A drawback of fuzzers is that they tend to keep rediscovering known bugs. Automated bug triaging techniques exist, but they are far from perfect, especially for bugs that don’t trigger crashes. Furthermore, there’s good reason to believe that perfect automated bug triaging is impossible, since the notion of “distinct bugs” is inherently a human construct. When there is a reasonable possibility that a bug you have discovered is already known and reported, it is best to just sit on your new bug report for a little while. Once the potential duplicates that are already in the bug tracker get fixed, it will be easy to see if your test case still triggers a failure. If so, you can report it. Otherwise, discard it.</p>



<p>Occasionally, your bug finding technique will come up with a trigger for a known bug that is considerably smaller or simpler than what is currently in the issue tracker. For example, if a web browser contains a race condition that people believe can only be triggered using five threads, and your test case makes it happen using just two threads, then this is an interesting new finding. In this case you should consider appending your test case to the existing issue. </p>



<h2 class="wp-block-heading">Can you write a convincing bug report?</h2>



<p>Writing good bug reports is a learned skill that requires significant care and patience. <a href="https://www.softwaretestinghelp.com/how-to-write-good-bug-report/">Here</a> are a <a href="https://developer.mozilla.org/en-US/docs/Mozilla/QA/Bug_writing_guidelines">couple</a> of good resources.</p>



<p>I only have a few points to add:</p>



<ul class="wp-block-list"><li>A bug report is not simply a collection of facts, it also forms an argument for why a developer &#8212; who surely already had plans for the day &#8212; should instead work on fixing the defect.</li><li>It is important to put yourself in the developers’ position. If you were reading this report, would you have enough information to act on it? Would you want to?</li><li>Avoid, at all costs, being rude or presumptuous.</li><li>Test cases found by fuzzers can be inherently less compelling to developers than test cases that come from real use cases. First, these test cases often just look funny. Second, developers often believe (no doubt correctly, sometimes) that fuzzer-discovered corner cases will not be triggered by human users. If you are submitting a bug report containing fuzzer-generated input, careful test case reduction must be performed, with an eye not only towards minimality but also towards readability and understandability.</li></ul>



<p>A major advantage of external bug finding is that since the people performing the testing are presumably submitting a lot of bug reports, they can do a really good job at it. In contrast, regular end users may not submit bug reports very often, and consequently we would not expect them to be as skilled. Treat the ability to write high-quality bug reports as a superpower that enables you and the system&#8217;s developers to collaboratively increase the quality of the code in an efficient and pleasant fashion.</p>



<h2 class="wp-block-heading">Is the bug important?</h2>



<p>Developers are almost always willing to fix important bugs that might have serious consequences for their users. On the other hand, they are often happy to delay fixing bugs that they believe are unimportant, so they work on more pressing things. As an external bug finder, it can be hard to tell which kind of bug you have discovered (and the numbers are not on your side: the large majority of bugs are not that important). One way to approach this problem is to look at bugs (both fixed and unfixed) that are already in the bug tracker: what common characteristics can you identify that caused developers to address defects rapidly? What common characteristics can you identify that caused developers to ignore bugs or mark them WONTFIX? You can also open a dialogue with developers: they may be willing to tell you the kinds of bugs they’re interested in, and not interested in. The developers of the CVC4 SMT solver recently added <a href="https://github.com/CVC4/CVC4/wiki/Fuzzing-CVC4">a public document to this effect</a>. I hope that many other projects will follow their lead.</p>



<p>Common sense is also useful: if the bug requires an obscure or unlikely combination of command-line options to be triggered, then perhaps it doesn’t matter that much. For example, when fuzzing compilers using Csmith we rarely invoked GCC and LLVM with flags other than -O0, -O1, -O2, -Os, and -O3.</p>



<h2 class="wp-block-heading">Have you tried to fix the bug?</h2>



<p>In some cases, fixing a bug in software you’re unfamiliar with is impractical because it’s something like a deep race condition in an OS kernel or a subtle emission of incorrect object code by a compiler. Years of specific experience with the system might be required to locate the right place to fix the bug. On the other hand, a lot of bugs end up being easy to fix, including typos, copy-and-paste errors, or incorrect order of arguments to a library call. If you are able to take a few minutes to check if a bug you are reporting is easy to fix, and to suggest a patch if it is, then you can potentially save project developers considerable time and effort. They will appreciate this.</p>



<h2 class="wp-block-heading">How many open issues do you have in the bug tracker?</h2>



<p>If you have already reported a few bugs and they have not been fixed (or, worse, have not even been acknowledged), then something is not working. Perhaps the developers are otherwise occupied, or perhaps they don’t find these bugs to be actionable or important. Regardless, at this point it is not helpful to continue reporting bugs. If you haven’t done so already, you might open a dialogue with developers to see what the situation is, or you might simply move on and find bugs in a different code base.</p>



<h2 class="wp-block-heading">Do the developers trust you?</h2>



<p>If the developers of the system you’re reporting bugs in have never seen or heard of you, they are very likely to at least take a look at the first issue you submit. If they trust you, because you’ve submitted a number of high-quality bug reports before, they’ll look at it closely and are likely to take it seriously. However, if they actively don’t trust you, for example because you’ve flooded their bug tracker with corner-case issues, then they’re not likely to ever listen to you again, and you probably need to move on to do bug finding somewhere else. Do not let this happen, it makes developers salty and that makes bug finding work more difficult for all of us.</p>



<h2 class="wp-block-heading">Are your tools open source?</h2>



<p>Internal bug finding &#8212; where project developers find bugs themselves &#8212; can be superior to external bug finding because developers are more in tune with their own project’s needs and also they can schedule bug finding efforts in such a way that they are most effective, for example after landing a major new feature. Thus, an alternative to external bug finding campaigns is to create good bug finding tools and make them available to developers, preferably as open source software. Drawbacks of this scheme include increasing the software engineering burden on bug-finding researchers (who must now create tools that other people can use, rather than creating tools solely for their own use) and decreasing social and academic credit given to researchers since some of the benefit of their work is now hidden, rather than being quantifiable and out in the open.</p>



<h2 class="wp-block-heading">Conclusions</h2>



<p>Although software correctness is at some level a technical problem (either the code implements its specification or it doesn&#8217;t), improving the quality of software is an intensely human activity. Developers and external bug finders are on the same team here, but there’s plenty of potential for friction because their costs and incentives differ. This friction can be reduced if bug finders ask themselves a series of questions listed in this post before submitting bug reports.</p>


guid: https://blog.regehr.org/?p=2109
title: llvm-reduce
author: regehr
url: https://blog.regehr.org/archives/2109
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>Test-case reduction is more or less a necessity when debugging failures of complex programs such as compilers. Automated test-case reduction is useful not only because it allows developers to avoid wasting time reducing inputs by hand, but also because it supports new techniques such as automatically triaging bulk failures seen in the field or during fuzzing campaigns.</p>



<p>llvm-reduce is a newish tool for reducing LLVM IR. Structurally, it is similar to C-Reduce, using a reducer core that calls out to a modular collection of reduction passes. Each pass implements a potentially-reducing transformation on LLVM IR, such as removing some instructions, in order to create variants: candidates for further reduction. The core inspects each variant, keeping only the ones that are &#8220;interesting&#8221; according to some user-defined criterion. A module full of LLVM IR might, for example, be interesting if it triggers some assertion violation in a backend.</p>



<p>In test-case reduction there&#8217;s a tension between creating generic tools that are intended to reduce all kinds of input, and creating specific tools that contain transformations that only work on one kind of input. llvm-reduce is all the way at one end of this spectrum: it can&#8217;t be used to reduce anything other than LLVM IR. This kind of specificity has advantages, such as reducing more rapidly by not creating illegal variants and reducing more effectively by supporting invariant-preserving transformations that are infeasible to perform for tools that don&#8217;t understand the relevant invariants. For example, when removing a parameter from a function, it is necessary to also remove the corresponding argument from each location where that function is called.</p>



<p>Something that I like quite a bit about llvm-reduce is that its interestingness tests are compatible with C-Reduce&#8217;s, so people can switch between the two tools to see if one of them works better than the other for some particular problem. </p>



<p>llvm-reduce is part of LLVM, so you will get it for free by building or downloading an LLVM. The implementation is super approachable if you&#8217;re interested in digging into it. You can learn more about this tool from <a href="https://www.youtube.com/watch?v=n1jDj7J9N8c">this talk by its creator</a>, and from <a href="https://llvm.org/docs/HowToSubmitABug.html">this page</a>.</p>


guid: https://blog.regehr.org/?p=2173
title: A Close Look at a Spinlock
author: regehr
url: https://blog.regehr.org/archives/2173
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p><style type="text/css">
pre { overflow: none; white-space: pre; font-family: "courier new", courier, monospace; font-size: 70%; font-weight: bold; }
</style></p>
<p>The spinlock is the most basic mutual exclusion primitive provided by a multiprocessor operating system. Spinlocks need to protect against preemption on the current CPU (typically by disabling interrupts, but we&#8217;ll ignore that aspect in this post) and also against attempts by other cores to concurrently access the critical section (by using atomic memory operations). As the name implies, attempts to acquire a locked spinlock simply spin: they burn CPU time. Thus, we don&#8217;t want to hold spinlocks for long and we certainly don&#8217;t want to be preempted while holding one.</p></p>



<p>In the old days, people came up with spinlock implementations that were based on standard memory operations: loads and stores. <a href="https://en.wikipedia.org/wiki/Eisenberg_%26_McGuire_algorithm">Eisenberg &amp; McGuire</a> and the <a href="https://en.wikipedia.org/wiki/Lamport%27s_bakery_algorithm">Lamport Bakery</a> are examples. Modern multicores with weak memory models (including x86 and x86-64) break naive implementations of these algorithms and, while they can be fixed by adding memory fences, the resulting code isn&#8217;t as efficient as what can be accomplished using hardware-supported atomic memory operations that are more powerful than loads and stores, such as <a href="https://en.wikipedia.org/wiki/Test-and-set">test-and-set</a>, <a href="https://en.wikipedia.org/wiki/Compare-and-swap">compare-and-swap</a>, and <a href="https://en.wikipedia.org/wiki/Load-link/store-conditional">load-linked / store-conditional</a>.</p>



<p>A spinlock does not need to be complicated, we can write a decently portable user-mode one using GCC&#8217;s <a href="https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html">atomic intrinsics</a>:</p>



<pre class="wp-block-preformatted">struct spin_lock_t {
  int lock;
};

void spin_lock(struct spin_lock_t *s) {
  while (1) {
    int zero = 0;
    int one = 1;
    if (__atomic_compare_exchange(&amp;s-&gt;lock, &amp;zero,
                                  &amp;one, 0,
                                  __ATOMIC_SEQ_CST,
                                  __ATOMIC_SEQ_CST))
      return;
  }
}

void spin_unlock(struct spin_lock_t *s) {
  int zero = 0;
  __atomic_store(&amp;s-&gt;lock, &amp;zero, __ATOMIC_SEQ_CST);
}</pre>



<p>The idea here is that the lock field holds a zero when the lock is free and a one when it is held. To acquire the lock, we use a compare-and-swap operation to try to change a zero in the lock field into a one &#8212; the key is that the comparison and the swap execute atomically. How the architecture accomplishes this is out of scope for this post! The __ATOMIC_SEQ_CST specifies that we want the strongest possible synchronization behavior from these atomic operations. This, and other aspects of this lock, could be optimized, but this implementation works.</p>



<p>An issue with this spinlock is that it permits unfairness under contention: if multiple cores are trying to get into a critical section, it is the processor&#8217;s memory subsystem that effectively chooses who gets in. On a random AMD machine that I have, it is easy to see some cores accessing the critical section several times more often than others when contention is heavy. To make access to the critical section more fair, we can create spinlocks that enforce FIFO access. Here&#8217;s a simple way to do that:</p>



<pre class="wp-block-preformatted">struct ticket_lock_t {
  unsigned front;
  unsigned back;
};

void ticket_lock(struct ticket_lock_t *s) {
  unsigned ticket =
    __atomic_add_fetch(&amp;s-&gt;back, 1, __ATOMIC_SEQ_CST) - 1;
  while (1) {
    unsigned front;
    __atomic_load(&amp;s-&gt;front, &amp;front, __ATOMIC_SEQ_CST);
    if (front == ticket)
      return;
  }
}

void ticket_unlock(struct ticket_lock_t *s) {
  __atomic_add_fetch(&amp;s-&gt;front, 1, __ATOMIC_SEQ_CST);
}</pre>



<p>Here the lock is free when front == back, and the number of threads waiting to enter the critical section is front &#8211; back &#8211; 1. The metaphor is the same as in the Lamport Bakery: threads receive increasing ticket numbers as they request access to the critical section, and they are granted access in order of increasing ticket number. Some experimentation on a largish multicore shows that this spinlock is highly fair, even when contention is high.</p>



<p>On most platforms, the Linux kernel currently uses a &#8220;queued spinlock&#8221; that has a pretty complicated implementation. It is spread across several files but <a href="https://github.com/torvalds/linux/blob/v5.15/kernel/locking/qspinlock.c">the bulk of the logic is here</a>. The rest of this post will ignore that code, however, and rather focus on Linux&#8217;s spinlock for 32-bit ARM platforms, which has a lot going on that&#8217;s fun to dissect. Here&#8217;s <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L56-L79">the code for acquiring a lock</a>; I&#8217;ll refer to it by line number in the rest of this post. But first, <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock_types.h#L11-L24">here&#8217;s the data structure for this spinlock</a>, the code is a bit messy but basically it&#8217;s just saying that we can look at the spinlock data either as a 32-bit int or else as a pair of 16-bit ints; these are going to work in the same way as front and back in the ticket spinlock above. Of course this spinlock will fail if anyone manages to create a machine containing enough 32-bit ARM cores that more than 65535 of them end up contending for the same spinlock. Seems safe enough.</p>



<p>The <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L62">prefetchw() call at line 62</a> turns into a <a href="https://developer.arm.com/documentation/dui0489/h/arm-and-thumb-instructions/pld--pldw--and-pli">pldw</a> instruction. I don&#8217;t have any idea why it is advantageous to prefetch a value so close to where it is actually needed. [<b>UPDATE:</b> Luke Wren and Paul Khuong pointed out on Twitter that this is to reserve the cache line for writing right off the bat, instead of reserving it for reading and then a bit later reserving it for writing.]</p>



<p>The <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L64-L71">inline assembly block at lines 64-71</a> is for grabbing a ticket number; it is functionally equivalent to the __atomic_add_fetch() in the ticket lock above. GCC inline assembly is never super easy so let&#8217;s dig in. <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L69">Line 69</a> specifies the outputs of the inline assembly block: it is going to write values into three variables, lockval, newval, and tmp, and these are going to be referred to respectively as %0, %1, and %2 in the assembly code. These are &#8220;virtual registers&#8221; that the compiler will map to physical registers as it sees fit. <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L70">Line 70</a> specifies the inputs to the inline assembly block: the address of the lock struct will be virtual register %3 and the constant 1&lt;&lt;16 will be stored in virtual register %4. Finally, <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L71">line 71</a> specifies the &#8220;clobbers&#8221; for this inline assembly block: machine state not mentioned anywhere else that is going to be overwritten by the assembly. &#8220;cc&#8221; tells the compiler that it may not assume that the condition code flags will have the same values after this block executes that they held on entry. </p>



<p>Next we have <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L64-L68">five ARM instructions to look at</a>. Together, <a href="https://developer.arm.com/documentation/dht0008/a/ch01s02s01">ldrex and strex are a load-linked / store-conditional pair</a>. This is a quirky but powerful mechanism supporting optimistic concurrency control &#8212; &#8220;optimistic&#8221; because it doesn&#8217;t prevent interference, but rather allows code to detect interference and recover, usually by just trying again, which is what happens here. Ok, into the details. The ldrex from %3 accomplishes two things:</p>



<ul class="wp-block-list"><li>the 32-bit contents of the lock struct (pointed to by %3) are loaded into virtual register %0, aka lockval</li><li>the memory location containing the lock struct is <a href="https://developer.arm.com/documentation/dht0008/a/arm-synchronization-primitives/exclusive-accesses/exclusive-monitors?lang=en">tagged for exclusive use</a></li></ul>



<p>Next, lockval is incremented by 1&lt;&lt;16 and the result is stored into newval (virtual register %1). It is easy to prove that this addition does not change the low 16 bits of this value, but rather increments the value stored in the high 16 bits by one. This gives us a new ticket value that will be used to determine when we get access to the critical section.</p>



<p>Next, the strex instruction either stores newval (%1) back into the lock struct, or doesn&#8217;t, depending on whether anyone has touched the lock struct since we marked it for exclusive use (in practice there are other conditions that can cause loss of exclusivity but they don&#8217;t matter for purposes of this spinlock). Additionally, the success or failure of the store is recorded in tmp (virtual register %2). We&#8217;re not used to seeing an error code for stores to RAM but that&#8217;s exactly how this works. Next, tmp is tested for equality against zero, and finally if tmp is non-zero we branch back to the ldrex, indicating that we lost exclusivity, the store failed, and we need to try this sequence of operations again.</p>



<p>So what have we seen so far? The five ARM assembly instructions implement a little spin loop that grabs a new ticket value, retrying until this can be done without interference. Since the race window is short, we can hope that the expected number of retries is very close to zero.</p>



<p><a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L73-L76">Lines 73-76</a> are comparatively easy: this is the actual spinlock where we wait for ourselves to be the customer who is currently being served. This happens when &#8220;next&#8221; and &#8220;owner&#8221; are the same value. In the expected (no-contention) case the loop body here never executes. Keep in mind that lockval holds the state of the spinlock struct before we incremented the next field, so a quiescent spinlock would have had those values being the same when we arrived.</p>



<p>wfe() is a macro for the <a href="https://developer.arm.com/documentation/ka001283/latest">wfe (wait for event) instruction</a>, which puts our core into a low power state until someone tells us it&#8217;s time to wake up &#8212; below we&#8217;ll look at how the unlock code makes that happen. Once awakened, our core uses Linux&#8217;s READ_ONCE() macro to load from the low half of the spinlock struct, which hopefully allows us to escape from the while loop. READ_ONCE() is shorthand for casting a pointer to &#8220;pointer to volatile&#8221; and then loading through that pointer &#8212; this lets the compiler know that it is not allowed to cache the value of &#8220;owner.&#8221; If that happened, this would turn into an infinite loop.</p>



<p>Finally, the smp_mb() at <a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L78">line 78</a> creates a memory barrier instruction that stops memory accesses inside the critical section from being reordered with respect to memory accesses outside of the critical section, from the point of view of other cores in this system.</p>



<p><a href="https://github.com/torvalds/linux/blob/v5.15/arch/arm/include/asm/spinlock.h#L107-L112">Releasing this spinlock</a> has three parts. First, another memory barrier to stop unfriendly reorderings. Second, increment the &#8220;owner&#8221; part of the spinlock struct. This is going to require three instructions: a load, an add, and a store, and none of these instructions are special synchronization instructions. This is OK because at this point we hold the lock and nobody else is allowed to change the owner. The high half of the word (containing next) can be changed behind our back &#8212; but that doesn&#8217;t matter because we&#8217;re only touching owner using 16-bit load/store instructions. Finally, dsb_sev() turns into a data synchronization barrier (ensuring that everyone can see our update to the owner field) and a sev (signal event) instruction, which causes all cores sleeping on a wfe instruction to wake up and continue executing. And we&#8217;re done!</p>


guid: https://blog.regehr.org/?p=2148
title: High-Throughput, Formal-Methods-Assisted Fuzzing for LLVM
author: regehr
url: https://blog.regehr.org/archives/2148
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p><em>[This piece is coauthored by <a href="https://spica.cloud/">Yuyou Fan</a> and John Regehr]</em></p>



<p>Mutation-based fuzzing is based on the idea that new, bug-triggering inputs can often be created by randomly modifying existing, non-bug-triggering inputs. For example, if we wanted to find bugs in a PDF reader, we could grab a bunch of PDF files off the web, mutate them by flipping a few bits or something, and feed the resulting mutants into our fuzzing target and wait for it to crash or otherwise malfunction.</p>



<p>To exploit this idea for fuzzing LLVM optimization passes, we can plug existing LLVM IR files such as those found in LLVM&#8217;s unit test suite into a workflow like this one:</p>



<figure class="wp-block-image size-large"><img decoding="async" width="1280" height="448" src="https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-1280x448.png" alt="" class="wp-image-2154" srcset="https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-1280x448.png 1280w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-450x158.png 450w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-150x53.png 150w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-768x269.png 768w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-1536x538.png 1536w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-2048x717.png 2048w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-10-at-2.33.28-PM-1568x549.png 1568w" sizes="(max-width: 1280px) 100vw, 1280px" /></figure>



<p>In other words, we create a fresh test case using mutation, optimize it, and then we use our formal methods tool <a href="https://alive2.llvm.org/ce/">Alive2</a> to either prove that the optimization was correct, or prove that it was buggy. This sort of collaboration between formal and informal methods ends up being powerful and useful!</p>



<p>The first thing we tried out is <a href="https://gitlab.com/akihe/radamsa">radamsa</a>, an open source mutation engine. The resulting testing loop worked poorly: the vast majority of mutated tests were either semantically identical to the original test case or else were not valid bitcode files. This isn&#8217;t radamsa&#8217;s fault: files in LLVM IR (intermediate representation) have strong validity constraints, making it difficult for a tool that is unaware of the file structure to effect meaningful changes. After running for a few days on a fast machine, we did not discover any bugs this way. It&#8217;s possible that a different domain-independent mutation engine, such as the one found in AFL, would work better, but this doesn&#8217;t seem likely and we didn&#8217;t pursue it. (Related, C-Reduce does a poor job reducing LLVM IR, whereas the domain-specific llvm-reduce is quite good.)</p>



<p>Next, we wrote a custom mutator for LLVM IR that always generates valid output. To do this, we used existing APIs for things like replacing all uses of an SSA value and computing the SSA dominator tree, in order to avoid generating outputs that violate IR invariants. Mutations that we can do include:</p>



<ul class="wp-block-list"><li>replacing an instruction with a different one, for example changing left-shift to subtraction</li><li>permuting an instruction&#8217;s operands</li><li>toggling flags attached to instructions such as the &#8220;exact&#8221; flag on division and the &#8220;no unsigned wrap&#8221; flag on addition and subtraction</li><li>toggling various attributes attached to functions and function parameters</li><li>changing constant values</li><li>adding a randomly generated instruction with randomly chosen inputs, and then replacing an SSA use with the value produced by this instruction</li><li>inserting a call to a function and then inlining its body using the LLVM inliner</li></ul>



<p>Of course you can probably think of additional ways to mutate a function &#8212; and hopefully we&#8217;ll get around to implementing those sooner or later. But this seems like a reasonable start.</p>



<p>Our original, radamsa-based fuzzing loop was structured as a collection of processes interacting through the filesystem. That setup incurred a lot of overhead due to repeated fork/exec, parse/print, etc. The new fuzzing loop with our custom mutator lives in a single unix process and has much higher fuzzing throughput. Basically, we invoke a program called alive-mutate on an IR file and let it know how long to run, and then it tests as many mutants as possible before reaching its time limit. If we want to fuzz each of the ~7000 unit tests in the <tt>llvm/test/Transforms</tt> directory for 10 minutes, this takes about 36 hours on a 32-core machine. We can create a mutant, optimize it, and prove the optimization correct up to about 16,000 times per second, though the loop can be much slower than this when the file is large or when Z3 has a hard time finding a proof. Here&#8217;s what the throughput looks like for some files randomly chosen from the LLVM unit test suite:</p>



<figure class="wp-block-image size-full"><img decoding="async" width="640" height="480" src="https://blog.regehr.org/wp-content/uploads/2022/05/Alive-mutateThroughput.png" alt="" class="wp-image-2304" srcset="https://blog.regehr.org/wp-content/uploads/2022/05/Alive-mutateThroughput.png 640w, https://blog.regehr.org/wp-content/uploads/2022/05/Alive-mutateThroughput-450x338.png 450w, https://blog.regehr.org/wp-content/uploads/2022/05/Alive-mutateThroughput-150x113.png 150w" sizes="(max-width: 640px) 100vw, 640px" /></figure>



<p>Let&#8217;s look at an example of how mutation works. We&#8217;ll start with this function that is found in the LLVM unit tests:</p>



<pre class="wp-block-preformatted has-extra-small-font-size">declare i8* @bar(i8*) readonly nounwind

define internal i8* @callee_with_explicit_control_flow(i8* %p) alwaysinline {
  %r = call i8* @bar(i8* %p)
  %cond = icmp ne i8* %r, null
  br i1 %cond, label %ret, label %orig

ret:
  ret i8* %r

orig:
  ret i8* %p
}</pre>



<p><a href="https://alive2.llvm.org/ce/z/BTKnfn">Here you can see</a> how the LLVM optimizer  transforms this code: it replaces the control flow with a ternary &#8220;select&#8221; instruction. Alive2 agrees with LLVM that that optimization is correct.</p>



<p>Now we invoke alive-mutate, asking it to run for 60 seconds and to dump its results into a directory called &#8220;output&#8221;:</p>



<pre class="wp-block-preformatted has-extra-small-font-size">$ alive-mutate -t 60 test.ll output
Unsound found! at 124th copies
Unsound found! at 140th copies
Unsound found! at 182th copies
Unsound found! at 269th copies
Unsound found! at 557th copies
program ended
Summary:
  818 correct transformations
  5 incorrect transformations
  0 failed-to-prove transformations
  0 Alive2 errors
$</pre>



<p>This is telling us that Alive2 believes that 818 mutants were optimized correctly by LLVM and that five were optimized incorrectly. Here&#8217;s one of them:</p>



<pre class="wp-block-preformatted has-extra-small-font-size">declare i8* @bar(i8*)

define internal i8* @callee_with_explicit_control_flow(i8* %p) {
  %r = call i8* @bar(i8* %p)
  %cond = icmp ne i8* %r, %p
  br i1 %cond, label %ret, label %orig

ret:
  ret i8* %r

orig:
  ret i8* %p
}</pre>



<p>What changed here? Just the second operand of the icmp instruction, which is testing two pointers for inequality. But now the optimizer does something different, it rewrites the function to this:</p>



<pre class="wp-block-preformatted has-extra-small-font-size">define i8* @callee_with_explicit_control_flow(i8* %p) {
  %r = call i8* @bar(i8* %p)
  ret i8* %r
}</pre>



<p>Before we look at why this is incorrect, let&#8217;s make sure we understand what the optimizer is thinking. The mutated function compares two pointers for inequality; if they are unequal it returns %r, if they are equal it returns %p. But if they&#8217;re equal, the optimizer reasons that returning %r is just as good as returning %p. In practice it&#8217;s a bit more complicated than that, a few different passes are involved, but the details aren&#8217;t all that important. The upshot is that <a href="https://alive2.llvm.org/ce/z/t7hQWR">this transformation is perfectly OK when performed on integer code</a>, but it is wrong for pointers because pointers carry provenance, and two arbitrary pointers cannot be interchanged just because they compare as equal. <a href="https://alive2.llvm.org/ce/z/dUei8C">Alive2 will even try to explain this to us if we read its counterexample carefully</a>. What it is telling us is that %p has offset 2 into a memory block of size 0 and that %r has offset 0 into a memory block of size 1. This explains how the pointers can compare as equal: %p is out-of-bounds (perfectly legal in LLVM as long as we don&#8217;t dereference it) and happens to refer to the same location as %r. But these are not interchangeable! In a different situation, this optimization would end up replacing an in-bounds pointer with an out-of-bounds pointer, which is not good; followed by other optimizations, this sort of thing could cascade into an end-to-end miscompile. Alas, this is a long-standing LLVM defect that is known to the community and there&#8217;s no point reporting it. There&#8217;s a bit of a broader lesson about formal methods here, which is that verification is just one small part of the larger enterprise of creating high-quality software. Happily, other bugs that we find and report do get fixed.</p>



<p>Here are some bugs we found using alive-mutate:</p>



<ul class="wp-block-list"><li><a href="https://bugs.llvm.org/show_bug.cgi?id=51351
">https://bugs.llvm.org/show_bug.cgi?id=51351</a></li><li><a href="https://bugs.llvm.org/show_bug.cgi?id=51618">https://bugs.llvm.org/show_bug.cgi?id=51618</a></li><li><a href="https://github.com/llvm/llvm-project/issues/53252
">https://github.com/llvm/llvm-project/issues/53252</a></li><li><a href="https://github.com/llvm/llvm-project/issues/53218
">https://github.com/llvm/llvm-project/issues/53218</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55003
">https://github.com/llvm/llvm-project/issues/55003</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55178
">https://github.com/llvm/llvm-project/issues/55178</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55201
">https://github.com/llvm/llvm-project/issues/55201</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55129
">https://github.com/llvm/llvm-project/issues/55129</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55271
">https://github.com/llvm/llvm-project/issues/55271</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55284
">https://github.com/llvm/llvm-project/issues/55284</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55287
">https://github.com/llvm/llvm-project/issues/55287</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55296
">https://github.com/llvm/llvm-project/issues/55296</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55342
">https://github.com/llvm/llvm-project/issues/55342</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55484
">https://github.com/llvm/llvm-project/issues/55484</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55490
">https://github.com/llvm/llvm-project/issues/55490</a></li><li><a href="https://github.com/llvm/llvm-project/issues/52884
">https://github.com/llvm/llvm-project/issues/52884</a></li></ul>



<p>In summary, alive-mutate&#8217;s hybrid fuzzing and formal verification approach works nicely and is also intellectually satisfying: we push the formal methods aspect as far as it can go, and then back off to randomized methods to explore individual cases in situations where we can&#8217;t yet formally reason about many cases at once.</p>



<p>So where&#8217;s this work going? We want to deploy it in two ways. First, we should have a few spare machines that just build an LLVM every day and then fuzz it, reporting any new bugs as they show up. The best time to discover a bug is before it gets pushed to the repo, but the second best time is immediately after it lands, while the logic is still fresh in someone&#8217;s mind. This kind of fuzzing offers a good chance to make that happen. Second, we want to use translation validation beyond bug-finding: we should use tools like Alive2 to monitor the compilation of codes whose correctness we care about, in order to ensure that nothing went wrong along the way. This is far more difficult, but we&#8217;re working on it.</p>


guid: https://blog.regehr.org/?p=2265
title: Formal-Methods-Based Bugfinding for LLVM’s AArch64 Backend
author: regehr
url: https://blog.regehr.org/archives/2265
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>[This piece is co-authored by Ryan Berger and Stefan Mada (both Utah CS undergrads), by Nader Boushehri, and by John Regehr.] </p>



<p>An optimizing compiler traditionally has three main parts: a frontend that translates a source language into an intermediate representation (IR), a &#8220;middle end&#8221; that rewrites IR into better IR, and then a backend that translates IR into assembly language. My group and our collaborators have spent a lot of time finding and reporting defects in the LLVM compiler&#8217;s middle end optimizers using <a href="https://alive2.llvm.org/ce/">Alive2</a>, a formal methods tool that we created. This piece is about extending that work to cover one of LLVM&#8217;s backends.</p>



<p>Why is it useful to try to prove that an LLVM backend did the right thing? It turns out that (despite the name) LLVM IR isn&#8217;t all that low-level &#8212; the backends need to do a whole lot of work in order to create efficient object code. In fact, there&#8217;s quite a bit of code in the backends, such as peephole optimizations that clean up local inefficiencies, that ends up duplicating analogous code in the LLVM middle end optimizers. And where there&#8217;s a lot of code, there&#8217;s a lot of potential for bugs.</p>



<p>The basic job performed by Alive2 is to prove that a function in LLVM IR refines another one, or else to provide a counterexample showing a violation of refinement. To understand &#8220;refinement&#8221; we need to know that due to undefined behaviors, an LLVM function can effectively act non-deterministically. For example, a function that returns a value that it loaded from uninitialized storage could return &#8220;1&#8221; one time we execute it and &#8220;2&#8221; the next time. An LLVM optimization pass is free to turn this function into one that always returns &#8220;1&#8221;. The refinement relation, then, holds when &#8212; for every possible circumstance (values of arguments, values stored in memory, etc.) in which that function could end up being called &#8212; the optimized function exhibits a subset of the behaviors of the original function. Non-trivial refinement, where the behaviors after optimization are a proper subset of the original behaviors, is ubiquitous in practice. If the optimized code contains any behavior not in the unoptimized code, this is a violation of refinement.</p>



<p>So how can we take a refinement checker for LLVM IR and use it to validate the correctness of an LLVM backend? We accomplish this by lifting the compiled AArch64 code back into Alive2 IR, which is very similar to LLVM IR. Then we check that the lifted IR refines the original code. A number of such binary lifters already exist and the techniques they employ are fairly well-understood: it&#8217;s basically a matter of converting each ARM instruction into one or more Alive2 instructions that implement that instruction, and then putting the resulting code into SSA form. We tried out some of the existing lifters, and for a few reasons chose not to base our project on any of them. First, fuzzing thrives on throughput, but forking and execing an external tool isn&#8217;t fast. Second, we ran into a lot of bugs in some of the lifters we tried &#8212; these bugs impede our work by hiding the LLVM bugs we&#8217;d have hoped to find. And third, we wanted control over the lifted code so we could tailor it to our needs.</p>



<p>So we wrote a new lifter, basically just by reading about the AArch64 ISA and ABI and implementing them. It&#8217;s very much a work in progress and also it&#8217;s not really general-purpose: it only needs to deal with the specific case of lifting AArch64 that was just generated from some LLVM IR that we have available to us. This allowed us to take some nice little shortcuts: we don&#8217;t have to worry about any instruction that the LLVM backend doesn&#8217;t emit, we get the types of function arguments and return values for free, and we didn&#8217;t need to write a parser for AArch64 assembly language because LLVM already has one.</p>



<p>Let&#8217;s work through a quick example. Here&#8217;s an LLVM function:</p>



<pre class="wp-block-preformatted has-small-font-size">define i32 @f(i32) {
  %c = icmp ult i32 %0, 777
  br i1 %c, label %t, label %f
t:
  %a = add i32 %0, 1
  ret i32 %a
f:
  ret i32 %0
}</pre>



<p>If f()&#8217;s argument is unsigned-smaller-than 777, the argument is incremented and returned, otherwise the argument is returned unmodified. The AArch64 backend translates this into:</p>



<pre class="wp-block-preformatted has-small-font-size">f:
        cmp     w0, #777
        cinc    w0, w0, lo
        ret</pre>



<p><a href="https://gcc.godbolt.org/z/qf979TrEr">Here&#8217;s this example in Compiler Explorer</a>. Do these three instructions faithfully implement the semantics of the LLVM code? It&#8217;s fairly clear that they do, but we want an automated proof. Here&#8217;s the Alive2 IR that we get from lifting the ARM code:</p>



<pre class="wp-block-preformatted has-small-font-size">define i32 @f-tgt(i32 X0) {
f:
  X0_1 = freeze i32 X0
  X0_2 = zext i32 X0_1 to i64
  WZR_2x1 = trunc i64 X0_2 to i32
  WZR_2x5 = icmp uge i32 WZR_2x1, 777
  X0_3x1 = trunc i64 X0_2 to i32
  X0_3x2 = trunc i64 X0_2 to i32
  X0_3x3 = add i32 X0_3x2, 1
  X0_3x4 = select i1 WZR_2x5, i32 X0_3x1, i32 X0_3x3
  ret i32 X0_3x4
}</pre>



<p>Does this lifted Alive2 IR refine the original LLVM IR? It does. Alive2 asks Z3 a series of questions and gets back answers that it likes and eventually it prints:</p>



<pre class="wp-block-preformatted has-small-font-size">Transformation seems to be correct!</pre>



<p><a href="https://alive2.llvm.org/ce/z/XCpfBu">Here&#8217;s this example online.</a></p>



<p>The lifted code isn&#8217;t very pretty, what&#8217;s going on? First, the &#8220;freeze&#8221; instruction &#8212; which tames LLVM&#8217;s poison and undef values &#8212; is required because those concepts do not exist at the AArch64 level. Second, there&#8217;s a good bit of extending and truncating going on, this happens because this platform has 64-bit registers but the instructions are only manipulating the bottom 32 bits of these registers. Third, there&#8217;s a bit of unnecessary code; we wrote a rudimentary optimizer for Alive2 IR, but this isn&#8217;t a place where we want to put much complexity, so we don&#8217;t try very hard to clean up the mess left behind by our lifter. The intent of our optimizer is simply to increase readability by removing as many irrelevant instructions as possible. For example, AArch64 assembly sometimes has to build constants using multiple instructions. This LLVM IR:</p>



<pre class="wp-block-preformatted has-small-font-size">define i64 @f() {
  ret i64 4183428261488279476
}</pre>



<p>turns into this AArch64:</p>



<pre class="wp-block-preformatted has-small-font-size">f:
	mov	x0, #49076
	movk	x0, #52566, lsl #16
	movk	x0, #34262, lsl #32
	movk	x0, #14862, lsl #48
	ret</pre>



<p>which we lift to this Alive2 IR:</p>



<pre class="wp-block-preformatted has-small-font-size">define i64 @f-tgt() {
f:
  %X0_2x1x0 = add i64 49076, 0
  %X0_3x1x0 = shl i64 52566, 16
  %X0_3x2x0 = and i64 %X0_2x1x0, -4294901761
  %X0_3x3x0 = or i64 %X0_3x2x0, %X0_3x1x0
  %X0_4x1x0 = shl i64 34262, 32
  %X0_4x2x0 = and i64 %X0_3x3x0, -281470681743361
  %X0_4x3x0 = or i64 %X0_4x2x0, %X0_4x1x0
  %X0_5x1x0 = shl i64 14862, 48
  %X0_5x2x0 = and i64 %X0_4x3x0, 281474976710655
  %X0_5x3x0 = or i64 %X0_5x2x0, %X0_5x1x0
  ret i64 %X0_5x3x0
}</pre>



<p>This is no fun to read. However, after some straightforward constant folding, we get this:</p>



<pre class="wp-block-preformatted has-small-font-size">define i64 @f-tgt() {<br>f:<br>  ret i64 4183428261488279476<br>}</pre>



<p> If we had lifted AArch64 to LLVM IR, instead of Alive2 IR, we could have gotten a lot of optimizations for free, but we didn&#8217;t do that since we try to avoid trusting LLVM in a tool that is designed to find bugs in LLVM.</p>



<p>So far we only have half of a fuzzing loop; if we&#8217;re going to look for bugs, we need to push a large number of test cases through the compile-decompile-verify cycle. We do this in three ways. First, using the LLVM unit test suite, which contains about 270,000 functions in LLVM IR. Second, by fuzzing with <a href="https://blog.regehr.org/archives/2148">the mutation engine we recently posted about</a>, using the LLVM unit tests as seeds. Third, using <a href="https://github.com/regehr/opt-fuzz">opt-fuzz</a>, a standalone LLVM generator that John wrote. So far the mutation approach is performing best in terms of finding bugs, probably because opt-fuzz generates a lot of stuff that&#8217;s just too simple to trigger bugs, but also because we&#8217;ve been leaning harder on the mutator. We&#8217;ve already seen several cases where the mutator and opt-fuzz can both find the same bug, which makes us feel warm and fuzzy.</p>



<p>So what do the bugs we find look like? Last week <a href="https://github.com/llvm/llvm-project/issues/55833">we reported that the AArch64 backend miscompiles this code</a>:</p>



<pre class="wp-block-preformatted has-small-font-size">define i64 @f(i64) {
  %2 = lshr i64 %0, 32
  %3 = shl i64 %2, 16
  %4 = trunc i64 %3 to i32
  %5 = ashr exact i32 %4, 16
  %6 = zext i32 %5 to i64
  ret i64 %6
}</pre>



<p>by translating it to this:</p>



<pre class="wp-block-preformatted has-small-font-size">_f:
	sbfx	x0, x0, #32, #16
	ret</pre>



<p>The bad behavior is seen both in the most recent release (LLVM 14) and also in the current sources.</p>



<p>Here are the bugs we&#8217;ve found so far (we&#8217;re not looking for crashes &#8212; these are all miscompilations):</p>



<ul class="wp-block-list"><li><a href="https://github.com/llvm/llvm-project/issues/55003
">https://github.com/llvm/llvm-project/issues/55003</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55129
">https://github.com/llvm/llvm-project/issues/55129</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55150
">https://github.com/llvm/llvm-project/issues/55150</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55178
">https://github.com/llvm/llvm-project/issues/55178</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55201
">https://github.com/llvm/llvm-project/issues/55201</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55271
">https://github.com/llvm/llvm-project/issues/55271</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55284
">https://github.com/llvm/llvm-project/issues/55284</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55287
">https://github.com/llvm/llvm-project/issues/55287</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55296
">https://github.com/llvm/llvm-project/issues/55296</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55342
">https://github.com/llvm/llvm-project/issues/55342</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55484
">https://github.com/llvm/llvm-project/issues/55484</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55490
">https://github.com/llvm/llvm-project/issues/55490</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55627
">https://github.com/llvm/llvm-project/issues/55627</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55644">https://github.com/llvm/llvm-project/issues/55644</a></li><li><a href="https://github.com/llvm/llvm-project/issues/55833">https://github.com/llvm/llvm-project/issues/55833</a></li></ul>



<p>It&#8217;s great to see that a number of these have been fixed already! Note that some of the bugs only happen when <a href="https://www.youtube.com/watch?v=S6SNs2ttdoA">global instruction selection</a> is enabled &#8212; this is not yet enabled by default.</p>



<p>What have we learned? Perhaps surprisingly, a number of the bugs we found are not ARM-specific, but rather are in shared, non-target-specific LLVM backend code, which means that some of these bugs end up affecting multiple backends. Something else we learned (predictable, in retrospect) is that testing a backend is in an important sense more difficult than testing the middle end. Whereas most middle-end optimizations don&#8217;t care about the bitwidth of the instructions involved, the backends most certainly do care about this: instruction selection will be different at 8 vs 16 vs 32 vs 64 bits, different still at 19 bits, and even more different at 91 bits. So we have to run a lot more test cases through a backend than the middle end to get roughly equivalent coverage, and also we&#8217;re going to need to implement mutations that specifically change bitwidths in a test case. Does anyone actually care if the LLVM backends can deal with non-power-of-2 bitwidths? Turns out yes: in an optimized compile of LLVM itself, using LLVM, every integer width from 1 through 64 can be found. The largest integer that occurs when compiling LLVM using LLVM is 320 bits wide.</p>



<p>One might ask: Why are we doing bugfinding work? Why not focus all of our effort on developing our tooling to the point where it can prove that application codes of interest are being compiled correctly? There are two answers. The first is personal: bugfinding is a rewarding activity that lets us engage with the LLVM community in ways that we enjoy. In contrast, true formal verification work can feel isolated and also the endpoint is very difficult to reach: real codes invariably contain features that are hostile to formal methods. The final result, if reached, tends to be a bit of a letdown (&#8220;wow, it&#8217;s correct&#8221;). The second reason that we like to do bugfinding work is that it benefits all LLVM users, as opposed to a pure verification effort that benefits only the small subset of users who are willing to put formal methods into their workflow. But, of course, in the long run, translation validation of real codes is something we&#8217;re pursuing.</p>



<p>The AArch64 lifter is part of our larger program of working towards developer tools that deserve to be trusted. There&#8217;s plenty of work left to do on our lifter, and also some follow-on projects suggest themselves. First, it would be nice to derive our lifter from an existing formal semantics for AArch64 rather than writing it afresh from the documentation. Second, we&#8217;d like to support additional architectures. For example, we think that RISC-V will be pretty easy to support given the infrastructure we now have in place.</p>



<p>[Our ARM lifter is not yet available on the <a href="https://alive2.llvm.org/ce/">Alive2 Compiler Explorer instance</a>.]</p>



<p>[This work is sponsored, in part, by <a href="https://www.woven-planet.global/en">Woven Planet</a>]</p>


guid: https://blog.regehr.org/?p=2485
title: Why Do Peephole Optimizations Work?
author: regehr
url: https://blog.regehr.org/archives/2485
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>In <a href="https://dl.acm.org/doi/10.1145/364995.365000">its original form</a>, a peephole optimization applied to a collection of instructions located close together in a program. For example, in a register transfer language we might find this sequence of instructions:</p>



<pre class="wp-block-preformatted">r0 = xor r8, -1
r1 = xor r9, -1
r0 = and r0, r1</pre>



<p>Here, assuming the two&#8217;s complement representation, -1 is just a convenient way to specify a register filled with ones. Following a De Morgan law, a peephole optimization could rewrite this sequence as:</p>



<pre class="wp-block-preformatted">r0 = or r8, r9
r0 = xor r0, -1</pre>



<p>But modern compilers don&#8217;t use RTL as their primary intermediate representation (IR), they&#8217;re more likely to have an SSA-based IR that represents the flow of data between instructions explicitly. You can think of SSA as teasing out a purely functional subset of a program that you write. For example in LLVM IR we might have:</p>



<pre class="wp-block-preformatted">%x = xor %a, -1
...
%y = xor %b, -1
...
%z = and %x, %y</pre>



<p>Here the instructions need not be located near each other, they could even live in different basic blocks; it is only required that %x and %y both <em>dominate</em> %z &#8212; this just means that there&#8217;s no way to get to %z without having run across definitions for both %x and %y. There&#8217;s a bit more to SSA than this, but the rest doesn&#8217;t matter to us right now. Here, again, we could use the De Morgan law to rewrite this computation and reduce the overall instruction count by one:</p>



<pre class="wp-block-preformatted">%tmp = or %a, %b
%z = xor %tmp, -1</pre>



<p>These kinds of localized rewrites are intuitively appealing: they look like they should work. And they do! But let&#8217;s make sure we understand why.</p>



<p>First, let&#8217;s talk about what a peephole optimization, in isolation, should do. A strong correctness requirement would be that the new code is equivalent to the old code. That is, for all possible values of the inputs to the code that&#8217;s going to be optimized, the unoptimized and optimized code have to have the same observable behavior. Here we have to be careful about what we mean by &#8220;observable behavior&#8221; &#8212; it should include the value actually computed by the code in addition to any side effects, for example on the state of memory. We usually <em>do not want</em> the execution time of the program, nor its code size, to be part of its observable behavior, since preserving those behaviors would preclude all optimizations for speed or size. We also don&#8217;t want microarchitectural effects, such as changes to the state of branch predictors and caches, to be observable; though in certain security-sensitive applications, we might change our minds about this. But anyway, however we define it, equivalence is a stronger criterion than we want optimizations to obey. Rather, we want the optimized code to <em>refine</em> the unoptimized code.</p>



<p>Refinement roughly means &#8220;take a description of a system and produce a more specific version of that system,&#8221; and it is an ubiquitous concept in programming. We often start with a rough design for a software system and then refine it into a specification, which we then refine into an implementation. A compiler then refines the source-level implementation when it converts it into object code. If the compiler ever produces a non-refinement, this is considered to be a compiler bug. (We can flip this around and say that producing a refinement is the top-level correctness requirement for a compiler.) At the level of the compiler, non-trivial refinements correspond to removing possible behaviors from the code. For example, consider a high-performance parallel sorting algorithm that isn&#8217;t stable: it doesn&#8217;t necessarily preserve the ordering of elements whose sort keys compare the same. Depending on how the parallel execution goes, this algorithm might produce different outputs different times it&#8217;s run, but the different unstably-sorted outputs are all considered to be correct. Now consider a compiler that takes this algorithm and compiles it to a sequential platform, meaning that the entire sorting algorithm runs without concurrency, on a single core. On this platform we would expect that, for a given unsorted input, the sorting algorithm will produce the same (unstable) result every time that it is run. The compiler has eliminated non-determinism, and we generally consider that to be an OK thing for a compiler to do. This is an example of a compiler performing a non-trivial refinement.</p>



<p>Refinement also happens for sequential code. For example, the size of the &#8220;int&#8221; type in the C++ language is implementation-defined. One C++ compiler might refine an int in the source code to a 32-bit value, whereas a different one might refine it to a 64-bit value. Unspecified behavior in C and C++, such as the order of evaluation of arguments to a function, is also refined into some specific order at compile time.</p>



<p>For peephole optimizations, refinement is generally about undefined behavior. This isn&#8217;t the evil kind of programmer-facing undefined behavior that we often read about; this is specific to the compiler IR and it allows the compiler to perform interesting optimizations even when, for example, we&#8217;re using LLVM to compile Swift or Rust. <a href="https://users.cs.utah.edu/~regehr/papers/undef-pldi17.pdf">We wrote somewhat extensively about this topic</a> a few years ago. <a href="https://gcc.godbolt.org/z/PjYee78j5">Here&#8217;s an example</a> of LLVM&#8217;s peephole pass &#8220;InstCombine&#8221; performing a non-trivial refinement where it takes an immediate undefined behavior, dividing by zero, and turns it into a function returning LLVM&#8217;s poison value. Then, we can see an LLVM backend <a href="https://gcc.godbolt.org/z/W56jxajsY">further refining the poison value</a> into code that returns whatever value happened to be sitting in w0, which is the register that the AArch64 ABI uses to return a 32-bit integer value. The backend could also have refined this function to return a constant such as zero, but that would have been less efficient.</p>



<p>So now we hopefully understand what peephole optimizations are supposed to do: the optimized code must refine the original code, in a context where &#8220;refinement&#8221; means something like &#8220;for defined inputs, preserve the behavior of the code; for undefined inputs, feel free to pick a subset of the behaviors.&#8221; A much more precise definition can be found in <a href="https://users.cs.utah.edu/~regehr/alive2-pldi21.pdf">the Alive2 paper</a>. But we still haven&#8217;t figured out why peephole optimizations work. The thing we&#8217;re going to need to do here is show that a refinement at the site of a peephole optimization (that is, buried somewhere deep in the code) leads to a refinement at the whole program level.</p>



<p>Let&#8217;s start with a function <em>f</em> that contains some peephole-optimizable code. Logically, we&#8217;ll split it into two functions: a tiny little one, <em>o</em>, that contains the instructions that are going to get optimized, and a larger residual function, <em>r</em>, that contains that rest of the code. Composing the two new functions together is equivalent to the original function: <em>f</em> = <em>r</em>(<em>o</em>). Now, by assumption, we have the optimized version of <em>o</em> refining <em>o</em>: that is, <em>o</em> → <em>o&#8217;</em>. (Read <em>o</em> → <em>o&#8217;</em> as &#8220;<em>o</em> is refined by <em>o</em>-prime.&#8221;) To move forward we need to know that refinement is compositional. In other words, if <em>x</em> → <em>y</em> then <em>g</em>(<em>x</em>) →<em> g</em>(<em>y</em>), for some arbitrary function <em>g</em>. We haven&#8217;t formally defined refinement, but any reasonable definition of it will have this property. Compositionality lets us prove that <em>r</em>(<em>o</em>) → <em>r</em>(<em>o</em>&#8216;). And now, since <em>f</em> = <em>r</em>(<em>o</em>), we get f → <em>r</em>(o&#8217;), and that is the function-level version of the result that we wanted in the first place. To get the whole program result we can keep applying the compositionality of refinement. And that&#8217;s why peephole optimizations work!</p>



<p>[My students Zhengyang Liu and Manasij Mukherjee contributed to this piece. It happened because one time I remarked to Nuno Lopes that I wasn&#8217;t sure I actually understood why peephole optimizations work and he said &#8220;Oh yeah refinement is compositional.&#8221;]</p>


guid: https://blog.regehr.org/?p=2578
title: Dataflow Analyses and Compiler Optimizations that Use Them, for Free
author: regehr
url: https://blog.regehr.org/archives/2578
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>Compilers can be improved over time, but this is a slow process. &#8220;Proebsting&#8217;s Law&#8221; is an old joke which suggested that advances in compiler optimization will double the speed of a computation every 18 years &#8212; but if anything this is optimistic. Slow compiler evolution is never a good thing, but this is particularly problematic in today&#8217;s environment of rapid innovation in GPUs, TPUs, and other entertaining platforms.</p>



<p>One of my research group&#8217;s major goals is to create technologies that enable self-improving compilers. Taking humans out of the compiler-improvement loop will make this process orders of magnitude faster, and also the resulting compilers will tend to be correct by construction. One such technology is superoptimization, where we use an expensive search procedure to discover optimizations that are missing from a compiler. Another is <a href="https://users.cs.utah.edu/~regehr/generalization-oopsla24.pdf">generalization</a>, which takes a specific optimization (perhaps, but not necessarily, discovered by a superoptimizer) and turns it into a broadly applicable form that is suitable for inclusion in a production compiler.</p>



<p>Together with a representative benchmark suite, superoptimization + generalization will result in a fully automated self-improvement loop for one part of an optimizing compiler: the peephole optimizer. In the rest of this piece I&#8217;ll sketch out an expanded version of this self-improvement loop that includes dataflow analyses.</p>



<p>The goal of a dataflow analysis is to compute useful facts that are true in every execution of the program being compiled. For example, if we can prove that x is always in the range [5..15], then we don&#8217;t need to emit an array bound check when x is used as an index into a 20-element array. This particular dataflow analysis is the integer range analysis and compilers such as GCC and LLVM perform it during every optimizing compile. Another analysis &#8212; one that LLVM leans on particularly heavily &#8212; is &#8220;known bits,&#8221; which tries to prove that individual bits of SSA values are zero or one in all executions.</p>



<p>Out in the literature we can find a huge number of dataflow analyses, some of which are useful to optimize some kinds of code &#8212; but it&#8217;s hard to know which ones to actually implement. We can try out different ones, but it&#8217;s a lot of work implementing even one new dataflow analysis in a production compiler. The effort can be divided into two major parts. First, implementing the analysis itself, which requires creating an abstract version of each instruction in the compiler&#8217;s IR: these are called dataflow transfer functions. For example, to implement the addition operation for integer ranges, we can use [lo1, hi1] + [lo2, hi2] = [lo1 + lo2, hi1 + hi2] as the transfer function. But even this particularly easy case will become tricker if we have to handle overflows, and then writing a correct and precise transfer function for bitwise operators is much less straightforward. Similarly, consider writing a correct and precise known bits transfer function for multiplication. This is not easy! Then, once we&#8217;ve finished this job, we&#8217;re left with the second piece of work which is to implement optimizations that take advantage of the new dataflow facts.</p>



<p>Can we automate both of these pieces of work? We can! There&#8217;s an initial bit of work in creating a representation for dataflow facts and formalizing their meaning that cannot be automated, but this is not difficult stuff. Then, to automatically create the dataflow transfer functions, we turn to this <a href="https://pages.cs.wisc.edu/~loris/papers/oopsla22.pdf">very nice paper</a> which synthesizes them basically by squeezing the synthesized code between a hard soundness constraint and a soft precision constraint. Basically, every dataflow analysis ends up making approximations, but these approximations can only be in one direction, or else analysis results can&#8217;t be used to justify compiler optimizations. The paper leaves some work to be done in making this all practical  in a production compiler, but it looks to me like this should mainly be a matter of engineering.</p>



<p>A property of dataflow transfer functions is that they lose precision across instruction boundaries. We can mitigate this by finding collections of instructions commonly found together (such as those implementing a minimum or maximum operation) and synthesizing a transfer function for the aggregate operation. We can also gain back precision by special-casing the situation where both arguments to an instruction come from the same source. We don&#8217;t tend to do these things when writing dataflow transfer functions by hand, but in an automated workflow they would be no problem at all. Another thing that we&#8217;d like to automate is creating efficient and precise <a href="https://arxiv.org/abs/1309.5146">product operators</a> that allow dataflow analyses to exchange information with each other.</p>



<p>Given a collection of dataflow transfer functions, creating a dataflow analysis is a matter of plugging them into a generic dataflow framework that applies transfer functions until a fixpoint is reached. This is all old hat. The result of a dataflow analysis is a collection of dataflow facts attached to each instruction in a file that is being compiled.</p>



<p>To automatically make use of dataflow facts to drive optimizations, we can use a superoptimizer. For example, we taught Souper to use several of LLVM&#8217;s dataflow results. This is easy stuff compared to creating a superoptimizer in the first place: basically, we can reuse the same formalization of the dataflow analysis that we already created in order to synthesize transfer functions. Then, the generalization engine also needs to fully support dataflow analyses; our <a href="https://users.cs.utah.edu/~regehr/generalization-oopsla24.pdf">Hydra</a> tool already does a great job at this, there are plenty of details in the paper.</p>



<p>Now that we&#8217;ve closed the loop, let&#8217;s ask whether there are interesting dataflow analyses missing from LLVM, that we should implement? Of course I don&#8217;t know for sure, but one such domain that I&#8217;ve long been interested in trying out is &#8220;congruences&#8221; where for a variable v, we try to prove that it always satisfies v = ax+b, for a pair of constants a and b. This sort of domain is useful for tracking values that point into an array of structs, where a is the struct size and b is the offset of one of its fields.</p>



<p>Our current generation of production compilers, at the implementation level, is somewhat divorced from the mathematical foundations of compilation. In the future we&#8217;ll instead derive parts of compiler implementations &#8212; such as dataflow analyses and peephole optimizations &#8212; directly from these foundations.</p>


guid: https://blog.regehr.org/?p=2124
title: Looking for Missed Alarm Bugs in a Formal Verification Tool
author: regehr
url: https://blog.regehr.org/archives/2124
feedurl: https://blog.regehr.org/feed
content_mime_type: text/html
content:
<p>[This piece is co-authored with Vsevolod Livinskii.]</p>



<p>Formal verification isn&#8217;t some sort of magic pixie dust that we sprinkle over a computer system to make it better. Real formal verification involves a lot of  the same kind of difficult, nasty, grungy engineering work that any other systems-level job involves. Furthermore, the verification tools themselves are very difficult to get right. They are subject to many kinds of defects, and they are if anything more difficult to debug than other software. They need to be rigorously tested if we&#8217;re going to trust them.</p>



<p><a href="https://users.cs.utah.edu/~regehr/alive2-pldi21.pdf">Alive2 is a translation validation tool</a>: given two versions of a function in LLVM IR&#8211;usually these correspond to some code before and after an optimization has been performed on it&#8211;Alive2 tries to either prove that the optimization was correct, or prove that it was incorrect. Alive2 is used in practice by compiler engineers: <a href="https://github.com/llvm/llvm-project/issues?q=is%3Aissue+alive2.llvm.org">more than 600 LLVM issues</a> link to our <a href="https://alive2.llvm.org/ce/">online Alive2 instance</a>.</p>



<p>Ignoring crashes and such, we can broadly categorize defects in Alive2 as:</p>



<ul class="wp-block-list">
<li>a false alarm: Alive2 signals an error when none was present</li>



<li>a missed alarm: Alive2 fails to signal an error when one was present</li>
</ul>



<p>The first kind of error, false alarms, are not too difficult to test for: we simply ask Alive2 to verify a large number of optimizations, and we look closely at the errors that it signals. Every such error is the result of either a bug in LLVM or in Alive2. The second kind of error, the ones in the bottom left quadrant, are much more difficult to test for:</p>



<figure class="wp-block-image size-large"><img fetchpriority="high" decoding="async" width="1280" height="817" src="https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-1280x817.png" alt="" class="wp-image-2165" srcset="https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-1280x817.png 1280w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-450x287.png 450w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-150x96.png 150w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-768x490.png 768w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-1536x981.png 1536w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM-1568x1001.png 1568w, https://blog.regehr.org/wp-content/uploads/2021/10/Screen-Shot-2021-10-20-at-12.16.35-PM.png 1688w" sizes="(max-width: 1280px) 100vw, 1280px" /></figure>



<p>The thing we&#8217;re missing is test cases that might trigger a missed-alarm bug: each such test requires a pair of functions in LLVM IR that have the same signatures, but which provably behave differently for at least one choice of inputs. We can&#8217;t rely on compiler bugs to create these test cases because LLVM is not a very buggy compiler! The vast majority of optimizations it performs are correct.</p>



<p>To find missed-alarm bugs, we tried two different ideas. In the rest of this piece we&#8217;ll describe both of them.</p>



<p>First, we started with <a href="https://github.com/intel/yarpgen">YARPGen</a>, a random program generator that has been used to find a large number of compiler bugs. The key guarantee that YARPGen provides is that the C or C++ function it generates is free of undefined behavior. Without this guarantee, we can&#8217;t really do randomized differential testing, since different compilers will tend to exploit undefined behaviors differently. However, this guarantee isn&#8217;t a sufficient basis for finding missed alarm bugs, so we modified YARPGen to suit our purposes. Our modified version generates a random function, as usual, and then it does something new: it randomly mutates this function slightly, while maintaining the guarantee that the new function is also free of undefined behavior.</p>



<p>Now we have two functions, very similar to each other, both guaranteed to be free of undefined behavior. Is this a basis for finding missed alarm bugs? Not quite&#8211;we still need to make sure that these functions produce different results when executed, for at least one set of inputs. To do this, we simply compile and run the pairs of functions, throwing away any pairs where our mutation happens to not change the observable behavior. Finally, we compile the pair of functions into LLVM IR and then ask Alive2 to see if one of them refines the other. By construction, this check must fail&#8211;if Alive2 does not signal an error, then we have found the kind of missed alarm bug in Alive2 that we were originally looking for.</p>



<p>The other method that we have for seeking out missed alarm bugs is one that we arrived at by accident. We realized that Zhengyang Liu&#8217;s LLVM superoptimizer, Minotaur, is basically passively looking for missed alarm bug every time that we use it. Here&#8217;s the <a href="https://github.com/minotaur-toolkit/minotaur">Minotaur</a> source code and here&#8217;s <a href="https://users.cs.utah.edu/~regehr/minotaur.pdf">a paper about it</a>.</p>



<p>For every LLVM instruction in the program being optimized, Minotaur tries to find a cheaper way to compute it. It does this by extracting that instruction, and some of its backwards data, control, and memory dependencies into a new LLVM function that returns the value computed by the target instruction. This new function serves as the <em>specification</em> for a program synthesis problem, where the goal is to find a cheaper way to compute the specification. Minotaur uses Alive2 to ensure that the new function refines the old one, and it uses <a href="https://llvm.org/docs/CommandGuide/llvm-mca.html">llvm-mca</a> to ensure that the new function is cheaper to compute than the old one.</p>



<p>Synthesis works by enumerating a large number of <em>partially symbolic</em> candidates, where instructions are represented concretely, but literal constants are represented symbolically. Zhengyang modified Alive2 in such a way that when a candidate contains at least one symbolic constant, it emits an exists-forall solver query, which asks the solver: &#8220;Do there exist values for the symbolic constants in the candidate, such that the candidate refines the specification?&#8221;</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"></blockquote>



<p>The details of Minotaur&#8217;s synthesis procedure don&#8217;t matter too much; the important point is that since there are a huge number of candidates, the vast majority of which do not refine the specification, we end up giving Alive2 many opportunities to miss an alarm. </p>



<p>But, if Alive2 misses an alarm when it is invoked by Minotaur, how will we know? Well, keep in mind that missing an alarm means that Alive2 claimed that a candidate refined the specification when in fact no refinement relation exists. Since refinement is the correctness criterion for an optimizer, these failures, by definition, lead to miscompilations. Since we routinely use Minotaur to compile large open source programs and then we run their test suites, we should have a decent chance of finding any miscompilations that it introduces.</p>



<p>We&#8217;ve looked at two different ways, one using randomized search and the other using a small-scale exhaustive search, to look for missed alarm bugs in Alive2. What have we found so far? Not much! It does not look like Alive2 and Z3 are in the habit of missing alarms. This means that it is fulfilling its top-level design goal, which is good, since people actually rely on Alive2 in practice.</p>



<p>So is this the end of the story? Are we now certain that Alive2 doesn&#8217;t miss alarms? Alas, no, we&#8217;re not sure. My suspicion is that if we really wanted to find missed alarm bugs, we would look at Alive2&#8217;s support for function attributes and similar constructs that neither YARPGen nor Minotaur stress in interesting ways. </p>


guid: http://crawl.develz.org/wordpress/?p=4829
title: Trunk Updates 5 August 2020
author: ebering
url: https://crawl.develz.org/wordpress/trunk-updates-5-august-2020
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p><em>Clouds of trunk updates condense around you!</em></p>
<p>Hello crawlers, and welcome to another round of trunk updates, fresh from the Citadel in the Iron City.</p>
<p>The biggest change this week is the introduction of the <strong>malevolent chronology of Zot</strong>. One of food&#8217;s many (badly executed) purposes was a forward progress clock; in the absence of any such clock tedious optimal strategies crept in, of the kind that we on the devteam categorically prefer to prevent (check out the philosophy section of the manual for an elaboration). The Zot clock is a rather loose clock, but should you tarry too long Zot will find you, and rapidly end you. The hope is that this clock is sufficiently loose that only the most egregious behavior gets caught by it. Since this is trunk there will be further tuning.</p>
<p>Read on for the remainder of the tweaks, shifts, and ever seething chaos of trunk</p>
<ul>
<li><strong>Contamination</strong> now decays three times faster.</li>
<li>Scarves of <strong>shadows</strong> now only reduce LOS by 1, instead of a multiplicative factor.</li>
<li>The <strong>rat skin cloak</strong> is changed from evocable +Rats to a passive *Rats, summoning rats when the wearer is hit.</li>
<li><strong>Ripostes</strong> no longer trigger hydra-head chopping, *Rage, or Minotaur headbutts.</li>
<li><strong>Phantom Mirrors</strong> and <strong>Boxes of Beasts</strong> now recharge on XP, mirrors have a single charge and boxes have three charges.</li>
<li>A new misc evoker <strong>condenser vanes</strong> replace <strong>wands of clouds</strong>. Condenser Vanes have 4 charges and create clouds on and around monsters in view, with the clouds scaling in power similar to the old wand of clouds.</li>
<li><strong>Wanderers</strong> now have a chance to start with any of the XP charged misc evokers.</li>
</ul>
<div>That&#8217;s all for now, but for a preview of what&#8217;s in the pipeline: <em>hellmonk </em>has promised us a good look at Demonspawn mutations, <em>chequers</em> continues to add maniacal and twisted powers to the lords of pandemonium, further tweaks and shifts to evocations are under consideration, and another round of unrand tweaks is starting to shape up. As always, if you&#8217;ve got some ideas come by ##crawl-dev or check out our <a href="https://github.com/crawl/crawl#how-you-can-help">GitHub for how to contribute</a>!</div>
<div style="text-align: right;"><em>The now-empty trunk updates crumble to dust.</em></div>


guid: http://crawl.develz.org/wordpress/?p=4831
title: Trunk Updates 13 August 2020
author: ebering
url: https://crawl.develz.org/wordpress/trunk-updates-13-august-2020
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p><em>The trunk update comes into view, it is surrounded by clouds of flame!</em></p>
<p>Hello crawlers, and welcome to another round of trunk updates from the frigid winter wastes of <del>Falls Creek</del> Coc:7.</p>
<p>This week has seen a fair share of bug fixing, including from community members <em>Goratrix</em> and <em>Naruni</em>. If you&#8217;d like to help out in the effort of developing crawl come by ##crawl-dev or check out our <a href="https://github.com/crawl/crawl#how-you-can-help">GitHub for how to contribute</a>!</p>
<p>Let&#8217;s take a look at what else has been woven into the next <del>illusion</del> version:</p>
<ul>
<li><strong>Gauntlet</strong> monsters now generate awake</li>
<li>A new round of <strong>unrand revisions</strong> has begun in earnest:</li>
<ul>
<li><strong>Wucad Mu</strong> no longer provides evocable channeling. Instead, with a chance depending on evocations, it refunds the MP cost of the spell. There is a chance of confusion or intelligence drain (which scales on spell cost and evocations skill) when a spell is refunded.</li>
<li>The <strong>cloak of the Thief</strong> is changed from +Fog to *Fog, creating fog when the player is hit.</li>
<li>The <strong>Majin-Bo</strong> loses Archmagi, and instead gives a chance for any cast spell to be vampiric, healing the caster for damage done to living creatures.</li>
<li>The <strong>shield of Resistance</strong> is now a +2 kite shield {rF++ rC++ MR++}.</li>
</ul>
<li><strong>Rampaging</strong> can now appear as an artefact property on all armour types. Enjoy your rampaging octopodes, courtesy of <em>Implojin</em>!</li>
<li><strong>Staves of energy</strong> are removed.</li>
<li><em>chequers</em> has finished his project to add <strong>cloud rings</strong> to <strong>Pandemonium lords</strong>. There&#8217;s now a chance that pan lords will come with a ring of one of the following clouds: flame, ice, thunder, chaos, mutation, draining, acid, or miasma. The chance is much higher for melee pan lords, and the cloud will often flavorfully match the pan lord&#8217;s melee brand. Watch out for Faroclos, armed with a chaotic touch and surrounded by clouds of chaos!</li>
<li>Not content to let chequers have all the fun, first-time contributor <em>amcnicky </em>sent in some <strong>new Pandemonium lord tiles</strong>.</li>
</ul>
<p>There&#8217;s much more planned before we arrive at 0.26, so stay tuned for the latest and don&#8217;t hesitate to get involved yourself. Until next time, happy crawling!</p>
<p style="text-align: right;"><em>The trunk update sneers, &#8220;Beg for mercy, thou defiled idle-headed gnoll-tail!&#8221;</em></p>


guid: http://crawl.develz.org/wordpress/?p=4837
title: Trunk Updates 21 August 2020
author: ebering
url: https://crawl.develz.org/wordpress/trunk-updates-21-august-2020
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p><em>You feel the power of the Orb delaying this trunk update!</em></p>
<p>Hello crawlers, and welcome to another update from the fires of Geh:7. This stretch has seen a mix of gameplay and infrastructure changes, both with contributions from the community, as well as bug fixes and tweaks from <em>Goratrix, chequers, </em>and <em>Implojin</em>. If you’d like to help out in the effort of developing crawl come by ##crawl-dev or check out our <a href="https://github.com/crawl/crawl#how-you-can-help">GitHub for how to contribute</a>!</p>
<p>On the gameplay side:</p>
<ul>
<li><strong>Palentonga&#8217;s </strong>HP aptitude has been increased to +1</li>
<li><strong>staves of earth</strong> now deal higher damage, but with the shrapnel effect of being affected 3 times by AC.</li>
<li><strong>staves of conjuration</strong> now deal irresistible damage that checks AC (taking Earth&#8217;s old effect).</li>
<li><strong>staves of summoning and wizardry</strong> are removed.</li>
<li>The <strong>Controlled Blink</strong> spell has blinked right out of the game, and is removed.</li>
<li><strong>Blink scrolls</strong> no longer cause Contam.</li>
<li>The <strong>Orb&#8217;s translocations interference</strong> is reduced: blink scrolls can be used in Zot and on the Orb run, as can Passage of Golubria. However, passages are shorter lived and less accurate under the Orb&#8217;s influence.</li>
<li>The various <strong>plants and fungi</strong> of the dungeon are now much weaker, once hit they begin to wither and die in 3-5 turns. The exceptions are bushes and briars, made of stouter stuff.</li>
</ul>
<p>On the infrastructure side:</p>
<ul>
<li><em>wjchen</em> sent us a community contribution to improve <strong>WebTiles glyphs mode</strong> font handling. This introduces two new options for customization, glyph_mode_font and glyph_mode_font_size, for all you WebTiles glyphs players.</li>
<li>The window drawing loop has seen some optimizations.</li>
</ul>
<div><em>The trunk update blinks out of view!</em></div>


guid: http://crawl.develz.org/wordpress/?p=4841
title: CBRO upgrade/transition plan
author: advil
url: https://crawl.develz.org/wordpress/cbro-upgradetransition-plan
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p>Sad news: CBRO will be retiring. But good news: CBRO will be replaced by CBR2, which will not run out of hard drive space for at least 10 years! Here&#8217;s the full plan:</p>
<ul>
<li><a href="https://cbro.berotato.org" title="CBR2 (cbro.berotato.org)">CBR2 (cbro.berotato.org)</a> is a completely new server with a much bigger disk and better cpu, administered by ZureaL, advil, and johnstein, hosted by ZureaL, and located in the midwest.</li>
<li>We have imported the CBRO login database from CBRO to CBR2, but most other data (including save files) will not be imported, for a fresh start.</li>
<li>CBR2 is currently open for test games only, and <strong>the server will be reset on 9/25 at 11:59 EDT</strong> &#8212; games played before then are not on the record. Games played after this reset will appear in CAO scoring and Sequell. (We will do one last pull of the username db around then as well.)</li>
<li>After CBR2 goes live on 9/26, CBRO will stop allowing new games to begin. CBRO will remain open for a while to allow finishing off any old games, and will continue hosting its current data until its shutdown (we are estimating final shutdown around 12/1/20). Any games finished until that point will also be on the record. At that point we will most likely archive long-term data away from the CBRO server into cloud storage.</li>
</ul>
<p>Thanks very much to johnstein for solo-running CBRO for so many years, and to ZureaL for taking the lead in setting up CBR2!</p>


guid: http://crawl.develz.org/wordpress/?p=4851
title: Trunk Updates 6 November 2020
author: PleasingFungus
url: https://crawl.develz.org/wordpress/trunk-updates-6-november-2020
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p>Hello, crawlers! It&#8217;s been a bit, but here&#8217;s the latest from Trunk:</p>
<ul>
<li>The Swamp branch has been drastically reworked:
<ul>
<li>New monster: <strong>will-o-the-wisp</strong>, an unearthly gas that spits out powerful Foxfires.</li>
<li>New monster: <strong>goliath frog</strong>, capable of licking players from afar with its acidic tongue.</li>
<li>New monster: <strong>eleionoma</strong>, a spirit of the swamp akin to water nymphs. It flows with the woods and shoots shrapnel at nearby enemies. It&#8217;s quite difficult to kill while it&#8217;s next to trees.</li>
<li>New monster: <strong>bloated husk</strong>, a shambling abomination that explodes with a powerful burst of putrescent gas.</li>
<li>New monster: <strong>bunyip</strong>, an Australian cryptid that hits with enormous force but a slow recovery. (Did you know DCSS is Australian?)</li>
<li>New monster: <strong>fenstrider witch</strong>, a rare top-tier threat casting Paralyse, melee-range Agony, and Throw Sludge, a new spell that splatters Eringya&#8217;s Noxious Bog all over the victim.</li>
<li>Bog bodies cast much more powerfully, and their freezing touch attack has been merged in with their normal attack so that it actually does something.</li>
<li>Many weak and irrelevant enemies no longer appear in Swamp.</li>
<li>Swamp levels are, on average, about 20-25% smaller.</li>
<li>For a limited testing period, <strong>Swamp will appear in every game of Dungeon Crawl</strong>, even sometimes alongside Shoals. Enjoy!</li>
</ul>
</li>
<li>Further monster changes:
<ul>
<li><strong>Boulder beetles are back!</strong> This time, their trademark rolling attack uses standard monster movement rules, rather than Orb of Destruction projectile movement. That means they can follow you as you move and around corners. They can also blink. On the plus side, they&#8217;re worth more XP!</li>
<li>Crocodiles (but not alligators), porcupines, and hippogrives have departed this mortal plane. None of them did anything particularly interesting. A service will be held on Sunday from 4 to 6.</li>
<li>To follow up porcupines&#8217; passage, hell sentinels and torturous demonspawn no longer have spines. They weren&#8217;t scaled high enough to do anything against late-game players, and it wouldn&#8217;t be particularly fun if they had.</li>
<li>Batty monsters, such as bats or harpies, used to re-focus on their target once per player turn. That meant that players taking fast actions (swinging with short blades) would make them attack more often, and taking slow actions (using the Dark Maul) would make them attack less. No longer! Batty monsters now act of their own free will, unconstrained by players.</li>
<li>Again in celebration of boulder beetles&#8217; return, insect glyphs have been reorganized. Ants, beetles, and other non-flying insects now live on &#8216;B&#8217;, with &#8216;s&#8217; reserved for arachnids.</li>
<li>Monsters&#8217; spells&#8217; damage is now shown when examining them with &#8216;xv&#8217;.</li>
</ul>
</li>
<li>Player species changes:
<ul>
<li>New Demonspawn mutation: <strong>sharp scales</strong>, giving 1/2/3 AC and slaying.</li>
<li>New Demonspawn mutation: <strong>big brain</strong>, giving 2/4/6 int, along with wizardry (spell failure reduction) at level 3.</li>
<li>Demonspawns&#8217; thin skeletal structure mutation has its stealth bonus doubled.</li>
<li>At XL 7 and higher, <strong>Palentonga now curl defensively when hit</strong>, getting an AC bonus against subsequent hits before their next action. Useful when in big fights, against hydras, etc.</li>
<li>Wheals made a large number of improvements to the Palentonga charge targeter. Thanks, wheals!</li>
<li>Palentonga can now mutate poisonous tails.</li>
</ul>
</li>
<li>Item changes:
<ul>
<li>Potions of stabbing have been replaced with <strong>potions of attraction</strong>, which pull nearby monsters toward the player while active. See if you can come up with some fun ways to use this!</li>
<li>Potions of brilliance now halve spell MP costs while active.</li>
<li>When targeting the polymorph wand, you&#8217;ll now see a set of three random monsters that the wand can turn the target into. Less guessing!</li>
<li>Wand noise is now shown in the UI.</li>
<li>Many XP-charged evocable items have had their charges reduced, their recharge time increased, and their shop prices raised. They&#8217;re still extremely strong.</li>
<li>The condensation vane now has a lovely new set of tiles. Credit, as ever, to canofworms.</li>
<li>Broad axe delay has increased from 1.6 to 1.7.</li>
<li>The black knight&#8217;s barding is back! Now naga and palentonga can wear it.</li>
</ul>
</li>
<li>Miscellaneous changes:
<ul>
<li>There&#8217;s now a selection menu when x-ving to examine spaces with multiple interesting things. (Monsters, items, features&#8230;)</li>
<li>Gozag no longer gives a free potion petition on worship, to pull his power a bit back towards other gods.</li>
<li>Similarly, Nemelex Xobeh now starts gifting decks only at higher piety.</li>
<li>Now that you can see the amount of gold you get from a scroll of acquirement before you pick it, the huge variance that made for exciting gambles before just makes for really boring &#8216;take the massive pile of gold over the cool item&#8217; decisions. As such, gold acquirement now produces much smaller and more consistent amounts, to encourage more fun decisions.</li>
<li>Some Lair and Snake level layouts have been tweaked for increased fun.</li>
<li>Many new vaults have been added, including several exciting boulder beetle vaults and several transporter vaults.</li>
<li>Zot clock changes: the maximum &#8216;bank&#8217; is much larger (27k turns per branch, not 15k), all Hells share a single clock, and the clock is now deterministic and displayed numerically. If this means nothing to you, don&#8217;t worry about it. You can now enable always_show_zot if you really want to, though.</li>
<li>Vampiric Draining&#8217;s damage is no longer limited by the amount that it heals the player.</li>
</ul>
</li>
<li>Bugfixes:
<ul>
<li>Searing Ray no longer tracks invisible monsters.</li>
<li>Withered plants no longer wither. (Now they crumble.).</li>
<li>You can&#8217;t crash the game by trying to (P)ut a non-jewellery item on.</li>
<li>Dungeon Crawl will no longer claim that non-lifesaving gods will save lives.</li>
<li>Cunning players can no longer use Yredmenul&#8217;s Enslave Soul targeter to deduce which illusory clones are fake and which are real.</li>
<li>Donald now deigns to talk to followers of the Wu Jian Council.</li>
<li>While in the Slime Pits, <strong>you can no longer be shafted next to slimy walls and die instantly.</strong></li>
</ul>
</li>
</ul>
<p>Have a lovely weekend, all! Happy crawling.</p>


guid: http://crawl.develz.org/wordpress/?p=4857
title: Trunk Updates 13 December 2020
author: ebering
url: https://crawl.develz.org/wordpress/trunk-updates-13-december-2020
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p><em>Quiver which trunk update?</em></p>
<p>Ho ho ho hello! Santaeus has rolled up from the North Pole with a box of beastly trunk changes for all you good crawlers out there on the net.</p>
<p>The major highlight is the new quiver system, written primarily by advil. A quick overview:</p>
<blockquote><p>Any ranged attack, spell, ability, or evokable action can be quivered! This change change lets a wide range of things appear in the quiver slot for quick access via <code>f</code>iring, autofiring (<code>p</code> and shift-tab) and cycling (<code>(</code> and <code>)</code>); use <code>Q</code> to select an action. At the same time it introduces a number of UI improvements and new options.</p></blockquote>
<p>For the gory details, check out the updated documentation or read the <a href="https://github.com/crawl/crawl/pull/1581">pull request summary and discussion</a>.</p>
<p>Here&#8217;s what else is new in trunk:</p>
<ul>
<li><strong>Net traps</strong> are now permanent, but the nets always mulch (no more net farming!).</li>
<li><strong>Translocations miscasts</strong> now prevent movement altogether for a short time (like Tree form!).</li>
<li>Player <strong>spell damage</strong> and <strong>spell accuracy</strong> are now displayed in the UI in various easy to find places.</li>
<li><strong>Hailstorm</strong> now hits automatically (instead of having a very small chance to miss).</li>
<li>Several <strong>ghost vaults</strong> have been revamped to have better loot and better theme.</li>
<li><strong>Palentonga</strong> can now mutate longer rolls.</li>
<li><strong>Skills</strong> can now be trained at any time, without item or spell restrictions.</li>
<li>The revamped <strong>Swamp</strong> has seen various balance tweaks and adjustments after play testing, including:</li>
<ul>
<li><strong>Goliath frogs</strong> are now a bit slower,</li>
<li><strong>Will-o-the-wisps</strong> cast marshlight more often.</li>
</ul>
<li>The chance for the <strong>distortion brand</strong> to banish has been reduced.</li>
<li>The ancient pharoahs have asserted their right to title, <strong>greater mummies</strong> are now <strong>royal mummies</strong>.</li>
<li><strong>Evocable invisibility</strong> now drains skills, whether or not you successfully vanish!</li>
<li><strong>Magic Resistance</strong> has been renamed to <strong>Willpower</strong> (with the abbreviation changing from <strong>MR</strong> to <strong>Will</strong>).</li>
</ul>
<div>That&#8217;s all for now! This development cycle has seen many large changes land, so as always your play testing feedback and bug reports are welcome as we draw closer to the time of a new release. Until next time, happy crawling.</div>
<div style="text-align: right;"><em>Aiming: blog post<br />
Press: ? &#8211; help, Q &#8211; select action, (/) &#8211; cycle</em><em> </em></div>


guid: http://crawl.develz.org/wordpress/?p=4862
title: Dec 30 Trunk Update and 0.26 Tournament and Release Dates
author: gammafunk
url: https://crawl.develz.org/wordpress/dec-30-trunk-update-and-0-26-tournament-and-release-dates
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p><em>gammafunk gestures. gammafunk spews toxic sludge! The Trunk Update festers in the toxic bog!</em></p>
<p>Hello crawlers and welcome to the last trunk update of the 0.26 cycle. We&#8217;ve been under <strong>feature freeze</strong> since last Friday, so the remaining time before the <strong>0.26 Release and Tournament</strong> will be devoted to bug fixes, balance tweaks, and polish. If you&#8217;ve been holding out and playing 0.25 due to concerns about stability, know that all major features are fixed in trunk, and only reasonable balance changes and bug fixes will be a thing from here until release. We&#8217;ll be making the 0.26 beta branch soon, at which time servers will begin installing 0.26. So please play-test trunk or the 0.26 branch when that&#8217;s available on your favorite server to help us find bugs! The 0.26 release will be either Thursday, January 7th, or Friday, January 8th, depending on timing. Look out for an official release post with download links to come.</p>
<p>The <strong>0.26 tournament</strong> will run from <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210108T20">Friday Jan 8th, 8pm UTC</a> to <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210124T20">Sunday Jan 24th, 8pm UTC</a>. We don&#8217;t don&#8217;t yet have a rules page installed on this server, but expect that link in a forthcoming post. Check out the pages for the previous tournament to get a sense of the rules and presentation. We&#8217;ll try to announce any rules changes in that subsequent post, but as always the final rules page will cover everything you need to know.</p>
<p>Now, onto the Trunk Update! Note that this is not the full set of 0.26 changes, just the ones since our last trunk update. For the full and official set of 0.26 changes, check out the <a href="https://github.com/crawl/crawl/blob/master/crawl-ref/docs/changelog.txt">changelog</a>. Here are the changes since last time:</p>
<ul>
<li><strong>Monster draining attacks </strong>now temporarily reduce player max HP instead of skills. The UI cost of skill drain was high&#8230;beware! See a more detailed discussion in <a href="https://github.com/crawl/crawl/commit/4cbea82fa3b6">this commit</a>. Right now, the new drain effect is a bit weak, but I plan on making a couple balance changes to it during the freeze. Don&#8217;t worry, you&#8217;ll be afraid of bolts of drain, but it&#8217;ll be the <em>good </em>kind of afraid. Note that drain inflicted by the player is unchanged.</li>
<li><strong>Frozen Ramparts</strong> has its effect end upon movement. Yes, another nerf to your favorite new ice spell. That&#8217;s ok, kiting isn&#8217;t very fun, and now it works much better with Ozocubu&#8217;s Armour!</li>
<li><strong>Eringya&#8217;s Noxious Bog</strong> now places bogs on all squares within a range of 4 where there is no adjacent solid feature. A much-needed power increase for this spell, and hopefully an interesting positional trade-off. Kudos to hellmonk for the idea!</li>
<li><strong>Irradiate </strong>now checks monster AC but does slightly more damage. Ignoring AC was not part of the original PleasingFungus vision (which is all-powerful).</li>
<li>The<strong> cloak of Starlight </strong>gains *Dazzle, which sometimes dazzles foes like the Dazzling Flash spell upon successful dodge. It no longer has rElec and rC+. This cloak wins you so many style points when you go swish that monsters are dumbfounded!</li>
<li>The<strong> glaive of the Guard</strong> gains a spectral weapon brand in addition to electricity, and loses sInv and +Rage. That&#8217;s right, dual branding!</li>
<li>The<strong> sword of Power </strong>is now +5, vorpal, and sometimes fires a powerful but low-accuracy beam with chance based on the player&#8217;s current HP. Have you played the lesser-known indie game <em>The Legend of Zelda</em>? It&#8217;s a bit like the sword from that game.</li>
<li><strong>New Vaults! </strong>Sadly I don&#8217;t have enough time to go into detail about most individual vaults, but here&#8217;s a quick summary.</li>
<ul>
<li>A big set of<strong> winding woodlands </strong>vaults from nikheizen that feature a forest/woodland theme with many variations.</li>
<li>A new<strong> Swamp rune vault</strong> by nikheizen featuring a coven of 13 witches, mages, sorcerers, and wizards who want tell you all about demons.</li>
<li>A <strong>ghost vault</strong>, two greek myth themed <strong>Shoals vaults</strong>, four <strong>Swamp vaults</strong>, and 7 <strong>altar vaults</strong> from nikheizen.</li>
<li>A set of 11 <strong>arrival vaults, </strong>4 <strong>minivaults</strong>, and 2 <strong>altar vaults</strong> from amcnicky.</li>
<li>A <strong>Wu Jian altar vault </strong>from pdpol.</li>
</ul>
<li><strong>Damage and accuracy </strong>for spells and damage for wands are now shown in the UI. No more looking up info on various pages and bots just to get an idea of how much damage you can do! Well, for the most part. Some spells like Tornado refuse to cooperate.</li>
<li><strong>Boulder beetles </strong>now stop rolling when they can&#8217;t move towards their target (that&#8217;s you) nor will they start rolling if this is the case. I know it was funny, but beetles rolling in place behind that poor little kobold in front of them was too traumatizing for the kobold.</li>
<li><strong>Gauntlet</strong> monsters now generate awake, and those other than the minotaur can&#8217;t pickup loot. Now characters in Gauntlets don&#8217;t have weird incentives to explore the map in certain ways to not wake up monsters or to see the loot before monsters do.</li>
<li><strong>Temporary allies</strong> all dismiss when you leave the level. Apparently there were corner cases where they didn&#8217;t do that right away!</li>
</ul>
<p>That wraps it up for this set of trunk updates. Stay tuned for more tournament details, and until next time, happy crawling!</p>
<p><em>The Trunk Update looks as sick as possible!</em></p>


guid: http://crawl.develz.org/wordpress/?p=4884
title: 0.26 Tournament Page
author: gammafunk
url: https://crawl.develz.org/wordpress/0-26-tournament-page
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p>Hello fellow Bogchamps! I have a quick update about the 0.26 tournament. Please bookmark the <strong><a href="https://crawl.develz.org/tournament/0.26/">official 0.26 tournament page</a> </strong>and use this page to track your progress throughout the tournament. As a reminder, the tournament is scheduled from <strong><a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210108T20">20:00 UTC Friday 8 January</a></strong> through <strong><a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210124T20">20:00 UTC Sunday 24 January</a></strong>. The 0.26 release will likely happen beforehand on January 7th.</p>
<p>The 0.26 branch is available and already installed on CBR2, CKO, CUE, and CXC. For other servers, you&#8217;ll see the 0.26 game links appear on the server lobby at some point before the tournament starts. Once 0.26 is available on your server, you can follow the instructions on the tournament page to define/join clans via your 0.26 RC file. Clans can be formed and changed until one week into the tournament on <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210115T20">20:00 UTC Friday 15 January</a>. See the <a href="https://crawl.develz.org/tournament/0.26/#how-to-play">How to Play section</a> of the tournament page for details.</p>
<p>We&#8217;ve made some rule changes relative to the 0.25 tournament, so see the <a href="https://crawl.develz.org/tournament/0.26/#changes">Changes</a> section of the tournament page for details. Once the tournament has started, the leaderboard will display the current standings. You may see the leaderboard, player, and clan pages populated with test data from 0.26 games. Don&#8217;t worry, the test pages and data will be wiped before the tournament begins. If you&#8217;re looking for people to play with, you can find them in <a href="https://crawl.develz.org/tavern/viewtopic.php?p=345308#p345308">the Tavern</a>, <a href="https://www.reddit.com/r/DCSStourney">reddit</a>, or our IRC channel ##crawl on <a href="http://freenode.net/irc_servers.shtml">Freenode</a>. Many thanks to Napkin for again letting us host the tournament on CDO, and to ebering for updating the tournament scripts for our rule changes.</p>
<p>See you all on Swamp:4, paralyzed by a fenstrider witch!</p>


guid: http://crawl.develz.org/wordpress/?p=4892
title: 0.26 “Roll Around the Clock”
author: ebering
url: https://crawl.develz.org/wordpress/0-26-roll-around-the-clock
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p>We are pleased to announce the release of Dungeon Crawl: Stone Soup 0.26 “Roll Around the Clock”!</p>
<p>DCSS 0.26 features a new Palentonga species that has a rolling attack and gets bonus AC after getting hit, a new Delver background that starts the player on D:5 with a variety of escape tools, a reworked Swamp branch with new monsters and other layout changes, and a vastly reworked quiver system that lets spells, wands, and abilities be quivered, fired, and autotargeted. Additionally, food is gone, replaced with a Zot clock that prevents players from lingering too long in one branch. The Kobold species now has the Nightstalker 3 mutation, the Ghoul species has chunk eating replaced with heal-on-kill, and there are a bunch of reworked spells, monsters, and items (including nine overhauled unrandart weapons). As always, there are dozens of new deadly vaults, and many more changes small and large alike!</p>
<p>Download DCSS 0.26 <a title="download" href="http://crawl.develz.org/download.htm">here</a> or <a title="play online" href="http://crawl.develz.org/play.htm">play it online</a> on one of many servers across the world! Packages for Windows and OS X are available now. Linux debs are coming soon (<em>Update: The Linux debs are now available)</em>. See the full list of changes on the release page <a href="https://github.com/crawl/crawl/releases/tag/0.26.0">here</a>.</p>
<p>The tournament starts today in just a few hours at <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210108T20">20:00 UTC Friday 8 January</a> and runs through <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210124T20">20:00 UTC Sunday 24 January</a>, with all online 0.26 games counting towards your score. See the <a href="https://crawl.develz.org/tournament/0.26/">tournament page</a> for more details, including how to join or set-up a clan. Clans membership can be changed until <a href="https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210115T20">20:00 UTC Friday 15 January</a>.</p>
<p>Many thanks to all those who have <a title="credits" href="https://raw.githubusercontent.com/crawl/crawl/master/crawl-ref/CREDITS.txt">contributed</a> to DCSS over the years. A special thanks to the following community members who contributed to DCSS 0.26 and a warm welcome to the first-time contributors, listed in italics. (Let us know if someone is missing, this data is tricky to collect in bulk!)</p>
<blockquote><p>Aaron Golden, AdamPG, Alex Jurkiewicz, <em>aMcNicky</em>, <em>Anders Papitto</em>,<br />
Byrel Mitchell, <em>CrawlCycle</em>, Elliot Dronebarger, Emily, Eugene Abramchuk,<br />
<em>Gittourarmy</em>, Goratrix, <em>Heinrich Ody</em>, hellmonk, Implojin, Joe<br />
Childers, Kieron Dunbar, <em>Marcelo Henrique Cerri</em>, mgdelmonte,<br />
<em>Midn8</em>, Naruni, nikheizen, Nikolai Lavsky, <em>Paul Pollack</em>, Peter<br />
Gerlagh, Peter Hurst, <em>Petri Salminen</em>, Quipyowert2, <em>randomMesh</em>,<br />
<em>RJ Cunningham</em>, Roderick Schertler, <em>Cebolla Sunbeam</em>, Ryan McNeive,<br />
sdynet, <em>Sebastian Łużyński</em>, <em>Shummie</em>, Skrybe, syranez, Umer<br />
Shaikh, <em>ukdong99</em>, wjchen</p></blockquote>


guid: http://crawl.develz.org/wordpress/?p=4906
title: 0.26 Tournament Results
author: ebering
url: https://crawl.develz.org/wordpress/0-26-tournament-results
feedurl: https://crawl.develz.org/wordpress/feed
content_mime_type: text/html
content:
<p>The <a title="0.26 tournament" href="https://crawl.develz.org/tournament/0.26/overview.html">0.26 tournament</a> is over. On behalf of the Dev Team, many thanks to all the server admins, outside contributors, bug reporters, and to the many DCSS players who made the 0.26 release and tournament possible! For 16 days, players could compete for tournament points and banners by playing 0.26 games on the public servers. This was the second tournament held after the major revision of the tournament rules done for 0.26. There were changes based on previous input, and once again there will be discussion threads on the tavern and dcss subreddit to collect feedback on the tournament rules.</p>
<p>Congratulations to <a href="https://crawl.develz.org/tournament/0.26/players/yermak.html">Yermak</a>, perennial champion, for once again taking first place in the <a href="https://crawl.develz.org/tournament/0.26/all-players-ranks.html">individual competition</a>, with 95,456 points! Over the course of the tournament, Yermak won 47 games, including a streak of 16 games, 29 Nemelex&#8217; Choice combos, an XL 13 win, 3 species high scores, and 3 background high scores. In second place was <a href="https://crawl.develz.org/tournament/0.26/players/enegeticocto.html">EnegeticOcto</a> with 80,792 points and 38 wins, including 29 Nemelex&#8217; Choice combos and a 28 minute 17 second real-time win, the fastest of the tournament. Rounding out the top 3 was <a href="https://crawl.develz.org/tournament/0.26/players/booing.html">booing</a> with 78,169 points and 38 wins, including the first win of the tournament, 50 minutes and 27 seconds after the start, an XL 16 win, 3 species high scores, and 2 background high scores; they also collected every banner!</p>
<p>Our top three took several of the top spots in the individual category rankings, but they weren&#8217;t the only players on the leaderboard. <a href="https://crawl.develz.org/tournament/0.26/players/acrobat.html">Acrobat</a> set a high standard in <a href="https://crawl.develz.org/tournament/0.26/win-rate.html">Win Rate</a> and <a href="https://crawl.develz.org/tournament/0.26/streak-length.html">Streaking</a>, playing and winning 22 games over the course of the tournament, for the top streak and an adjusted win rate of 95.65%. Behind Acrobat in Win Rate was <a href="https://crawl.develz.org/tournament/0.26/players/sunset.html>Sunset</a>, who won 11 Barachi Venom Mages for an adjusted win rate of 91.67%. In the Streak category the runner-up streak was <a href=">DrMan</a> with a 20 game streak. For much of the tournament Acrobat and DrMan were engaged in a high-stakes back and forth at the top of this category. The <a href="https://crawl.develz.org/tournament/0.26/nemelex-choice.html">Nemelex&#8217; Choice</a> competition ended in an impressive 3-way tie, with Yermak, EnegeticOcto, and <a href="https://crawl.develz.org/tournament/0.26/players/shummie.html">Shummie</a>all winning 29 Nemelex combos for points over the course of the tournament. Yermak blew away the competition for <a href="https://crawl.develz.org/tournament/0.26/combo-high-scores.html">Combo High Scores</a>, with 329 points in the category; booing was the only other player to earn more than 200 points in the category, finishing second with a score of 226. The <a href="https://crawl.develz.org/tournament/0.26/best-high-score.html">Best High Score</a> of the tournament was the <a href="https://underhound.eu/crawl/morgue/iamserjio/morgue-iamserjio-20210117-115503.txt">72,401,124</a> point run by <a href="https://crawl.develz.org/tournament/0.26/players/iamserjio.html">iamserjio</a>. In the <a href="https://crawl.develz.org/tournament/0.26/lowest-turncount-win.html">Lowest Turncount Category</a>, <a href="https://crawl.develz.org/tournament/0.26/players/caminho.html">Caminho</a> won with 3 runes in <a href="https://underhound.eu/crawl/morgue/Caminho/morgue-Caminho-20210123-182710.txt">8,617 turns</a>, the 7th lowest turncount of all time! EnegeticOcto&#8217;s <a href="http://crawl.xtahua.com/crawl/morgue/EnegeticOcto/morgue-EnegeticOcto-20210111-114306.txt">28:17 win</a> was the <a href="https://crawl.develz.org/tournament/0.26/fastest-real-time-win.html">Fastest Realtime Win</a>, but Shummie was only 33 seconds behind with a <a href="https://crawl.kelbi.org/crawl/morgue/shummie/morgue-shummie-20210113-161431.txt">28:50 win</a>. Felids were the weapon of choice for the <a href="https://crawl.develz.org/tournament/0.26/lowest-xl-win.html">Low XL Win</a> category; Yermak&#8217;s <a href="http://crawl.xtahua.com/crawl/morgue/Yermak/morgue-Yermak-20210117-152839.txt">XL 13 cat</a> was followed by <a href="https://crawl.develz.org/tournament/0.26/players/curlybrace.html">CurlyBrace</a> who won at <a href="https://crawl.kelbi.org/crawl/morgue/CurlyBrace/morgue-CurlyBrace-20210118-055831.txt">XL 15</a>. The tournament was herladed by booing, who posted the <a href="https://crawl.develz.org/tournament/0.26/tournament-win-order.html">first win</a> 50 minutes after the tournament start. DrMan followed an hour later with the first <a href="https://crawl.develz.org/tournament/0.26/tournament-all-rune-win-order.html">all rune win</a>. The <a href="https://crawl.develz.org/tournament/0.26/ziggurat-diving.html">Ziggurat Dive</a> was an exciting back-and-forth for much of the tournament, until <a href="https://crawl.develz.org/tournament/0.26/players/sentinel.html">Sentinel</a> decided to clear a definitive 100 Ziggurats. Finally, in seeking the favor of the gods, booing was the only player to <a href="https://crawl.develz.org/tournament/0.26/banner-collection.html">collect every banner</a> over the course of the tournament.</p>
<p>The <a href="https://crawl.develz.org/tournament/0.26/teams.html">clan competition</a> was won by <a href="https://crawl.develz.org/tournament/0.26/clans/irradiated-dongers-enegeticocto.html">Irradiated Dongers</a> (captained by <a href="https://crawl.develz.org/tournament/0.26/players/enegeticocto.html">EnegeticOcto</a>) with 88,667 points. Following very close on their heels were <a href="https://crawl.develz.org/tournament/0.26/clans/gozag-or-stay-home-ge0ff.html">Gozag or Stay Home</a> (captained by <a href="https://crawl.develz.org/tournament/0.26/players/ge0ff.html">Ge0ff</a>) with 88,333 points. Rounding out the top three was <a href="https://crawl.develz.org/tournament/0.26/clans/teamsplat-shummie.html">TeamSplat</a> (captained by <a href="https://crawl.develz.org/tournament/0.26/players/shummie.html">Shummie</a>) with 72,456 points.</p>
<p>Most of the clan categories take the best score of the member, so I won&#8217;t recap those. The <a href="https://crawl.develz.org/tournament/0.26/clan-nemelex-choice.html">Nemelex&#8217; Choice</a> competition was a collective effort, and each of the three-way individual tie members were on different clans, which led to a very high scoring clan competition. Gozag or Stay Home came out on top, with other clan members contributing 30 more Nemelex Combo wins on top of Yermak&#8217;s 29, for a total of 59 clan nemelex points; following were Irradiated Dongers with 52 and TeamSplat with 50. The <a href="https://crawl.develz.org/tournament/0.26/clan-combo-high-scores.html">Combo High Scores</a> competition was similarly astronomical at the top; Gozag or Stay Home earned 533 points in the category, followed by Irradiated Dongers with 449, and <a href="https://crawl.develz.org/tournament/0.26/clans/make-beogh-great-again-sdynet.html">Make Beogh Great Again</a> with 392 points. These three clans were also the only three to <a href="https://crawl.develz.org/tournament/0.26/clan-banner-collection.html">collectively earn every banner</a>.</p>
<p>Thank you to everyone for playing. The rules were adjusted after 0.25 based on collected feedback, and once again there are feedback threads on the <a href="https://crawl.develz.org/tavern/viewtopic.php?f=23&amp;t=27054">tavern</a> and <a href="https://www.reddit.com/r/dcss/comments/l62hdd/026_tournament_results_and_feedback_post/?">reddit</a>.</p>
<p>To close out, here are some overall tournament statistics (with the 0.25 numbers in brackets for comparison). 2,760 players started a game at some point during the tournament; of those 2,333 completed a game in a non-boring way. This definition of “player” (players who completed a non-boring game) is used for subsequent calculations and was used for the 0.25 statistics.</p>
<ul>
<li>Players: 2,333 (0.25: 2,772)</li>
<li>Total player time: 29,392 hrs (0.25: 34,433 hrs)</li>
<li>Average player time: 12.6 hrs (0.25: 12.4 hrs)</li>
<li>Games played: 54,701 (0.25: 62,125)</li>
<li>Players who got a rune: 886 (0.25: 1010)</li>
<li>476 winners and 1767 wins (0.25: 548 winners and 1871 wins)</li>
<li>Win rate: 3.23% (0.25: 3.01%)</li>
<li>Proportion of players using WebTiles: 96% (0.25: 95.9%)</li>
</ul>
<p>The 0.27 season has begun, and it again looks to be an active trunk season. Watch this space for further information as the 0.27 trunk update posts begin to flow. Until next time, happy crawling!</p>


