# rss_item

guid: https://andrewkelley.me/post/troubleshooting-zig-regression-apitrace.html
title: Troubleshooting a Zig Regression with apitrace
url: https://andrewkelley.me/post/troubleshooting-zig-regression-apitrace.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|goto|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>Troubleshooting a Zig Regression with apitrace</h1>
<p>
The past three months I have spent rewriting <a href="http://ziglang.org/">Zig</a> internals.
</p>
<p>
Previously, the compiler looked like this:
</p>
<p>
Source → Tokenize → Abstract Syntax Tree → Semantic Analysis → LLVM Codegen → Binary
</p>
<p>
Now the compiler looks like this:
</p>
<p>
Source → Tokenize → Abstract Syntax Tree → Intermediate Representation Code →
Evaluation and Analysis → Intermediate Representation Code → LLVM Codegen → Binary
</p>
<p>
This was a significant amount of work:
</p>
<pre>
92 files changed, 22307 insertions(+), 15357 deletions(-)
</pre>
<p>
It took a while to get all 361 tests passing before I could merge back into master.
</p>
<p>
Part of my testing process is making sure this
<a href="https://github.com/andrewrk/tetris">Tetris game</a>
continues to work. Here's a screenshot of it working, from master branch:
</p>
<img src="http://superjoe.s3.amazonaws.com/blog-files/troubleshooting-zig-regression-apitrace/tetris-working.png">
<p>
Unfortunately, after my changes to Zig, the game looked like this:
</p>
<img src="http://superjoe.s3.amazonaws.com/blog-files/troubleshooting-zig-regression-apitrace/tetris-not-working.png">
<p>
So, I ran both versions of the game with <a href="http://apitrace.github.io/">apitrace</a>.
This resulted in a handy diff:
</p>
<img src="http://superjoe.s3.amazonaws.com/blog-files/troubleshooting-zig-regression-apitrace/apitrace-diff.png">
<p>
Here, it looks like both programs are issuing the same OpenGL commands except for different values to
<code>glUniform4fv</code>. Aha! Let's go see what's going on there.
</p>
<p>
After investigating this, it turned out that the <code>glUniform4fv</code> was simply for the piece color
and since the game uses a random number for each piece, the two instances of the game started with different
pieces.
</p>
<p>
So, I made a small change to the Random Number Generator code...
</p>
<p>
Before:
</p>
<pre>
<code class="language-zig">fn getRandomSeed() -&gt; %u32 {
    var seed : u32 = undefined;
    const seed_bytes = (&amp;u8)(&amp;seed)[0...4];
    %return std.os.getRandomBytes(seed_bytes);
    return seed;
 }</code>
</pre>
<p>
After:
</p>
<pre>
<code class="language-zig">fn getRandomSeed() -&gt; %u32 {
    return 4;
 }</code>
</pre>
<p>
After
<a href="http://xkcd.com/221/">this change</a>,
the <code>glUniform4fv</code> commands were sending the same data. Therefore, the difference
<strong>must</strong> be in the "data blob" parameters sent in initialization.
</p>
<p>
This led me to scrutinize this code:
</p>
<pre>
<code class="language-zig">const rect_2d_vertexes = [][3]c.GLfloat {
    []c.GLfloat{0.0, 0.0, 0.0},
    []c.GLfloat{0.0, 1.0, 0.0},
    []c.GLfloat{1.0, 0.0, 0.0},
    []c.GLfloat{1.0, 1.0, 0.0},
};
c.glGenBuffers(1, &amp;sg.rect_2d_vertex_buffer);
c.glBindBuffer(c.GL_ARRAY_BUFFER, sg.rect_2d_vertex_buffer);
c.glBufferData(c.GL_ARRAY_BUFFER, 4 * 3 * @sizeOf(c.GLfloat), (&amp;c_void)(&amp;rect_2d_vertexes[0][0]), c.GL_STATIC_DRAW);</code>
</pre>
<p>
I discovered the problem was <code>&amp;rect_2d_vertexes[0][0]</code>.
The compiler noticed that <code>rect_2d_vertexes</code> was a compile-time constant and therefore
generated the 2D array data structure as static data. It therefore evaluated <code>&amp;rect_2d_vertexes[0][0]</code>
as a compile-time known expression as well.
</p>
<p>
The problem was that each element in the <code>rect_2d_vertexes</code> referenced another array.
The compile-time constant generation code emitted an independent array for the inner arrays, whereas
we are expecting the pointer to point to a 2D array that contains all the data contiguously.
</p>
<p>
So I updated the data structure of constant arrays to refer to their parents, added a test case to cover the change,
and now the tetris game works again. Huzzah!
</p>


guid: https://andrewkelley.me/post/zig-programming-language-blurs-line-compile-time-run-time.html
title: Zig Programming Language Blurs the Line Between Compile-Time and Run-Time
url: https://andrewkelley.me/post/zig-programming-language-blurs-line-compile-time-run-time.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|goto|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime|or|and)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
  Prism.languages['rust'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|const|var|pub|struct|enum|break|return|continue|if|else|match|while|for|true|false)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|str|static)\b/,
  });
</script>
<h1>Zig Programming Language Blurs the Line Between Compile-Time and Run-Time</h1>
<p>
Zig places importance on the concept of whether an expression is known at compile-time.
There are a few different places this concept is used, and these building blocks are used
to keep the language small, readable, and powerful.
</p>
<ul>
  <li><a href="#introducing-compile-time-concept">Introducing the Compile-Time Concept</a></li>
  <ul>
    <li><a href="#compile-time-parameters">Compile-time parameters</a></li>
    <li><a href="#compile-time-variables">Compile-time variables</a></li>
    <li><a href="#compile-time-expressions">Compile-time expressions</a></li>
  </ul>
  <li><a href="#generic-data-structures">Generic Data Structures</a></li>
  <li><a href="#case-study-printf">Case Study: printf in C, Rust, and Zig</a></li>
  <li><a href="#conclusion">Conclusion</a></li>
</ul>
<h3 id="introducing-compile-time-concept">Introducing the Compile-Time Concept</h3>
<h4 id="compile-time-parameters">Compile-Time Parameters</h4>
<p>
Compile-time parameters is how Zig implements generics. It is compile-time duck typing
and it works mostly the same way that C++ template parameters work. Example:
</p>
<pre>
<code class="language-zig">fn max(comptime T: type, a: T, b: T) -&gt; T {
    if (a &gt; b) a else b
}
fn gimmeTheBiggerFloat(a: f32, b: f32) -&gt; f32 {
    max(f32, a, b)
}
fn gimmeTheBiggerInteger(a: u64, b: u64) -&gt; u64 {
    max(u64, a, b)
}</code>
</pre>
<p>
In Zig, types are first-class citizens. They can be assigned to variables, passed as parameters to functions,
and returned from functions. However, they can only be used in expressions which are known at <em>compile-time</em>,
which is why the parameter <code>T</code> in the above snippet must be marked with <code>comptime</code>.
</p>
<p>
A <code>comptime</code> parameter means that:
</p>
<ul>
  <li>At the callsite, the value must be known at compile-time, or it is a compile error.</li>
  <li>In the function definition, the value is known at compile-time.</li>
</ul>
<p>
</p>
<p>
For example, if we were to introduce another function to the above snippet:
</p>
<pre>
<code class="language-zig">fn max(comptime T: type, a: T, b: T) -&gt; T {
    if (a &gt; b) a else b
}
fn letsTryToPassARuntimeType(condition: bool) {
    const result = max(
        if (condition) f32 else u64,
        1234,
        5678);
}</code>
</pre>
<p>
Then we get this result from the compiler:
</p>
<pre>
<code>./test.zig:6:9: error: unable to evaluate constant expression
        if (condition) f32 else u64,
        ^</code>
</pre>
<p>
This is an error because the programmer attempted to pass a value only known at run-time
to a function which expects a value known at compile-time.
</p>
<p>
Another way to get an error is if we pass a type that violates the type checker when the
function is analyzed. This is what it means to have <em>compile-time duck typing</em>.
</p>
<p>
For example:
</p>
<pre>
<code class="language-zig">fn max(comptime T: type, a: T, b: T) -&gt; T {
    if (a &gt; b) a else b
}
fn letsTryToCompareBools(a: bool, b: bool) -&gt; bool {
    max(bool, a, b)
}</code>
</pre>
<p>
The code produces this error message:
</p>
<pre>
<code>./test.zig:2:11: error: operator not allowed for type 'bool'
    if (a &gt; b) a else b
          ^
./test.zig:5:8: note: called from here
    max(bool, a, b)
       ^</code>
</pre>
<p>
On the flip side, inside the function definition with the <code>comptime</code> parameter, the
value is known at compile-time. This means that we actually could make this work for the bool type
if we wanted to:
</p>
<pre>
<code class="language-zig">fn max(comptime T: type, a: T, b: T) -&gt; T {
    if (T == bool) {
        return a or b;
    } else if (a &gt; b) {
        return a;
    } else {
        return b;
    }
}
fn letsTryToCompareBools(a: bool, b: bool) -&gt; bool {
    max(bool, a, b)
}</code>
</pre>
<p>
This works because Zig implicitly inlines <code>if</code> expressions when the condition
is known at compile-time, and the compiler guarantees that it will skip analysis of
the branch not taken.
</p>
<p>
This means that the actual function generated for <code>max</code> in this situation looks like
this:
</p>
<pre>
<code class="language-zig">fn max(a: bool, b: bool) -&gt; bool {
    return a or b;
}</code>
</pre>
<p>
All the code that dealt with compile-time known values is eliminated and we are left with only
the necessary run-time code to accomplish the task.
</p>
<p>
This works the same way for <code>switch</code> expressions - they are implicitly inlined
when the target expression is compile-time known.
</p>
<h4 id="compile-time-variables">Compile-Time Variables</h4>
<p>
In Zig, the programmer can label variables as <code>comptime</code>. This guarantees to the compiler
that every load and store of the variable is performed at compile-time. Any violation of this results in a
compile error.
</p>
<p>
This combined with the fact that we can <code>inline</code> loops allows us to write
a function which is partially evaluated at compile-time and partially at run-time.
</p>
<p>
For example:
</p>
<pre>
<code class="language-zig">const CmdFn = struct {
    name: []const u8,
    func: fn(i32) -&gt; i32,
};

const cmd_fns = []CmdFn{
    CmdFn {.name = "one", .func = one},
    CmdFn {.name = "two", .func = two},
    CmdFn {.name = "three", .func = three},
};
fn one(value: i32) -&gt; i32 { value + 1 }
fn two(value: i32) -&gt; i32 { value + 2 }
fn three(value: i32) -&gt; i32 { value + 3 }

fn performFn(comptime prefix_char: u8, start_value: i32) -&gt; i32 {
    var result: i32 = start_value;
    comptime var i = 0;
    inline while (i &lt; cmd_fns.len) : (i += 1) {
        if (cmd_fns[i].name[0] == prefix_char) {
            result = cmd_fns[i].func(result);
        }
    }
    return result;
}

fn testPerformFn() {
    @setFnTest(this);

    assert(performFn('t', 1) == 6);
    assert(performFn('o', 0) == 1);
    assert(performFn('w', 99) == 99);
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}</code>
</pre>
<p>
This example is a bit contrived, because the compile-time evaluation component is unnecessary;
this code would work fine if it was all done at run-time. But it does end up generating
different code. In this example, the function <code>performFn</code> is generated three different times,
for the different values of <code>prefix_char</code> provided:
</p>
<pre>
<code class="language-zig">// From the line:
// assert(performFn('t', 1) == 6);
fn performFn(start_value: i32) -&gt; i32 {
    var result: i32 = start_value;
    result = two(result);
    result = three(result);
    return result;
}

// From the line:
// assert(performFn('o', 0) == 1);
fn performFn(start_value: i32) -&gt; i32 {
    var result: i32 = start_value;
    result = one(result);
    return result;
}

// From the line:
// assert(performFn('w', 99) == 99);
fn performFn(start_value: i32) -&gt; i32 {
    var result: i32 = start_value;
    return result;
}</code>
</pre>
<p>
Note that this happens even in a debug build; in a release build these generated functions still
pass through rigorous LLVM optimizations. The important thing to note, however, is not that this
is a way to write more optimized code, but that it is a way to make sure that what <em>should</em> happen
at compile-time, <em>does</em> happen at compile-time. This catches more errors and as demonstrated
later in this article, allows expressiveness that in other languages requires using macros,
generated code, or a preprocessor to accomplish.
</p>
<h4 id="compile-time-expressions">Compile-Time Expressions</h4>
<p>
In Zig, it matters whether a given expression is known at compile-time or run-time. A programmer can
use a <code>comptime</code> expression to guarantee that the expression will be evaluated at compile-time.
If this cannot be accomplished, the compiler will emit an error. For example:
</p>
<pre>
<code class="language-zig">extern fn exit() -&gt; unreachable;

fn foo() {
    comptime {
        exit();
    }
}</code>
</pre>
<pre>
<code>./test.zig:5:9: error: unable to evaluate constant expression
        exit();
        ^</code>
</pre>
<p>
It doesn't make sense that a program could call <code>exit()</code> (or any other external function)
at compile-time, so this is a compile error. However, a <code>comptime</code> expression does much
more than sometimes cause a compile error.
</p>
<p>
Within a <code>comptime</code> expression:
</p>
<ul>
  <li>All variables are <code>comptime</code> variables.</li>
  <li>All <code>if</code>, <code>while</code>, <code>for</code>, <code>switch</code>, and <code>goto</code>
    expressions are evaluated at compile-time, or emit a compile error if this is not possible.</li>
  <li>All function calls cause the compiler to interpret the function at compile-time, emitting a
    compile error if the function tries to do something that has global run-time side effects.</li>
</ul>
<p>
This means that a programmer can create a function which is called both at compile-time and run-time, with
no modification to the function required.
</p>
<p>
Let's look at an example:
</p>
<pre>
<code class="language-zig">fn fibonacci(index: u32) -&gt; u32 {
    if (index &lt; 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    // test fibonacci at run-time
    assert(fibonacci(7) == 13);

    // test fibonacci at compile-time
    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}</code>
</pre>
<pre>
<code>$ zig test test.zig
Test 1/1 testFibonacci...OK</code>
</pre>
<p>
Imagine if we had forgotten the base case of the recursive function and tried to run the tests:
</p>
<pre>
<code class="language-zig">fn fibonacci(index: u32) -&gt; u32 {
    //if (index &lt; 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}</code>
</pre>
<pre>
<code>$ zig test test.zig
./test.zig:3:28: error: operation caused overflow
    return fibonacci(index - 1) + fibonacci(index - 2);
                           ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:14:25: note: called from here
        assert(fibonacci(7) == 13);
                        ^</code>
</pre>
<p>
The compiler produces an error which is a stack trace from trying to evaluate the
function at compile-time.
</p>
<p>
Luckily, we used an unsigned integer, and so when we tried to subtract 1 from 0, it triggered
undefined behavior, which is always a compile error if the compiler knows it happened.
But what would have happened if we used a signed integer?
</p>
<pre>
<code class="language-zig">fn fibonacci(index: i32) -&gt; i32 {
    //if (index &lt; 2) return index;
    return fibonacci(index - 1) + fibonacci(index - 2);
}

fn testFibonacci() {
    @setFnTest(this);

    comptime {
        assert(fibonacci(7) == 13);
    }
}

fn assert(ok: bool) {
    if (!ok) unreachable;
}</code>
</pre>
<pre>
<code>./test.zig:3:21: error: evaluation exceeded 1000 backwards branches
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^
./test.zig:3:21: note: called from here
    return fibonacci(index - 1) + fibonacci(index - 2);
                    ^</code>
</pre>
<p>
The compiler noticed that evaluating this function at compile-time took a long time,
and thus emitted a compile error and gave up. If the programmer wants to increase
the budget for compile-time computation, they can use a built-in function called
<code>@setEvalBranchQuota</code> to change the default number 1000 to something else.
</p>
<p>
What if we fix the base case, but put the wrong value in the <code>assert</code> line?
</p>
<pre>
<code class="language-zig">comptime {
    assert(fibonacci(7) == 99999);
}</code>
</pre>
<pre>
<code>./test.zig:15:14: error: unable to evaluate constant expression
    if (!ok) unreachable;
             ^
./test.zig:10:15: note: called from here
        assert(fibonacci(7) == 99999);
              ^</code>
</pre>
<p>
What happened is Zig started interpreting the <code>assert</code> function with the
parameter <code>ok</code> set to <code>false</code>. When the interpreter hit
<code>unreachable</code> it emitted a compile error, because reaching unreachable
code is undefined behavior, and undefined behavior causes a compile error if it is detected
at compile-time.
</p>

<p>
In the global scope (outside of any function), all expressions are implicitly
<code>comptime</code> expressions. This means that we can use functions to
initialize complex static data. For example:
</p>
<pre>
<code class="language-zig">const first_25_primes = firstNPrimes(25);
const sum_of_first_25_primes = sum(first_25_primes);

fn firstNPrimes(comptime n: usize) -&gt; [n]i32 {
    var prime_list: [n]i32 = undefined;
    var next_index: usize = 0;
    var test_number: i32 = 2;
    while (next_index &lt; prime_list.len) : (test_number += 1) {
        var test_prime_index: usize = 0;
        var is_prime = true;
        while (test_prime_index &lt; next_index) : (test_prime_index += 1) {
            if (test_number % prime_list[test_prime_index] == 0) {
                is_prime = false;
                break;
            }
        }
        if (is_prime) {
            prime_list[next_index] = test_number;
            next_index += 1;
        }
    }
    return prime_list;
}

fn sum(numbers: []i32) -&gt; i32 {
    var result: i32 = 0;
    for (numbers) |x| {
        result += x;
    }
    return result;
}</code>
</pre>
<p>
When we compile this program, Zig generates the constants
with the answer pre-computed. Here are the lines from the generated LLVM IR:
</p>
<pre>
<code>@0 = internal unnamed_addr constant [25 x i32] [i32 2, i32 3, i32 5, i32 7, i32 11, i32 13, i32 17, i32 19, i32 23, i32 29, i32 31, i32 37, i32 41, i32 43, i32 47, i32 53, i32 59, i32 61, i32 67, i32 71, i32 73, i32 79, i32 83, i32 89, i32 97]
@1 = internal unnamed_addr constant i32 1060</code>
</pre>
<p>
Note that we did not have to do anything special with the syntax of these functions. For example,
we could call the <code>sum</code> function as is with a slice of numbers whose length and values were
only known at run-time.
</p>
<h3 id="generic-data-structures">Generic Data Structures</h3>
<p>
Zig uses these capabilities to implement generic data structures without introducing any
special-case syntax. If you followed along so far, you may already know how to create a
generic data structure.
</p>
<p>
Here is an example of a generic <code>List</code> data structure, that we will instantiate with
the type <code>i32</code>. Whereas in C++ or Rust we would refer to the instantiated type as
<code>List&lt;i32&gt;</code>, in Zig we refer to the type as <code>List(i32)</code>.
</p>
<pre>
<code class="language-zig">fn List(comptime T: type) -&gt; type {
    struct {
        items: []T,
        len: usize,
    }
}</code>
</pre>
<p>
That's it. It's a function that returns an anonymous <code>struct</code>. For the purposes of error messages
and debugging, Zig infers the name <code>"List(i32)"</code> from the function name and parameters invoked when creating
the anonymous struct.
</p>
<p>
To keep the language small and uniform, all aggregate types in Zig are anonymous. To give a type
a name, we assign it to a constant:
</p>
<pre>
<code class="language-zig">const Node = struct {
    next: &amp;Node,
    name: []u8,
};</code>
</pre>
<p>
This works because all top level declarations are order-independent, and as long as there isn't
an actual infinite regression, values can refer to themselves, directly or indirectly. In this case,
<code>Node</code> refers to itself as a pointer, which is not actually an infinite regression, so
it works fine.
</p>
<h3 id="case-study-printf">Case Study: printf in C, Rust, and Zig</h3>
<p>
Putting all of this together, let's compare how <code>printf</code> works in C, Rust, and Zig.
</p>
<p>
Here's how <code>printf</code> work in C:
</p>
<pre>
<code class="language-c">#include &lt;stdio.h&gt;

static const int a_number = 1234;
static const char * a_string = "foobar";

int main(int argc, char **argv) {
    fprintf(stderr, "here is a string: '%s' here is a number: %d\n", a_string, a_number);
    return 0;
}</code>
</pre>
<pre>
<code>here is a string: 'foobar' here is a number: 1234</code>
</pre>
<p>
What happens here is the <code>printf</code> implementation iterates over the format string
at run-time, and when it encounters a format specifier such as <code>%d</code> it looks at
the next argument which is passed in an architecture-specific way, interprets the argument as
a type depending on the format specifier, and attempts to print it. If the types are incorrect
or not enough arguments are passed, undefined behavior occurs - it may crash, print garbage
data, or access invalid memory.
</p>
<p>
Luckily, the compiler defines an attribute that you can use like this:
</p>
<pre>
<code class="language-c">__attribute__ ((format (printf, x, y)));</code>
</pre>
<p>
Where x and y are the 1-based indexes of the argument parameters that correspond to the
format string and the first var args parameter, respectively.
</p>
<p>
This attribute adds type checking to the function it decorates, to prevent the above problems,
and the <code>printf</code> function from <code>stdio.h</code> has this attribute on it, so these
problems are solved.
</p>
<p>
But what if you want to invent your own format string syntax and have the compiler check
it for you?
</p>
<p>
You can't.
</p>
<p>
That's how it works in C. It is hard-coded into the compiler. If you wanted to write your own
format string printing code and have it checked by the compiler, you would have to use the
preprocessor or metaprogramming - generate C code as output from some other code.
</p>
<p>
Zig is a programming language which is intended to replace C. We can do better than this.
</p>
<p>
Here's the equivalent program in Zig:
</p>

<pre>
<code class="language-zig">const io = @import("std").io;

const a_number: i32 = 1234;
const a_string = "foobar";

pub fn main(args: [][]u8) -&gt; %void {
    %%io.stderr.printf("here is a string: '{}' here is a number: {}\n", a_string, a_number);
}</code>
</pre>
<pre>
<code>here is a string: 'foobar' here is a number: 1234</code>
</pre>

<p>
Let's crack open the implementation of this and see how it works:
</p>

<pre>
<code class="language-zig">/// Calls print and then flushes the buffer.
pub fn printf(self: &amp;OutStream, comptime format: []const u8, args: ...) -&gt; %void {
    const State = enum {
        Start,
        OpenBrace,
        CloseBrace,
    };

    comptime var start_index: usize = 0;
    comptime var state = State.Start;
    comptime var next_arg: usize = 0;

    inline for (format) |c, i| {
        switch (state) {
            State.Start =&gt; switch (c) {
                '{' =&gt; {
                    if (start_index &lt; i) %return self.write(format[start_index...i]);
                    state = State.OpenBrace;
                },
                '}' =&gt; {
                    if (start_index &lt; i) %return self.write(format[start_index...i]);
                    state = State.CloseBrace;
                },
                else =&gt; {},
            },
            State.OpenBrace =&gt; switch (c) {
                '{' =&gt; {
                    state = State.Start;
                    start_index = i;
                },
                '}' =&gt; {
                    %return self.printValue(args[next_arg]);
                    next_arg += 1;
                    state = State.Start;
                    start_index = i + 1;
                },
                else =&gt; @compileError("Unknown format character: " ++ c),
            },
            State.CloseBrace =&gt; switch (c) {
                '}' =&gt; {
                    state = State.Start;
                    start_index = i;
                },
                else =&gt; @compileError("Single '}' encountered in format string"),
            },
        }
    }
    comptime {
        if (args.len != next_arg) {
            @compileError("Unused arguments");
        }
        if (state != State.Start) {
            @compileError("Incomplete format string: " ++ format);
        }
    }
    if (start_index &lt; format.len) {
        %return self.write(format[start_index...format.len]);
    }
    %return self.flush();
}</code>
</pre>
<p>
This is a proof of concept implementation; it will gain more formatting capabilities before
Zig reaches its first release.
</p>
<p>
Note that this is not hard-coded into the Zig compiler; this userland code in the standard library.
</p>
<p>
When this function is analyzed from our example code above, Zig partially evaluates the function
and emits a function that actually looks like this:
</p>
<pre>
<code class="language-zig">pub fn printf(self: &amp;OutStream, arg0: i32, arg1: []const u8) -&gt; %void {
    %return self.write("here is a string: '");
    %return self.printValue(arg0);
    %return self.write("' here is a number: ");
    %return self.printValue(arg1);
    %return self.write("\n");
    %return self.flush();
}</code>
</pre>
<p>
<code>printValue</code> is a function that takes a parameter of any type, and does different things depending
on the type:
</p>
<pre>
<code class="language-zig">pub fn printValue(self: &amp;OutStream, value: var) -&gt; %void {
    const T = @typeOf(value);
    if (@isInteger(T)) {
        return self.printInt(T, value);
    } else if (@isFloat(T)) {
        return self.printFloat(T, value);
    } else if (@canImplicitCast([]const u8, value)) {
        const casted_value = ([]const u8)(value);
        return self.write(casted_value);
    } else {
        @compileError("Unable to print type '" ++ @typeName(T) ++ "'");
    }
}</code>
</pre>
<p>
And now, what happens if we give too many arguments to <code>printf</code>?
</p>
<pre>
<code class="language-zig">%%io.stdout.printf("here is a string: '{}' here is a number: {}\n",
        a_string, a_number, a_number);</code>
</pre>
<pre>
<code>.../std/io.zig:147:17: error: Unused arguments
                @compileError("Unused arguments");
                ^
./test.zig:7:23: note: called from here
    %%io.stdout.printf("here is a number: {} and here is a string: {}\n",
                      ^</code>
</pre>
<p>
Zig gives programmers the tools needed to protect themselves against their own mistakes.
</p>
<p>
Let's take a look at how <a href="https://www.rust-lang.org/en-US/">Rust</a> handles this
problem. Here's the equivalent program:
</p>
<pre>
<code class="language-rust">const A_NUMBER: i32 = 1234;
const A_STRING: &amp;'static str = "foobar";

fn main() {
    print!("here is a string: '{}' here is a number: {}\n",
        A_STRING, A_NUMBER);
}</code>
</pre>
<pre>
<code>here is a string: 'foobar' here is a number: 1234</code>
</pre>
<p>
<code>print!</code>, as evidenced by the exclamation point, is a macro. Here is the definition:
</p>
<pre>
<code class="language-rust">#[macro_export]
#[stable(feature = "rust1", since = "1.0.0")]
#[allow_internal_unstable]
macro_rules! print {
    ($($arg:tt)*) =&gt; ($crate::io::_print(format_args!($($arg)*)));
}
#[stable(feature = "rust1", since = "1.0.0")]
#[macro_export]
macro_rules! format_args { ($fmt:expr, $($args:tt)*) =&gt; ({
/* compiler built-in */
}) }</code>
</pre>
<p>
Rust accomplishes the syntax that one would want from a var args print implementation, but
it requires using a macro to do so.
</p>
<p>
Macros have some limitations. For example, in this case, if you move the format string to
a global variable, the Rust example can no longer compile:
</p>
<pre>
<code class="language-rust">const A_NUMBER: i32 = 1234;
const A_STRING: &amp;'static str = "foobar";
const FMT: &amp;'static str = "here is a string: '{}' here is a number: {}\n";

fn main() {
    print!(FMT, A_STRING, A_NUMBER);
}</code>
</pre>
<pre>
<code>error: format argument must be a string literal.
 --&gt; test.rs:6:12
  |
6 |     print!(FMT, A_STRING, A_NUMBER);
  |            ^^^</code>
</pre>
<p>
On the other hand, Zig doesn't care whether the format argument is a string literal,
only that it is a compile-time known value that is implicitly castable to a <code>[]const u8</code>:
</p>
<pre>
<code class="language-zig">const io = @import("std").io;

const a_number: i32 = 1234;
const a_string = "foobar";
const fmt = "here is a string: '{}' here is a number: {}\n";

pub fn main(args: [][]u8) -&gt; %void {
    %%io.stderr.printf(fmt, a_string, a_number);
}</code>
</pre>
<p>
This works fine.
</p>
<p>
A macro is a reasonable solution to this problem, but it comes at the cost of readability. From
<a href="https://doc.rust-lang.org/beta/book/macros.html">Rust's own documentation</a>:
</p>
<blockquote>
  <p>
The drawback is that macro-based code can be harder to understand, because fewer of the built-in rules apply. Like an ordinary function, a well-behaved macro can be used without understanding its implementation. However, it can be difficult to design a well-behaved macro! Additionally, compiler errors in macro code are harder to interpret, because they describe problems in the expanded code, not the source-level form that developers use.
  </p>
  <p>
These drawbacks make macros something of a "feature of last resort". That’s not to say that macros are bad; they are part of Rust because sometimes they’re needed for truly concise, well-abstracted code. Just keep this tradeoff in mind.
  </p>
</blockquote>
<p>
One of the goals of Zig is to avoid these drawbacks while still providing enough of the power that
macros provide in order to make them unnecessary.
</p>
<p>
There is another thing I noticed, and I hope someone from the Rust community can correct me if I'm wrong,
but it looks like Rust also special cased <code>format_args!</code> in the compiler by making it a built-in.
If my understanding is correct, this would make Zig stand out as the only language of the three mentioned here
which does not special case string formatting in the compiler and instead exposes enough power to accomplish this
task in userland.
</p>
<p>
But more importantly, it does so without introducing another language on top of Zig, such as
a macro language or a preprocessor language. It's Zig all the way down.
</p>

<h3 id="conclusion">Conclusion</h3>
<p>
Thank you for following along and checking out what I've been working on lately.
</p>
<p>
As always, I welcome discussion, criticism, and users. Please keep in mind that this is alpha software;
I am working toward a first beta release, but the project is not there yet.
</p>


guid: https://andrewkelley.me/post/zig-already-more-knowable-than-c.html
title: Zig: Already More Knowable Than C
url: https://andrewkelley.me/post/zig-already-more-knowable-than-c.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|goto|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>Zig: Already More Knowable Than C</h1>
<p>
There is a nifty article created back in 2015 that made its way onto Hacker News today:
<a href="http://kukuruku.co/hub/programming/i-do-not-know-c">I Do Not Know C</a>.
</p>
<p>
The author creates a delightful set of source code samples in which the reader is
intended to determine the correctness, and if correct, predict the output of
the provided code. If you have not done the exercises, I encourage you to take
a moment to do so.
</p>
<p>
What follows are some of the examples translated into <a href="http://ziglang.org/">Zig</a> for comparison.
The numbers correspond to the numbers from the original post linked above,
but the C code is embedded here for comparison.
</p>
<h2>1. Declaring the same variable twice</h2>
<pre>
<code class="language-c">int i;
int i = 10;</code>
</pre>
<p>
In Zig:
</p>
<pre>
<code class="language-zig">var i = undefined;
var i = 10;</code>
</pre>
<p>
Output:
</p>
<pre>./test.zig:2:1: error: redefinition of 'i'
var i = 10;
^
./test.zig:1:1: note: previous definition is here
var i = undefined;
^</pre>

<h2>2. Null pointer</h2>
<pre>
<code class="language-c">extern void bar(void);
void foo(int *x) {
    int y = *x;  /* (1) */
    if (!x) {    /* (2) */
        return;  /* (3) */
    }
    bar();
}</code>
</pre>
<p>
When this example is translated to Zig, you have to explicitly decide if the pointer
is nullable. For example, if you used a bare pointer in Zig:
</p>
<pre>
<code class="language-zig">extern fn bar();

export fn foo(x: &amp;c_int) {
    var y = *x;  // (1)
    if (x == null) {     // (2)
        return;    // (3)
    }
    bar();
}</code>
</pre>
<p>
Then it wouldn't even compile:
</p>
<pre>./test.zig:7:11: error: operator not allowed for type '?&amp;c_int'
    if (x == null) {     // (2)
          ^</pre>
<p>
I think this error message can be improved, but even so it prevents a possible
null related bug here.
</p>
<p>
If the code author makes the pointer nullable, then the natural way to port the
C code to Zig would be:
</p>
<pre>
<code class="language-zig">extern fn bar();

export fn foo(x: ?&amp;c_int) {
    var y = x ?? return;  // (1), (2), (3)
    bar();
}</code>
</pre>
<p>
This compiles to:
</p>
<pre>
<code>0000000000000000 &lt;foo&gt;:
   0:	48 85 ff             	test   %rdi,%rdi
   3:	74 05                	je     a &lt;foo+0xa&gt;
   5:	e9 00 00 00 00       	jmpq   a &lt;foo+0xa&gt;
   a:	c3                   	retq   </code>
</pre>
<p>
This does a null check, returns if null, and otherwise calls bar.
</p>
<p>
Perhaps a more faithful way to port the C code to Zig would be this:
</p>
<pre>
<code class="language-zig">extern fn bar();

fn foo(x: ?&amp;c_int) {
    var y = ??x;  // (1)
    if (x == null) {     // (2)
        return;    // (3)
    }
    bar();
}

pub fn main() -&gt; %void {
    foo(null);
}</code>
</pre>
<p>
The <code>??</code> operator unwraps the nullable value. It asserts
that the value is not null, and returns the value. If the value is null
then the behavior is undefined, just like in C.
</p>
<p>
However, in Zig, undefined behavior causes a crash in debug mode:
</p>
<pre>
<code>$ ./test
attempt to unwrap null
/home/andy/dev/zig/build/lib/zig/std/special/zigrt.zig:16:35: 0x0000000000203395 in ??? (test)
        @import("std").debug.panic("{}", message_ptr[0..message_len]);
                                  ^
/home/andy/tmp/test.zig:4:13: 0x000000000020a61d in ??? (test)
    var y = ??x;  // (1)
            ^
/home/andy/tmp/test.zig:12:8: 0x00000000002046fd in ??? (test)
    foo(null);
       ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:60:21: 0x0000000000203697 in ??? (test)
    return root.main();
                    ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203420 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:34:25: 0x0000000000203290 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted</code>
</pre>
<p>
This is a half-finished traceback implementation (lots of TODO items to
complete before the
<a href="https://github.com/andrewrk/zig/milestone/1">0.1.0 milestone</a>),
but the point is that Zig detected the undefined behavior and aborted.
</p>
<p>
In release mode, this example invokes undefined behavior just like in C.
To avoid this, programmers are expected to choose one of these options:
</p>
<ul>
  <li>Test code sufficiently in debug mode to catch undefined behavior abuse.</li>
  <li>Utilize the safe-release option which includes the runtime undefined behavior
  safety checks.</li>
</ul>
<h2>5. strlen</h2>
<pre>
<code class="language-c">int my_strlen(const char *x) {
    int res = 0;
    while(*x) {
        res++;
        x++;
    }
    return res;
}</code>
</pre>
<p>
In Zig, pointers generally point to single objects, while <em>slices</em> are used to
refer to ranges of memory. So in practice you wouldn't need a strlen function,
you would use <code>some_bytes.len</code>. But we can port this code over anyway:
</p>
<pre>
<code class="language-zig">export fn my_strlen(x: &amp;const u8) -&gt; c_int {
    var res: c_int = 0;
    while (x[res] != 0) {
        res += 1;
    }
    return res;
}</code>
</pre>
<p>
Here we must use pointer indexing because Zig does not support direct pointer
arithmetic.
</p>
<p>The compiler catches this problem:</p>
<pre>
<code>./test.zig:3:14: error: expected type 'usize', found 'c_int'
    while (x[res] != 0) {
             ^</code>
</pre>
<h2>6. Print string of bytes backwards</h2>
<pre>
<code class="language-c">#include &lt;stdio.h&gt;
#include &lt;string.h&gt;
int main() {
    const char *str = "hello";
    size_t length = strlen(str);
    size_t i;
    for(i = length - 1; i &gt;= 0; i--) {
        putchar(str[i]);
    }
    putchar('\n');
    return 0;
}</code>
</pre>
<p>
Ported to Zig:
</p>
<pre>
<code class="language-zig">const c = @cImport({
    @cInclude("stdio.h");
    @cInclude("string.h");
});

export fn main(argc: c_int, argv: &amp;&amp;u8) -&gt; c_int {
    const str = c"hello";
    const length: c.size_t = c.strlen(str);
    var i: c.size_t = length - 1;
    while (i &gt;= 0) : (i -= 1) {
        _ = c.putchar(str[i]);
    }
    _ = c.putchar('\n');
    return 0;
}</code>
</pre>
<p>It compiles fine but produces this output when run:</p>
<pre>
<code>integer overflow
test.zig:10:25: 0x000000000020346d in ??? (test)
    while (i &gt;= 0) : (i -= 1) {
                        ^
Aborted</code>
</pre>
<h2>8. Weirdo syntax</h2>
<pre>
<code class="language-c">#include &lt;stdio.h&gt;
int main() {
    int array[] = { 0, 1, 2 };
    printf("%d %d %d\n", 10, (5, array[1, 2]), 10);
}</code>
</pre>
<p>
There is no way to express this code in Zig. Good riddance.
</p>

<h2>9. Unsigned overflow</h2>
<pre>
<code class="language-c">unsigned int add(unsigned int a, unsigned int b) {
    return a + b;
}</code>
</pre>
<p>In Zig:</p>
<pre>
<code class="language-zig">const io = @import("std").io;

export fn add(a: c_uint, b: c_uint) -&gt; c_uint {
    return a + b;
}

pub fn main() -&gt; %void {
    %%io.stdout.printf("{}\n", add(@maxValue(c_uint), 1));
}</code>
</pre>
<p>Output:</p>
<pre>
<code>$ ./test
integer overflow
test.zig:4:14: 0x00000000002032b4 in ??? (test)
    return a + b;
             ^
test.zig:8:35: 0x0000000000204747 in ??? (test)
    %%io.stdout.printf("{}\n", add(@maxValue(c_uint), 1));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x00000000002036d7 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203460 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x00000000002032d0 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted</code>
</pre>
<p>
The <code>+</code> operator asserts that there will be no overflow.
If you want twos complement wraparound behavior, that is possible
with the <code>+%</code> operator instead:
</p>
<pre>
<code class="language-zig">export fn add(a: c_uint, b: c_uint) -&gt; c_uint {
    return a +% b;
}</code>
</pre>
<p>
Now the output is:
</p>
<pre>
<code>$ ./test 
0</code>
</pre>

<h2>10. Signed overflow</h2>
<pre>
<code class="language-c">int add(int a, int b) {
    return a + b;
}</code>
</pre>
<p>In C signed and unsigned integer overflow work differently. In Zig,
they work the same. <code>+</code> asserts that no overflow occurs,
and <code>+%</code> performs twos complement wraparound behavior.</p>

<h2>11. Negation overflow</h2>
<pre>
<code class="language-c">int neg(int a) {
    return -a;
}</code>
</pre>
<p>By now you can probably predict how this works in Zig.</p>
<pre>
<code class="language-zig">const io = @import("std").io;

export fn neg(a: c_int) -&gt; c_int {
    return -a;
}

pub fn main() -&gt; %void {
    %%io.stdout.printf("{}\n", neg(@minValue(c_int)));
}</code>
</pre>
<p>Output:</p>
<pre>
<code>$ ./test
integer overflow
test.zig:4:12: 0x00000000002032b0 in ??? (test)
    return -a;
           ^
test.zig:8:35: 0x0000000000204742 in ??? (test)
    %%io.stdout.printf("{}\n", neg(@minValue(c_int)));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x00000000002036d7 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203460 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x00000000002032d0 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted</code>
</pre>
<p>The <code>-%</code> wraparound variant of the negation operator works
here too:
<pre>
<code class="language-zig">export fn neg(a: c_int) -&gt; c_int {
    return -%a;
}</code>
</pre>
<p>Output:</p>
<pre>
<code>$ ./test 
-2147483648</code>
</pre>

<h2>12. Division overflow</h2>
<pre>
<code class="language-c">int div(int a, int b) {
    assert(b != 0);
    return a / b;
}</code>
</pre>
<p>Different operation, same deal.</p>
<pre>
<code class="language-zig">const io = @import("std").io;

fn div(a: i32, b: i32) -&gt; i32 {
    return a / b;
}

pub fn main() -&gt; %void {
    %%io.stdout.printf("{}\n", div(@minValue(i32), -1));
}</code>
</pre>
<p>First of all, Zig doesn't let us do this operation because
it's unclear whether we want floored division or truncated division:</p>
<pre><code>
test.zig:4:14: error: division with 'i32' and 'i32': signed integers must use @divTrunc, @divFloor, or @divExact
    return a / b;
             ^</code></pre>
<p>Some languages use truncation division (C) while others (Python) use floored division.
Zig makes the programmer choose explicitly.</p>
<pre><code class="language-zig">fn div(a: i32, b: i32) -&gt; i32 {
    return @divTrunc(a, b);
}</code></pre>
<p>Output:</p>
<pre><code>$ ./test
integer overflow
test.zig:4:12: 0x000000000020a683 in ??? (test)
    return @divTrunc(a, b);
           ^
test.zig:8:35: 0x0000000000204707 in ??? (test)
    %%io.stdout.printf("{}\n", div(@minValue(i32), -1));
                                  ^
lib/zig/std/special/bootstrap.zig:60:21: 0x0000000000203697 in ??? (test)
    return root.main();
                    ^
lib/zig/std/special/bootstrap.zig:47:13: 0x0000000000203420 in ??? (test)
    callMain(argc, argv, envp) %% std.os.posix.exit(1);
            ^
lib/zig/std/special/bootstrap.zig:34:25: 0x0000000000203290 in ??? (test)
    posixCallMainAndExit()
                        ^
Aborted</code></pre>
<p>
Notably, if you execute the division operation at compile time,
the overflow becomes a compile error (same for the other operations):
</p>
<pre>
<code class="language-zig">    %%io.stdout.printf("{}\n", comptime div(@minValue(i32), -1));</code>
</pre>
<p>Output:</p>
<pre>
<code>./test.zig:4:14: error: operation caused overflow
    return a / b;
             ^
./test.zig:8:44: note: called from here
    %%io.stdout.printf("{}\n", comptime div(@minValue(i32), -1));
                                           ^</code>
</pre>

<h2>Conclusion</h2>
<p>
Zig is on track to boot out C as the simple, straightforward way to write system code.
</p>


guid: https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html
title: A Better Way to Implement Bit Fields
url: https://andrewkelley.me/post/a-better-way-to-implement-bit-fields.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|packed|volatile|export|pub|noalias|inline|struct|enum|goto|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|i13|u1|u2|u3|u4|u5|u7|u12|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>A Better Way to Implement Bit-Fields</h1>
<p>
One of the main use cases for the <a href="https://ziglang.org/">Zig Programming Language</a>
is operating system development and embedded development. So, what better way to make sure the
language is suitable than to work on a project in this field?
</p>
<p>
This is why I am creating a
<a href="https://github.com/andrewrk/clashos">4-player arcade game that runs directly on the Raspberry Pi 3 hardware</a>.
The project has only just begun, but already it is revealing important fixes and features in Zig,
and I am updating the compiler to incorporate these things as I work.
</p>
<p>
The next thing I'm working on is adding a USB controller driver, so that I can use a gamepad to
test the game. I noticed when looking at
<a href="https://github.com/Chadderz121/csud/blob/e13b9355d043a9cdd384b335060f1bc0416df61e/include/hcd/dwc/designware20.h#L164">some reference code</a>
that there are large structs full of bit-fields, and these turn out to be extremely convenient:
</p>
<pre>
<code class="language-c">extern volatile struct CoreGlobalRegs {
	volatile struct {
		volatile bool sesreqscs : 1;
		volatile bool sesreq : 1;
		volatile bool vbvalidoven:1;
		volatile bool vbvalidovval:1;
		volatile bool avalidoven:1;
		volatile bool avalidovval:1;
		volatile bool bvalidoven:1;
		volatile bool bvalidovval:1;
		volatile bool hstnegscs:1;
		volatile bool hnpreq:1;
		volatile bool HostSetHnpEnable : 1;
		volatile bool devhnpen:1;
		volatile unsigned _reserved12_15:4;
		volatile bool conidsts:1;
		volatile unsigned dbnctime:1;
		volatile bool ASessionValid : 1;
		volatile bool BSessionValid : 1;
		volatile unsigned OtgVersion : 1;
		volatile unsigned _reserved21:1;
		volatile unsigned multvalidbc:5;
		volatile bool chirpen:1;
		volatile unsigned _reserved28_31:4;
	} __attribute__ ((__packed__)) OtgControl; // +0x0
	volatile struct {
		volatile unsigned _reserved0_1 : 2; // @0
		volatile bool SessionEndDetected : 1; // @2
		volatile unsigned _reserved3_7 : 5; // @3
		volatile bool SessionRequestSuccessStatusChange : 1; // @8
		volatile bool HostNegotiationSuccessStatusChange : 1; // @9
		volatile unsigned _reserved10_16 : 7; // @10
		volatile bool HostNegotiationDetected : 1; // @17
		volatile bool ADeviceTimeoutChange : 1; // @18
		volatile bool DebounceDone : 1; // @19
		volatile unsigned _reserved20_31 : 12; // @20
	} __attribute__ ((__packed__)) OtgInterrupt; // +0x4
// ...</code>
</pre>
<p>
I'll stop there, but this struct goes on for pages and pages. You can see that in C,
bit-fields have special syntax. The fields have a type like normal, and then an extra
colon and a number of bits.
</p>
<p>
Bit-fields in C have acquired a poor reputation in the programming community, for a few reasons:
</p>
<ul>
  <li><a href="https://www.flamingspork.com/blog/2014/11/14/c-bitfields-considered-harmful/">Poor performance due to loading more bytes than necessary</a></li>
  <li><a href="https://gcc.gnu.org/onlinedocs/gcc/Structures-unions-enumerations-and-bit-fields-implementation.html">Largely implementation-defined behavior</a></li>
  <li><a href="http://stackoverflow.com/a/4240989/432">Platform dependence</a></li>
  <li><a href="http://yarchive.net/comp/linux/bitfields.html">Linus Torvalds: bitfields make it harder to work with combinations of flags</a></li>
</ul>
<p>
Many experienced developers have given up on the usefulness of bit-fields and prefer
to manually wrangle their binary data.
</p>
<p>
Some of these problems I can address in Zig, by loading the minimal number of bytes necessary,
defining how bit-fields are laid out in memory, and ensuring that bit-fields work the same
or at least predictably on all platforms. Some of the problems are inherently tricky, such
as the question of how to deal with endianness when a bit-field has more than 8 bits and
crosses a byte boundary.
</p>
<p>
With these things in mind, I set out to implement bit-fields in Zig, and
now I am pleased to announce that work is complete. Zig has bit-fields, and it required
no syntax additions.
</p>
<p>
Zig takes advantage of the fact that types are first-class values at compile-time. There is a
built-in function which returns an integer type with the specified signness and bit count,
and it can be used to get access to uncommon integer types like this:
</p>
<pre>
<code class="language-zig">const i13 = @intType(true, 13);</code>
</pre>
<p>
These uncommon integer types work like normal integer types. Arithmetic, casting, and overflow
are generalized to work with any integer type.
</p>
<p>
These integer types are one component to the way bit-fields are implemented in Zig.
The other component takes advantage of the fact that Zig has 3 different
kinds of <code>struct</code> layouts:
</p>
<ul>
  <li>Default - compiler may re-arrange fields and insert padding</li>
  <li><code>extern</code> - compatible with the target environment C ABI</li>
  <li><code>packed</code> - fields in exact order specified, no padding</li>
</ul>
<p>
In a <code>packed</code> struct, the programmer is directly in charge of the memory layout
of the struct. Fields with integer types take up exactly as many bits as the integer type specifies.
If a field greater than 8 bits is byte-aligned, it is represented in memory with
the endianness of the host. If the field is not byte-aligned, it is represented in memory in
big-endian. Boolean values are represented as 1 bit in packed structs.
</p>
<p>
So that's it. To make a bit-field, you have a packed struct with fields that are
integers with the bit sizes you want.
</p>
<p>
For illustration, here is the above code translated into Zig:
</p>
<pre>
<code class="language-zig">const u1 = @intType(false, 1);
const u12 = @intType(false, 12);

const CoreGlobalRegs = packed struct {
    OtgControl: struct {
        sesreqscs: bool,
        sesreq: bool,
        vbvalidoven: bool,
        vbvalidovval: bool,
        avalidoven: bool,
        avalidovval: bool,
        bvalidoven: bool,
        bvalidovval: bool,
        hstnegscs: bool,
        hnpreq: bool,
        HostSetHnpEnable: bool,
        devhnpen: bool,
        _reserved12_15: u4,
        conidsts: bool,
        dbnctime: u1,
        ASessionValid: bool,
        BSessionValid: bool,
        OtgVersion: u1,
        _reserved21: u1,
        multvalidbc: u5,
        chirpen: bool,
        _reserved28_31: u4,
    }, // +0x0
    OtgInterrupt: packed struct {
        _reserved0_1: u2, // @0
        SessionEndDetected: bool, // @2
        _reserved3_7: u5, // @3
        SessionRequestSuccessStatusChange: bool, // @8
        HostNegotiationSuccessStatusChange: bool, // @9
        _reserved10_16: u7, // @10
        HostNegotiationDetected: bool, // @17
        ADeviceTimeoutChange: bool, // @18
        DebounceDone: bool, // @19
        _reserved20_31: u12, // @20
    }, // +0x4
// ...</code>
</pre>
<p>
By the way it's nice to know that the USB protocol has a bit to indicate "a valid oven".
I wonder when that is used.
</p>
<p>
You may notice that we do a bit of setup before declaring the struct by creating
these integer types. Currently, only integer types of size <code>8</code>, <code>16</code>,
<code>32</code>, and <code>64</code> are provided by the compiler globally.
But perhaps more integer types can be provided as primitive types, or perhaps there will
be a standard library file to import and get more integer types.
</p>
<p>
You may also notice that the <code>volatile</code> keyword is gone. This is a different
issue, but Zig handles volatile at the pointer level. So instead of putting the keyword
on every field, you make sure the pointer to the whole struct is volatile, and then
any loads and stores done via the pointer become volatile, and any pointers derived from
the volatile pointer are also volatile.
</p>
<p>
One point of comparison with C bit-fields. Take a look at this code:
</p>
<pre>
<code class="language-c">struct Foo {
    unsigned a : 3;
    unsigned b : 3;
    unsigned c : 2;
} __attribute__ ((__packed__));

struct Foo foo = {1, 2, 3};

unsigned f(void) {
    unsigned *ptr = &amp;foo.b;
    return *ptr;
}</code>
</pre>
<p>
Here we try to take the address of a bit-field, and clang doesn't like that idea so much:
</p>
<pre><code>test.c:10:12: error: address of bit-field requested
    return &amp;foo.b;
           ^~~~~~</code>
</pre>
<p>
Meanwhile, in Zig, this works just fine:
</p>
<pre>
<code class="language-zig">const Foo = packed struct {
    a: u3,
    b: u3,
    c: u2,
};

var foo = Foo {
    .a = 1,
    .b = 2,
    .c = 3,
};

fn f() u3 {
    const ptr = &amp;foo.b;
    return ptr.*;
}</code>
</pre>
<p>
The bit offset and length is carried in the type of the pointer, so you get an error if you
try to pass such a pointer to a function expecting a normal, byte-aligned value:
</p>
<pre>
<code class="language-zig">const BitField = packed struct {
    a: u3,
    b: u3,
    c: u2,
};

fn foo(bit_field: *const BitField) u3 {
    return bar(&amp;bit_field.b);
}

fn bar(x: *const u3) u3 {
    return x.*;
}</code>
</pre>
<p>
In this case the compiler catches the mistake:
</p>
<pre>
<code>./test.zig:8:26: error: expected type '*const u3', found '*:3:6 const u3'
    return bar(&amp;bit_field.b);
                         ^</code>
</pre>
<p>
There are bound to be some edge cases and bugs as I polish this feature, but I am
pleased that it turned out to integrate so cleanly into Zig's minimal design.
</p>


guid: https://andrewkelley.me/post/zig-december-2017-in-review.html
title: Zig: December 2017 in Review
url: https://andrewkelley.me/post/zig-december-2017-in-review.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>Zig: December 2017 in Review</h1>

<p>
I figured since I <a href="https://github.com/users/andrewrk/sponsorship">ask people to donate monthly</a>, 
I will start giving a monthly progress report to provide accountability.
</p>

<p>So here's everything that happened in <a href="http://ziglang.org/">Zig land</a>
in December 2017:</p>

<h2>enum tag types</h2>

<p>You can now specify the tag type for an enum (<a href="https://github.com/zig-lang/zig/issues/305">#305</a>):</p>

<pre><code class="language-zig">const Small2 = enum (u2) {
    One,
    Two,
};</code></pre>

<p>If you specify the tag type for an enum, you can put it in a packed struct:</p>

<pre><code class="language-zig">const A = enum (u3) {
    One,
    Two,
    Three,
    Four,
    One2,
    Two2,
    Three2,
    Four2,
};

const B = enum (u3) {
    One3,
    Two3,
    Three3,
    Four3,
    One23,
    Two23,
    Three23,
    Four23,
};

const C = enum (u2) {
    One4,
    Two4,
    Three4,
    Four4,
};

const BitFieldOfEnums = packed struct {
    a: A,
    b: B,
    c: C,
};

const bit_field_1 = BitFieldOfEnums {
    .a = A.Two,
    .b = B.Three3,
    .c = C.Four4,
};</code></pre>

<p>
You can no longer cast from an enum to an arbitrary integer. Instead you must
cast to the enum tag type and vice versa:
</p>

<pre><code class="language-zig">const Small2 = enum (u2) {
    One,
    Two,
};
test "casting enum to its tag type" {
    testCastEnumToTagType(Small2.Two);
}

fn testCastEnumToTagType(value: Small2) {
    assert(u2(value) == 1);
}</code></pre>

<h2>enum tag values</h2>
<p>
Now you can set the tag values of enums:
</p>

<pre><code class="language-zig">const MultipleChoice = enum(u32) {
    A = 20,
    B = 40,
    C = 60,
    D = 1000,
};</code></pre>

<h2>Complete enum and union overhaul</h2>
<p>
Related issue: <a href="https://github.com/zig-lang/zig/issues/618">#618</a>
</p>
<p>
Enums are now a simple mapping between a symbol and a number. They can
no longer contain payloads.
</p>

<p>
Unions have been upgraded and can now accept an enum as an argument:
</p>

<pre><code class="language-zig">const TheTag = enum {A, B, C};
const TheUnion = union(TheTag) { A: i32, B: i32, C: i32 };
test "union field access gives the enum values" {
    assert(TheUnion.A == TheTag.A);
    assert(TheUnion.B == TheTag.B);
    assert(TheUnion.C == TheTag.C);
}</code></pre>

<p>
If you want to auto-create an enum for a union, you can use the <code>enum</code>
keyword like this:
</p>

<pre><code class="language-zig">const TheUnion2 = union(enum) {
    Item1,
    Item2: i32,
};</code></pre>

<p>
You can switch on a union-enum just like you could previously with an
enum:
</p>

<pre><code class="language-zig">const SwitchProngWithVarEnum = union(enum) {
    One: i32,
    Two: f32,
    Meh: void,
};
fn switchProngWithVarFn(a: &amp;const SwitchProngWithVarEnum) {
    switch(*a) {
        SwitchProngWithVarEnum.One =&gt; |x| {
            assert(x == 13);
        },
        SwitchProngWithVarEnum.Two =&gt; |x| {
            assert(x == 13.0);
        },
        SwitchProngWithVarEnum.Meh =&gt; |x| {
            const v: void = x;
        },
    }
}</code></pre>

<p>
However, if you do not give an enum to a union, the tag value is not
visible to the programmer:
</p>

<pre><code class="language-zig">const Payload = union {
    A: i32,
    B: f64,
    C: bool,
};
export fn entry() {
    const a = Payload { .A = 1234 };
    foo(a);
}
fn foo(a: &amp;const Payload) {
    switch (*a) {
        Payload.A =&gt; {},
        else =&gt; unreachable,
    }
}</code></pre>

<pre><code>test.zig:11:13: error: switch on union which has no attached enum
    switch (*a) {
            ^
test.zig:1:17: note: consider 'union(enum)' here
const Payload = union {
                ^
test.zig:12:16: error: container 'Payload' has no member called 'A'
        Payload.A =&gt; {},
               ^</code></pre>

<p>
There is still debug safety though!
</p>

<pre><code class="language-zig">const Foo = union {
    float: f32,
    int: u32,
};

pub fn main() -&gt; %void {
    var f = Foo { .int = 42 };
    bar(&amp;f);
}

fn bar(f: &amp;Foo) {
    f.float = 12.34;
}
</code></pre>

<pre><code>access of inactive union field
lib/zig/std/special/panic.zig:12:35: 0x0000000000203674 in ??? (test)
        @import("std").debug.panic("{}", msg);
                                  ^
test.zig:12:6: 0x0000000000217bd7 in ??? (test)
    f.float = 12.34;
     ^
test.zig:8:8: 0x0000000000217b7c in ??? (test)
    bar(&amp;f);
       ^
Aborted</code></pre>

<p>
However, if you make an <code>extern union</code> to be compatible with C code,
there is no debug safety, just like a C union.
</p>

<p>
Other tidbits:
</p>

<ul>
  <li> <code>@enumTagName</code> is renamed to <a href="http://ziglang.org/documentation/master/#builtin-tagName">@tagName</a></li>
 <li> <code>@EnumTagType</code> is renamed to <a href="http://ziglang.org/documentation/master/#builtin-TagType">@TagType</a>, and it works on both enums and
   union-enums.</li>
 <li>There is no longer an <code>EnumTag</code> type</li>
 <li>It is now an error for enums and unions to have 0 fields. However
   you can still have a struct with 0 fields.</li>
 <li> union values can implicitly cast to enum values when the enum type is
   the tag type of the union and the union value tag is comptime known to
   have a void field type. likewise, enum values can implicitly cast to
   union values. See <a href="https://github.com/zig-lang/zig/issues/642">#642</a>.</li>
</ul>

<pre><code class="language-zig">test "cast tag type of union to union" {
    var x: Value2 = Letter2.B;
    assert(Letter2(x) == Letter2.B);
}
const Letter2 = enum { A, B, C };
const Value2 = union(Letter2) { A: i32, B, C, };

test "implicit cast union to its tag type" {
    var x: Value2 = Letter2.B;
    assert(x == Letter2.B);
    giveMeLetterB(x);
}
fn giveMeLetterB(x: Letter2) {
    assert(x == Value2.B);
}</code></pre>

<h2>Update LLD fork to 5.0.1rc2</h2>

<p>
We have a fork of LLD in the zig project because of several upstream issues, all
of which I have filed bugs for:
</p>

<ul>
  <li>LDD calls exit after successful link. Patch to fix sent upstream and accepted.
  </li>
  <li>LDD crashes on a linker script with empty sections. Fixed upstream.</li>
  <li>Buggy MACH-O code causes assertion failure in simple object. We have a
    hacky workaround for this bug in zig's fork of LLD, but the workaround is
    not good enough to send upstream. See
    <a href="https://github.com/zig-lang/zig/issues/662">#662</a> for more details.
  </li>
  <li>MACH-O: Bad ASM code generated for __stub_helpers section
    Thanks to Patricio V. for sending the fix upstream, which has been accepted.</li>
</ul>

  <p>
When LLVM 6.0.0 comes out, Zig will have to keep its fork because of the one
issue, but we can drop all the other patches since they have been accepted
upstream.
  </p>

  <h2>Self-hosted compiler progress</h2>

<p>
The self-hosted compiler effort has begun.
</p>

<p>
So far we have a tokenizer, and an incomplete parser and formatter.
The code uses no recursion and therefore has compile-time
known stack space usage. See <a href="https://github.com/zig-lang/zig/issues/157">#157</a>
</p>

<p>
The self-hosted compiler works on every supported platform, is built using
the zig build system, tested with <code>zig test</code>, links against LLVM,
and can import <strong>100%</strong> of the LLVM symbols from the LLVM
C-API .h files - <em>even the inline functions</em>.
</p>

<p>
There is one C++ file in Zig which uses the more powerful LLVM C++ API
(for example to create debug information) and exposes a C API. This file
is now shared between the C++ self-hosted compiler and the self-hosted compiler.
In stage1, we create a static library with this one file in it, and then use
that library in both the C++ compiler and the self-hosted compiler.
</p>

<h2>Higher level arg-parsing API</h2>

<p>
It's really a shame that Windows command line parsing requires you to
allocate memory. This means that to have a cross-platform API for command
line arguments, even though in POSIX it can never fail, we have to handle
the possibility because of Windows. This lead to a command line args
API like this:
</p>

<pre><code class="language-zig">pub fn main() -&gt; %void {
    var arg_it = os.args();
    // skip my own exe name
    _ = arg_it.skip();
    while (arg_it.next(allocator)) |err_or_arg| {
        const arg = %return err_or_arg;
        defer allocator.free(arg);
        // use the arg...
    }
}</code></pre>

<p>
Yikes, a bit cumbersome. I added a higher level API. Now you can call
<code>std.os.argsAlloc</code> and get a <code>%[]const []u8</code>, and you just have to call
<code>std.os.argsFree</code> when you're done with it.
</p>

<pre><code class="language-zig">pub fn main() -&gt; %void {
    const allocator = std.heap.c_allocator;

    const args = %return os.argsAlloc(allocator);
    defer os.argsFree(allocator, args);

    var arg_i: usize = 1;
    while (arg_i &lt; args.len) : (arg_i += 1) {
        const arg = args[arg_i];
        // do something with arg...
    }
}</code></pre>

<p>Better! Single point of failure.</p>

<p>
For now this uses the other API under the hood, but it could be
reimplemented with the same API to do a single allocation.
</p>

<p>
I added a new kind of test to make sure command line argument parsing works.
</p>

<h2>Automatic C-to-Zig translation</h2>

<ul>
  <li> Translation now understands enum tag types.</li>
  <li> I refactored the C macro parsing, and made it understand pointer casting.
   Now some kinds of int-to-ptr code in .h files for embedded programming
   just work:</li>
</ul>

<pre><code class="languuage-c">#define NRF_GPIO ((NRF_GPIO_Type *) NRF_GPIO_BASE)</code></pre>

<p>Zig now understands this C macro.</p>

   <h2>std.mem</h2>

   <ul>
     <li>add <code>aligned</code> functions to Allocator interface
</li>
<li><code>mem.Allocator</code> initializes bytes to undefined. This does nothing in ReleaseFast
  mode. In Debug and ReleaseSafe modes, it initializes bytes to <code>0xaa</code> which helps catch
   memory errors.
   </li>
   <li>add <code>mem.FixedBufferAllocator</code>
 </li>
   </ul>

 <h2>std.os.ChildProcess</h2>

 I added <code>std.os.ChildProcess.exec</code> for when you want to spawn a child process, wait for it
to complete, and then capture the stdandard output into a buffer.

<pre><code class="language-zig">pub fn exec(self: &amp;Builder, argv: []const []const u8) -&gt; []u8 {
    const max_output_size = 100 * 1024;
    const result = os.ChildProcess.exec(self.allocator, argv, null, null, max_output_size) %% |err| {
        std.debug.panic("Unable to spawn {}: {}", argv[0], @errorName(err));
    };
    switch (result.term) {
        os.ChildProcess.Term.Exited =&gt; |code| {
            if (code != 0) {
                warn("The following command exited with error code {}:\n", code);
                printCmd(null, argv);
                warn("stderr:{}\n", result.stderr);
                std.debug.panic("command failed");
            }
            return result.stdout;
        },
        else =&gt; {
            warn("The following command terminated unexpectedly:\n");
            printCmd(null, argv);
            warn("stderr:{}\n", result.stderr);
            std.debug.panic("command failed");
        },
    }
}</code></pre>

<h2>std.sort</h2>

<p>
<a href="https://github.com/Hejsil">Hejsil</a> <a href="https://github.com/zig-lang/zig/issues/657">pointed out</a>
that the quicksort implementation in the standard library failed a simple test case.
</p>

<p>
There was another problem with the implementation of sort in the standard library, 
which is that it used <code>O(n)</code> stack space via recursion. This is fundamentally
insecure, especially if you consider that the length of an array you might want to sort could be
user input. It prevents <a href="https://github.com/zig-lang/zig/issues/157">#157</a>
from working as well.
</p>

<p>
I had a look at
<a href="https://en.wikipedia.org/wiki/Sorting_algorithm#Comparison_of_algorithms">Wikipedia's Comparison of Sorting Algorithms</a> and only 1 sorting algorithm checked all the boxes:
</p>

<ul>
  <li>Best case <code>O(n)</code> complexity (adaptive sort)</a>
  <li>Average case <code>O(n * log(n))</code> complexity</a>
  <li>Worst case <code>O(n * log(n))</code> complexity</a>
  <li><code>O(1)</code> memory</a>
  <li>Stable sort</a>
</ul>

<p>
And that algorithm is <a href="https://en.wikipedia.org/wiki/Block_sort">Block sort</a>.
</p>

<p>
I found a
<a href="https://github.com/BonzaiThePenguin/WikiSort/blob/master/WikiSort.c">high quality implementation of block sort in C</a>,
which is licensed under the public domain.
</p>

<p>
I ported the code from C to Zig, integrated it into the standard library, and it passed all tests first try. Amazing.
</p>

<p>
Surely, I thought, there must be some edge case. So I created a simple fuzz tester:
</p>

<pre><code class="language-zig">test "sort fuzz testing" {
    var rng = std.rand.Rand.init(0x12345678);
    const test_case_count = 10;
    var i: usize = 0;
    while (i &lt; test_case_count) : (i += 1) {
        fuzzTest(&amp;rng);
    }
}

var fixed_buffer_mem: [100 * 1024]u8 = undefined;

fn fuzzTest(rng: &amp;std.rand.Rand) {
    const array_size = rng.range(usize, 0, 1000);
    var fixed_allocator = mem.FixedBufferAllocator.init(fixed_buffer_mem[0..]);
    var array = %%fixed_allocator.allocator.alloc(IdAndValue, array_size);
    // populate with random data
    for (array) |*item, index| {
        item.id = index;
        item.value = rng.range(i32, 0, 100);
    }
    sort(IdAndValue, array, cmpByValue);

    var index: usize = 1;
    while (index &lt; array.len) : (index += 1) {
        if (array[index].value == array[index - 1].value) {
            assert(array[index].id &gt; array[index - 1].id);
        } else {
            assert(array[index].value &gt; array[index - 1].value);
        }
    }
}</code></pre>

<p>
This test passed as well. And so I think this problem is solved.
</p>

<h2>@export</h2>

<p>
There is now an <a href="http://ziglang.org/documentation/master/#builtin-export">@export</a> builtin function which can be used in a comptime block
to conditionally export a function:
</p>

<pre><code class="language-zig">const builtin = @import("builtin");

comptime {
    const strong_linkage = builtin.GlobalLinkage.Strong;
    if (builtin.link_libc) {
        @export("main", main, strong_linkage);
    } else if (builtin.os == builtin.Os.windows) {
        @export("WinMainCRTStartup", WinMainCRTStartup, strong_linkage);
    } else {
        @export("_start", _start, strong_linkage);
    }
}</code></pre>

<p>
It can also be used to create aliases:
</p>


<pre><code class="language-zig">const builtin = @import("builtin");
const is_test = builtin.is_test;

comptime {
    const linkage = if (is_test) builtin.GlobalLinkage.Internal else builtin.GlobalLinkage.Weak;
    const strong_linkage = if (is_test) builtin.GlobalLinkage.Internal else builtin.GlobalLinkage.Strong;

    @export("__letf2", @import("comparetf2.zig").__letf2, linkage);
    @export("__getf2", @import("comparetf2.zig").__getf2, linkage);

    if (!is_test) {
        // only create these aliases when not testing
        @export("__cmptf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__eqtf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__lttf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__netf2", @import("comparetf2.zig").__letf2, linkage);
        @export("__gttf2", @import("comparetf2.zig").__getf2, linkage);
    }
}</code></pre>

<p>
Previous export syntax is still allowed. See <a href="https://github.com/zig-lang/zig/issues/462 ">#462</a> and <a href="https://github.com/zig-lang-zig/issues/420">#420</a>.
</p>

<h2>Labeled loops, blocks, break, and continue, and R.I.P. goto</h2>

<p>
We used to have labels and goto like this:
</p>

<pre><code class="language-zig">export fn entry() {
    label:
    goto label;
}</code></pre>

<p>
Now this does not work, because goto is gone.
</p>

<pre><code>test.zig:2:10: error: expected token ';', found ':'
    label:
         ^</code></pre>

<p>
There are a few reasons to use goto, but all of the use cases are better served
with other zig control flow features:
</p>

<ul>
  <li>cleanup pattern. Use <code>defer</code> and <code>%defer</code> instead.</li>
  <li>goto backward</li>
  <li>goto forward</li>
</ul>

<h3>goto backward</h3>

<pre><code class="language-zig">export fn entry() {
    start_over:

    while (some_condition) {
        // do something...
        goto start_over;
    }
}</code></pre>

<p>
Instead, use a loop!
</p>

<pre><code class="language-zig">export fn entry() {
    outer: while (true) {

        while (some_condition) {
            // do something...
            continue :outer;
        }

        break;
    }
}</code></pre>

<h3>goto forward</h3>

<pre><code class="language-zig">pub fn findSection(elf: &amp;Elf, name: []const u8) -&gt; %?&amp;SectionHeader {
    var file_stream = io.FileInStream.init(elf.in_file);
    const in = &amp;file_stream.stream;

    section_loop: for (elf.section_headers) |*elf_section| {
        if (elf_section.sh_type == SHT_NULL) continue;

        const name_offset = elf.string_section.offset + elf_section.name;
        %return elf.in_file.seekTo(name_offset);

        for (name) |expected_c| {
            const target_c = %return in.readByte();
            if (target_c == 0 or expected_c != target_c) goto next_section;
        }

        {
            const null_byte = %return in.readByte();
            if (null_byte == 0) return elf_section;
        }
next_section:
    }

    return null;
}</code></pre>

<p>
Looks like the use case is breaking out of an outer loop:
</p>

<pre><code class="language-zig">pub fn findSection(elf: &amp;Elf, name: []const u8) -&gt; %?&amp;SectionHeader {
    var file_stream = io.FileInStream.init(elf.in_file);
    const in = &amp;file_stream.stream;

    section_loop: for (elf.section_headers) |*elf_section| {
        if (elf_section.sh_type == SHT_NULL) continue;

        const name_offset = elf.string_section.offset + elf_section.name;
        %return elf.in_file.seekTo(name_offset);

        for (name) |expected_c| {
            const target_c = %return in.readByte();
            if (target_c == 0 or expected_c != target_c) continue :section_loop;
        }

        {
            const null_byte = %return in.readByte();
            if (null_byte == 0) return elf_section;
        }
    }

    return null;
}</code></pre>

<p>
You can also break out of arbitrary blocks:
</p>

<pre><code class="language-zig">export fn entry() {
    outer: {

        while (some_condition) {
            // do something...
            break :outer;
        }
    }
}</code></pre>

<p>
This can be used to return a value from a block in the same way you
can return a value from a function:
</p>

<pre><code class="language-zig">export fn entry() {
    const value = init: {
        for (slice) |item| {
            if (item &gt; 100)
                break :init item;
        }
        break :init 0;
    };
}</code></pre>

<p>
Omitting a semicolon no longer causes the value to be returned by the block.
Instead you must use explicit block labels to return a value from a block.
I'm considering a keyword such as <code>result</code> which defaults to the
current block.
</p>

<p>
Removal of goto caused a regression in C-to-Zig translation: Switch statements
no longer can be translated. However this code will be resurrected
soon using labeled loops and labeled break instead of goto.
</p>

<p>
See <a href="https://github.com/zig-lang/zig/issues/346">#346</a>, <a href="https://github.com/zig-lang-zig/issues/630">#630</a>, and <a href="https://github.com/zig-lang-zig/issues/629">#629</a>.
</p>

<h2>New IR pass iteration strategy</h2>

<p>
Before:
</p>

<ul>
  <li> IR basic blocks are in arbitrary order</li>
    <li> when doing an IR pass, when a block is encountered, code
    must look at all the instructions in the old basic block,
    determine what blocks are referenced, and queue up those
    old basic blocks first.
    </li>
    <li> This had a bug</li>
</ul>

<pre><code class="language-zig">while (cond) {
    if (false) { }
    break;
}</code></pre>

<p>
Pretty crazy right? Something as simple as this would crash the compiler.
</p>
<p>
Now:
</p>

<ul>
  <li> IR basic blocks are required to be in an order that guarantees
    they will be referenced by a branch, before any instructions
    within are referenced.
    ir pass1 is updated to meet this constraint.</li>
    <li>hen doing an IR pass, we iterate over old basic blocks
    in the order they appear. Blocks which have not been
    referenced are discarded.</li>
    <li>fter the pass is complete, we must iterate again to look
    for old basic blocks which now point to incomplete new
    basic blocks, due to comptime code generation.</li>
    <li>his last part can probably be optimized - most of the time
      we don't need to iterate over the basic block again.</li>
</ul>

    <p>
This improvement deletes a lot of messy code:
    </p>

<pre> 5 files changed, 288 insertions(+), 1243 deletions(-) </pre>

<p>
And it also fixes comptime branches not being respected sometimes:
</p>

<pre><code class="language-zig">export fn entry() {
    while (false) {
        @compileError("bad");
    }
}</code></pre>

<p>Before, this would cause a compile error. Now the while loop respects the
implicit compile-time.
</p>

<p>
See <a href="https://github.com/zig-lang/zig/issues/667">#667</a>.
</p>

<h2>Bug Fixes</h2>

<ul>
  <li> fix const and volatile qualifiers being dropped sometimes.
   in the expression <code>&amp;const a.b</code>, the const (and/or volatile)
   qualifiers would be incorrectly dropped. See <a href="https://github.com/zig-lang/zig/issues/655">#655</a>.
  </li>
  <li> fix compiler crash in a nullable if after an if in a switch
 prong of a switch with 2 prongs in an else. See <a href="https://github.com/zig-lang/zig/issues/656">#656</a>.
  </li>
  <li>fix assert when wrapping zero bit type in nullable. See <a href="https://github.com/zig-lang/zig/issues/659">#659</a>.
  </li>
  <li>fix crash when implicitly casting array of len 0 to slice. See <a href="https://github.com/zig-lang/zig/issues/660">#660</a>.
  </li>
  <li>fix endianness of sub-byte integer fields in packed structs. In the future
 packed structs will require specifying endianness. See <a href="https://github.com/zig-lang/zig/issues/307">#307</a>.
  </li>
  <li>fix <code>std.os.path.resolve</code> when the drive is missing.
  </li>
  <li>fix automatically C-translated functions not having debug information.
  </li>
  <li>fix crash when passing union enum with sub-byte field to const slice parameter.
 See <a href="https://github.com/zig-lang/zig/issues/664">#664</a>.
 </li>
</ul>
    
   <h2>Miscellaneous changes</h2>

<ul>
  <li>Rename <code>builtin.is_big_endian</code> to <code>builtin.endian</code>. This is in preparation for
   having endianness be a pointer property, which is related to packed structs.
   See <a href="https://github.com/zig-lang/zig/issues/307">#307</a>.
  </li>
  <li>Tested Zig with LLVM debug mode and fixed some bugs that were causing LLVM
   assertions.
  </li>
  <li>Add
    <a href="http://ziglang.org/documentation/master/#builtin-noInlineCall">@noInlineCall</a>.
    See <a href="https://github.com/zig-lang/zig/issues/640">#640</a>.
    This fixes a crash in <code>--release-safe</code> and <code>--release-fast</code> modes
    where the optimizer inlines everything into <code>_start</code> and
   clobbers the command line argument data.
   If we were able to verify that the user's code never reads
   command line args, we could leave off this "no inline"
   attribute. This might call for a patch to LLVM. It seems like inlining
   into a naked function should correctly bump the stack pointer.
  </li>
  <li>add <code>i29</code> and <code>u29</code> primitive types. <code>u29</code> is the type of alignment,
   so it makes sense to be a primitive.
   probably in the future we'll make any <code>i</code> or <code>u</code> followed by
   digits into a primitive.
  </li>
  <li>add implicit cast from enum tag type of union to const ptr to the union. closes <a href="https://github.com/zig-lang/zig/issues/654">#654</a>
  </li>
  <li>ELF stack traces support <code>DW_AT_ranges</code>, so sometimes when you would see "???"
   you now get a useful stack trace instead.
   </li>
   <li>add <code>std.sort.min</code> and <code>std.sort.max</code> functions
 </li>
 <li><code>std.fmt.bufPrint</code> returns a possible <code>error.BufferTooSmall</code> instead of 
   asserting that the buffer is large enough.
   </li>
   <li>Remove unnecessary inline calls in <code>std.math</code>.
 </li>
 <li><code>zig build</code> now has a <code>--search-prefix</code> option. Any number of search prefixes can be
   specified.
   </li>
   <li>add some utf8 parsing utilities to the standard library.
 </li>
</ul>

 <h2>Thank you contributors!</h2>

 <ul>
   <li><strong>MIURA Masahiro</strong> fixed the color of compiler messages for light-themed terminals.
     (See <a href="https://github.com/zig-lang/zig/issues/644">#644</a>)</li>
   <li><strong>Peter Rönnquist</strong> added format for floating point numbers.
     <code>{.x}</code> where <code>x</code> is the number of decimals. (See <a href="https://github.com/zig-lang/zig/issues/668">#668</a>)</li>
 </ul>

   <h2>Thank you financial supporters!</h2>

<p>
Special thanks to those who <a href="https://github.com/users/andrewrk/sponsorship">donate monthly</a>:
</p>

<ul>
<li>Lauren Chavis</li>
<li>Andrea Orru</li>
<li>Adrian Sinclair</li>
<li>David Joseph</li>
<li>jeff kelley</li>
<li>Hasan Abdul-Rahman</li>
<li>Wesley Kelley</li>
<li>Jordan Torbiak</li>
<li>Richard Ohnemus</li>
<li>Martin Schwaighofer</li>
<li>Matthew </li>
<li>Mirek Rusin</li>
<li>Brendon Scheinman</li>
<li>Pyry Kontio</li>
<li>Thomas Ballinger</li>
<li>Peter Ronnquist</li>
<li>Robert Paul Herman</li>
<li>Audun Wilhelmsen</li>
<li>Marko Mikulicic</li>
<li>Anthony J. Benik</li>
<li>Caius </li>
<li>Tyler Philbrick</li>
<li>Jeremy Larkin</li>
<li>Rasmus Rønn Nielsen</li>
</ul>


guid: https://andrewkelley.me/post/unsafe-zig-safer-than-unsafe-rust.html
title: Unsafe Zig is Safer Than Unsafe Rust
url: https://andrewkelley.me/post/unsafe-zig-safer-than-unsafe-rust.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
  Prism.languages['rust'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|const|var|pub|struct|enum|break|return|continue|if|else|match|while|for|true|false)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|str|static)\b/,
  });
  Prism.languages['llvm'] = Prism.languages.extend('clike', {
    'keyword': /\b(private|constant|declare|define|c|noreturn|nounwind|alloca|br|store|load|getelementptr|and|icmp|eq|zext|call|unreachable|add|target|datalayout|triple|unnamed_addr|align|inbounds|uwtable|sext|ret|ne|phi|global|to|zeroinitializer|shl|or|ult|switch)\b/,
    'number': null,
    'comment': /;.*/,
    'operator': /\w[\w\d_]*:/,
    'regex': /[@%]\.?\w+/,
    'property': /\b(i8|i1|i32|i64|i16|void|label)\*?\b/,
    'punctuation': null,
    'function': null,
  });
</script>
<style type="text/css">
.red {
  color: red;
}
.green {
  color: green;
}
</style>
<h1>Unsafe Zig is Safer than Unsafe Rust</h1>

<p>Consider the following Rust code:</p>

<pre><code class="language-rust">struct Foo {
    a: i32,
    b: i32,
}

fn main() {
    unsafe {
        let mut array: [u8; 1024] = [1; 1024];
        let foo = std::mem::transmute::&lt;&amp;mut u8, &amp;mut Foo&gt;(&amp;mut array[0]);
        foo.a += 1;
    }
}</code></pre>

<p>
This pattern is pretty common if you are <a href="https://gist.github.com/andrewrk/182ace5dee6c4025d8c4b0ca22ca98ca">interacting with Operating System APIs</a>. <a href="https://github.com/andrewrk/libsoundio/blob/fc96baf8130b52ba6fe928e5f629afd55ecc7321/src/alsa.c#L802">Another example</a>.
</p>

<p>Can you spot the problem with the code?</p>

<p>It's pretty subtle, but there is actually undefined behavior going on here. Let's take a look at the LLVM IR:</p>

<pre><code class="language-llvm">define internal void @_ZN4test4main17h916a53db53ad90a1E() unnamed_addr #0 {
start:
  %transmute_temp = alloca %Foo*
  %array = alloca [1024 x i8]
  %0 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i32 0, i32 0
  call void @llvm.memset.p0i8.i64(i8* %0, i8 1, i64 1024, i32 1, i1 false)
  br label %bb1

bb1:                                              ; preds = %start
  %1 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i64 0, i64 0
  %2 = bitcast %Foo** %transmute_temp to i8**
  store i8* %1, i8** %2, align 8
  %3 = load %Foo*, %Foo** %transmute_temp, !nonnull !1
  br label %bb2

bb2:                                              ; preds = %bb1
  %4 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0
  %5 = load i32, i32* %4
  %6 = call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %5, i32 1)
  %7 = extractvalue { i32, i1 } %6, 0
  %8 = extractvalue { i32, i1 } %6, 1
  %9 = call i1 @llvm.expect.i1(i1 %8, i1 false)
  br i1 %9, label %panic, label %bb3

bb3:                                              ; preds = %bb2
  %10 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0
  store i32 %7, i32* %10
  ret void

panic:                                            ; preds = %bb2
; call core::panicking::panic
  call void @_ZN4core9panicking5panic17hfecc01813e436969E({ %str_slice, [0 x i8], %str_slice, [0 x i8], i32, [0 x i8], i32, [0 x i8] }* noalias readonly dereferenceable(40) bitcast ({ %str_slice, %str_slice, i32, i32 }* @panic_loc.2 to { %str_slice, [0 x i8], %str_slice, [0 x i8], i32, [0 x i8], i32, [0 x i8] }*))
  unreachable
}</code></pre>

<p>
That's the code for the main function.
This is using rustc version 1.21.0.
Let's zoom in on the problematic parts:
</p>

<pre><code class="language-llvm">  %array = alloca [1024 x i8]
  ; loading foo.a in order to do + 1
  %5 = load i32, i32* %4
  ; storing the result of + 1 into foo.a
  store i32 %7, i32* %10
</code></pre>

<p>
None of these <a href="http://llvm.org/docs/LangRef.html#alloca-instruction">alloca</a>,
<a href="http://llvm.org/docs/LangRef.html#load-instruction">load</a>, or
<a href="http://llvm.org/docs/LangRef.html#store-instruction">store</a>
instructions have alignment attributes on them, so they use the ABI alignment of the respective types.
</p>

<p>
That means the i8 array gets alignment of 1, since the ABI alignment of i8 is 1,
and the load and store instructions get alignment 4, since the ABI alignment of i32 is 4.
This is undefined behavior:
</p>

<ul>
<blockquote>
the store has undefined behavior if the alignment is not set to a value which is at least the size in bytes of the pointee
</blockquote>
<blockquote>
the load has undefined behavior if the alignment is not set to a value which is at least the size in bytes of the pointee
</blockquote>
</ul>

<p>
It's a nasty bug, because besides being an easy mistake to make, on some architectures it will only
cause mysterious slowness, while on others it can cause an illegal instruction exception on the CPU.
Regardless, it's undefined behavior, and we are professionals, and so we do not accept undefined
behavior.
</p>

<p>
Let's try writing the equivalent code in <a href="http://ziglang.org/">Zig</a>:
</p>

<pre><code class="language-zig">const Foo = struct {
    a: i32,
    b: i32,
};

pub fn main() {
    var array = []u8{1} ** 1024;
    const foo = @ptrCast(&amp;Foo, &amp;array[0]);
    foo.a += 1;
}</code></pre>

<p>And now we compile it:</p>

<pre>/home/andy/tmp/test.zig:8:17: error: cast increases pointer alignment
    const foo = @ptrCast(&amp;Foo, &amp;array[0]);
                ^
/home/andy/tmp/test.zig:8:38: note: '&amp;u8' has alignment 1
    const foo = @ptrCast(&amp;Foo, &amp;array[0]);
                                     ^
/home/andy/tmp/test.zig:8:27: note: '&amp;Foo' has alignment 4
    const foo = @ptrCast(&amp;Foo, &amp;array[0]);
                          ^</pre>

<p>
Zig knows not to compile this code. Here's how to fix it:
</p>

<pre><code class="language-diff">@@ -4,7 +4,7 @@
 };
 
 pub fn main() {
<span class="red">-    var array = []u8{1} ** 1024;</span>
<span class="green">+    var array align(@alignOf(Foo)) = []u8{1} ** 1024;</span>
     const foo = @ptrCast(&amp;Foo, &amp;array[0]);
     foo.a += 1;
 }</code></pre>

<p>
Now it compiles fine. Let's have a look at the LLVM IR:
</p>

<pre><code class="language-llvm">define internal fastcc void @main() unnamed_addr #0 !dbg !8911 {
Entry:
  %array = alloca [1024 x i8], align 4
  %foo = alloca %Foo*, align 8
  %0 = bitcast [1024 x i8]* %array to i8*, !dbg !8923
  call void @llvm.memcpy.p0i8.p0i8.i64(i8* %0, i8* getelementptr inbounds ([1024 x i8], [1024 x i8]* @266, i32 0, i32 0), i64 1024, i32 4, i1 false), !dbg !8923
  call void @llvm.dbg.declare(metadata [1024 x i8]* %array, metadata !8914, metadata !529), !dbg !8923
  %1 = getelementptr inbounds [1024 x i8], [1024 x i8]* %array, i64 0, i64 0, !dbg !8924
  %2 = bitcast i8* %1 to %Foo*, !dbg !8925
  store %Foo* %2, %Foo** %foo, align 8, !dbg !8926
  call void @llvm.dbg.declare(metadata %Foo** %foo, metadata !8916, metadata !529), !dbg !8926
  %3 = load %Foo*, %Foo** %foo, align 8, !dbg !8927
  %4 = getelementptr inbounds %Foo, %Foo* %3, i32 0, i32 0, !dbg !8927
  %5 = load i32, i32* %4, align 4, !dbg !8927
  %6 = call { i32, i1 } @llvm.sadd.with.overflow.i32(i32 %5, i32 1), !dbg !8929
  %7 = extractvalue { i32, i1 } %6, 0, !dbg !8929
  %8 = extractvalue { i32, i1 } %6, 1, !dbg !8929
  br i1 %8, label %OverflowFail, label %OverflowOk, !dbg !8929

OverflowFail:                                     ; preds = %Entry
  tail call fastcc void @panic(%"[]u8"* @88, %StackTrace* null), !dbg !8929
  unreachable, !dbg !8929

OverflowOk:                                       ; preds = %Entry
  store i32 %7, i32* %4, align 4, !dbg !8929
  ret void, !dbg !8930
}</code></pre>

<p>Zooming in on the relevant parts:</p>

<pre><code class="language-llvm">  %array = alloca [1024 x i8], align 4
  %5 = load i32, i32* %4, align 4, !dbg !8927
  store i32 %7, i32* %4, align 4, !dbg !8929
</code></pre>

<p>
Notice that the alloca, load, and store all agree on the alignment.
</p>

<p>
In Zig the problem of alignment is solved completely; the compiler catches all
possible alignment issues.
In the situation where you need to assert to the compiler that something is more aligned than Zig thinks it is, you can use <a href="http://ziglang.org/documentation/master/#alignCast">@alignCast</a>.
This inserts a cheap safety check in debug mode to make sure the alignment assertion is correct.
</p>


guid: https://andrewkelley.me/post/zig-january-2018-in-review.html
title: Zig: January 2018 in Review
url: https://andrewkelley.me/post/zig-january-2018-in-review.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|catch|try|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime|noinline)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>Zig: January 2018 in Review</h1>

<p>
One month (and a few days, sorry I'm late!) has passed since I did the
<a href="zig-december-2017-in-review.html">December 2017 writeup</a>, and so it's time for
another month-in-review for <a href="https://github.com/users/andrewrk/sponsorship">my esteemed sponsors</a>.
</p>

<h2>LLVM 6 Readiness</h2>
<p><a href="http://prereleases.llvm.org/6.0.0/">LLVM 6.0.0rc2</a> was just announced on the mailing list.
It's scheduled to be released on February 21, and Zig is ready.
I plan to have Zig release 0.2.0 one week after LLVM 6 comes out.
We already have all tests passing with debug builds of LLVM 6 in the
<a href="https://github.com/zig-lang/zig/tree/llvm6">llvm6 branch</a>,
but that extra week is for a bug-stomping rampage.
</p>
<p>
After that, all the 0.3.0 milestone issues get postponed to 0.4.0, and all the
0.2.0 milestone issues get moved to 0.3.0.
</p>
<p>
0.2.0 will be an exciting release because, among many other things, it enables
source-level debugging with MSVC on Windows.
</p>
<p>
Zig is once again in the <a href="http://prereleases.llvm.org/6.0.0/rc2/docs/ReleaseNotes.html#zig-programming-language">release notes of LLVM</a>, so we should see a slight increase in community size when LLVM release notes hit the tech news headlines.
</p>
<h2>Error Syntax Cleanup</h2>
<p>
One of the biggest complaints newcomers to Zig had was about its sigils regarding error handling.
Given this, I made an effort to choose friendlier syntax.
</p>
<ul>
  <li><code>%return</code> is replaced with <code>try</code></li>
  <li><code>%defer</code> is replaced with <code>errdefer</code></li>
  <li><code>a %% b</code> is replaced with <code>a catch b</code></li>
  <li><code>%%x</code> is removed entirely to discourage its use.
    You can get an equivalent effect with <code>x catch unreachable</code>,
    which has been updated to understand that it was attempting to unwrap an error union.
    See <a href="https://github.com/zig-lang/zig/issues/545">#545</a>  and <a href="https://github.com/zig-lang/zig/issues/510">#510</a></li>
</ul>
<p>
After these changes, there is a strong pattern that only keywords can modify control flow.
For example we have <code>and</code> and <code>or</code> instead of <code>&amp;&amp;</code> and <code>||</code>. There is one last exception, which is <code>a ?? b</code>. Maybe it's okay, since <a href="https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/operators/null-conditional-operator">C# set a precedent</a>.
</p>
<p>
An even bigger change is coming soon which I'm calling
<a href="https://github.com/zig-lang/zig/issues/632">Error Sets</a>.
</p>
<h2>Error Return Traces</h2>
<p>
I'm really excited about this one. I invented a new kind of debugging tool and integrated it into Debug and ReleaseSafe builds.
</p>
<p>
One of the concerns with removing the <code>%%</code> prefix operator was that it was
just so gosh darn convenient to get a stack trace right at the moment where you
asserted that a value did not have an error. I wanted to make it so that programmers
could use <code>try</code> everywhere and still get the debuggability benefit when
an error occurred.
</p>
<p>
Watch this:
</p>
<pre><code class="language-zig">const std = @import("std");

pub fn main() !void {
    const allocator = std.debug.global_allocator;

    const args = try std.os.argsAlloc(allocator);
    defer std.os.argsFree(allocator, args);

    const count = try parseFile(allocator, args[1]);

    if (count &lt; 10) return error.NotEnoughItems;
}

fn parseFile(allocator: &amp;std.mem.Allocator, file_path: []const u8) !usize {
    const contents = std.io.readFileAlloc(allocator, file_path) catch return error.UnableToReadFile;
    defer allocator.free(contents);

    return contents.len;
}</code></pre>
<p>
Here's a simple program with a bunch of different ways that errors could get returned
from <code>main</code>. In our test example, we're going to open a bogus file that
does not exist.
</p>
<pre>
$ zig build-exe test2.zig
$ ./test2 bogus-does-not-exist.txt
error: UnableToReadFile
/home/andy/dev/zig/build/lib/zig/std/os/index.zig:301:33: 0x000000000021acd0 in ??? (test2)
                posix.ENOENT =&gt; return PosixOpenError.PathNotFound,
                                ^
/home/andy/dev/zig/build/lib/zig/std/os/file.zig:25:24: 0x00000000002096f6 in ??? (test2)
            const fd = try os.posixOpen(allocator, path, flags, 0);
                       ^
/home/andy/dev/zig/build/lib/zig/std/io.zig:267:16: 0x000000000021ebec in ??? (test2)
    var file = try File.openRead(allocator, path);
               ^
/home/andy/dev/zig/build/test2.zig:15:71: 0x000000000021ce72 in ??? (test2)
    const contents = std.io.readFileAlloc(allocator, file_path) catch return error.UnableToReadFile;
                                                                      ^
/home/andy/dev/zig/build/test2.zig:9:19: 0x000000000021c1f9 in ??? (test2)
    const count = try parseFile(allocator, args[1]);
                  ^
</pre>
<p>
I'm going to include a picture of the above here, because it looks a lot better with
terminal colors:
</p>
<img src="https://superjoe.s3.amazonaws.com/blog-files/zig-january-2018-in-review/error-return-traces.png">
<p>
This is not a stack trace snapshot from when an error was "created". This is a <strong>return trace</strong> of all the points in the code where an error was returned from a function.
</p>
<p>
Note that, if it only told you the origin of the error that we ultimately received - <code>UnableToReadFile</code> - we would only see the bottom 2 items in the trace.
Not only do we have this information, we have all the information about the origin of
the error, right up to the fact that we received <code>ENOENT</code> from <code>open</code>.
</p>
<p>
With this in place, programmers can comfortably use <code>try</code> everywhere, safe
in the knowledge that it will be straightforward to troubleshoot the origin of any error
bubbling up through the system.
</p>
<p>
I hope you're skeptically wondering, OK, what's the tradeoff in terms of binary size, performance, and memory?
</p>
<p>
First of all, this feature is disabled in ReleaseFast mode. So the answer is,
literally no cost, in this case. But what about Debug and ReleaseSafe builds?
</p>
<p>
To analyze performance cost, there are two cases:
</p>
<ul>
  <li>when no errors are returned</li>
  <li>when returning errors</li>
</ul>
<p>
For the case when no errors are returned, the cost is a single memory write operation, only in the first non-failable function in the call graph that calls a failable function, i.e. when a function returning <code>void</code> calls a function returning <code>error</code>.
This is to initialize this struct in the stack memory:
</p>
<pre><code class="language-zig">pub const StackTrace = struct {
    index: usize,
    instruction_addresses: [N]usize,
};</code></pre>
<p>
Here, N is the maximum function call depth as determined by call graph analysis. Recursion is ignored and counts for 2.
</p>
<p>
A pointer to <code>StackTrace</code> is passed as a secret parameter to every function that can return an error, but it's always the first parameter, so it can likely sit in a register and stay there.
</p>
<p>
That's it for the path when no errors occur. It's practically free in terms of performance.
</p>
<p>
When generating the code for a function that returns an error, just before the <code>return</code> statement (only for the <code>return</code> statements that return errors), Zig generates a call to this function:
</p>
<pre><code class="language-zig">noinline fn __zig_return_error(stack_trace: &amp;StackTrace) void {
    stack_trace.instruction_addresses[stack_trace.index] = @returnAddress();
    stack_trace.index = (stack_trace.index + 1) % N;
}</code></pre>
<p>
The cost is 2 math operations plus some memory reads and writes. The memory accessed is constrained and should remain cached for the duration of the error return bubbling.
</p>
<p>
As for code size cost, 1 function call before a return statement is no big deal. Even so,
I have <a href="https://github.com/zig-lang/zig/issues/690">a plan</a> to make the call to
<code>__zig_return_error</code> a tail call, which brings the code size cost down to actually zero. What is a return statement in code without error return tracing can become a jump instruction in code with error return tracing.
</p>
<p>
There are a few ways to activate this error return tracing feature:
</p>
<ul>
  <li>Return an error from main</li>
  <li>An error makes its way to <code>catch unreachable</code> and you have not overridden the default panic handler</li>
  <li>Use <a href="http://ziglang.org/documentation/master/#errorReturnTrace">@errorReturnTrace</a> to access the current return trace. You can use <code>std.debug.dumpStackTrace</code> to print it. This function returns comptime-known <code>null</code> when building without error return tracing support.</li>
</ul>

<p>Related issues: <a href="https://github.com/zig-lang/zig/issues/651">#651</a>  <a href="https://github.com/zig-lang/zig/issues/684">#684</a>
</p>

<h2>Documentation</h2>
<p>
Big news on the documentation front.
</p>
<p>
All the outdated docs are fixed, and we have automatic
<a href="https://github.com/zig-lang/zig/blob/46aa416c48c283849059292267ac25a6d0db76d6/doc/docgen.zig">docgen tool</a>
which:
</p>
<ul>
  <li>Automatically generates the table of contents</li>
  <li>Validates all internal links</li>
  <li>Validates all code examples</li>
  <li>Turns terminal coloring of stack traces and compile errors into HTML</li>
</ul>
<p>The tool is, of course, written in Zig. <a href="https://github.com/zig-lang/zig/issues/465">#465</a></p>
<p>In addition to the above, the following improvements were made to the documentation:</p>
<ul>
  <li>Added documentation for <a href="http://ziglang.org/documentation/master/#noInlineCall">@noInlineCall</a></li>
  <li>Added documentation for <a href="http://ziglang.org/documentation/master/#extern-enum">extern enum</a></a>
  <li>Improved the documentation styling</li>
  <li>Made the documentation a single file that has no external dependencies</li>
  <li>Add the documentation to appveyor build artifacts as <code>langref.html</code>. In other words we ship with the docs now.</li>
</ul>
<p><strong>Marc Tiehuis</strong> improved documentation styling for mobile devices. <a href="https://github.com/zig-lang/zig/issues/729">#729</a></p>
<ul>
  <li> No overscrolling on small screens</li>
  <li> Font-size is reduced for more content per screen</li>
  <li> Tables + Code blocks scroll within a block to avoid page-widenening</li>
</ul>
<p>
There is still much more to document, before we have achieved <a href="https://github.com/zig-lang/zig/issues/367">basic documentation for everything</a>.
</p>

<h2>Self-Hosted Compiler</h2>
<p>The self-hosted compiler now fully successfully builds on Windows and MacOS.</p>
<p>The main test suite builds the self-hosted compiler.</p>
<p>The self-hosted build inherits the std lib file list and C header file list from the stage1 cmake build, as well as the <code>llvm-config</code> output. So if you get stage1 to build,
stage2 will reliably build as well.</p>

<h2>Windows 32-bit Support Status</h2>
<p>
Windows 32-bit mostly works, but there are some failing tests. The number of failing tests
grew and it didn't seem fair to claim that we supported it officially.
</p>
<p>
So I removed the claims that we support Windows 32-bit from the README, and removed
32-bit Windows from the testing matrix.
</p>
<p>
<a href="https://github.com/zig-lang/zig/issues/537">We still want to support Windows 32-bit</a>.
</p>

<h2>Syntax: Mandatory Function Return Type </h2>
<p><code>-&gt;</code> is removed, and all functions require an explicit return type.</p>
<p>The purpose of this is:</p>
<ul>
  <li> Only one way to do things</li>
  <li> Changing a function with void return type to return a possible
       error becomes a 1 character change, subtly encouraging
       people to use errors.</li>
</ul>
  <p>
    Here are some imperfect sed commands for performing this update:
  </p>
  <p>
    remove arrow:
  </p>
  <pre>sed -i 's/\(\bfn\b.*\)-&gt; /\1/g' $(find . -name "*.zig")</pre>
  <p>
    add void:
  </p>
  <pre>sed -i 's/\(\bfn\b.*\))\s*{/\1) void {/g' $(find . -name "*.zig")</pre>
  <p>
    Some cleanup may be necessary, but this should do the bulk of the work.
  </p>

    <p>This has been a controversial change, and <a href="https://github.com/zig-lang/zig/issues/760">may be reverted</a>.</p>

<h2>Generating .h Files</h2>
<ul>
  <li>Now Zig emits compile errors for non-extern, non-packed struct, enum, unions in <code>extern</code> fn signatures.</li>
  <li>Zig generates .h file content for <code>extern</code> struct, enum, unions</li>
  <li>.h file generation is now tested in the main test suite.</li>
</ul>
<p>Marc Tiehuis added array type handling:</p>
<pre><code class="language-zig">const Foo = extern struct {
    A: [2]i32,
    B: [4]&amp;u32,
};
export fn entry(foo: Foo, bar: [3]u8) void { }</code></pre>
<p>This generates:</p>
<pre><code class="language-c">struct Foo {
    int32_t A[2];
    uint32_t * B[4];
};

TEST_EXPORT void entry(struct Foo foo, uint8_t bar[]);</code></pre>

<h2>Translating C to Zig</h2>
<p><strong>Jimmi Holst Christensen</strong> <a href="https://github.com/zig-lang/zig/pull/695">improved translate-c</a>:</p>
<ul>
  <li> output "undefined" on uninitialized variables</li>
  <li> correct translation of if statements on integers and floats</li>
</ul>

<h2>Crypto Additions to Zig std lib</h2>
<p><strong>Marc Tiehuis</strong> added a bunch of crypto functions:</p>
<ul>
<li>added hardware sqrt for x86_64.
  (See <a href="https://github.com/zig-lang/zig/issues/681">#681</a>)</li>
<li>fixed bitrotted endian swapping std lib code
  (See <a href="https://github.com/zig-lang/zig/issues/682">#682</a>)</li>
</ul>
<h3>Integer Rotation Functions</h3>
<pre><code class="language-zig">/// Rotates right. Only unsigned values can be rotated.
/// Negative shift values results in shift modulo the bit count.
pub fn rotr(comptime T: type, x: T, r: var) -&gt; T {
test "math.rotr" {
    assert(rotr(u8, 0b00000001, usize(0))  == 0b00000001);
    assert(rotr(u8, 0b00000001, usize(9))  == 0b10000000);
    assert(rotr(u8, 0b00000001, usize(8))  == 0b00000001);
    assert(rotr(u8, 0b00000001, usize(4))  == 0b00010000);
    assert(rotr(u8, 0b00000001, isize(-1)) == 0b00000010);
}
/// Rotates left. Only unsigned values can be rotated.
/// Negative shift values results in shift modulo the bit count.
pub fn rotl(comptime T: type, x: T, r: var) -&gt; T {
test "math.rotl" {
    assert(rotl(u8, 0b00000001, usize(0))  == 0b00000001);
    assert(rotl(u8, 0b00000001, usize(9))  == 0b00000010);
    assert(rotl(u8, 0b00000001, usize(8))  == 0b00000001);
    assert(rotl(u8, 0b00000001, usize(4))  == 0b00010000);
    assert(rotl(u8, 0b00000001, isize(-1)) == 0b10000000);
}</code></pre>
<h3>MD5 and SHA1 Hash Functions</h3>
<p><a href="https://github.com/zig-lang/zig/pull/686">Marc writes</a>:</p>
<p>Some performance comparisons to C.</p>

<p>We take the fastest time measurement taken across multiple runs.</p>

<p>The block hashing functions use the same md5/sha1 methods.</p>

<pre><code>Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
Gcc: 7.2.1 20171224
Clang: 5.0.1
Zig: 0.1.1.304f6f1d
</code></pre>

<p>See <a href="https://www.nayuki.io/page/fast-md5-hash-implementation-in-x86-assembly">https://www.nayuki.io/page/fast-md5-hash-implementation-in-x86-assembly</a>:</p>

<pre><code>gcc -O2
    661 Mb/s
clang -O2
    490 Mb/s
zig --release-fast and zig --release-safe
    570 Mb/s
zig
    50 Mb/s
</code></pre>

<p>See <a href="https://www.nayuki.io/page/fast-sha1-hash-implementation-in-x86-assembly">https://www.nayuki.io/page/fast-sha1-hash-implementation-in-x86-assembly</a>
:</p>

<pre><code>gcc -O2
    588 Mb/s
clang -O2
    563 Mb/s
zig --release-fast and zig --release-safe
    610 Mb/s
zig
    21 Mb/s
</code></pre>

<p>In short, zig provides pretty useful tools for writing this sort of
code. We are in the lead against clang (which uses the same LLVM
backend) with us being slower only against md5 with GCC.</p>

<h3>SHA-2 Functions</h3>
<p><a href="https://github.com/zig-lang/zig/pull/687">Marc writes</a>:</p>
<p>We take the fastest time measurement taken across multiple runs. Tested
across multiple compiler flags and the best chosen.</p>

<pre><code>Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
Gcc: 7.2.1 20171224
Clang: 5.0.1
Zig: 0.1.1.304f6f1d
</code></pre>

<p>See <a href="https://www.nayuki.io/page/fast-sha2-hashes-in-x86-assembly">https://www.nayuki.io/page/fast-sha2-hashes-in-x86-assembly</a>.</p>

<pre><code>Gcc -O2
    219 Mb/s
Clang -O2
    213 Mb/s
Zig --release-fast
    284 Mb/s
Zig --release-safe
    211 Mb/s
Zig
    6 Mb/s
</code></pre>

<pre><code>Gcc -O2
    350 Mb/s
Clang -O2
    354 Mb/s
Zig --release-fast
    426 Mb/s
Zig --release-safe
    300 Mb/s
Zig
    11 Mb/s
</code></pre>

<h3>Blake2 Hash Functions</h3>
<p><a href="https://github.com/zig-lang/zig/pull/689">Marc writes</a>:</p>

<p>Blake performance numbers for reference:</p>

<pre><code>Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
</code></pre>

<p>-- Blake2s</p>

<pre><code>Zig --release-fast
    485 Mb/s
Zig --release-safe
    377 Mb/s
Zig
    11 Mb/s
</code></pre>

<p>-- Blake2b</p>

<pre><code>Zig --release-fast
    616 Mb/s
Zig --release-safe
    573 Mb/s
Zig
    18 Mb/s
</code></pre>

<h3>Sha3 Hashing Functions</h3>
<p><a href="https://github.com/zig-lang/zig/pull/696">Marc writes:</a></p>
<p>These are on the slower side and could be improved. No performance optimizations
yet have been done.</p>

<pre><code>Cpu: Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
</code></pre>

<p>-- Sha3-256</p>

<pre><code>Zig --release-fast
    93 Mb/s
Zig --release-safe
    99 Mb/s
Zig
    4 Mb/s
</code></pre>

<p>-- Sha3-512</p>

<pre><code>Zig --release-fast
    49 Mb/s
Zig --release-safe
    54 Mb/s
Zig
    2 Mb/s
</code></pre>

<p>Interestingly, release-safe is producing slightly better code than
release-fast.</p>

<h2>Improvements</h2>
<ul>
  <li>The return type of <code>main</code> can now be <code>void</code>, <code>noreturn</code>, <code>u8</code>, or an error union. <a href="https://github.com/zig-lang/zig/issues/535">#535</a></li>
  <li>Implemented bigint div and rem. <a href="https://github.com/zig-lang/zig/issues/405">#405</a></li>
  <li>Removed coldcc keyword and added <a href="http://ziglang.org/documentation/master/#setCold">@setCold</a>. <a href="https://github.com/zig-lang/zig/issues/661">#661</a></li>
  <li>Renamed "debug safety" to "runtime safety". <a href="https://github.com/zig-lang/zig/issues/437">#437</a></li>
  <li>Updated windows build to use llvm 5.0.1.
    <a href="http://lists.llvm.org/pipermail/llvm-dev/2018-January/120153.html">Reported usability issue regarding diaguids.lib to llvm-dev.</a></li>
  <li>Implemented <code>std.os.selfExePath</code> and <code>std.os.selfExeDirPath</code> for windows.</li>
  <li>Added more test coverage.</li>
  <li>
    The same string literal codegens to the same constant.
    This makes it so that you can send the same string literal
    as a comptime slice and get the same type.
  </li>
</ul>
<p>Zig now supports structs defined inside a function that reference local constants:</p>
<pre><code class="language-zig">const assert = @import("std").debug.assert;

test "struct inside function" {
    const BlockKind = u32;

    const Block = struct {
        kind: BlockKind,
    };

    var block = Block { .kind = 1234 };

    block.kind += 1;

    assert(block.kind == 1235);
}</code></pre>
<p>This fixed <a href="https://github.com/zig-lang/zig/issues/672">#672</a>  and <a href="https://github.com/zig-lang/zig/issues/552">#552</a>. However there is still issue <a href="https://github.com/zig-lang/zig/issues/675">#675</a>  which is that structs inside
functions get named after the function they are in:</p>
<pre><code class="language-zig">test "struct inside function" {
    const Block = struct { kind: u32 };
    @compileLog(@typeName(Block));
}</code></pre>
<p>When executed gives</p>
<pre>
| "struct inside function()"
</pre>
<p>
Moving on, Zig now allows enum tag values to not be in parentheses:
</p>
<pre><code class="language-zig">const EnumWithTagValues = enum(u4) {
    A = 1 &lt;&lt; 0,
    B = 1 &lt;&lt; 1,
    C = 1 &lt;&lt; 2,
    D = 1 &lt;&lt; 3,
};</code></pre>
<p>Previously this required <code>A = (1 &lt;&lt; 0)</code>.
</code></pre>

<h2>Bug Fixes</h2>
<ul>
  <li>fix exp1m implementation by using <code>@setFloatMode</code> and using modular arithmetic</li>
  <li>fix compiler crash related to <code>@alignOf</code></li>
  <li>fix null debug info for 0-length array type. <a href="https://github.com/zig-lang/zig/issues/702">#702</a></li>
  <li>fix compiler not able to rename files into place on windows if the file already existed</li>
  <li>fix crash when switching on enum with 1 field and no switch prongs. <a href="https://github.com/zig-lang/zig/issues/712">#712</a></li>
  <li>fix crash on union-enums with only 1 field. <a href="https://github.com/zig-lang/zig/issues/713">#713</a></li>
  <li>fix crash when align 1 field before self referential align 8 field as slice return type. <a href="https://github.com/zig-lang/zig/issues/723">#723</a></li>
  <li>fix error message mentioning <code>unreachable</code> instead of <code>noreturn</code></li>
  <li>fix std.io.readFileAllocExtra incorrectly returning <code>error.EndOfStream</code></li>
  <li>workaround for <a href="http://ziglang.org/documentation/master/#extern-enum">microsoft releasing windows SDK with the wrong version</a></li>
  <li>Found a bug in NewGVN. Disabled it to match clang and filed an llvm bug.</li>
  <li>emit compile error for @panic called at compile time. <a href="https://github.com/zig-lang/zig/issues/706">#706</a></li>
  <li>emit compile error for shifting by negative comptime integer. <a href="https://github.com/zig-lang/zig/issues/698">#698</a></li>
  <li>emit compile error for calling naked function. @ptrCast a naked function first to call it.</li>
  <li>emit compile error for duplicate struct, enum, union fields. <a href="https://github.com/zig-lang/zig/issues/730">#730</a></li>
</ul>

 
 <h2>Thank you contributors!</h2>
 <ul>
   <li><strong>Jimmi Holst Christensen</strong> fixed bitrotted code: `std.Rand.scalar` and `std.endian.swap`.
     (See <a href="https://github.com/zig-lang/zig/issues/674">#674</a>)</li>
   <li><strong>Andrea Orru</strong> removed the deprecated Darwin target and added Zen OS target.
     (See <a href="https://github.com/zig-lang/zig/issues/438">#438</a>)</li>
   <li><strong>Andrea Orru</strong> added intrusive linked lists to the standard library.
     (See <a href="https://github.com/zig-lang/zig/issues/680">#680</a>)</li>
   <li><strong>Jimmi Holst Christensen</strong> fixed bigint xor with zero.
     (See <a href="https://github.com/zig-lang/zig/issues/701">#701</a>)</li>
   <li><strong>Jimmi Holst Christensen</strong> implemented windows versions of seekTo and getPos
     (See <a href="https://github.com/zig-lang/zig/issues/710">#710</a>)</li>
   <li><strong>Jeff Fowler</strong> sent a pull request to GitHub/linguist to add Zig syntax highlighting.
     (See <a href="https://github.com/github/linguist/pull/4005">github/linguist/pull/4005</a>)</li>
   <li><strong>Jeff Fowler</strong> Added Zig 0.1.1 to Homebrew.
     (See <a href="https://github.com/Homebrew/homebrew-core/pull/23189">Homebrew/homebrew-core/pull/23189</a>)</li>
 </ul>

   <h2>Thank you financial supporters!</h2>

<p>
Special thanks to those who <a href="https://github.com/users/andrewrk/sponsorship">donate monthly</a>. We're now at $207 of the $3,000 goal.
</p>

<ul>
  <li>Lauren Chavis</li>
  <li>Andrea Orru</li>
  <li>Adrian Sinclair</li>
  <li>David Joseph</li>
  <li>jeff kelley</li>
  <li>Hasen Judy</li>
  <li>Wesley Kelley</li>
  <li>Harry Eakins</li>
  <li>Richard Ohnemus</li>
  <li>Brendon Scheinman</li>
  <li>Martin Schwaighofer</li>
  <li>Matthew </li>
  <li>Mirek Rusin</li>
  <li>Jordan Torbiak</li>
  <li>Pyry Kontio</li>
  <li>Thomas Ballinger</li>
  <li>Peter Ronnquist</li>
  <li>Luke McCarthy</li>
  <li>Robert Paul Herman</li>
  <li>Audun Wilhelmsen</li>
  <li>Marko Mikulicic</li>
  <li>Jimmi Holst Christensen</li>
  <li>Caius </li>
  <li>Don Poor</li>
  <li>Anthony J. Benik</li>
  <li>David Hayden</li>
  <li>Tanner Schultz</li>
  <li>Tyler Philbrick</li>
  <li>Eduard Nicodei</li>
  <li>Christopher A. Butler</li>
  <li>Colleen Silva-Hayden</li>
  <li>Jeremy Larkin</li>
  <li>Rasmus Rønn Nielsen</li>
  <li>Brian Lewis</li>
  <li>Tom Palmer</li>
  <li>Josh McDonald</li>
  <li>Chad Russell</li>
  <li>Alexandra Gillis</li>
  <li>david karapetyan</li>
  <li>Zi He Goh</li>
</ul>


guid: https://andrewkelley.me/post/full-time-zig.html
title: I Quit My Cushy Job at OkCupid to Live on Donations to Zig
url: https://andrewkelley.me/post/full-time-zig.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
</script>
<h1>I Quit My Cushy Job at OkCupid to Live on Donations to Zig</h1>
<p>
I am fortunate to be one of those people who started tinkering with code
in their teens, and then found themselves in the position where their hobby
could not only pay rent, but afford a comfortable standard of living, even
in New York City.
</p>
<p>
For a little over a year now, I worked on the backend team at
<a href="https://www.okcupid.com/">OkCupid</a>, maintaining a large C++
codebase that has not been reworked since 2005 when
<a href="https://twitter.com/maxtaco">Maxwell Krohn</a>
published the original
<a href="https://pdos.csail.mit.edu/papers/okws:krohn-ms/okws_krohn-ms.pdf">OKWS paper</a>
on which OkCupid is still based.
</p>
<p>
As satisfying as it is to delete dead code, rework abstractions to remove footguns,
and migrate a monolithic codebase to
<a href="https://en.wikipedia.org/wiki/Service-oriented_architecture">Service Oriented Architecture</a>,
my coworkers were well aware that my real passion was spent on nights and weekends,
where I poured all the energy I could muster into <a href="https://ziglang.org/">Zig</a> -
if not from my incessant comparisons between snippets of C++ code and how it could be
better expressed in Zig, then from the tea mug that my lovely girlfriend Alee got me
as a celebratory gift for reaching
<a href="https://ziglang.org/download/#release-0.1.1">release 0.1.0</a>:
</p>
<img src="https://andrewkelley.me/img/zig-mug.jpg" style="width: 300px">
<h2>Zig is Picking Up Steam</h2>
<p>
Ever since I gave this
<a href="https://www.recurse.com/events/localhost-andrew-kelley">Software Should be Perfect</a> talk, it seems that the Zig community has seen a steady influx of new members. 
</p>
<img src="https://andrewkelley.me/img/david-zig.jpg" style="width: 200px">
<p>David keeps sending me selfies with this hat that he made.</p>
<p>
The Zig community grew in size so much, that some weekends I found myself
with only enough time to merge pull requests and respond to issues, and no
time leftover to make progress on big roadmap changes.
</p>
<p>
I found this both exciting, and frustrating. Here I was working on legacy code
40 hours per week, while multiple Zig issues needed my attention.
</p>
<p>
And so I took the plunge. I gave up my senior backend software engineer salary,
and will attempt to live on <a href="https://github.com/users/andrewrk/sponsorship">monthly donations</a>.
</p>
<h2>Roadmap</h2>
<p>Here are some of the bigger items that are coming up now that I have more time:</p>
<ul>
  <li><a href="https://github.com/ziglang/zig/issues/1023">remove more sigils from the language</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/208">add tuples and remove var args</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/89">self-hosted compiler</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/910">http server and http client based on async/await</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/943">decentralized package manager</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/21">generate html documentation</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/68">hot code swapping</a></li>
</ul>
<p>I can't wait to make significant progress on these items. That said, I will also
dedicate time each day for bug fixes and writing documentation.
Every weekday except Friday, this will be my wakeup process:
</p>
<ol>
  <li>Make tea</li>
  <li>Fix a bug</li>
  <li>Write some documentation</li>
  <li>Deal with all open pull requests, whether that is by merging, rejecting, requesting
    changes, or solving the use case a different way.</li>
  <li>Proceed with working on a big feature item.</li>
</ol>
<p>
On Fridays, I will spend the day working on a project other than Zig, but one that
is implemented in Zig. For example:
</p>
<ul>
  <li>Rewrite <a href="https://github.com/andrewrk/groovebasin">Groove Basin</a> in Zig.</li>
  <li>Work on my <a href="https://github.com/andrewrk/clashos">game that the raspberry pi boots directly into</a> (this explores Zig's ability to be used for embedded and Operating System development) </li>
  <li>Start reworking my <a href="http://genesisdaw.org/">digital audio workstation</a> in Zig.</li>
</ul>
<p>
I'm beyond excited to get Zig to a place where it can reasonably be used for
high quality and practical software projects. My goal is to make Zig so useful
and practical that people will find themselves using it without intending to.
</p>
<p>And so it comes down to this - will you fund my efforts?
</p>

<h2>Why Donate to Zig?</h2>
<p>
Consider the basic premise of a for-profit business:
</p>
<ul>
  <li>Customers pay currency in exchange for goods or services.</li>
</ul>
<p>
If all goes well, both parties benefit from the exchange. But for-profit
businesses are motivated, at the end of the day, not by customer satisfaction,
but by profit.
</p>
<p>
It's fine, this is how the world works. But it's not how Zig works.
</p>
<p>
Zig is created by the open source community, for the open source community.
One of our main <a href="https://ziglang.org/documentation/master/#Zen">tenets</a>
is</p>
<blockquote>Together we serve end users.</blockquote>
<p>
The Zig project is a public service, entirely motivated by improving the technical
landscape of open source tools. There is no board to please, no stock shareholders
demanding higher quarterly earnings, no office politics or career progressions on
the line. 100% of donations I receive go towards paying rent, buying food, and
generally attempting to live a modest, but healthy life.
</p>
<p>
<a href="https://github.com/users/andrewrk/sponsorship">Will you pledge $5/month?</a>
</p>


guid: https://andrewkelley.me/post/string-matching-comptime-perfect-hashing-zig.html
title: String Matching based on Compile Time Perfect Hashing in Zig
url: https://andrewkelley.me/post/string-matching-comptime-perfect-hashing-zig.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(test|fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|while|for|null|undefined|true|false|comptime)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong)\b/,
  });
  Prism.languages['llvm'] = Prism.languages.extend('clike', {
    'keyword': /\b(private|constant|declare|define|c|noreturn|nounwind|alloca|br|store|load|getelementptr|and|icmp|eq|zext|call|unreachable|add|target|datalayout|triple|unnamed_addr|align|inbounds|uwtable|sext|ret|ne|phi|global|to|zeroinitializer|shl|or|ult|switch)\b/,
    'number': null,
    'comment': /;.*/,
    'operator': /\w[\w\d_]*:/,
    'regex': /[@%]\.?\w+/,
    'property': /\b(i8|i1|i32|i64|i16|void|label)\*?\b/,
    'punctuation': null,
    'function': null,
  });
</script>
<h1>String Matching based on Compile Time Perfect Hashing in Zig</h1>
<p>Inspired by <a href="https://smilingthax.github.io/slides/cttrie/">cttrie - Compile time TRIE based string matching</a>,
I decided to see what this solution would look like in Zig.
</p>
<p>
Here's the API I came up with:
</p>
<pre><code class="language-zig">const ph = perfectHash([][]const u8{
    "a",
    "ab",
    "abc",
});
switch (ph.hash(target)) {
    ph.case("a") =&gt; std.debug.warn("handle the a case"),
    ph.case("ab") =&gt; std.debug.warn("handle the ab case"),
    ph.case("abc") =&gt; std.debug.warn("handle the abc case"),
    else =&gt; unreachable,
}</code></pre>
<p>
It notices at compile-time if you forget to declare one of the cases. For example, if I
comment out the last item:
</p>
<pre><code class="language-zig">const ph = perfectHash([][]const u8{
    "a",
    "ab",
    //"abc",
});</code></pre>
<p>When compiled this gives:</p>
<pre>perfect-hashing.zig:147:13: error: case value 'abc' not declared
            @compileError("case value '" ++ s ++ "' not declared");
            ^
perfect-hashing.zig:18:16: note: called from here
        ph.case("abc") =&gt; std.debug.warn("handle the abc case\n"),
               ^</pre>
<p>
It also has runtime safety if you pass in a string that was not prepared for:
</p>
<pre><code class="language-zig">const std = @import("std");
const assert = std.debug.assert;

test "perfect hashing" {
    basedOnLength("zzz");
}

fn basedOnLength(target: []const u8) void {
    const ph = perfectHash([][]const u8{
        "a",
        "ab",
        "abc",
    });
    switch (ph.hash(target)) {
        ph.case("a") =&gt; @panic("wrong one a"),
        ph.case("ab") =&gt; {}, // test pass
        ph.case("abc") =&gt; @panic("wrong one abc"),
        else =&gt; unreachable,
    }
}</code></pre>
<p>When run this gives:</p>
<pre>Test 1/1 perfect hashing...attempt to perfect hash zzz which was not declared
perfect-hashing.zig:156:36: 0x205a21 in ??? (test)
                    std.debug.panic("attempt to perfect hash {} which was not declared", s);
                                   ^
perfect-hashing.zig:15:20: 0x2051bd in ??? (test)
    switch (ph.hash(target)) {
                   ^
perfect-hashing.zig:5:18: 0x205050 in ??? (test)
    basedOnLength("zzz");
                 ^
/home/andy/dev/zig/build/lib/zig/std/special/test_runner.zig:13:25: 0x2238ea in ??? (test)
        if (test_fn.func()) |_| {
                        ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:96:22: 0x22369b in ??? (test)
            root.main() catch |err| {
                     ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:70:20: 0x223615 in ??? (test)
    return callMain();
                   ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:64:39: 0x223478 in ??? (test)
    std.os.posix.exit(callMainWithArgs(argc, argv, envp));
                                      ^
/home/andy/dev/zig/build/lib/zig/std/special/bootstrap.zig:37:5: 0x223330 in ??? (test)
    @noInlineCall(posixCallMainAndExit);
    ^

Tests failed. Use the following command to reproduce the failure:
/home/andy/dev/zig/build/zig-cache/test</pre>
<p>
So there's the API. How does it work?
</p>
<p>
Here's the implementation of the <code>perfectHash</code> function:
</p>
<pre><code class="language-zig">fn perfectHash(comptime strs: []const []const u8) type {
    const Op = union(enum) {
        /// add the length of the string
        Length,

        /// add the byte at index % len
        Index: usize,

        /// right shift then xor with constant
        XorShiftMultiply: u32,
    };
    const S = struct {
        fn hash(comptime plan: []Op, s: []const u8) u32 {
            var h: u32 = 0;
            inline for (plan) |op| {
                switch (op) {
                    Op.Length =&gt; {
                        h +%= @truncate(u32, s.len);
                    },
                    Op.Index =&gt; |index| {
                        h +%= s[index % s.len];
                    },
                    Op.XorShiftMultiply =&gt; |x| {
                        h ^= x &gt;&gt; 16;
                    },
                }
            }
            return h;
        }

        fn testPlan(comptime plan: []Op) bool {
            var hit = [1]bool{false} ** strs.len;
            for (strs) |s| {
                const h = hash(plan, s);
                const i = h % hit.len;
                if (hit[i]) {
                    // hit this index twice
                    return false;
                }
                hit[i] = true;
            }
            return true;
        }
    };

    var ops_buf: [10]Op = undefined;

    const plan = have_a_plan: {
        var seed: u32 = 0x45d9f3b;
        var index_i: usize = 0;
        const try_seed_count = 50;
        const try_index_count = 50;

        while (index_i &lt; try_index_count) : (index_i += 1) {
            const bool_values = if (index_i == 0) []bool{true} else []bool{ false, true };
            for (bool_values) |try_length| {
                var seed_i: usize = 0;
                while (seed_i &lt; try_seed_count) : (seed_i += 1) {
                    comptime var rand_state = std.rand.Xoroshiro128.init(seed + seed_i);
                    const rng = &amp;rand_state.random;

                    var ops_index = 0;

                    if (try_length) {
                        ops_buf[ops_index] = Op.Length;
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];

                        ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];
                    }

                    ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                    ops_index += 1;

                    if (S.testPlan(ops_buf[0..ops_index]))
                        break :have_a_plan ops_buf[0..ops_index];

                    const before_bytes_it_index = ops_index;

                    var byte_index = 0;
                    while (byte_index &lt; index_i) : (byte_index += 1) {
                        ops_index = before_bytes_it_index;

                        ops_buf[ops_index] = Op{ .Index = rng.scalar(u32) % try_index_count };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];

                        ops_buf[ops_index] = Op{ .XorShiftMultiply = rng.scalar(u32) };
                        ops_index += 1;

                        if (S.testPlan(ops_buf[0..ops_index]))
                            break :have_a_plan ops_buf[0..ops_index];
                    }
                }
            }
        }

        @compileError("unable to come up with perfect hash");
    };

    return struct {
        fn case(comptime s: []const u8) usize {
            inline for (strs) |str| {
                if (std.mem.eql(u8, str, s))
                    return hash(s);
            }
            @compileError("case value '" ++ s ++ "' not declared");
        }
        fn hash(s: []const u8) usize {
            if (std.debug.runtime_safety) {
                const ok = for (strs) |str| {
                    if (std.mem.eql(u8, str, s))
                        break true;
                } else false;
                if (!ok) {
                    std.debug.panic("attempt to perfect hash {} which was not declared", s);
                }
            }
            return S.hash(plan, s) % strs.len;
        }
    };
}</code></pre>
<p>
Here's what this is doing:
</p>
<ul>
  <li>Create the concept of a hashing operation. The operations that can happen are:
    <ul>
      <li><code>Length</code> - use wraparound addition to add the length of the string to the hash value.</li>
      <li><code>Index</code> - use wraparound addition to add the byte from the string at the specified index.
      Use modulus arithmetic in case the index is out of bounds.</li>
      <li><code>XorShiftMultiply</code> - perform a right shift by 16 bits of the hash value and then xor with
        the specified value (which will come from a random number generator executed at compile time).</li>
    </ul>
  </li>
  <li>Next, we're going to try a series of "plans" which is a sequence of operations strung together.
    We have this function <code>testPlan</code> which performs the hash operations on all the strings
    and sees if there are any collisions. If we ever find a plan that results in no collisions, we have
    found a perfect hashing strategy.
  </li>
  <li>
    First we test a plan that is simply the <code>Length</code> operation. If this works, then all the hash
    function has to do is take the length of the string mod the number of strings. Easy. You may notice
    this is true for the above example. Don't worry, I have a more complicated example below.
  </li>
  <li>
    Next we iterate over the count of how many different bytes we are willing to look at in the hash function.
    We start with 0. If we can xor the length with a random number and it fixes the collisions, then we're done.
    We try 50 seeds before giving up and deciding to inspect a random byte from the string in the hash function.
    If the addition of inspecting a random byte from the string to hash doesn't solve the problem,
    we try 50 seeds in order to choose different random byte indexes. If that still doesn't work,
    we look at 2 random bytes, again trying 50 different seeds with these 2 bytes.
    And so on until every combination of 50 seeds x 50 bytes inspected, at which point we give up and
    emit a compile error "unable to come up with perfect hash".
  </li>
</ul>
<p>We can use <code>@compileLog</code> to see the plan that the function came up with:</p>
<pre><code class="language-zig">for (plan) |op| {
    @compileLog(@TagType(Op)(op));
}</code></pre>
<p>This outputs:</p>
<pre>
| @TagType(Op).Length
</pre>
<p>So this means that the hash function only has to look at the length for this example.</p>
<p>The nice thing about this is that the plan is all known at compile time. Indeed, you can see
that the <code>hash</code> function uses <a href="https://ziglang.org/documentation/master/#inline-for">inline for</a>
to iterate over the operations in the plan.
This means that LLVM is able to fully optimize the hash function.
</p>
<p>Here's what it gets compiled to in release mode for x86_64:</p>
<pre>
0000000000000000 &lt;basedOnLength&gt;:
   0:	89 f0                	mov    %esi,%eax
   2:	b9 ab aa aa aa       	mov    $0xaaaaaaab,%ecx
   7:	48 0f af c8          	imul   %rax,%rcx
   b:	48 c1 e9 21          	shr    $0x21,%rcx
   f:	8d 04 49             	lea    (%rcx,%rcx,2),%eax
  12:	29 c6                	sub    %eax,%esi
  14:	48 89 f0             	mov    %rsi,%rax
  17:	c3                   	retq   
  18:	0f 1f 84 00 00 00 00 	nopl   0x0(%rax,%rax,1)
  1f:	00 
</pre>
<p>You can see there is not even a jump instruction in there. And it will output numbers in sequential order from 0,
so that the switch statement can be a jump table. Here is the LLVM IR of the switch:</p>
<pre><code class="language-llvm"> %2 = call fastcc i64 @"perfectHash((struct []const []const u8 constant))_hash"(%"[]u8"* %0), !dbg !736
  store i64 %2, i64* %1, align 8, !dbg !736
  %3 = load i64, i64* %1, align 8, !dbg !740
  switch i64 %3, label %SwitchElse [
    i64 1, label %SwitchProng
    i64 2, label %SwitchProng1
    i64 0, label %SwitchProng2
  ], !dbg !740</code></pre>
<p>
How about a harder example?
</p>
<pre><code class="language-zig">@setEvalBranchQuota(100000);
const ph = perfectHash([][]const u8{
    "one",
    "two",
    "three",
    "four",
    "five",
});
switch (ph.hash(target)) {
    ph.case("one") =&gt; std.debug.warn("handle the one case"),
    ph.case("two") =&gt; std.debug.warn("handle the two case"),
    ph.case("three") =&gt; std.debug.warn("handle the three case"),
    ph.case("four") =&gt; std.debug.warn("handle the four case"),
    ph.case("five") =&gt; std.debug.warn("handle the five case"),
    else =&gt; unreachable,
}</code></pre>
<p>
This example is interesting because there are 2 pairs of length collisions (one/two, four/five) and 2 pairs of byte collisions (two/three, four/five).
</p>
<p>
Here we have to use <a href="https://ziglang.org/documentation/master/#setEvalBranchQuota">@setEvalBranchQuota</a>
because it takes a bit of computation to come up with the answer.
</p>
<p>
Again, the hash function comes up with a mapping to 0, 1, 2, 3, 4 (but not necessarily in the same order as specified):
</p>
<pre><code class="language-llvm"> %2 = call fastcc i64 @"perfectHash((struct []const []const u8 constant))_hash.11"(%"[]u8"* %0), !dbg !749
  store i64 %2, i64* %1, align 8, !dbg !749
  %3 = load i64, i64* %1, align 8, !dbg !753
  switch i64 %3, label %SwitchElse [
    i64 4, label %SwitchProng
    i64 2, label %SwitchProng1
    i64 3, label %SwitchProng2
    i64 0, label %SwitchProng3
    i64 1, label %SwitchProng4
  ], !dbg !753</code></pre>
<p>
And the optimized assembly:
</p>
<pre>0000000000000020 &lt;basedOnOtherStuff&gt;:
  20:	48 83 fe 22          	cmp    $0x22,%rsi
  24:	77 0b                	ja     31 &lt;basedOnOtherStuff+0x11&gt;
  26:	b8 22 00 00 00       	mov    $0x22,%eax
  2b:	31 d2                	xor    %edx,%edx
  2d:	f7 f6                	div    %esi
  2f:	eb 05                	jmp    36 &lt;basedOnOtherStuff+0x16&gt;
  31:	ba 22 00 00 00       	mov    $0x22,%edx
  36:	0f b6 04 17          	movzbl (%rdi,%rdx,1),%eax
  3a:	05 ef 3d 00 00       	add    $0x3def,%eax
  3f:	b9 cd cc cc cc       	mov    $0xcccccccd,%ecx
  44:	48 0f af c8          	imul   %rax,%rcx
  48:	48 c1 e9 22          	shr    $0x22,%rcx
  4c:	8d 0c 89             	lea    (%rcx,%rcx,4),%ecx
  4f:	29 c8                	sub    %ecx,%eax
  51:	c3                   	retq   </pre>
<p>
We have a couple of jumps here, but no loop. This code looks at only 1 byte from
the target string to determine the corresponding case index.
We can see that more clearly with the <code>@compileLog</code> snippet from earlier:
</p>
<pre>| @TagType(Op).XorShiftMultiply
| @TagType(Op).Index</pre>
<p>
So it initializes the hash with a constant value of predetermined random bits, then adds
the byte from a randomly chosen index of the string. Using <code>@compileLog(plan[1].Index)</code>
I determined that it is choosing the value 34, which means that:
</p>
<ul>
  <li>For <code>one</code>, 34 % 3 == 1, it looks at the 'n'</li>
  <li>For <code>two</code>, 34 % 3 == 1, it looks at the 'w'</li>
  <li>For <code>three</code>, 34 % 5 == 4, it looks at the 'e'</li>
  <li>For <code>four</code>, 34 % 4 == 2, it looks at the 'u'</li>
  <li>For <code>five</code>, 34 % 4 == 2, it looks at the 'i'</li>
</ul>
<p>
So the perfect hashing exploited that these bytes are all different, and when combined with
the random constant that it found, the final modulus spreads out the value across the full
range of 0, 1, 2, 3, 4.
</p>
<h3>Can we do better?</h3>
<p>
There are lots of ways this can be improved - this is just a proof of concept.
For example, we could start by looking for a perfect hash in the subset of bytes that are
within the smallest string length. For example, if we used index 1 in the above example,
we could avoid the jumps and remainder division instructions in the hash function.
</p>
<p>
Another way this can be improved, to reduce compile times, is to have
the <code>perfectHash</code> function accept the RNG seed as a parameter.
If the seed worked first try, great. Otherwise the function would find the seed
that does work, and emit a compile error instructing the programmer to switch
the seed argument to the good one, saving time in future compilations.
</p>
<h3>Conclusion</h3>
<p>
If you like this, you should check out <a href="https://ziglang.org/">Zig</a>.
Consider <a href="https://github.com/users/andrewrk/sponsorship">becoming a sponsor</a>.
</p>


guid: https://andrewkelley.me/post/zig-stack-traces-kernel-panic-bare-bones-os.html
title: Using Zig to Provide Stack Traces on Kernel Panic for a Bare Bones Operating System
url: https://andrewkelley.me/post/zig-stack-traces-kernel-panic-bare-bones-os.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(test|fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|try|catch|while|for|null|undefined|true|false|comptime|setCold|ptrToInt|returnAddress)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong|noreturn)\b/,
  });
</script>
<h1>Using Zig to Provide Stack Traces on Kernel Panic for a Bare Bones Operating System</h1>
<p>Last week, I reached an exciting milestone in my career as a programmer.
<a href="https://www.youtube.com/watch?v=gihdpLtHi9Q">For the first time in my life, I ran code directly on hardware</a>, with no Operating System sitting between the bare metal and my code.
</p>
<p>
For some context, the goal of this side project is to create a 2-4 player arcade game
running directly on a Raspberry Pi 3+.
</p>
<img src="https://andrewkelley.me/img/clashos-button-small.jpg">
<p>
Having just gotten Hello World working, this software can do
little more than send a message over the serial uART on bootup:
</p>
<pre>
Hello World! ClashOS 0.0
</pre>
<p>
So what's the next step? Bootloader? File system? <em>Graphics?!</em>
</p>
<p>
Well, when coding all of these things, I'm inevitably going to run into bugs
and crashes. And when this happens, I want to quickly understand what's gone
wrong. Normally, in Zig, when something goes wrong, you get a nice
<a href="https://ziglang.org/download/0.3.0/release-notes.html#stack-traces">stack trace</a>,
like this:
</p>
<img src="https://ziglang.org/download/0.3.0/stack-traces-linux.png">
<p>This example, however, is targeting Linux, whereas this arcade game project is
<em>freestanding</em>. The equivalent code in freestanding actually just hangs
the CPU:
</p>
<pre><code class="language-diff">--- a/src/main.zig
+++ b/src/main.zig
     serial.log("Hello World! ClashOS 0.0\n");
<span class="diff1">+    var x: u8 = 255;
+    x += 1;
+    serial.log("got here\n");</span></code></pre>
<p>When run, you'll see we never get to the "got here" message, but also
we don't get any kind of error message printed or anything.</p>
<pre>
Hello World! ClashOS 0.0
</pre>
<p>To understand this, we can look at the <strong>default panic handler</strong>.
In Zig, you can provide a <code class="language-zig">pub fn panic</code> in your
<a href="https://ziglang.org/documentation/master/#Root-Source-File">root source file</a>
alongside <code class="language-zig">pub fn main</code>. But if you do not provide this
function, <a href="https://github.com/ziglang/zig/blob/0.3.0/std/special/panic.zig">the default</a>
is used:
</p>
<pre><code class="language-zig">pub fn panic(msg: []const u8, error_return_trace: ?*builtin.StackTrace) noreturn {
    @setCold(true);
    switch (builtin.os) {
        builtin.Os.freestanding =&gt; {
            while (true) {}
        },
        else =&gt; {
            const first_trace_addr = @ptrToInt(@returnAddress());
            std.debug.panicExtra(error_return_trace, first_trace_addr, "{}", msg);
        },
    }
}</code></pre>
<p>
Here we can see why the code earlier is hanging - the default panic handler for the
freestanding target is simply <code class="language-zig">while(true) {}</code>.
</p>
<p>
So we can make an immediate improvement by creating our own panic handler:
</p>
<pre><code class="language-zig">pub fn panic(message: []const u8, stack_trace: ?*builtin.StackTrace) noreturn {
    serial.write("\n!KERNEL PANIC!\n");
    serial.write(message);
    serial.write("\n");
    while(true) {}
}</code></pre>
<p>And now, the output of booting the Raspberry Pi:</p>
<pre>
Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow
</pre>
<p>
Already this is much better. We can see that an integer overflow caused a kernel panic.
But an integer overflow could occur anywhere. Wouldn't it be nice to have a full stack
trace printed?
</p>
<p>
Yes, yes it would. The first thing I needed to make this work is access to the DWARF
debugging info from inside the kernel. But I don't even have a file system. How can that work?
</p>
<p>
Easy! Just put the DWARF info directly into the kernel's memory. I modified my linker script
to do just that:
</p>
<pre><code class="language-ld">    .rodata : ALIGN(4K) {
        *(.rodata)
        __debug_info_start = .;
        KEEP(*(.debug_info))
        __debug_info_end = .;
        __debug_abbrev_start = .;
        KEEP(*(.debug_abbrev))
        __debug_abbrev_end = .;
        __debug_str_start = .;
        KEEP(*(.debug_str))
        __debug_str_end = .;
        __debug_line_start = .;
        KEEP(*(.debug_line))
        __debug_line_end = .;
        __debug_ranges_start = .;
        KEEP(*(.debug_ranges))
        __debug_ranges_end = .;
    }</code></pre>
<p>
And to my dismay, these error messages stared back at me:
</p>
<pre>
lld: error: incompatible section flags for .rodata
&gt;&gt;&gt; /home/andy/dev/clashos/zig-cache/clashos.o:(.debug_info): 0x0
&gt;&gt;&gt; output section .rodata: 0x12

lld: error: incompatible section flags for .rodata
&gt;&gt;&gt; &lt;internal&gt;:(.debug_str): 0x30
&gt;&gt;&gt; output section .rodata: 0x12

lld: error: incompatible section flags for .rodata
&gt;&gt;&gt; /home/andy/dev/clashos/zig-cache/clashos.o:(.debug_line): 0x0
&gt;&gt;&gt; output section .rodata: 0x32
</pre>
<p>
After a <a href="https://bugs.llvm.org/show_bug.cgi?id=39862">back and forth on a bug report</a>, 
George Rimar suggested simply deleting that particular check, as it might have been an
overly strict enforcement. I tried this in my LLD fork, and it worked! Debug information
was now linked into my kernel images. After completing the rest of the steps in this blog post,
I <a href="https://reviews.llvm.org/D55276">submitted a patch upstream</a>,
which Rui has already merged into LLD. This will be released with
LLVM 8, and in the meantime Zig's LLD fork has the patch.
</p>
<p>
At this point it was a simple matter of writing the glue code between my kernel and the
Zig Standard Library's stack trace facilities. In Zig, you don't have to intentionally support
freestanding mode. Code which has no dependencies on a particular operating system will work
in freestanding mode thanks to Zig's <em>lazy top level declaration analysis</em>.
Because the standard library stack trace code does not call any OS API, it therefore
supports freestanding mode.
</p>
<p>
The Zig std lib API for opening debug information from an ELF file looks like this:
</p>
<pre><code class="language-zig">pub fn openElfDebugInfo(
    allocator: *mem.Allocator,
    elf_seekable_stream: *DwarfSeekableStream,
    elf_in_stream: *DwarfInStream,
) !DwarfInfo</code></pre>
<p>
But this kernel is so bare bones, it's not even in an ELF file. It's booting directly from a binary blob.
We just have the debug info sections mapped directly into memory. For that we can use
a lower level API:
<p>
<pre><code class="language-zig">/// Initialize DWARF info. The caller has the responsibility to initialize most
/// the DwarfInfo fields before calling. These fields can be left undefined:
/// * abbrev_table_list
/// * compile_unit_list
pub fn openDwarfDebugInfo(di: *DwarfInfo, allocator: *mem.Allocator) !void</code></pre>
<p>And <code class="language-zig">DwarfInfo</code> is defined like this:</p>
<pre><code class="language-zig">pub const DwarfInfo = struct {
    dwarf_seekable_stream: *DwarfSeekableStream,
    dwarf_in_stream: *DwarfInStream,
    endian: builtin.Endian,
    debug_info: Section,
    debug_abbrev: Section,
    debug_str: Section,
    debug_line: Section,
    debug_ranges: ?Section,
    abbrev_table_list: ArrayList(AbbrevTableHeader),
    compile_unit_list: ArrayList(CompileUnit),
};</code></pre>
<p>
To hook these up, the glue code needs to initialize the fields of
<code class="language-zig">DwarfInfo</code> with the offsets into a
<code class="language-zig">std.io.SeekableStream</code>, which can
be implemented as a simple pointer to memory. By declaring <strong>external variables</strong>
and then looking at their <em>addresses</em>, we can find out where in memory the symbols
defined in the linker script are.
</p>
<p>
For <code>.debug_abbrev</code>, <code>.debug_str</code>, and <code>.debug_ranges</code>, I had to
set the offset to 0 to
<a href="https://bugs.llvm.org/show_bug.cgi?id=39862#c6">workaround</a>
LLD thinking that the sections start at 0 for some reason.
</p>
<pre><code class="language-zig">var kernel_panic_allocator_bytes: [100 * 1024]u8 = undefined;
var kernel_panic_allocator_state = std.heap.FixedBufferAllocator.init(kernel_panic_allocator_bytes[0..]);
const kernel_panic_allocator = &amp;kernel_panic_allocator_state.allocator;

extern var __debug_info_start: u8;
extern var __debug_info_end: u8;
extern var __debug_abbrev_start: u8;
extern var __debug_abbrev_end: u8;
extern var __debug_str_start: u8;
extern var __debug_str_end: u8;
extern var __debug_line_start: u8;
extern var __debug_line_end: u8;
extern var __debug_ranges_start: u8;
extern var __debug_ranges_end: u8;

fn dwarfSectionFromSymbolAbs(start: *u8, end: *u8) std.debug.DwarfInfo.Section {
    return std.debug.DwarfInfo.Section{
        .offset = 0,
        .size = @ptrToInt(end) - @ptrToInt(start),
    };
}

fn dwarfSectionFromSymbol(start: *u8, end: *u8) std.debug.DwarfInfo.Section {
    return std.debug.DwarfInfo.Section{
        .offset = @ptrToInt(start),
        .size = @ptrToInt(end) - @ptrToInt(start),
    };
}

fn getSelfDebugInfo() !*std.debug.DwarfInfo {
    const S = struct {
        var have_self_debug_info = false;
        var self_debug_info: std.debug.DwarfInfo = undefined;

        var in_stream_state = std.io.InStream(anyerror){ .readFn = readFn };
        var in_stream_pos: usize = 0;
        const in_stream = &amp;in_stream_state;

        fn readFn(self: *std.io.InStream(anyerror), buffer: []u8) anyerror!usize {
            const ptr = @intToPtr([*]const u8, in_stream_pos);
            @memcpy(buffer.ptr, ptr, buffer.len);
            in_stream_pos += buffer.len;
            return buffer.len;
        }

        const SeekableStream = std.io.SeekableStream(anyerror, anyerror);
        var seekable_stream_state = SeekableStream{
            .seekToFn = seekToFn,
            .seekForwardFn = seekForwardFn,

            .getPosFn = getPosFn,
            .getEndPosFn = getEndPosFn,
        };
        const seekable_stream = &amp;seekable_stream_state;

        fn seekToFn(self: *SeekableStream, pos: usize) anyerror!void {
            in_stream_pos = pos;
        }
        fn seekForwardFn(self: *SeekableStream, pos: isize) anyerror!void {
            in_stream_pos = @bitCast(usize, @bitCast(isize, in_stream_pos) +% pos);
        }
        fn getPosFn(self: *SeekableStream) anyerror!usize {
            return in_stream_pos;
        }
        fn getEndPosFn(self: *SeekableStream) anyerror!usize {
            return @ptrToInt(&amp;__debug_ranges_end);
        }
    };
    if (S.have_self_debug_info) return &amp;S.self_debug_info;

    S.self_debug_info = std.debug.DwarfInfo{
        .dwarf_seekable_stream = S.seekable_stream,
        .dwarf_in_stream = S.in_stream,
        .endian = builtin.Endian.Little,
        .debug_info = dwarfSectionFromSymbol(&amp;__debug_info_start, &amp;__debug_info_end),
        .debug_abbrev = dwarfSectionFromSymbolAbs(&amp;__debug_abbrev_start, &amp;__debug_abbrev_end),
        .debug_str = dwarfSectionFromSymbolAbs(&amp;__debug_str_start, &amp;__debug_str_end),
        .debug_line = dwarfSectionFromSymbol(&amp;__debug_line_start, &amp;__debug_line_end),
        .debug_ranges = dwarfSectionFromSymbolAbs(&amp;__debug_ranges_start, &amp;__debug_ranges_end),
        .abbrev_table_list = undefined,
        .compile_unit_list = undefined,
    };
    try std.debug.openDwarfDebugInfo(&amp;S.self_debug_info, kernel_panic_allocator);
    return &amp;S.self_debug_info;
}</code></pre>
<p>
You can see that the Zig common practice of accepting an allocator as an argument when
allocation is needed comes in extremely handy for kernel development. We simply statically
allocate a 100 KiB buffer and pass that along as the debug info allocator. If this ever
becomes too small, it can be adjusted.
</p>
<p>
And now it's just a matter of wiring up the <code>panic</code> function:
</p>
<pre><code class="language-zig">pub fn panic(message: []const u8, stack_trace: ?*builtin.StackTrace) noreturn {
    serial.log("\n!KERNEL PANIC! {}\n", message);
    const dwarf_info = getSelfDebugInfo() catch |err| {
        serial.log("unable to get debug info: {}\n", @errorName(err));
        hang();
    };
    const first_trace_addr = @ptrToInt(@returnAddress());
    var it = std.debug.StackIterator.init(first_trace_addr);
    while (it.next()) |return_address| {
        std.debug.printSourceAtAddressDwarf(
            dwarf_info,
            serial_out_stream,
            return_address,
            true, // tty color on
            printLineFromFile,
        ) catch |err| {
            serial.log("missed a stack frame: {}\n", @errorName(err));
            continue;
        };
    }
    hang();
}

fn hang() noreturn {
    while (true) {}
}

fn printLineFromFile(out_stream: var, line_info: std.debug.LineInfo) anyerror!void {
    serial.log("TODO print line from the file\n");
}</code></pre>
<p>
Here it is in action:
</p>
<pre>
Hello World! ClashOS 0.0

!KERNEL PANIC! integer overflow
/home/andy/dev/clashos/src/main.zig:166:7: 0x15e0 in ??? (clashos)
TODO print line from the file
      ^
???:?:?: 0x1c in ??? (???)
</pre>
<p>
Great progress.
</p>
<p>That last line is coming from the startup assembly code, which has no
source mapping. But we can see that it's correct, by looking at
the disassembly of the kernel:</p>
<pre>
0000000000000000 &lt;_start&gt;:
       0:	d53800a0 	mrs	x0, mpidr_el1
       4:	d2b82001 	mov	x1, #0xc1000000
       8:	8a210000 	bic	x0, x0, x1
       c:	b4000040 	cbz	x0, 14 &lt;master&gt;
      10:	14000003 	b	1c &lt;__hang&gt;

0000000000000014 &lt;master&gt;:
      14:	b26503ff 	mov	sp, #0x8000000
      18:	9400054b 	bl	1544 &lt;kernel_main&gt;

000000000000001c &lt;__hang&gt;:
      1c:	d503205f 	wfe
      20:	17ffffff 	b	1c &lt;__hang&gt;
</pre>
<p>You can see that <code>0x1c</code> is indeed the return address
(the next instruction that will be executed when the function returns)
of the function call to <code>kernel_main</code>.
</p>
<p>
Let's shuffle the code around into more files and make that trace longer.
</p>
<pre>
Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow
/home/andy/dev/clashos/src/serial.zig:42:7: 0x1b10 in ??? (clashos)
TODO print line from the file
      ^
/home/andy/dev/clashos/src/main.zig:58:16: 0x1110 in ??? (clashos)
TODO print line from the file
               ^
/home/andy/dev/clashos/src/main.zig:67:18: 0xecc in ??? (clashos)
TODO print line from the file
                 ^
???:?:?: 0x1c in ??? (???)
</pre>
<p>
It's looking good. But can we add a cherry on top, and make it print the source
lines?
</p>
<p>
Again we don't have a file system. How can the <code>printLineFromFile</code> function
have access to source files?
</p>
<p>
Sometimes the simplest solution is the best one. How about the kernel just has its own
source code in memory?
</p>
<p>
That's really easy to hook up in Zig:
</p>
<pre><code class="language-zig">const source_files = [][]const u8{
    "src/debug.zig",
    "src/main.zig",
    "src/mmio.zig",
    "src/serial.zig",
};
fn printLineFromFile(out_stream: var, line_info: std.debug.LineInfo) anyerror!void {
    inline for (source_files) |src_path| {
        if (std.mem.endsWith(u8, line_info.file_name, src_path)) {
            const contents = @embedFile("../" ++ src_path);
            try printLineFromBuffer(out_stream, contents[0..], line_info);
            return;
        }
    }
    try out_stream.print("(source file {} not added in std/debug.zig)\n", line_info.file_name);
}
</code></pre>
<p>
Here we take advantage of <a href="https://ziglang.org/documentation/master/#inline-for">inline for</a>
as well as <a href="https://ziglang.org/documentation/master/#embedFile">@embedFile</a>, and all the sudden
we can print lines of code from our own source files. <code>printLineFromBuffer</code>
<a href="https://github.com/andrewrk/clashos/blob/bf8e57ac220715d0698ab910d337ea590c4b4e33/src/debug.zig#L133-L152">left as an exercise for the reader</a>.
</p>
<p>
So let's see that output again:
</p>
<pre>
Hello World! ClashOS 0.0

!KERNEL PANIC!
integer overflow
/home/andy/dev/clashos/src/serial.zig:42:7: 0x1b10 in ??? (clashos)
    x += 1;
      ^
/home/andy/dev/clashos/src/main.zig:58:16: 0x1110 in ??? (clashos)
    serial.boom();
               ^
/home/andy/dev/clashos/src/main.zig:67:18: 0xecc in ??? (clashos)
    some_function();
                 ^
???:?:?: 0x1c in ??? (???)
</pre>
<p>Beautiful. Here's a picture so you can see it with color:</p>
<img src="https://andrewkelley.me/img/clashos-stack-trace.png">
<p>
And now all of the
<a href="https://ziglang.org/documentation/master/#Undefined-Behavior">protections that Zig offers against undefined behavior</a>
will result in output like this.
</p>

<h3>Conclusion</h3>
<p>
One of my big goals with Zig is to improve the embedded and OS development
process. I hope you're as excited as I am about the potential here.
</p>
<p>
If this blog post captured your interest, maybe you would like to check out this
<a href="https://github.com/andrewrk/HellOS">Hello World x86 Kernel</a>, which comes with a
<a href="https://www.youtube.com/watch?v=yUge-ujPxzQ">video of me live coding it</a>.
Thanks to an insightful Twitch comment, it works with only Zig code, no assembly required.
</p>


guid: https://andrewkelley.me/post/why-donating-to-musl-libc-project.html
title: Why I'm donating $150/month (10% of my income) to the musl libc project
url: https://andrewkelley.me/post/why-donating-to-musl-libc-project.html
feedurl: https://andrewkelley.me/rss.xml
content:
<style>
#prog {
  background-color: #171717;
  width: 350px;
  height: 30px;
  border: 1px solid black;
}
#done {
  background-color: #309e07;
  width: 100%;
  height: 30px;
  line-height: 30px;
  text-align: center;
  color: black;
  font-weight: bold;
}
</style>
<h1>Why I'm donating $150/month (10% of my income) to the musl libc project</h1>
<p>
One year ago, <a href="/post/full-time-zig.html">I quit my day job to work on Zig full time</a>.
Since then, the project has seen a steady growth in funding. Thanks to the people donating,
Zig is on track to become fully sustainable before my savings run out.
</p>
<p>
This support has allowed me to focus on steady improvements to the language and tooling. In the last
year, Zig has released two versions:
</p>
<ul>
  <li>0.3.0 (<a href="https://ziglang.org/download/0.3.0/release-notes.html">release notes</a>)</li>
  <li>0.4.0 (<a href="https://ziglang.org/download/0.4.0/release-notes.html">release notes</a>)</li>
</ul>
<p>
These release notes serve as my accountability to people who donate, and if you have a look at them, I hope you can agree with me that they speak for themselves.
</p>
<h2>The V language drama</h2>
<p>
I know that open-source project funding has been on people's minds lately, as we've watched the Internet's reaction to the
bizarre open-sourcing of the V language unfold. As to how this relates to musl libc, stick with me - let me give you the talking
points of this situation:
</p>
<ol>
  <li>
    The V language website was published several months ago, claiming that it could already do incredible things, such as automatically
    translate C++ code to V code, and compile 1.2 million lines of code per second per CPU core by directly outputting x64 machine code.
    These features already work today, it said, and they will be released soon. Please donate. Meanwhile, the project was closed-source.
    Binaries and release notes were behind a paywall.
  </li>
  <li>
    Over the next few months, people saw these amazing things it was claimed that V could do, and started donating money.
    It even looked like the language was already available, with download links including file sizes and file names, but if you
    clicked the download link, it did an alert() pop-up box saying "June 20". The project climbed up to $927/month in donations.
  </li>
  <li>
    On 2019-06-20, the V author released a macOS binary of the V language, and it immediately became apparent that the many fantastic
    claims on the V website were unsubstantiated. People were quick to point this out,
    <a href="https://news.ycombinator.com/item?id=20230351">including myself</a>.
  </li>
  <li>
    In response to the backlash, the V author redacted the macOS binary, and said that he would do a full source release in 2 days.
  </li>
  <li>
    On 2019-06-22, he updated the V language website with "WIP" labels to point out which features of the project are not available, and
    within an hour after that, published the V project source code. At this point, I made the <a href="https://github.com/vlang/v/issues/292#issuecomment-504689104">following statement</a>:

    <blockquote>
      <p>Now that the website has the "WIP" label to communicate which features are not available, and the source is released, I no longer consider this project to be fraudulent. The information is available to everyone, and people who donate on Patreon are making an informed choice.</p>
      <p>I'm genuinely glad it turned out this way. Good luck on your endeavor, and welcome to the programming languages club.</p>
    </blockquote>
  </li>
  <li>
    At this point, the Internet discovered the source release. <a href="http://web.archive.org/web/20190623042334/https://github.com/vlang/v/issues/319">Shitposters, trolls, and genuine people all started paying
      attention to the project at once</a>. Most people were unaware that the V website had been modified to retract the
    false claims, which is quite understandable given that they had been there just yesterday. Now that it
    was crystal clear that the V project had been intentionally misleading, the backlash intensified. In 2 days, V lost $100/month in
    donations.
  </li>
  <li>
    Finally, there was a backlash on the backlash. To some people, $927/month was "table scraps". From this perspective,
    people were piling on a hate-train over a paltry sum of cash. Maybe other open-source projects should take a hint
    from V and do a little bit more marketing. Donations to open-source projects are not a
    <a href="https://en.wikipedia.org/wiki/Zero-sum_game">zero sum game</a>.
  </li>
</ol>
<p>
One thing that is crystal clear is that the V author succeeded in creating hype. He got people excited about ambitious features - so
excited that they were willing to donate some cash.
This <a href="https://lobste.rs/s/rh1pbo/v_source_code_released#c_jyakpy">made me think</a> of another open source project in the opposite category.
</p>
<h2>musl libc: a project with no hype but huge impact</h2>
<p>
<a href="https://www.musl-libc.org/">musl libc</a> is an alternative to GNU libc for Linux, created by
<a href="http://ewontfix.com/">Rich Felker</a>, and with a healthy community of high-quality contributors.
It's been around for years, yet making less than V in donations.
</p>
<p>
The Zig project owes a lot to musl, for many reasons:
</p>

<h3>Mentorship</h3>
<p>Especially in the early stages of the Zig project, but still to this day, I asked question after question in the <code>#musl</code>
IRC channel, and the musl community patiently and <em>expertly</em> indulged them. Sometimes my questions were not even musl-related,
but just more about how the Linux kernel works.
</p>
<p>
I didn't start off as an expert systems programmer when I started Zig, but
the musl community has mentored me over the years.</p>

<h3>Zig bundles musl</h3>
<p>
Zig ships with musl source code. This allows Zig projects to cross compile for Linux. For example, on Windows, you can:
</p>
<pre><code>&gt; zig.exe build-exe --c-source hello.c -target x86_64-linux-musl</code></pre>
<p>
Copy the resulting ELF binary to a Linux machine and it will run. (Or run that in the Windows Subsystem for Linux).
</p>
<p>
This works because Zig lazily builds musl from source on demand, for the selected target. It is in no small part thanks to musl's
simplicity and well-designed nature that this is possible.
</p>
<p>
It's also useful to create static builds on Linux where the native libc is glibc.
</p>

<h3>Much of the Zig standard library is ported from musl</h3>
<p>
musl is such a high quality codebase, that most of the Zig standard library's interface to Linux is a direct port of musl code.
</p>
<p>
This has prevented countless bugs and made things "just work" in general. Without this head start, the Zig project would have
had to spend more time on Linux system interface and less on everything else.
</p>
<p>
For example, thanks to <a href="https://tiehu.is/">Marc Tiehuis</a> contributions, the Zig standard library has
all the math functions you would expect to find in libm, and they are available <a href="https://ziglang.org/documentation/master/#Introducing-the-Compile-Time-Concept">at compile-time</a> as well as runtime.
</p>

<h3>Float literal parsing adapted and ported from musl</h3>
<p>
Zig has 128-bit floating point literals, so that compile-time computations can be done in a higher level of precision,
before being casted to the floating point type of choice.
</p>
<p>
This is tricky to implement in the C++ non-self-hosted compiler, however, because libc does not have a 128-bit <code>strtod</code>
function. So I took musl's <code>strtold</code> function, which works with the type <code>long double</code>, and then
<a href="https://github.com/ziglang/zig/commit/4615ed5ea003516c8235728ac3f5f0ee2ccea8a7">ported the code into Zig</a>,
making all the #ifdefs assume that <code>long double</code> is 128-bits (which is only true on some architectures), and replacing all the
math with <a href="http://www.jhauser.us/arithmetic/SoftFloat.html">SoftFloat</a> so that it would work on any architecture, no matter
what <code>long double</code> maps to.
</p>

<h3>Alpine Linux is based on musl</h3>
<p>
Many people are aware of <a href="https://www.alpinelinux.org/">Alpine Linux</a> because it has become a popular Linux distribution
to use with <a href="https://www.docker.com/">Docker</a>, mainly due to its simplicity and small binary size. This is in large part
thanks to the fact that the system uses musl as its libc rather than glibc.
</p>
<p>
The Zig project uses Alpine Linux to create the static Linux builds on <a href="https://ziglang.org/download/">ziglang.org/download</a>.
</p>
<hr>
<p>
And so I have decided to <a href="https://www.patreon.com/musl">donate $150/month to the musl project</a>, even though that represents 10% of my income.
I'm putting my money where my mouth is. But there's more - please read on...
</p>
<h2>This is a marketing stunt</h2>
<p>Now if I'm being honest about my motivations for this blog post, it's that I want to prove that
<strong>open source funding is not a zero-sum game</strong>. If there's anything we've learned from the V language Internet drama that has unfolded over the past few days, it's that open source projects have to do marketing if they want to get financial support.</p>
<p>
But I want to set a better example of what that might look like. I don't think you have to trick people.
This blog post is a marketing stunt intended to show that you can do things to get attention and funding, without being dishonest.
</p>
<p>
<strong>Will you help me prove my point?</strong>
</p>
<p>If enough people start <a href="https://github.com/users/andrewrk/sponsorship">pledging donations to Zig</a> when reading this,
and the $150/month is recovered, then this little stunt will have been a proof-of-concept of one open source project
running a fund-raiser for another one. (To be clear my donations to musl are not conditional; regardless of the outcome
I will continue to donate.)
</p>
<p>
Worth a shot, right?
</p>
<p>
Since making news headlines, several generous people have taken me up on this experiment:
</p>
<ul>
  <li>Marko Mikulicic - $4</li>
  <li>Tristan Hume - $5</li>
  <li>Christine Dodrill - $15</li>
  <li>komu - $1</li>
  <li>Dylan Baker - $3</li>
  <li>Dave Voutila - $6.25</li>
  <li>Tyler Philbrick - $3</li>
  <li>Ben - $1</li>
  <li>Tanner Schultz - $3</li>
  <li>Martin DeMello - $5</li>
  <li>Thomas Ballinger - $2</li>
  <li>Steven Branson - $5</li>
  <li>Zachary Feldman - $10</li>
  <li>Dan Gallagher - $10</li>
  <li>Quetzal Bradley - $10</li>
  <li>Brian Orr - $5</li>
  <li>YVT - $5</li>
  <li>Dexter Haslem - $5</li>
  <li>Wes - $2.40</li>
  <li>Aeron Avery - $10</li>
  <li>Carsten Dreesbach - $5</li>
  <li>Ville Tuulos - $5</li>
  <li>Curtis Fenner - $3</li>
  <li>Nathan Sculli - $5</li>
  <li>Alec Nunn - $5</li>
  <li>Gurpreet Singh - $5</li>
  <li>Haze Booth - $20</li>
  <li>Don Harris - $20</li>
</ul>
<p>
That's a total of 178.65, well beyond the $150 that I started donating to musl.
</p>
</style>
<div id="prog">
  <div id="done">100%</div>
</div>
<p>
Thank you so much! I'm really grateful for all your support. This experiment was a success!
Today, the Zig project ran a fundraiser for the musl libc project, and it worked!
</p>


guid: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html
title: `zig cc`: a Powerful Drop-In Replacement for GCC/Clang
url: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html
feedurl: https://andrewkelley.me/rss.xml
content:
<h1>`zig cc`: a Powerful Drop-In Replacement for GCC/Clang</h1>
<p>
If you have heard of <a href="https://ziglang.org/">Zig</a> before, you may know it as
a promising new programming language which is ambitiously trying to overthrow C as the
de-facto systems language. But did you know that it also can straight up compile C code?
</p>
<p>
This has been possible for a while, and you can see some
<a href="https://ziglang.org/#Zig-is-also-a-C-compiler">examples of this on the home page</a>.
What's new is that the <code>zig cc</code> sub-command is available, and it supports
the same options as <a href="https://clang.llvm.org/">Clang</a>, which, in turn, supports
the same options as <a href="https://gcc.gnu.org/">GCC</a>.
</p>
<p>
Now, I'm sure you're feeling pretty skeptical right about now, so let me hook you real
quick before I get into the juicy details.
</p>
<h2>Clang and GCC cannot do this:</h2>
<pre>
<strong>andy@ark ~/tmp&gt; cat hello.c</strong>
#include &lt;stdio.h&gt;

int main(int argc, char **argv) {
    fprintf(stderr, "Hello, World!\n");
    return 0;
}
<strong>andy@ark ~/tmp&gt; clang -o hello.exe hello.c -target x86_64-windows-gnu</strong>
clang-7: warning: argument unused during compilation: '--gcc-toolchain=/nix/store/ificps9si1nvz85f9xa7gjd9h6r5lzg6-gcc-9.2.0' [-Wunused-command-line-argument]
/nix/store/7bhi29ainf5rjrk7k7wyhndyskzyhsxh-binutils-2.31.1/bin/ld: unrecognised emulation mode: i386pep
Supported emulations: elf_x86_64 elf32_x86_64 elf_i386 elf_iamcu elf_l1om elf_k1om
clang-7: <span style="color:#cc0000;font-weight:bold;">error</span>: linker command failed with exit code 1 (use -v to see invocation)
<strong>andy@ark ~/tmp&gt; clang -o hello hello.c -target mipsel-linux-musl</strong>
In file included from hello.c:1:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/stdio.h:27:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/bits/libc-header-start.h:33:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/features.h:452:
/nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/gnu/stubs.h:7:11: <span style="color:#cc0000;font-weight:bold;">fatal error</span>: 
      'gnu/stubs-32.h' file not found
# include &lt;gnu/stubs-32.h&gt;
          ^~~~~~~~~~~~~~~~
1 error generated.
<strong>andy@ark ~/tmp&gt; clang -o hello hello.c -target aarch64-linux-gnu</strong>
In file included from hello.c:1:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/stdio.h:27:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/bits/libc-header-start.h:33:
In file included from /nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/features.h:452:
/nix/store/8pp3i3hcp7bv0f8jllzqq7gcp9dbzvp9-glibc-2.27-dev/include/gnu/stubs.h:7:11: <span style="color:#cc0000;font-weight:bold;">fatal error</span>: 
      'gnu/stubs-32.h' file not found
# include &lt;gnu/stubs-32.h&gt;
          ^~~~~~~~~~~~~~~~
1 error generated.
</pre>

<h2>`zig cc` can:</h2>
<pre>
<strong>andy@ark ~/tmp&gt; zig cc -o hello.exe hello.c -target x86_64-windows-gnu</strong>
<strong>andy@ark ~/tmp&gt; wine64 hello.exe</strong>
Hello, World!
<strong>andy@ark ~/tmp&gt; zig cc -o hello hello.c -target mipsel-linux-musl</strong>
<strong>andy@ark ~/tmp&gt; qemu-mipsel ./hello</strong>
Hello, World!
<strong>andy@ark ~/tmp&gt; zig cc -o hello hello.c -target aarch64-linux-gnu</strong>
<strong>andy@ark ~/tmp&gt; qemu-aarch64 -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu ./hello</strong>
Hello, World!
</pre>
<h2>Features of `zig cc`</h2>
<p>
<code>zig cc</code> is <em>not the main purpose of the Zig project</em>. It merely
exposes the already-existing capabilities of the Zig compiler via a small frontend layer
that parses C compiler options.
</p>
<h3>Install simply by unzipping a tarball</h3>
<p>
Zig is an open source project, and of course can be
<a href="https://github.com/ziglang/zig/#building-from-source">built and installed from source the usual way</a>. However, the Zig project also has tarballs available on
<a href="https://ziglang.org/download/">the download page</a>.
You can download a 45 MiB tarball, unpack it, and you're done.
You can even have multiple versions at the same time, no problem.
</p>
<p>
Here, rather than downloading the x86_64-linux version, which matches the computer I am
currently using, I'll download the Windows version and run it in
<a href="https://www.winehq.org/">Wine</a> to show how simple installation is:
</p>
<pre>
<strong>andy@ark ~/tmp&gt; wget --quiet https://ziglang.org/builds/zig-windows-x86_64-0.5.0+13d04f996.zip</strong>
<strong>andy@ark ~/tmp&gt; unzip -q zig-windows-x86_64-0.5.0+13d04f996.zip </strong>
<strong>andy@ark ~/tmp&gt; wine64 ./zig-windows-x86_64-0.5.0+13d04f996/zig.exe cc -o hello hello.c -target x86_64-linux</strong>
<strong>andy@ark ~/tmp&gt; ./hello</strong>
Hello, World!
</pre>
<p>
Take a moment to appreciate what just happened here - I downloaded a Windows build of Zig,
ran it in Wine, using it to cross compile for Linux, and then ran the binary natively.
Computers are fun!
</p>
<p>
Compare this to
<a href="https://github.com/llvm/llvm-project/releases/tag/llvmorg-9.0.1">downloading Clang</a>,
which has 380 MiB Linux-distribution-specific tarballs. Zig's Linux tarballs are fully statically
linked, and therefore work correctly on all Linux distributions. The size difference here
comes because the Clang tarball ships with more utilities than a C compiler, as well as
pre-compiled static libraries for both LLVM and Clang. Zig does not ship with any pre-compiled
libraries; instead it ships with source code, and builds what it needs on-the-fly.
</p>
<h3 id="caching-system">Caching System</h3>
<p>
The Zig compiler uses a sophisticated caching system to avoid needlessly rebuilding
artifacts. I carefully designed this caching system to
make optimal use of the file system while maintaining correct semantics - which is
<a href="https://apenwarr.ca/log/20181113">trickier than you might think</a>!
</p>
<p>
The caching system uses a combination of hashing inputs and checking the fstat values
of file paths, while being mindful of mtime granularity. This makes it avoid
needlessly hashing files, while at the same time detecting when a modified file has
the same contents. It always has correct behavior, whether the file system has nanosecond
mtime granularity, second granularity, always sets mtime to zero, or anything in between.
</p>
<p>
You can find a
<a href="https://ziglang.org/download/0.4.0/release-notes.html#Build-Artifact-Caching">detailed description of the caching system in the 0.4.0 release notes</a>.
</p>
<p>
<code>zig cc</code> makes this caching system available when compiling C code. For simple
enough projects, this obviates the need for a Makefile or other build system.
</p>
<pre>
<strong>andy@ark ~/tmp&gt; cat foo.c</strong>
#include &lt;stdio.h&gt;

#include "another_file.c"

int main(int argc, char **argv) {
#include "printf_many_times.c"
}
<strong>andy@ark ~/tmp&gt; cat another_file.c </strong>
void another(void) {}
<strong>andy@ark ~/tmp&gt; time zig cc -c foo.c</strong>
0.12
<strong>andy@ark ~/tmp&gt; time zig cc -c foo.c</strong>
0.01
<strong>andy@ark ~/tmp&gt; touch another_file.c </strong>
<strong>andy@ark ~/tmp&gt; time zig cc -c foo.c</strong>
0.01
<strong>andy@ark ~/tmp&gt; echo "/* add a comment */" &gt;&gt;another_file.c</strong>
<strong>andy@ark ~/tmp&gt; time zig cc -c foo.c</strong>
0.12
<strong>andy@ark ~/tmp&gt; time zig cc -c foo.c</strong>
0.01
</pre>
<p>
Here you can see the caching system is smart enough to find dependencies that are
included with the preprocessor, and smart enough to avoid a full rebuild when the
mtime of another_file.c was updated.
</p>
<p>
One last thing before I move on. I want to point out that this caching system is not
some fluffy bloated feature - rather it is an absolutely critical component to making
cross-compiling work in a usable manner. As we'll see below, other compilers ship with 
pre-compiled, target-specific binaries, while Zig ships with <em>source code only</em>
and cross-compiles on-the-fly, caching the result.
</p>

<h3>Cross Compiling</h3>
<p>I have carefully designed Zig since the very beginning to treat cross compilation
as a first class use case. Now that the <code>zig cc</code> frontend is available,
it brings these capabilities to C code.
</p>
<p>
I showed you above cross-compiling some simple "Hello, World!" programs. But now let's
try a real-world C project.
</p>
<p>
Let's try <a href="https://luajit.org/">LuaJIT</a>!
</p>
<pre>
[~/Downloads]$ <strong>git clone https://github.com/LuaJIT/LuaJIT</strong>
[~/Downloads]$ <strong>cd LuaJIT</strong>
[~/Downloads/LuaJIT]$ <strong>ls</strong>
COPYRIGHT  doc  dynasm  etc  Makefile  README  src
</pre>
<p>
OK so it uses standard Makefiles. Here we go, first let's make sure it works natively
with <code>zig cc</code>.
</p>
<pre>
[~/Downloads/LuaJIT]$ <strong>export CC="zig cc"</strong>
[~/Downloads/LuaJIT]$ <strong>make CC="$CC"</strong>
==== Building LuaJIT 2.1.0-beta3 ====
make -C src
make[1]: Entering directory '/home/andy/Downloads/LuaJIT/src'
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
DYNASM    host/buildvm_arch.h
HOSTCC    host/buildvm.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
HOSTLINK  host/buildvm
BUILDVM   lj_vm.S
ASM       lj_vm.o
CC        lj_gc.o
BUILDVM   lj_ffdef.h
CC        lj_err.o
CC        lj_char.o
BUILDVM   lj_bcdef.h
CC        lj_bc.o
CC        lj_obj.o
CC        lj_buf.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_dispatch.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_strfmt.o
CC        lj_strfmt_num.o
CC        lj_api.o
CC        lj_profile.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
BUILDVM   lj_folddef.h
CC        lj_opt_fold.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_record.o
CC        lj_crecord.o
BUILDVM   lj_recdef.h
CC        lj_ffrecord.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
BUILDVM   lj_libdef.h
CC        lib_base.o
CC        lib_math.o
CC        lib_bit.o
CC        lib_string.o
CC        lib_table.o
CC        lib_io.o
CC        lib_os.o
CC        lib_package.o
CC        lib_debug.o
CC        lib_jit.o
CC        lib_ffi.o
CC        lib_init.o
AR        libluajit.a
CC        luajit.o
BUILDVM   jit/vmdef.lua
DYNLINK   libluajit.so
LINK      luajit
warning: unsupported linker arg: -E
OK        Successfully built LuaJIT
make[1]: Leaving directory '/home/andy/Downloads/LuaJIT/src'
==== Successfully built LuaJIT 2.1.0-beta3 ====

[~/Downloads/LuaJIT]$ <strong>ls</strong>
COPYRIGHT  doc  dynasm  etc  Makefile  README  src

[~/Downloads/LuaJIT]$ <strong>./src/</strong>
host/         jit/          libluajit.so  luajit        zig-cache/    

[~/Downloads/LuaJIT]$ <strong>./src/luajit </strong>
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON SSE2 SSE3 SSE4.1 BMI2 fold cse dce fwd dse narrow loop abc sink fuse
&gt; <strong>print(3 + 4)</strong>
7
&gt; 
</pre>
<p>
OK so that worked. Now for the real test - can we make it cross compile?
</p>
<pre>
[~/Downloads/LuaJIT]$ <strong>git clean -xfdq</strong>
[~/Downloads/LuaJIT]$ <strong>export CC="zig cc -target aarch64-linux-gnu"</strong>
[~/Downloads/LuaJIT]$ <strong>export HOST_CC="zig cc"</strong>
[~/Downloads/LuaJIT]$ <strong>make CC="$CC" HOST_CC="$HOST_CC" TARGET_STRIP="echo"</strong>
==== Building LuaJIT 2.1.0-beta3 ====
make -C src
make[1]: Entering directory '/home/andy/Downloads/LuaJIT/src'
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
DYNASM    host/buildvm_arch.h
HOSTCC    host/buildvm.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
HOSTLINK  host/buildvm
BUILDVM   lj_vm.S
ASM       lj_vm.o
CC        lj_gc.o
BUILDVM   lj_ffdef.h
CC        lj_err.o
CC        lj_char.o
BUILDVM   lj_bcdef.h
CC        lj_bc.o
CC        lj_obj.o
CC        lj_buf.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_dispatch.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_strfmt.o
CC        lj_strfmt_num.o
CC        lj_api.o
CC        lj_profile.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
BUILDVM   lj_folddef.h
CC        lj_opt_fold.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_record.o
CC        lj_crecord.o
BUILDVM   lj_recdef.h
CC        lj_ffrecord.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
BUILDVM   lj_libdef.h
CC        lib_base.o
CC        lib_math.o
CC        lib_bit.o
CC        lib_string.o
CC        lib_table.o
CC        lib_io.o
CC        lib_os.o
CC        lib_package.o
CC        lib_debug.o
CC        lib_jit.o
CC        lib_ffi.o
CC        lib_init.o
AR        libluajit.a
CC        luajit.o
BUILDVM   jit/vmdef.lua
DYNLINK   libluajit.so
libluajit.so
LINK      luajit
warning: unsupported linker arg: -E
luajit
OK        Successfully built LuaJIT
make[1]: Leaving directory '/home/andy/Downloads/LuaJIT/src'
==== Successfully built LuaJIT 2.1.0-beta3 ====

[~/Downloads/LuaJIT]$ <strong>file ./src/luajit </strong>
./src/luajit: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 2.0.0, with debug_info, not stripped
</pre>
<p>
It worked! Will it run in <a href="https://www.qemu.org/">QEMU</a> though?
</p>
<pre>
[~/Downloads/LuaJIT]$ <strong>qemu-aarch64 -L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu ./src/luajit</strong>
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON fold cse dce fwd dse narrow loop abc sink fuse
&gt; <strong>print(4 + 3)</strong>
7
&gt; 
</pre>
<p>
Amazing. QEMU never fails to impress me.
</p>
<p>
Before we move on, I want to show one more thing. You can see above, in order to run the
foreign-architecture binary, I had to pass
<code>-L ~/Downloads/glibc/multi-2.31/install/glibcs/aarch64-linux-gnu</code>. This is
due to the binary being dynamically linked. You can confirm this with the output from
<code>file</code> above where it says: <code>dynamically linked, interpreter /lib/ld-linux-aarch64.so.1</code>
</p>
<p>
Often, when cross-compiling, it is useful to make a <em>static</em> binary.
In the case of Linux, for example, this will make the resulting binary able to run on
any Linux distribution, rather than only ones with a hard-coded glibc dynamic linker path
of <code>/lib/ld-linux-aarch64.so.1</code>.
</p>
<p>
We can accomplish this by targeting musl rather than glibc:
</p>
<pre>
[~/Downloads/LuaJIT]$ git clean -qxfd
[~/Downloads/LuaJIT]$ export CC="zig cc -target aarch64-linux-musl"
[~/Downloads/LuaJIT]$ make CC="$CC" CXX="$CXX" HOST_CC="$HOST_CC" TARGET_STRIP="echo"
==== Building LuaJIT 2.1.0-beta3 ====
(same output)
==== Successfully built LuaJIT 2.1.0-beta3 ====
[~/Downloads/LuaJIT]$ file src/luajit
src/luajit: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, not stripped
[~/Downloads/LuaJIT]$ qemu-aarch64 ./src/luajit
LuaJIT 2.1.0-beta3 -- Copyright (C) 2005-2020 Mike Pall. http://luajit.org/
JIT: ON fold cse dce fwd dse narrow loop abc sink fuse
&gt; print(11 + 22)
33
</pre>
<p>
Here you can see the <code>file</code> command reported <em>statically linked</em>,
and in the qemu command, the <code>-L</code> parameter was not needed.
</p>

<h2>Use Cases of `zig cc`</h2>
<p>
Alright, so I've given you a taste of what <code>zig cc</code> can do, but now I will
list explicitly what I consider to be the use cases:
</p>
<h3>Experimentation</h3>
<p>
Sometimes you just want a tool that you can use to try out different things. It can quickly
answer questions such as "What assembly does this code generate on MIPS vs ARM?". The widely
popular <a href="https://godbolt.org/">Compiler Explorer</a> serves this purpose.
</p>
<p>
<code>zig cc</code> provides a lightweight tool which can also answer questions such as,
"What happens if I swap out glibc for <a href="https://musl.libc.org/">musl</a>?" and
"How big is this executable when cross-compiled for Windows?".
<a href="https://twitter.com/andy_kelley/status/1242183564512366595">Here's me using Zig to
quickly find out what the maximum UDP packet size is on Linux</a>.
</p>
<p>
Since Zig is so easy to install - and it actually works everywhere without patches,
even Linux distributions such as <a href="https://nixos.org/">NixOS</a> -
it can often be a more convenient tool for running quick C test programs on your computer.
</p>
<p>
At the time of this writing, LLVM 10 was just released two hours ago.
It will take days or weeks for it to become available in various system package managers.
But you can already
<a href="https://ziglang.org/download/">download a master branch build of Zig</a>
and play with the new features of Clang/LLVM 10. For example, improved RISC-V support!
</p>
<pre>
andy@ark ~/tmp&gt; <strong>zig cc -o hello hello.c -target riscv64-linux-musl</strong>
andy@ark ~/tmp&gt; <strong>qemu-riscv64 ./hello</strong>
Hello, World!
</pre>

<h3>Bundling a C compiler as part of a larger project</h3>
<p>
With Zig tarballs weighing in at under 45 MiB, zero system dependencies, no configuration,
and MIT license, it makes for an ideal candidate when you need to bundle a C compiler along
with another project.
</p>
<p>
For example, maybe you have
<a href="https://www.acton-lang.org/">a programming language that compiles to C</a>.
Zig is an obvious choice for what C compiler to ship with your language.
</p>
<p>
Or maybe you want to make a batteries-included
<a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a>
that ships with a compiler.
</p>

<h3>Lightweight alternative to a cross compilation environment</h3>
<p>
If you're trying to build something with a large dependency tree, you'll probably want to
use a full cross compilation environment, such as <a href="https://mxe.cc/">mxe.cc</a>
or <a href="http://musl.cc/">musl.cc</a>.
</p>
<p>
But if you don't need such a sledgehammer, <code>zig cc</code> could be a useful alternative,
especially if your goal is to compile for N different targets. Consider that musl.cc lists different
tarballs for each architecture, each weighing in at roughly 85 MiB. Meanwhile Zig weighs in at 45 MiB
and it supports all those architectures, plus glibc and Windows.
</p>

<h3>An alternative to installing MSVC on Windows</h3>
<p>
You could spend days - literally! - waiting for Microsoft Visual Studio to install,
or you could install Zig and
<a href="https://code.visualstudio.com/">VS Code</a> in a matter of minutes.
</p>

<h2>Under the Hood</h2>
<p>If <code>zig cc</code> is built on top of Clang, why doesn't Clang just do this?
What exactly is Zig doing on top of Clang to make this work?</p>
<p>
The answer is, <em>a lot</em>, actually. I'll go over how it works here.
</p>
<h3>compiler-rt</h3>
<p>
compiler-rt is a library that provides "polyfill" implementations of language-supported features
when the target does not have machine code instructions for it. For example, compiler-rt has
the function <code>__muldi3</code> to perform signed 64-bit integer multiplication on architectures
that do not have a 64-bit wide integer multiplication instruction.
</p>
<p>
In the GNU world, compiler-rt is named <strong>libgcc</strong>.
</p>
<p>
Most C compilers ship with this library pre-built for the target.
For example, on an Ubuntu (Bionic) system, with the <code>build-essential</code> package installed,
you can find this at <code>/lib/x86_64-linux-gnu/libgcc_s.so.1</code>.
</p>
<p>
If you download <a href="https://github.com/llvm/llvm-project/releases/download/llvmorg-9.0.1/clang+llvm-9.0.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz">clang+llvm-9.0.1-x86_64-linux-gnu-ubuntu-16.04.tar.xz</a> and take a look
around, clang actually does not even ship with compiler-rt. Instead, it relies on the system libgcc
noted above. This is one reason that this tarball is Ubuntu-specific and does not work on other
Linux distributions, 
<a href="https://www.leidinger.net/blog/2010/09/28/the-freebsd-linuxulator-explained-for-users/">FreeBSD's Linuxulator</a>,
or <a href="https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux">WSL</a>,
which have system files in different locations.
</p>
<p>
Zig's strategy with compiler-rt is that we have
<a href="https://github.com/ziglang/zig/blob/0.5.0/lib/std/special/compiler_rt.zig">our own implementation of this library</a>,
written in Zig. Most of it is ported from
<a href="https://github.com/llvm/llvm-project/tree/llvmorg-10.0.0-rc6/compiler-rt">LLVM's compiler-rt project</a>,
but we also have some of our own improvements on top of this.
</p>
<p>
Anyway, rather than depending on system compiler-rt being installed, or shipping a pre-compiled
library, Zig ships its compiler-rt <em>in source form</em>, and lazily builds compiler-rt 
for the compilation target, and then caches the result using
<a href="#caching-system">the caching system discussed above</a>.
</p>
<p>
Zig's compiler-rt is <a href="https://github.com/ziglang/zig/issues/1290">not yet complete</a>.
However, completing it is a prerequisite for releasing Zig version 1.0.0.
</p>

<h3>libc</h3>
<p>
When C code calls <code>printf</code>, <code>printf</code> has to be implemented <em>somewhere</em>,
and that somewhere is libc.
</p>
<p>
Some operating systems, such as <a href="https://www.freebsd.org/">FreeBSD</a> and macOS, have a
designated system libc, and it is the kernel syscall interface. On others, such as
Windows and Linux, libc is optional, and therefore there are multiple options of which
libc to use, if any.
</p>
<p>
As of the time of this writing, Zig can provide libcs for the following targets:
</p>
<pre>
andy@ark ~&gt; zig targets | jq .libc
[
  "aarch64_be-linux-gnu",
  "aarch64_be-linux-musl",
  "aarch64_be-windows-gnu",
  "aarch64-linux-gnu",
  "aarch64-linux-musl",
  "aarch64-windows-gnu",
  "armeb-linux-gnueabi",
  "armeb-linux-gnueabihf",
  "armeb-linux-musleabi",
  "armeb-linux-musleabihf",
  "armeb-windows-gnu",
  "arm-linux-gnueabi",
  "arm-linux-gnueabihf",
  "arm-linux-musleabi",
  "arm-linux-musleabihf",
  "arm-windows-gnu",
  "i386-linux-gnu",
  "i386-linux-musl",
  "i386-windows-gnu",
  "mips64el-linux-gnuabi64",
  "mips64el-linux-gnuabin32",
  "mips64el-linux-musl",
  "mips64-linux-gnuabi64",
  "mips64-linux-gnuabin32",
  "mips64-linux-musl",
  "mipsel-linux-gnu",
  "mipsel-linux-musl",
  "mips-linux-gnu",
  "mips-linux-musl",
  "powerpc64le-linux-gnu",
  "powerpc64le-linux-musl",
  "powerpc64-linux-gnu",
  "powerpc64-linux-musl",
  "powerpc-linux-gnu",
  "powerpc-linux-musl",
  "riscv64-linux-gnu",
  "riscv64-linux-musl",
  "s390x-linux-gnu",
  "s390x-linux-musl",
  "sparc-linux-gnu",
  "sparcv9-linux-gnu",
  "wasm32-freestanding-musl",
  "x86_64-linux-gnu",
  "x86_64-linux-gnux32",
  "x86_64-linux-musl",
  "x86_64-windows-gnu"
]
</pre>
<p>
In order to provide libc on these targets, Zig ships with a subset of the source files
for these projects:
</p>
<ul>
  <li>musl v1.2.0</li>
  <li><a href="https://mingw-w64.org/">mingw-w64</a> v7.0.0</li>
  <li>glibc 2.31</li>
</ul>
<p>
For each libc, there is a
<a href="https://github.com/ziglang/zig/wiki/Updating-libc">process for upgrading to a new release</a>.
This process is a sort of pre-processing step. We still end up with source files, but we
de-duplicate non-multi-arch source files into multi-arch source files.
</p>

<h4>glibc</h4>
<p>
glibc is the most involved. The first step is building glibc for every target that it supports,
which takes upwards of 24 hours and 74 GiB of disk space.
</p>
<p>
From here, the
<a href="https://github.com/ziglang/zig/blob/dc44fe053c609f389e375f6857f96b6bb3794897/tools/process_headers.zig">process_headers tool</a>
inspects all the header files from all the targets, and identifies which files are the same across
all targets, and which header files are target-specific. They are then sorted into the
corresponding directories in Zig's source tree, in:
</p>
<ul>
  <li>lib/libc/include/generic-glibc/</li>
  <li>lib/libc/include/$ARCH-linux-$ABI/ (there are multiple of these directories)</li>
</ul>
<p>
Additionally, Linux header files are not included in glibc, and so the same process is applied to
Linux header files, with the directories:
</p>
<ul>
  <li>lib/libc/include/any-linux-any/</li>
  <li>lib/libc/include/$ARCH-linux-any/</li>
</ul>
<p>
That takes care of the header files, but now we have the problem of dynamic linking against
glibc, without touching any system files.
</p>
<p>
For this, we have the
<a href="https://github.com/ziglang/zig/blob/dc44fe053c609f389e375f6857f96b6bb3794897/tools/update_glibc.zig">update_glibc tool</a>.
Given the path to the glibc source directory, it finds all the <code>.abilist</code> text files
and uses them to produce 3 simple but crucial files:
</p>
<ul>
  <li><a href="https://github.com/ziglang/zig/blob/master/lib/libc/glibc/vers.txt">vers.txt</a>
    - the list of all glibc versions.
  </li>
  <li><a href="https://github.com/ziglang/zig/blob/master/lib/libc/glibc/fns.txt">fns.txt</a>
    - the list of all symbols that glibc provides, followed by the library it appears in
    (for example libm, libpthread, libc, librt).
  </li>
  <li><a href="https://github.com/ziglang/zig/blob/master/lib/libc/glibc/abi.txt">abi.txt</a>
    - for each target, for each function, tells which versions of glibc, if any, it appears in.
  </li>
</ul>
<p>
Together, these files amount to only 192 KB (27 KB gzipped), and they allow Zig to target any
version of glibc.
</p>
<p>
Yes, I did not make a typo there. Zig can target any of the 42 versions of glibc for any of the
architectures listed above. I'll show you:
</p>
<pre>
andy@ark ~/tmp&gt; <strong>cat rand.zig </strong>
const std = @import("std");

pub fn main() anyerror!void {
    var buf: [10]u8 = undefined;
    _ = std.c.getrandom(&amp;buf, buf.len, 0);
    std.debug.warn("random bytes: {x}\n", .{buf});
}
andy@ark ~/tmp&gt; <strong>zig build-exe rand.zig -lc -target native-native-gnu.2.25</strong>
andy@ark ~/tmp&gt; <strong>./rand</strong>
random bytes: e2059382afb599ea6d29
andy@ark ~/tmp&gt; <strong>zig build-exe rand.zig -lc -target native-native-gnu.2.24</strong>
lld: error: undefined symbol: getrandom
&gt;&gt;&gt; referenced by rand.zig:5 (/home/andy/tmp/rand.zig:5)
&gt;&gt;&gt;               ./rand.o:(main.0)
</pre>
<p>
Sure enough, if you look at the
<a href="http://man7.org/linux/man-pages/man2/getrandom.2.html">man page for getrandom</a>,
it says:
</p>
<blockquote>
Support was added to glibc in version 2.25.
</blockquote>
<p>
When no explicit glibc version is requested, and the target OS is the native (host) OS,
Zig detects the native glibc version by inspecting the Zig executable's own dynamically
linked libraries, looking for glibc, and checking the version. It turns out you can look for
<code>libc.so.6</code> and then <code>readlink</code> on that, and it will look something
like <code>libc-2.27.so</code>. When this strategy does not work, Zig looks at
<code>/usr/bin/env</code>, looking for the same thing. Since this file path is hard-coded
into countless shebang lines, it's a pretty safe bet to find out the dynamic linker path and
glibc version (if any) of the native system!
</p>
<p>
<code>zig cc</code> currently does not provide a way to choose a specific glibc version
(because C compilers do not provide a way), and so Zig chooses the native version for
compiling natively, and the default (2.17) for cross-compiling.
However, I'm sure this problem can be solved, even when using <code>zig cc</code>. For example,
maybe it could support an environment variable, or simply introduce an extra command line
option that does not conflict with any Clang options.
</p>
<p>
When you request a certain version of glibc, Zig uses those text files noted above to
create dummy <code>.so</code> files to link against, which contain exactly the correct
set of symbols (with appropriate name mangling) based on the requested version.
The symbols will be resolved at runtime, by the dynamic linker on the target platform.
</p>
<p>
In this way, most of libc in the glibc case resides on the target file system. But not all of it!
There are still the "C runtime start files":
</p>
<ul>
  <li>Scrt1.o</li>
  <li>crti.o</li>
  <li>crtn.o</li>
</ul>
<p>
These are statically compiled into every binary that dynamically links glibc, and their
<a href="https://en.wikipedia.org/wiki/Application_binary_interface">ABI</a> is therefore
Very Very Stable.
</p>
<p>
And so, Zig bundles a small subset of glibc's source files needed to build these object
files from source for every target. The total size of this comes out to 1.4 MiB (252 KB gzipped).
I do think there is some room for improvement here, but I digress.
</p>
<p>
There are a couple of patches to this small subset of glibc source files, which simplify them
to avoid including too many .h files, since the end result that we need is some bare bones object
files, and not all of glibc.
</p>
<p>
And finally, we certainly do not ship the build system of glibc with Zig! I manually inspected,
audited, and analyzed glibc's build system, and then by hand wrote code in the Zig
compiler which hooks into Zig's <a href="#caching-system">caching system</a> and performs a minimal
build of only these start files, as needed.
</p>

<h4>musl</h4>
<p>
The process for preparing musl to ship with Zig is much simpler by comparison.
</p>
<p>
It still involves building musl for every target architecture that it supports,
but in this case only the <code>install-headers</code> target has to be run,
and it takes less than a minute, even to do it for all targets.
</p>
<p>
The same
<a href="https://github.com/ziglang/zig/blob/dc44fe053c609f389e375f6857f96b6bb3794897/tools/process_headers.zig">process_headers tool</a>
tool used for glibc headers is used on the musl headers:
</p>
<ul>
  <li>lib/libc/include/generic-musl/</li>
  <li>lib/libc/include/$ARCH-linux-$ABI/ (there are multiple of these directories)</li>
</ul>
<p>
Unlike glibc, musl supports building statically. Zig currently assumes a static libc
when musl is chosen, and does not support dynamically linking against musl, although
that could potentially be added in the future.
</p>
<p>
And so for musl, zig actually bundles most - but still not all - of musl's source files.
Everything in <code>arch</code>, <code>crt</code>, <code>compat</code>, <code>src</code>, and <code>include</code> gets copied in.
</p>
<p>
Again much like glibc, I carefully studied musl's build system, and then hand-coded logic
in the Zig compiler to build these source files. In musl's case it is simpler - just a bit
of logic having to do with the file extension, and whether to override files with an
architecture-specific file. The only file that needs to be patched (by hand) is
<code>version.h</code>, which is normally generated during the configure phase in musl's build
system.
</p>
<p>
I really appreciate Rich Felker's efforts to make musl simple to utilize in this way,
and he has been incredibly helpful in the <code>#musl</code> IRC channel when I ask
questions.
<a href="why-donating-to-musl-libc-project.html">I proudly sponsor Rich Felker for $150/month</a>.
</p>

<h4>mingw-w64</h4>
<p>
mingw-w64 was an absolute joy to support in Zig. The beautiful thing about this project is that
they have already been transitioning into having one set of header files that applies to all
architectures (using <code>#ifdefs</code> only where needed). One set of header files
is sufficient to support all four architectures: arm, aarch64, x86, and x86_64. 
</p>
<p>
So for updating headers, all we have to do is build mingw-w64, then:
</p>
<pre>
mv $INSTALLPREFIX/include $ZIGSRC/lib/libc/include/any-windows-any
</pre>
<p>
After doing this for all 3 libcs, the libc/include directory looks like this:
</p>
<pre>
aarch64_be-linux-any   i386-linux-musl           powerpc-linux-any
aarch64_be-linux-gnu   mips64el-linux-any        powerpc-linux-gnu
aarch64-linux-any      mips64el-linux-gnuabi64   powerpc-linux-musl
aarch64-linux-gnu      mips64el-linux-gnuabin32  riscv32-linux-any
aarch64-linux-musl     mips64-linux-any          riscv64-linux-any
any-linux-any          mips64-linux-gnuabi64     riscv64-linux-gnu
any-windows-any        mips64-linux-gnuabin32    riscv64-linux-musl
armeb-linux-any        mips64-linux-musl         s390x-linux-any
armeb-linux-gnueabi    mipsel-linux-any          s390x-linux-gnu
armeb-linux-gnueabihf  mipsel-linux-gnu          s390x-linux-musl
arm-linux-any          mips-linux-any            sparc-linux-gnu
arm-linux-gnueabi      mips-linux-gnu            sparcv9-linux-gnu
arm-linux-gnueabihf    mips-linux-musl           x86_64-linux-any
arm-linux-musl         powerpc64le-linux-any     x86_64-linux-gnu
generic-glibc          powerpc64le-linux-gnu     x86_64-linux-gnux32
generic-musl           powerpc64-linux-any       x86_64-linux-musl
i386-linux-any         powerpc64-linux-gnu
i386-linux-gnu         powerpc64-linux-musl
</pre>
<p>
When Zig generates a C command line to send to clang, it puts the appropriate
include paths using <code>-I</code> depending on the target. For example, if the
target is <code>aarch64-linux-musl</code>, then the following command line parameters
are appended:
</p>
<ul>
  <li><code>-I$LIB/libc/include/aarch64-linux-musl</code></li>
  <li><code>-I$LIB/libc/include/aarch64-linux-any</code></li>
  <li><code>-I$LIB/libc/include/generic-musl</code></li>
</ul>
<p>
Anyway back to mingw-w64.
</p>
<p>
Again, Zig includes a subset of source files from mingw-w64 with a few patches applied
to make things compile successfully.
</p>
<p>
The Zig compiler code that builds mingw-w64 from source files emulates only the parts of
the build system that are needed for this subset. This includes preprocessing <code>.def.in</code>
files to get <code>.def</code> files, and then in-turn using LLD to generate <code>.lib</code> files
from the <code>.def</code> files, which allows Zig to provide <code>.lib</code> files for
any Windows DLL, such as kernel32.dll or even opengl32.dll.
</p>

<h3>Invoking Clang Without a System Dependency</h3>
<p>
Since Zig already links against Clang libraries for the
<a href="https://ziglang.org/#Integration-with-C-libraries-without-FFIbindings">translate-c feature</a>,
it was not much more cost to expose the <code>main()</code> entry point from Zig.
So that's exactly what we do:
</p>
<ul>
  <li><code>llvm-project/clang/tools/driver/driver.cpp</code> is copied to <code>$ZIGGIT/src/zig_clang_driver.cpp</code></li>
  <li><code>llvm-project/clang/tools/driver/cc1_main.cpp</code> is copied to <code>$ZIGGIT/src/zig_clang_cc1_main.cpp</code></li>
  <li><code>llvm-project/clang/tools/driver/cc1as_main.cpp</code> is copied to <code>$ZIGGIT/src/zig_clang_cc1as_main.cpp</code></li>
</ul>
<p>
The following patch is applied:
</p>
<pre>
--- a/src/zig_clang_driver.cpp
+++ b/src/zig_clang_driver.cpp
@@ -206,8 +205,6 @@
                     void *MainAddr);
 extern int cc1as_main(ArrayRef&lt;const char *&gt; Argv, const char *Argv0,
                       void *MainAddr);
<span class="diff0">-extern int cc1gen_reproducer_main(ArrayRef&lt;const char *&gt; Argv,</span>
<span class="diff0">-                                  const char *Argv0, void *MainAddr);</span>
 
 static void insertTargetAndModeArgs(const ParsedClangName &amp;NameParts,
                                     SmallVectorImpl&lt;const char *&gt; &amp;ArgVector,
@@ -330,19 +327,18 @@
   if (Tool == "-cc1as")
     return cc1as_main(makeArrayRef(ArgV).slice(2), ArgV[0],
                       GetExecutablePathVP);
<span class="diff0">-  if (Tool == "-cc1gen-reproducer")</span>
<span class="diff0">-    return cc1gen_reproducer_main(makeArrayRef(ArgV).slice(2), ArgV[0],</span>
<span class="diff0">-                                  GetExecutablePathVP);</span>
   // Reject unknown tools.
   llvm::errs() &lt;&lt; "error: unknown integrated tool '" &lt;&lt; Tool &lt;&lt; "'. "
                &lt;&lt; "Valid tools include '-cc1' and '-cc1as'.\n";
   return 1;
 }
 
<span class="diff0">-int main(int argc_, const char **argv_) {</span>
<span class="diff1">+extern "C" int ZigClang_main(int argc_, const char **argv_);</span>
<span class="diff1">+int ZigClang_main(int argc_, const char **argv_) {</span>
   noteBottomOfStack();
   llvm::InitLLVM X(argc_, argv_);
<span class="diff0">-  SmallVector&lt;const char *, 256&gt; argv(argv_, argv_ + argc_);</span>
<span class="diff1">+  size_t argv_offset = (strcmp(argv_[1], "-cc1") == 0 || strcmp(argv_[1], "-cc1as") == 0) ? 0 : 1;</span>
<span class="diff1">+  SmallVector&lt;const char *, 256&gt; argv(argv_ + argv_offset, argv_ + argc_);</span>
 
   if (llvm::sys::Process::FixupStandardFileDescriptors())
     return 1;
</pre>
<p>
This disables some cruft, and then renames <code>main</code> to <code>ZigClang_main</code> so that
it can be called like any other function. Next, in Zig's actual <code>main</code>, it looks
for <code>clang</code> as the first parameter, and calls it.
</p>
<p>
So, <code>zig clang</code> is low-level undocumented API that Zig exposes for directly invoking Clang.
But <code>zig cc</code> is much higher level than that. When Zig needs to compile C code,
it invokes itself as a child process, taking advantage of <code>zig clang</code>. <code>zig cc</code>
on the other hand, has a more difficult job: it must parse Clang's command line options and
map those to the Zig compiler's settings, so that ultimately <code>zig clang</code> can be invoked
as a child process.
</p>

<h3>Parsing Clang Command Line Options</h3>
<p>
When using <code>zig cc</code>, Zig acts as a proxy between the user and Clang. It does not need
to understand all the parameters, but it does need to understand some of them, such as
the target. This means that Zig must understand when a C command line parameter expects
to "consume" the next parameter on the command line.
</p>
<p>
For example, <code>-z -target</code> would mean to pass <code>-target</code> to the linker,
whereas <code>-E -target</code> would mean that the next parameter specifies the target.
</p>
<p>Clang has a
<a href="https://clang.llvm.org/docs/ClangCommandLineReference.html">long list of command line options</a> and so it would be foolish to try to hard-code all of them.
</p>
<p>
Fortunately, LLVM has a file "options.td" which describes all of its command line parameter options
in some obscure format. But fortunately again, LLVM comes with the <code>llvm-tblgen</code> tool
that can dump it as JSON format.
</p>
<p>
Zig has an
<a href="https://github.com/ziglang/zig/blob/dc44fe053c609f389e375f6857f96b6bb3794897/tools/update_clang_options.zig">update_clang_options tool</a>
which processes this JSON dump and produces a
<a href="https://github.com/ziglang/zig/blob/dc44fe053c609f389e375f6857f96b6bb3794897/src-self-hosted/clang_options_data.zig">big sorted list of Clang's command line options</a>.
</p>
<p>
Combined with a list of "known options" which correspond to Zig compiler options,
this is used to make an iterator API that <code>zig cc</code> uses to parse command line
parameters and instantiate a Zig compiler instance. Any Clang options that Zig is not
aware of are forwarded to Clang directly. Some parameters are handled specially.
</p>

<h3>Linking</h3>
<p>
This part is pretty straightforward. Zig depends on LLD for linking rather than
shelling out to the system linker, like GCC and Clang do.
</p>
<p>
When you use <code>-o</code> with <code>zig cc</code>, Clang is not actually acting as
a linker driver here. Zig is still the linker driver.
</p>

<h2>Everybody Wins</h2>
<p>
Now that I've spent this entire blog article comparing Zig and Clang as if they are
competitors, let me make it absolutely clear that both of these are harmonious,
mutually beneficial open-source projects. It's pretty obvious how Clang and the entire
LLVM project are massively beneficial to the Zig project, since Zig builds on top of them.
</p>
<p>
But it works the other way, too.
</p>
<p>
With Zig's focus on cross-compiling, its test suite has been expanding rapidly to cover
a large number of architectures and operating systems, leading to
<a href="https://github.com/ziglang/zig/issues?q=is%3Aissue+label%3Aupstream+is%3Aclosed">dozens of bugs reported upstream and patches sent</a>, including, for example:
</p>
<ul>
  <li><a href="https://bugs.llvm.org/show_bug.cgi?id=43268">Regression discovered in LLVM 9 release candidate</a></li>
  <li><a href="https://bugs.winehq.org/show_bug.cgi?id=47979">Bug fixed in Wine's NtDll</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/3338#issuecomment-536771508">Directly working with RISC-V target developers</a></li>
  <li><a href="https://bugs.llvm.org/show_bug.cgi?id=43768#c3">Bug fixes in LLVM's MIPS code generation</a></li>
</ul>
<p>Everybody wins.</p>

<h2>This is still experimental!</h2>
<p>
I have only recently landed <code>zig cc</code> support last week, and it is still experimental.
Please do not expect it to be production quality yet.
</p>
<p>
Zig's 0.6.0 release is right around the corner, scheduled for April 13th. I will be sure to provide
an update on the release notes on how stable and robust you can expect <code>zig cc</code> to be
in the 0.6.0 release.
</p>
<p>
There are some follow-up issues related to <code>zig cc</code> which are still open:
</p>
<ul>
  <li><a href="https://github.com/ziglang/zig/issues/4784">improve zig cc flag integration</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/4785">using zig as a drop in replacement for msvc</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/4786">support compiling and linking c++ code</a></li>
  <li><a href="https://github.com/ziglang/zig/issues/4787">use case: directly symlink zig binary to /usr/bin/cc</a></li>
</ul>
As always, <a href="https://github.com/ziglang/zig/blob/master/CONTRIBUTING.md">Contributions are most welcome</a>.

<h2>💖 Sponsor Zig 💖</h2>
<p><a href="https://github.com/sponsors/andrewrk">Sponsor Andrew Kelley on GitHub</a></p>
<p>
If you're reading this and you already sponsor me, thank you so much! I wake up every day
absolutely thrilled that I get to do this for my full time job.
</p>
<p>
As Zig has been gaining popularity, demands for my time have been growing faster than
funds to hire another full-time programmer. Every recurring donation helps, and if the funds keep
growing then soon enough the Zig project will have two full-time programmers.
</p>
<p>
That's all folks. I hope you and your loved ones are well.
</p>


guid: https://andrewkelley.me/post/goodbye-twitter-reddit.html
title: So Long, Twitter and Reddit
url: https://andrewkelley.me/post/goodbye-twitter-reddit.html
feedurl: https://andrewkelley.me/rss.xml
content:
<h1>So Long, Twitter and Reddit</h1>

<p>It's been over three years since my last blog post. I think that
the website code started to feel like it had bitrotted, and so making new blog
posts became onerous. But I've taken the time to redo this website
using <a href="https://ziglang.org/">Zig</a>, so it's quite easy to add blog
posts now, and plus I added
<a href="https://developer.mozilla.org/en-US/docs/Web/CSS/@media/prefers-color-scheme">dark mode</a>.</p>

<h2>Twitter</h2>

<p>Over the past few years I amassed a healthy number of followers on Twitter by posting
screenshots, demos, and small progress updates for the Zig programming language. It
felt good to gain an audience and have my perspective and efforts weighed more heavily in
the court of public opinion.</p>

<p>However, I always knew that it would need to end. It's not really possible to use Twitter
without at least a little bit of doomscrolling, and that activity took a toll on me emotionally.
I think that the core concept of a "tweet" is fundamentally unhealthy, because by design it
promotes angry and extreme content. Nuance and subtlety is impossible to distinguish from
dog whistling. Most users don't understand that by "dunking" on someone, they are actually
promoting the content. In summary, it gave me a darker feeling about the world in general,
and in exchange, did not offer much to enrich my life.</p>

<p>So, when the <a href="https://knowyourmeme.com/memes/enshittification">enshittification</a>
process kicked into high gear with the
<a href="https://en.wikipedia.org/wiki/Acquisition_of_Twitter_by_Elon_Musk">Acquisition of Twitter by Elon Musk</a>,
this was the perfect opportunity for me to make my exit.</p>

<p>It has now been a few months since I downloaded all my data and deleted my
account, and in retrospect it was so, so worth it. I feel more optimistic about
the world and human society in general. I interact with people around me with fewer preconceived
judgements about their belief systems. And finally, I have more nuanced conversations with my wife
and friends about politics and other hot topics, without worrying that I'm going to come off
as dog whistling or naysaying.</p>

<p>One thing I do miss is to have the ability to correct misinformation that I stumble upon in
the wild. For example, somebody linked me to <a href="https://twitter.com/ThePrimeagen/status/1681343593208856576">this tweet</a>:</p>

<blockquote>
I cannot stop thinking about this.  Andrew created Zig and ~15 years ago asked on SO how to open a file in C++
</blockquote>

<p>(screenshot of this <a href="https://stackoverflow.com/questions/7880/how-do-you-open-a-file-in-c">StackOverflow question</a>)</p>

<blockquote>
if you take this in any other way than this is incredible and motivating that you can accomplish anything with time + effort then you nuts/sad
</blockquote>

<p>It's a sweet sentiment! Unfortunately it's based on a bit of a misunderstanding.
If you look at the date the question was asked, it was August 11th, 2008.
<a href="https://en.wikipedia.org/wiki/Stack_Overflow">According to Wikipedia</a>, StackOverflow launched on September 15th, 2008. How is this possible?</p>

<p>I'll tell you - if you look at
<a href="https://en.wikipedia.org/wiki/Stack_Overflow#cite_note-launches-1">the citation</a>,
it's a link to
<a href="https://www.joelonsoftware.com/2008/09/15/stack-overflow-launches/">the launch announcement</a> on <a href="https://www.joelonsoftware.com">Joel Spolsky's blog</a> -
a blog that I was a huge fan of when I was 20 years old. The fact is that Stack Overflow actually was annouced before that, on a podcast episode with
<a href="https://blog.codinghorror.com/">Jeff Atwood</a>.</p>

<p>Being a bored intern at Lockheed Martin, because they failed to find me anything to do besides "learn XML", I enthusiastically participated in the launch of Stack Overflow, including asking basic questions such as "How do you open a file in C++?" in order to farm karma points. And it definitely worked! I have edit powers on that website now, and you can bet your ass I use them to clean up
misleading suggestions and stinky opinions whenever I see them.</p>

<p>Now, I don't want to poop on ThePrimeagen's parade, so I can find him some replacement lore.
Here you go, enjoy!
<a href="https://programmersheaven.com/discussion/142689/binary-files">Binary files - Programmers Heaven</a></p>

<blockquote>
 I don't have a clue about binary files. I'm a good programmer, have read more programming books than I can count, but still have no knowlege about binary files. Can you read them? Will they look like text? Is it good to use them? How can I print to a binary file, and then retreive from a binary file?
</blockquote>

<p>I was 14 years old when I wrote <em>this</em> cringefest. I think ThePrimeagen's take can survive transplanted from the Stack Overflow question to this forum post.</p>

<h2>Reddit</h2>

<p>Unlike Twitter, where I find the elemental unit of socializing problematic in and of itself,
I think the Reddit formula is solid gold. This one is a case of malevolent platform stewards.
I expect this from every platform run by for-profit entities, where the goal is not to serve
the users, but to milk them for capital gains. The grim reaper of capitalism has come
for Reddit.</p>

<p>Right now, venture capitalists are freaking out about "AI". Publicly-traded
businesses with access to structured user speech that can be used as
training data are rushing to build a moat around that data. That's why Twitter
prevented viewing tweets from logged out users, and Reddit locked down its APIs
despite it causing a massive protest from the moderators who do the actual
labor to keep the website running.</p>

<p>Detecting that Reddit enshittification was reaching terminal levels, I decided that during
these protests was the ideal time to close the /r/zig subreddit, which I was
currently a moderator of. I closed the subreddit, downloaded all my
user data, and permanently deleted my reddit account, leaving moderation to
<a href="https://kristoff.it/">Loris Cro</a>, VP of Community of
<a href="https://ziglang.org/zsf/">Zig Software Foundation</a>.</p>

<p>Once Loris was the sole moderator, he decided to change the subreddit to read-only mode.
However, this was thwarted today when a troll (check their comment history)
<a href="https://old.reddit.com/r/redditrequest/comments/15rtit7/requesting_rzig/">performed a hostile takeover in broad daylight</a> and
<a href="https://old.reddit.com/r/Zig/comments/15yoxok/rzig_is_reopened/">reopened the sub</a>.
It appears that Reddit admins are perfectly willing to forcibly replace moderators
in an effort to keep the system churning out user data. Those Large Language Models are hungry!</p>

<p>Anyway, I don't consider this to be a problem; it is outside of my control what Reddit does
with its own data. I just want to make it crystal clear that
<strong>Zig, Zig Software Foundation, and myself are not in any way affiliated
with /r/zig</strong>. That subreddit is now run by a third party and all signs
point to a high likelihood of problematic behavior happening there
which I definitely want to avoid being associated with.</p>

<p>Update (2023-08-30): As of today, the moderator of /r/zig is now Jens
Goldberg, a well-regarded member of the Zig community.</p>

<p>If you want an alternative to a zig subreddit, give <a href="https://ziggit.dev/">Ziggit</a>
a try. This is the <a href="https://www.discourse.org/">Discourse forum
software</a> run by dude_the_builder,
a friendly Zig community member, and I find it to be a healthy and rewarding way of socializing
with other community members.</p>

<h2>Moving Forward</h2>

<p>I have a Mastodon account, but I don't love it, for the same reasons I
didn't like Twitter. In addition, I think that way of consuming content
is generally like watching mainstream TV or listening to radio with ads. You're
letting a bunch of people who aren't really that important to you, or qualified
to do the job, be the content curators for you.</p>

<p>Discord has been decent for a while. I suspect enshittification will commence soon, so we
should be on the lookout for an alternative over the next five years.</p>

<p>I'm going to look into setting up an RSS reader for myself and start hunting for
high quality blogs.</p>

<p>And finally, I will be redirecting my micro-blogging energy that was previously wasted on
Twitter into actual-blogging energy here.</p>


guid: https://andrewkelley.me/post/the-techno-optimist-manifesto.html
title: The Techno-Optimist Manifesto
url: https://andrewkelley.me/post/the-techno-optimist-manifesto.html
feedurl: https://andrewkelley.me/rss.xml
content:
<h1>The Techno-Optimist Manifesto</h1>
<p>
Technology is a tool that enhances the power of the wielder.
</p>
<p>
Our tools are getting more and more powerful. Some tools, such as nuclear fission, are
so powerful that they pose an existential threat to all of humanity, because
human society has a history of wielding tools to maintain power over each other at any cost.
</p>
<p>
The dark desire to subjugate one another is the ultimate enemy of innovation,
because the conquerors are content to have their needs met by human labor,
while the conquered are too busy laboring to become scientists.
</p>
<p>
Human societies are systems. Systems are themselves tools. Free Market
Capitalism is a system. It has tech specs just like any computer - features,
limitations, and tradeoffs. We will not settle for a deprecated system when
there is a new and improved one that can be created, discovered, and iterated
upon.
</p>
<p>
Venture capital is an outdated tool. It leads to monopolies, anti-competitive
practices, generational wealth, and environmental externalities.
Non-donation-based non-profit organizations are an underexplored alternative,
especially for tech companies where the number of users is large and size of
the organization is relatively small.
</p>
<p>
Religion is an outdated tool, used to control, weaken, and exploit the masses.
It causes societies to maintain unscientific practices, especially in healthcare and
justice, and leads to war.
</p>
<p>
War is a means to destroy the wealth created by technology.
</p>
<p>
War turns technology against mankind.
</p>
<p>
War is opportunity cost for every scientist and engineer who works on national defense.
</p>
<p>
War makes it dangerous to research advanced technology.
</p>
<p>
As techno-optimists, we believe that we <em>must</em>, <strong>and we
will</strong>, create more advanced human systems to prevent war forevermore.
Human society <em>will</em> mature and overcome its
barbaric history, learning to cooperate with each other so that we can create technological
works of global magnitude, such as orbital rings.
</p>
<p>
We cannot meaningfully explore space while borders exist on Earth.
</p>
<p>
Every dollar spent on the military is a blight on the country's honor. We
believe that military budget reductions are a cause for celebration because
they symbolize progress towards the space age. We believe that the atomic
bombings of Hiroshima and Nagasaki were the most horrifying, shameful thing
ever done by humanity, to itself. Importantly, <strong>we believe that it will
never, ever happen again.</strong>
</p>
<p>
We believe that there are irreversible destructive actions. While some amount of
species extinction is a natural process, the outlier extinction event currently happening
due to the emergence of homo sapiens is a tragedy. We believe that it is our duty to
not only improve technology to enhance our own lives, but also enhance technology
to preserve natural diversity. We believe that Free Market Capitalism is an outdated
technology that has failed to prevent irreversible destruction to our home planet. However,
we believe that human society will migrate to a better system without these flaws,
and will prevent the ultimate irreversible action of self-destruction.
</p>
<p>
We believe in the simple, physical fact that endless growth is unsustainable,
and we refuse to resolve this problem with war. Instead, we will improve
efficiency and work within the limits of the system, whatever they may be.
We will upgrade our economic system to one that is capable of supporting these
requirements.
</p>
<p>
Therefore, finally, we believe that it is justified to spend one's life or
career in pursuit of science, engineering, and technology, because we believe
in humanity's ability to mature and improve to the point where it can wield
ever more powerful tools responsibly.
</p>


guid: https://andrewkelley.me/post/why-we-cant-have-nice-software.html
title: Why We Can't Have Nice Software
url: https://andrewkelley.me/post/why-we-cant-have-nice-software.html
feedurl: https://andrewkelley.me/rss.xml
content:
<h1>Why We Can't Have Nice Software</h1>
<p>
The problem with software is that it's too powerful. It creates so much wealth
so fast that it's virtually impossible to not distribute it.
</p>
<p>
Think about it: sure, it takes a while to make useful software. But then you
make it, and then it's done. It keeps working with no maintenance whatsoever,
and just a <em>trickle</em> of electricity to run it.
</p>
<p>
Immediately, this poses a problem: how can a small number of people keep all
that wealth for themselves, and not let it escape in the dirty, dirty fingers
of the general populace?
</p>
<p>
This is a question that the music industry faced head-on, and they came up
with EULAs, enforced via the state's monopoly on violence, and DRM, a way for
software to act antagonistically against its own users. Software can do useful
things like encode media into bits, and then copy those bits. That's
dangerously useful, and it had to be stopped.
</p>

<h2>The True Cause of Bitrot</h2>
<p>
What about bitrot, you say? It takes ongoing maintenance to keep software working, doesn't it?
</p>
<p>
Let's think critically about bitrot for a moment because, as a reminder, bits
don't actually rot - that's kinda the point of bits. In the best case scenario,
bitrot happens due to progress - perhaps a dependency has made improvements but
requires breaking API compatibility, or better hardware comes out and the
software needs to be recompiled for that hardware. In this case, it's kind of a
happy outcome. Some labor is needed to enhance the software in response, but
then, once again, it's <em>done</em>; ripples disappearing from the surface of
a lake hours after a stone is thrown into it.
</p>
<p>
The darker side of bitrot is due to businesses trying to make <strong>more profit
than last year</strong>, and launching marketing initiatives. For example, Microsoft
shipped a Windows Update that puts advertisements into the start menu,
advertisements into the task bar, and changed the control panel's user
interface to unify it with their business incentives - namely a superficial
makeover to justify customers paying additional money for what is effectively
worse software - it has new bugs and is now ridden with advertisements. This
caused a bunch of churn in their own codebase, as well as other software trying
to use native user interfaces on Windows.
</p>
<p>
It's all so incredibly <em>wasteful</em>.
And that's the point, isn't it?
</p>
<p>
The programmers at Microsoft could have done less work, or worked on bug fixes
instead. The UI designers could have done less work, or tweaked their existing
design instead of making a new one. The managers could have done less work.
Customers could have paid little to no additional money for a Windows Update.
This all would have culminated in a more robust version of Windows that
customers preferred, instead of one that is effectively boycotted like Vista
and Windows 11.
</p>
<p>
It's actually a problem that software is too efficient and has this nasty tendency
of being <em>completed</em>.
Software offers us a glimpse into a post-scarcity society, but it is being
actively sabotaged by those who seek to turn a profit.
</p>

<h2>Platform Waste</h2>
<p>
Consumers love standards. Standards allow multiple parties, perhaps even competitors,
perhaps <em>especially</em> competitors, to have interchangeable components with each
other, which gives consumers options, and negotiation power.
</p>
<p>
For-profit companies hate standards. They would rather have their own special cable,
for example, that only works for their devices, and only they are allowed to manufacture
them. To be more specific, underdog companies like standards because it
lets them compete. The established players don't want to have to play fair.
</p>
<p>
You can see this playing out right now with the
<a href="https://www.cnn.com/2022/10/24/tech/eu-law-charging-standard/index.html">EU formally adopting a law requiring Apple to support USB-C chargers</a>. At the time of writing there is no such law in the United States, but it is <a href="https://techcrunch.com/2022/06/17/senators-call-for-us-to-adopt-common-charger">being discussed by politicians</a>.
</p>
<p>
Standards allow software to be more efficient. By sticking with a standard for a period of time
and then coordinating an upgrade to a newer one, software churn is minimized, resulting in
a fixed amount of software development labor needed.
</p>
<p>
On the other hand, without a standard, for-profit companies are incentivized to
fiddle with their product in a wasteful manner. For example, Apple has in the past made
insignificant changes to their charging cable, making it not compatible with
the one from the previous year. This resulted in more profit for Apple since
consumers found their existing cables useless and had to buy new ones. Ultimately
this resulted in more money being spent in the economy, increasing the country's GDP.
Economists rejoice; the Earth weeps.
</p>
<p>
Think about how many messaging apps have come and go and how much programmer
hours have been wasted on them. We almost had XMPP be mainstream, but then
Google outgrew their "don't be evil" diaper and put on their "make profit" big
boy pants. If your goal is to turn a profit, it's obviously the correct choice
to invest into a platform that you own. So then we got a half-dozen buggy
messaging apps from Google that didn't even work with each other, let alone
Apple's platform or the other contemporary players.
</p>
<p>
The new hotness is Discord, which is already starting, predictably, 
<a href="https://www.eff.org/deeplinks/2023/04/platforms-decay-lets-put-users-first">to decay</a>. I can't believe a human sat down and wasted hours of their life coding "super reactions". It's
not something that really needed to happen.
</p>
<p>
Imagine if all these programmer hours spent on all these products actually centered around
a proper standard, which evolved along with consumers' needs rather than
these companies' ongoing need to fiddle with the knobs and sliders until
profit comes out. The thing is, if this actually happened, then
<em>what would these employees spend their time on?</em> At some point society
would be pretty much done implementing messaging software. Messaging app updates
would be rare, and bugs in messaging apps would be rare. We would reach peak messaging.
</p>

<h2>Peak Dishwasher</h2>
<p>Decades ago, we already did it. We reached peak dishwasher. Dishwashers
achieved perfection, and it was no longer possible to improve them. The mechanics
were optimal, the user interface was ideal, and consumers had no desire for any
changes.
</p>
<p>
One would, naively, think of this as an accomplishment. But how is a company supposed
to make <em>more profit than last year</em>? By any means necessary, of course.
</p>
<p>
They invented these dishwasher detergent pods that are actually a downgrade - slightly
more time consuming to use than powder, more expensive to manufacture and purchase,
worse for the environment, and most offensive of all -
<strong>actively sabotage the dishwasher's prewash feature</strong> making the product
actually function worse than before!
</p>
<p>
And from a business perspective, it is a critical success. They found a way to
make consumers spend more money on dishwashing. The line goes up, for one more
year. But it's not enough. It has to go up every year. What else can we do?
</p>
<p>
I found myself in a position where I needed to buy a new dishwasher last month, and,
already being aware of this problem, did my very best to buy one that worked well.
I picked one that had 5/5 stars on Consumer Reports. Unfortunately, the
dishwasher that I ended up with is my worst nightmare.
</p>
<p>
It takes 30 seconds to boot up, presumably because of the Bluetooth and WiFi driver
in it. Many of the configuration options are hidden behind a proprietary app.
The buttons are hidden and touch based instead of being visible and depressing
with natural tactile feedback. I still haven't yet done the chore of going into
my router and disabling it from accessing the Internet. I had to give it access to
use the app to find out why it was broken. Until I do that chore, there's a chance it
could auto update and have a firmware bug and stop working, or just waste my bandwidth.
Who knows what it's up to?
</p>
<p>
Meanwhile, I  had to call a repair technician to fix the door latch already, as well
as the soap dispenser latch. Both things have since failed to work properly again and
I still need to do the chore of calling the company to get a repair done a third time.
</p>
<p>
Before we moved, we had an older dishwasher that worked perfectly. No Internet,
no Bluetooth, and the door latches worked flawlessly through thousands of runs.
</p>
<p>
The problem with the requirement for each year to be more profitable than the
last is that once you reach the peak, once it's not possible to actually
improve your product any more, <em>you still have to change something</em>.
Since you can't change it to make it better, you therefore will change it to
make it worse.
</p>

<h2>What Blockchains and LLMs Have in Common</h2>
<p>
Plenty of people roll their eyes at blockchain being the new buzzword, or about
how tech is overobsessed with AI (LLMs) right now. It's easy to chock it up to
it being a silly, harmless fad perpetuated by uneducated or misguided people,
but in reality it's a lot more intentional than that.
</p>
<p>
Most tech workers work 40 hours per week at some company. That's a lot of
collective hours spent on something. What factors go into deciding, as a whole,
what that effort is spent on? Employees have some choices in the matter, but in
the end those choices are limited to job offers. Job offers are created by the
owners of companies who decide what they want to invest their money into.
</p>
<p>
In other words, venture capitalists decide what is the current hotness
precisely by directing large amounts of labor towards whatever they want.
Empirically, VCs are primarily motivated by seeking a return on
investment. The goal is to turn a big sum of money into an even bigger sum of
money. In theory, this is because with an even bigger sum of money, you can
then start to spend that money on directing an even larger amount of labor
towards whatever you want, bypassing democracy to influence the future of
humanity, but in practice, most VCs get fixated on that return on investment
until they die. 
</p>
<p>
If you were to criticize blockchain technology from a purely technical perspective,
you might point out the flaw that proof of work requires an exponentially increasing
amount of computational power, and thus electricity, in order to keep the blockchain
database alive over time. You might point out how inefficient of a database it is.
But you would be missing the point.
<strong>Blockchain technology excites investors precisely because of how wasteful it is</strong>.
Even if we had fusion (!!) it would eat up all that energy and more. It's difficult to express
the magnitude of how wasteful this is, and the fact that it's built into the system intentionally
is sinister.
</p>
<p>
Blockchain technology is a kind of software that doesn't get completed or
perfected. Rather it's the opposite; the longer it is in existence the more
work it creates for everyone to do. The waste is a feature; it's how the <a
href="https://www.youtube.com/watch?v=YQ_xWvX1n9g">line goes up</a>.
</p>

<p>Reminds me of this scene in The Fifth Element where Zorg says:</p>
<blockquote>
Life, which you so nobly serve, comes from destruction, disorder, and chaos.
Now take this empty glass. Here it is: peaceful, serene, <em>boring</em>. But if
it is destroyed... look at all these little things! So busy now. Notice how each one
of them is useful. What a lovely ballet ensues, so full of form and color. Now, think
about all those people that created them. Technicians, engineers, hundreds of people
who will be able to feed their children tonight so those children can grow up big and
strong and have little teeny children of their own and so on and so forth. Thus adding
to the great chain of life.
</blockquote>
<p>You can find this scene on YouTube but I won't link it for fear of accidentally
causing someone to view an advertisement.</p>
<p>
LLMs offer an even more ideal kind of software to the investor. First of all
they require an enormous amount of capital to train, and specialized hardware to run,
making them suitable to offer as a service, where the amount of profit can be made to go
up in a controlled manner. What a delicious idea.
</p>
<p>
More to my point, they offer a host of subjective, ill-defined tasks that are
immune to being completed. They've managed to take something well-defined,
well-scoped, and completable, and turn it into an untameable monster that will
be sure to offer software churn for decades to come.
</p>
<p>
Have a peek at this blog post that is going around lately:
<a href="https://austinhenley.com/blog/copilotpainpoints.html">The pain points of building a copilot</a>
</p>
<p>
These people are brimming with excitement about all the new problems that LLMs are bringing
to the table. Some choice quotes:
</p>
<blockquote>
Prompt engineering is time consuming and requires considerable trial and error...As one developer said, "it's more of an art than a science".
</blockquote>
<blockquote>
Testing is fundamental to software development but arduous when LLMs are involved. Every test is a flaky test. 
</blockquote>
<blockquote>
The field is moving fast, and it requires developers to "throw away everything that they've learned and rethink it."
</blockquote>
<blockquote>
Developers are having to learn and compare many new tools rather than focusing on the customer problem. They then have to glue these tools together.
</blockquote>
<blockquote>
It is still the wild, wild west ... It will be interesting to see how software engineering will evolve, either through new processes or tools, over the next several years.
</blockquote>

<p>
LLMs are a way to make software take orders of magnitude more computational
power, electricity, and human labor, while delivering a product whose extremely
volatile quality is impossible to assure. The work will never be completed; it
will only create the need for ever more labor.
</p>
<p>
For investors, all this churn is attractive. It's <em>disruptive</em>.
</p>
<p>
It's why we can't have nice software.
</p>

<h2>Conclusion</h2>
<p>Technology, and in particular, software, offers a glimpse of magic; a
perpetual motion machine; wealth created from nothing. It offers us a chance to
work together on something beautiful; to achieve perfection by ratcheting
improvements over time.</p>
<p>In the end, this opportunity is squandered in a doomed quest for endless growth.</p>


guid: https://andrewkelley.me/post/redis-renamed-to-redict.html
title: Redis Renamed to Redict
url: https://andrewkelley.me/post/redis-renamed-to-redict.html
feedurl: https://andrewkelley.me/rss.xml
content:
<h1>Redis Renamed to Redict</h1>
<p><a href="https://redict.io/">Redict</a> was originally
created by Salvatore Sanfilippo under the name "Redis".
Around 2018 he started losing interest in the project to pursue a science fiction career and gave
stewardship of the project to <a
href="https://en.wikipedia.org/wiki/Redis_(company)">Redis Labs</a>.</p>
<p>
I think that was an unfortunate move because their goal is mainly to extract profit from
the software project rather than to uphold the ideals of Free and Open Source Software.
On March 20, 2024, they
<a href="https://github.com/redis/redis/pull/13157">changed the license to be proprietary</a>,
a widely unpopular action. You know someone is up to no good when they write
"Live long and prosper 🖖" directly above a meme of Darth Vader.
</p>
<img class="light" alt="screenshot of the PR merging with many downvotes and darth vader" src="/img/redis-screenshot-light.png">
<img class="dark" alt="screenshot of the PR merging with many downvotes and darth vader" src="/img/redis-screenshot-dark.png">

<h2 id="license-problems">What are the actual problems with these licenses?</h2>
<p>In short summary, the licenses limit the freedoms of what one can do with
the software in order for Redis Labs to be solely enriched, while asking for
volunteer labor, and having <a
href="https://github.com/redis/redis/pull/13157#issuecomment-2014737480">already
benefitted from volunteer labor</a>, that is and was <a
href="https://news.ycombinator.com/item?id=39775468">generally offered only
because it enriches <em>everybody</em></a>.</p>
<p><a href="https://ssplisbad.com/">SSPL is BAD</a></p>

<h2 id="legality">Are they allowed to do this?</h2>
<p>All the code before the license change is available under the previous license (BSD-3),
however it is perfectly legal to make further changes to the project under a different license.</p>
<p>This means that it is also legal to <em>fork</em> the project from before the license change,
and continue maintaining the project without the proprietary license change. The only problem
there would be, the project would be missing out on all those juicy future
contributions from Redis Labs... wait a minute, isn't the project already <em>done</em>?
</p>

<h2 id="already-completed">Redict is a Finished Product</h2>
<p>Redict already works great. Lots of companies already use it in production and have
been doing so for many years.</p>
<p>In <a href="/post/why-we-cant-have-nice-software.html">Why We Can't Have Nice Software</a>,
I point out this pattern of needless software churn in the mindless quest for
profit. This is a perfect example occurring right now. Redict has already reached its
peak; it does not need any more serious software development to occur.
It does not need to <a href="https://redis.com/blog/the-future-of-redis/">pivot
to AI</a>. It can be maintained for decades to come with minimal effort. It can
continue to provide a high amount of value for a low amount of labor. That's
the entire point of software!</p>
<p><strong>Redict does not have any profit left to offer</strong>. It no longer
needs a fund-raising entity behind it anymore. It just needs a good project steward.</p>

<h2 id="drew">Drew DeVault is a Good Steward</h2>
<p><a href="https://drewdevault.com/">Drew</a> is a controversial person, I
think for two reasons.</p>
<p>One, is that he has a record of being rude to many people in the past -
including myself. However, in a <a href="https://lore.kernel.org/lkml/CA+55aFy+Hv9O5citAawS+mVZO+ywCKd9NQ2wxUmGsz9ZJzqgJQ@mail.gmail.com/">similar manner as Linus Torvalds</a>,
Drew has expressed what I can only interpret as sincere regret for such interactions, as
well as a pattern of improved behavior. I was poking through his blog to try to find
example posts of what I mean, and it's difficult to pick them out because he's such a
prolific writer, but perhaps <a href="https://drewdevault.com/2022/05/30/bleh.html">this one</a>
or maybe <a href="https://drewdevault.com/2023/05/01/2023-05-01-Burnout.html">this one</a>.
I'm a strong believer in applying
<a href="https://en.wikipedia.org/wiki/Tit_for_tat">the best game theory strategy</a>
to society: people should have consequences for the harm that they do, but then they should
get a chance to start cooperating again. I can certainly think of "cancelable" things
I have done in the past, that I am thankful are not public, and I cringe every
time I remember them.
</p>

<p>Secondly, and I think this is actually the more important point, Drew has been
an uncompromising advocate of Free and Open Source Software his entire life,
walking the walk more than anyone else I can think of. It's crystal clear that
this is the driving force of his core ideology that determines all of his decision
making. He doesn't budge on any of these principles and it creates conflicts
with people who are trying to exploit FOSS for their own gains. For example, when
you <a href="https://drewdevault.com/2023/07/04/Dont-sign-a-CLA-2.html">call out SourceGraph</a>
you basically piss off everyone who has SourceGraph stock. Do enough of these callouts,
and you've pissed off enough people that there's an entire meme subculture around hating you.</p>

<p>Meanwhile, Drew created and maintained <a
href="https://gitlab.freedesktop.org/wlroots/wlroots">wlroots</a> and <a
href="https://swaywm.org/">Sway</a>, successfully appointing a successor
maintainer to carry the torch, and runs <a href="https://sourcehut.org/">a
sustainable business</a> on top of
<a href="https://sr.ht/~sircmpwn/sourcehut/">Free and Open Source Software</a>.
SourceHut has a dependency on Redict, so it naturally follows that Drew wants
to keep his supply chain FOSS.</p>

<h2 id="redis-is-the-fork">Redis is the Fork</h2>
<p>
The only thing Redis has going for it, as a software project, is the brand name.
Salvatore is long gone. The active contributors who are working on it are, like I said,
pivoting to AI. Seriously, here's a quote from <a href="https://redis.com/blog/the-future-of-redis/">The Future of Redis</a>:
<blockquote>
<p>Making Redis the Go-To for Generative AI</p>
<p>we’re staying at the forefront of the GenAI wave</p>
</blockquote>
<p>
Meanwhile, Redict has an actual Free and Open Source Software movement behind
it, spearheaded by Drew DeVault, who has a track record of effective open
source project management.
</p>
<p>
In other words, Redict is the true spiritual successor to what was once Redis.
The title of this blog post is not spicy or edgy; it reflects reality.
</p>


guid: https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html
title: Zig's New CLI Progress Bar Explained
url: https://andrewkelley.me/post/zig-new-cli-progress-bar-explained.html
feedurl: https://andrewkelley.me/rss.xml
content:
<script>
  Prism.languages['zig'] = Prism.languages.extend('clike', {
    'keyword': /\b(test|fn|import|cImport|const|var|extern|volatile|export|pub|noalias|inline|struct|enum|break|return|continue|asm|defer|if|else|switch|try|catch|while|for|null|undefined|true|false|comptime|setCold|ptrToInt|returnAddress)\b/,
    'property': /\b(bool|i8|u8|i16|u16|i32|u32|i64|u64|isize|usize|f32|f64|void|unreachable|type|error|c_short|c_ushort|c_int|c_uint|c_long|c_ulong|c_longlong|c_ulonglong|noreturn)\b/,
  });
</script>
<h1>Zig's New CLI Progress Bar Explained</h1>

<p>
Sometimes, programming projects are too easy and boring. Sometimes, they're too
hard, never ending or producing subpar results.
</p><p>
This past week I had the pleasure of completing a project that felt like
maximum difficulty - only possible because I am at the top of my game, using a
programming language designed for making perfect software. This problem threw
everything it had at me, but I rose to the challenge and emerged victorious.
</p><p>
What a rush.
</p><p>
In this blog post I'll dig into the technical implementation as well as provide the
<a href="#zig-progress-protocol-spec">Zig Progress Protocol Specification</a>.
</p>

<h2 id="demo">Demo</h2>

<p>
Before we take a deep dive, let's look at the final results by building my <a
href="https://codeberg.org/andrewrk/player">music player side project</a> while recording with Asciinema:
</p><p>
<a href="https://asciinema.org/a/MfJdqRHlMaHeNY8KJSnHBUP2I">Old</a> vs
<a href="https://asciinema.org/a/661404">New</a>
</p><p>
The usage code looks basically like this:
</p>
<pre><code class="language-zig">const parent_progress_node = std.Progress.start(.{});

// ...

const progress_node = parent_progress_node.start("sub-task name", 10);
defer progress_node.end();

for (0..10) |_| progress_node.completeOne();</code></pre>
<p>To include a child process's progress under a particular node, it's a single assignment
before calling <code>spawn</code>:</p>
<pre><code class="language-zig">child_process.progress_node = parent_progress_node;</code></pre>
<p>
For me the most exciting thing about this is its ability to visualize what the
Zig Build System is up to after you run <code>zig build</code>. Before this
feature was even merged into master branch, it led to discovery, diagnosis, and
<a href="https://github.com/ziglang/zig/commit/389181f6be8810b5cd432e236a962229257a5b59">resolution</a>
of a subtle bug that has hidden in Zig's standard library child process
spawning code for years. Not included in this blog post: a rant about how much
I hate the fork() API.
</p>

<h2 id="motivation">Motivation</h2>
<p>
The previous implementation was more conservative. It had the design limitation
that it could not assume ownership of the terminal. This meant that it had to
assume another process or thread could print to stderr at any time, and it was
not allowed to register a <code>SIGWINCH</code> signal handler to learn about when the
terminal size changed. It also did not rely on knowledge of the terminal size,
or spawn any threads.
</p><p>
This new implementation represents a more modern philosophy: when a CLI
application is spawned with stderr being a terminal, then that application owns
the terminal and owes its users the best possible user experience, taking advantage of
all the terminal features available, working around their limitations. I'm
excited for projects such as <a
href="https://mitchellh.com/ghostty">Ghostty</a> which are expanding
the user interface capabilities of terminals, and lifting those limitations.
</p><p>
With this new set of constraints in mind, it becomes possible to design a much
more useful progress bar. We gain a new requirement: since only one process
owns the terminal, child processes must therefore report their progress
semantically so it can be aggregated and displayed by the terminal owner.
</p>

<h2 id="implementation">Implementation</h2>
<p>
The whole system is designed around the public API, which must be thread-safe, lock-free, and avoid
contention as much as possible in order to prevent the progress system itself from harming performance,
particularly in a multi-threaded environment.
</p><p>
The key insight I had here is that, since the end result must be displayed on a
terminal screen, there is a reasonably small upper bound on how much memory is
required, beyond which point the extra memory couldn't be utilized because it
wouldn't fit on the terminal screen anyway.
</p><p>
By statically pre-allocating 200 nodes, the API is made infallible and non-heap-allocating. Furthermore,
it makes it possible to implement a thread-safe, lock-free node allocator based on a free list.
</p><p>
The shared data is split into several arrays:
</p><p>
<pre><code class="language-zig">node_parents: [200]Node.Parent,
node_storage: [200]Node.Storage,
node_freelist: [200]Node.OptionalIndex,</code></pre>
<p>
The freelist is used to ensure that two racing <code>Node.start()</code> calls obtain
different indexes to operate on, as well as a mechanism for <code>Node.end()</code> calls
to return nodes back to the system for reuse. Meanwhile, the parents array is
used to indicate which nodes are actually allocated, and to which parent node
they should be attached to. Finally, the other storage contains the number of
completed items, estimated total items, and task name for each node.
</p><p>
Each <code>Node.Parent</code> is 1 byte exactly. It has 2 special values, which can be
represented with type safety in Zig:
</p>
<pre><code class="language-zig">const Parent = enum(u8) {
    /// Unallocated storage.
    unused = std.math.maxInt(u8) - 1,
    /// Indicates root node.
    none = std.math.maxInt(u8),
    /// Index into `node_storage`.
    _,

    fn unwrap(i: @This()) ?Index {
        return switch (i) {
            .unused, .none =&gt; return null,
            else =&gt; @enumFromInt(@intFromEnum(i)),
        };
    }
};</code></pre>
<p>
The data mentioned above is mutated in the implementation of the thread-safe public API.
</p><p>
Meanwhile, the update thread is running on a timer. After an initial delay, it
wakes up at regular intervals to either draw progress to the terminal, or send
progress information to another process through a pipe.
</p><p>
In either case, when the update thread wakes up, the first thing it does is "serialize" the
shared data into a separate preallocated location. After carefully copying the
shared data using atomic primitives, the copied, serialized data can then be
operated on in a single-threaded manner, since it is not shared with any other
threads.
</p><p>
The update thread performs this copy by iterating over the full 200 shared
parents array, atomically loading each value and checking if it is not the
special "unused" value (0xfe). In most programming projects, a linear scan to
find allocated objects is undesirable, however in this case 200 bytes for node
parent indexes is practically free to iterate over since it's 4 cache lines
total, and the only contention comes from actual updates that must be observed.
This full scan in the update thread buys us a cheap, lock-free implementation
for <code>Node.start()</code> and <code>Node.end()</code> via popping and
pushing the freelist, respectively.
</p><p>
Next, IPC nodes are expanded. More on this point later.
</p><p>
Once the serialization process is complete, we are left with a subset of the shared data from
earlier, this time with no gaps - only used nodes are present here:
</p><p>
<pre><code class="language-zig">serialized_parents: [200]Node.Parent,
serialized_storage: [200]Node.Storage,
serialized_len: usize,</code></pre>
<p>
At this point the behavior diverges depending on whether the current process owns the terminal, or
must send progress updates via a pipe. A process that fits neither of these categories would not
have spawned an update thread to begin with. Any parent process that wants to
track progress from children creates a pipe with <code>O_NONBLOCK</code> enabled, passing
it to the child as if it were a fourth I/O stream after stdin, stdout, and
stderr. To indicate to the process that the file descriptor is in fact a
progress pipe, it sets the <code>ZIG_PROGRESS</code> environment variable. For example, in
the Zig standard library implementation, this ends up being <code>ZIG_PROGRESS=3</code>.
</p><p>
A process that is given a progress pipe sends the serialized data over the
pipe, while a process that owns the terminal draws it directly.
</p>
<p>For example, this source code:</p>
<pre><code class="language-zig">const std = @import("std");

pub fn main() !void {
    const root_node = std.Progress.start(.{
        .root_name = "preparing assets",
    });
    defer root_node.end();

    const sub_node = root_node.start("reticulating splines", 100);
    defer sub_node.end();

    for (0..50) |_| sub_node.completeOne();
    std.time.sleep(1000 * std.time.ns_per_ms);
    for (0..50) |_| sub_node.completeOne();
}</code></pre>
<p>At the sleep() call, would display this to the terminal:</p>
<pre>preparing assets
└─ [50/100] reticulating splines</pre>
<p>But if it were a child process, would send this message over the pipe instead:</p>
<pre>0000  02 00 00 00 00 00 00 00  00 70 72 65 70 61 72 69  .........prepari
00a0  6E 67 20 61 73 73 65 74  73 00 00 00 00 00 00 00  ng assets.......
00b0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00c0  00 32 00 00 00 64 00 00  00 72 65 74 69 63 75 6C  .2...d...reticul
00d0  61 74 69 6E 67 20 73 70  6C 69 6E 65 73 00 00 00  ating splines...
00e0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................
00f0  00 FF 00                                          ...</pre>

<h3 id="drawing-to-the-terminal">Drawing to the Terminal</h3>
<p>Drawing to the terminal begins with the serialized data copy. This data contains only edges
pointing to parents, and therefore cannot be used to walk the tree starting from the root. The first
thing done here is compute sibling and children edges into preallocated buffers so that we can
then walk the tree top-down.</p>

<pre><code class="language-zig">child: [200]Node.OptionalIndex,
sibling: [200]Node.OptionalIndex,</code></pre>

<p>Next, the draw buffer is computed, but not written to the terminal yet. It looks like this:</p>
<ol>
  <li><a href="https://gist.github.com/christianparpart/d8a62cc1ab659194337d73e399004036">Start sync sequence</a>. This makes high framerate terminals not blink rapidly.</li>
  <li>Clear previous update by moving cursor up times the number of newlines
  outputted last time, and then a clear to end of screen escape sequence.</li>
  <li>Recursively walk the tree of nodes, which which is now possible since we
  have computed children and sibling edges, outputting tree-drawing sequences,
  node names, and counting newlines.</li>
  <li>End sync sequence.</li>
</ol>

<p>This draw buffer is only computed - it is not sent to the write() syscall yet. At this point,
we try to obtain the stderr lock. If we get it, then we write the buffer to the terminal. Otherwise,
this update is dropped.</p>

<p>If any attempt to write to the terminal fails, the update thread exits so
that no further attempt is made.</p>

<h3 id="ipc">Inter-Process Communication</h3>
<p>Earlier, I said "IPC nodes are expanded" without further explanation. Let's dig into that a little bit.</p>

<p>As a final step in the serialization process, the update thread iterates the data, looking for
special nodes. Special nodes store the progress pipe file descriptor of a child process rather than
<code>completed_items</code> and <code>estimated_total_items</code>. The update thread reads progress
data from this pipe, and then grafts the child's sub-tree onto the parent's
tree, plugging the root node of the child into the special node of the parent.
The main storage data can be directly memcpy'd, but the parents array must be
relocated based on the offset within the serialized data arrays.
</p><p>
In case of a big-endian system, the 2 integers within each node storage must be
byte-swapped. Not relying on host endianness means that edge cases continue to
work, such as the parent process running on an x86_64 host, with a child
process running a mips program in QEMU user mode. This is a real use case when
testing the Zig compiler, for example.
</p><p>
The parent process ignores all but the last message from the pipe fd, and
references a copy of the data from the last update in case there is no message
in the pipe for a particular update.
</p>
<h2 id="performance">Performance</h2>
<p>
Here are some performance data points I took while working on this.
</p><p>
Building the Zig compiler as a sub-process. This measures the cost of the new
API implementations, primarily <code>Node.start</code>, <code>Node.end</code>,
and the disappearance of <code>Node.activate</code>. In this case, standard
error is not a terminal, and thus an update thread is never spawned:
</p>
<pre>Benchmark 1 (3 runs): old zig build-exe self-hosted compiler
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          51.8s  ±  138ms    51.6s  … 51.9s           0 ( 0%)        0%
  peak_rss           4.57GB ±  286KB    4.57GB … 4.57GB          0 ( 0%)        0%
  cpu_cycles          273G  ±  360M      273G  …  274G           0 ( 0%)        0%
  instructions        487G  ±  105M      487G  …  487G           0 ( 0%)        0%
  cache_references   19.0G  ± 31.8M     19.0G  … 19.1G           0 ( 0%)        0%
  cache_misses       3.87G  ± 12.0M     3.86G  … 3.88G           0 ( 0%)        0%
  branch_misses      2.22G  ± 3.44M     2.21G  … 2.22G           0 ( 0%)        0%
Benchmark 2 (3 runs): new zig build-exe self-hosted compiler
  measurement          mean ± σ            min … max           outliers         delta
  wall_time          51.5s  ±  115ms    51.4s  … 51.7s           0 ( 0%)          -  0.4% ±  0.6%
  peak_rss           4.58GB ±  190KB    4.58GB … 4.58GB          0 ( 0%)          +  0.1% ±  0.0%
  cpu_cycles          272G  ±  494M      272G  …  273G           0 ( 0%)          -  0.4% ±  0.4%
  instructions        487G  ± 63.5M      487G  …  487G           0 ( 0%)          -  0.1% ±  0.0%
  cache_references   19.1G  ± 16.9M     19.1G  … 19.1G           0 ( 0%)          +  0.3% ±  0.3%
  cache_misses       3.86G  ± 17.1M     3.84G  … 3.88G           0 ( 0%)          -  0.2% ±  0.9%
  branch_misses      2.23G  ± 5.82M     2.22G  … 2.23G           0 ( 0%)          +  0.4% ±  0.5%</pre>

<p>
Building the Zig compiler, with <code>time zig build-exe -fno-emit-bin
...</code> so that the progress is updating the terminal on a regular interval:
</p>
<ul>
<li>Old:<ul>
  <li>4.115s</li>
  <li>4.216s</li>
  <li>4.221s</li>
  <li>4.227s</li>
  <li>4.234s</li></ul></li>
<li>New (1% slower):<ul>
  <li>4.231s</li>
  <li>4.240s</li>
  <li>4.271s</li>
  <li>4.339s</li>
  <li>4.340s</li></ul></li>
</ul>
<p>
Building my music player application with <code>zig build</code>, with the
project-local cache cleared. This displays a lot of progress information to the
terminal. This data point accounts for many sub processes sending progress
information over a pipe to the parent process for aggregation:
</p>
<ul>
<li>Old<ul>
  <li>65.74s</li>
  <li>66.39s</li>
  <li>71.09s</li></ul></li>
<li>New (1% faster)<ul>
  <li>65.51s</li>
  <li>65.88s</li>
  <li>66.09s</li></ul></li>
</ul>
<p>
Conclusion? It appears that I succeeded in making this progress-reporting
system have no significant effect on the performance of the software that uses
it.
</p>
<h2 id="zig-progress-protocol-spec">The Zig Progress Protocol Specification</h2>
<p>Any programming language can join the fun! Here is a specification so that any software project
can participate in a standard way of sharing progress information between parent and child processes.
</p><p>
When the <code>ZIG_PROGRESS=X</code> environment variable is present, where <code>X</code> is an
unsigned decimal integer in range 0...65535, a process-wide progress reporting channel is
available.
</p><p>
The integer is a writable file descriptor opened in non-blocking mode.
Subsequent messages to the stream supersede previous ones.
</p><p>
The stream supports exactly one kind of message that looks like this:
</p>
<ol>
<li><code>len: u8</code> - number of nodes, limited to 253 max, reserving 0xfe and 0xff for
   special meaning.</li>
<li>48 bytes for every <code>len</code>:<ol>
   <li><code>completed: u32le</code> - how many items already done</li>
   <li><code>estimated_total: u32le</code> - guessed number of items to be completed, or 0 (unknown)</li>
   <li><code>name: [40]u8</code> - task description; remaining bytes zeroed out</li>
   </ol></li>
<li>1 byte for every <code>len</code>:<ol>
   <li><code>parent: u8</code> - creates an edge to a parent node in the tree, or 0xff (none)</li>
   </ol></li>
</ol>
<p>
Future versions of this protocol, if necessary, will use different environment
variable names.
</p>

<h2 id="c">Bonus: Using Zig Standard Library in C Code</h2>
<p>Much of the Zig standard library is available to C programs with minimal integration pain. Here's
a full, working example:</p>

<h3 id="example.c">example.c</h3>

<pre><code class="language-c">#include "zp.h"
#include &lt;string.h&gt;
#include &lt;unistd.h&gt;

int main(int argc, char **argv) {
    zp_node root_node = zp_init();

    const char *task_name = "making orange juice";
    zp_node sub_node = zp_start(root_node, task_name, strlen(task_name), 5);
    for (int i = 0; i &lt; 5; i += 1) {
        zp_complete_one(sub_node);
        sleep(1);
    }
    zp_end(sub_node);
    zp_end(root_node);
}</code></pre>

<h3 id="zp.h">zp.h</h3>

<pre><code class="language-c">#include &lt;stdint.h&gt;
#include &lt;stddef.h&gt;

typedef uint8_t zp_node;

zp_node zp_init(void);
zp_node zp_start(zp_node parent, const char *name_ptr, size_t name_len, size_t estimated_total);
zp_node zp_end(zp_node node);
zp_node zp_complete_one(zp_node node);</code></pre>

<h3 id="zp.zig">zp.zig</h3>

<pre><code class="language-zig">const std = @import("std");

export fn zp_init() std.Progress.Node.OptionalIndex {
    return std.Progress.start(.{}).index;
}

export fn zp_start(
    parent: std.Progress.Node.OptionalIndex,
    name_ptr: [*]const u8,
    name_len: usize,
    estimated_total_items: usize,
) std.Progress.Node.OptionalIndex {
    const node: std.Progress.Node = .{ .index = parent };
    return node.start(name_ptr[0..name_len], estimated_total_items).index;
}

export fn zp_end(node_index: std.Progress.Node.OptionalIndex) void {
    const node: std.Progress.Node = .{ .index = node_index };
    node.end();
}

export fn zp_complete_one(node_index: std.Progress.Node.OptionalIndex) void {
    const node: std.Progress.Node = .{ .index = node_index };
    node.completeOne();
}

pub const _start = void;</code></pre>

<p>Compile with <code>zig cc -o example example.c zp.zig</code></p>

<p>
<a href="https://asciinema.org/a/dJ8iLKpje6fsJygM8r1qw5e6M">Asciinema Demo</a>
</p>

<p>This same executable will also work correctly as a child process, reporting
progress over the <code>ZIG_PROGRESS</code> pipe if provided!</p>

<h2 id="follow-up-work">Follow-Up Work</h2>
<p>Big thanks to <a href="https://www.ryanliptak.com/">Ryan Liptak</a> for helping me with the
Windows console printing code, and <a href="https://github.com/jacobly0/">Jacob Young</a> for
<a href="https://github.com/ziglang/zig/pull/20114">working on the Windows IPC logic</a>, which isn't done yet.</p>


guid: 2009-10-20-css-box-model-differences-in-firefox-and-internet-explorer
title: CSS Box Model differences in Firefox and Internet Explorer
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2009-10-20-css-box-model-differences-in-firefox-and-internet-explorer

content:
Here is the bane of web developers existence, Internet Explorer incorrectly displaying the CSS box model. What is the box model? This describes all elements used in HTML documents.


guid: 2009-11-04-introduction-to-photoshop-blending-options
title: Introduction to Photoshop Blending Options
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-04-introduction-to-photoshop-blending-options
feedurl: https://thomashunter.name/feed.rss
content:
Here we will describe several of the popular blending options used in photoshop, with screenshots and explanations of each setting.


guid: 2009-11-06-web-image-formats-size-and-quality-comparison
title: Web Image Formats Size and Quality Comparison
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-06-web-image-formats-size-and-quality-comparison
feedurl: https://thomashunter.name/feed.rss
content:
This is a demonstration of the different image formats used for the web, with an explanation of which situation to use one or the other.


guid: 2009-11-07-create-windows-vista-like-buttons-using-photoshop
title: Create Windows Vista like buttons using Photoshop
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-07-create-windows-vista-like-buttons-using-photoshop
feedurl: https://thomashunter.name/feed.rss
content:
This is a step by step guide on how to make attractive buttons in Photoshop which look just like the Windows Vista buttons.


guid: 2009-11-13-simple-ajax
title: Simple AJAX
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-13-simple-ajax
feedurl: https://thomashunter.name/feed.rss
content:
AJAX is not a language. It is simply giving a name to something that has existed for years. Using the DOM of the web browser you are allowed to pass XML between the server and the client without having to reload the page. You don't necessarily have to pass XML; in this example we will be passing simple text.


guid: 2009-11-15-introduction-to-css
title: Introduction to CSS: Selectors and Relationships Explained
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-15-introduction-to-css
feedurl: https://thomashunter.name/feed.rss
content:
This article will provide an introduction to CSS, which also requires an explanation of HTML (even if you know HTML you may want to read through it as it explains the relationship model). This article won't explain every single CSS attribute, but will explain everything you need to know to fully comprehend the language.


guid: 2009-11-16-fl-studio-interface-overview
title: FL Studio Interface Overview
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-16-fl-studio-interface-overview
feedurl: https://thomashunter.name/feed.rss
content:
This is an overview of the FL Studio application interface. This describes every window used in FL Studio.


guid: 2009-11-17-fl-studio-automations
title: FL Studio Automations
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-17-fl-studio-automations
feedurl: https://thomashunter.name/feed.rss
content:
An explanation of Automation Clips in FL Studio, how to create and modify them and assign them to different properties in FL Studio.


guid: 2009-11-17-polymorphism-abstract-classes-and-interfaces-in-php
title: Polymorphism, Abstract Classes, and Interfaces in PHP
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-17-polymorphism-abstract-classes-and-interfaces-in-php
feedurl: https://thomashunter.name/feed.rss
content:
This article will explain the advanced topics of PHP classes, including polymorphism, abstract classes, and interfaces.


guid: 2009-11-18-fl-studio-comprehensive-mixer-interface-overview
title: FL Studio Comprehensive Mixer Interface Overview
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-18-fl-studio-comprehensive-mixer-interface-overview
feedurl: https://thomashunter.name/feed.rss
content:
This tutorial will provide you with a comprehensive understanding of the mixer interface for FL Studio. We describe every single knob/option on the interface.


guid: 2009-11-26-web-spidering
title: Web Spidering with PHP
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-11-26-web-spidering
feedurl: https://thomashunter.name/feed.rss
content:
Spidering, in its simplest form is the act of transferring data from one database to another. Spidering requires the use of Regular Expressions, the cURL library (if POST data or cookies are used), and the cron libraries (if we need to download information with a schedule).


guid: 2009-12-29-convert-ascii-to-utf-8-using-vba
title: Convert ASCII to UTF-8 using VBA
author: Thomas Hunter II
url: https://thomashunter.name/posts/2009-12-29-convert-ascii-to-utf-8-using-vba
feedurl: https://thomashunter.name/feed.rss
content:
A function for converting characters into a UTF-8 format, required by Google Maps for making API calls.


guid: 2010-01-17-php-curl-cookies-not-saving-on-windows
title: PHP cURL cookies not saving on Windows
author: Thomas Hunter II
url: https://thomashunter.name/posts/2010-01-17-php-curl-cookies-not-saving-on-windows
feedurl: https://thomashunter.name/feed.rss
content:
Why cURL doesn't work well with relative paths that PHP works fine with, and a workaround for the issue.


guid: 2010-02-20-accessing-codeigniter-session-data-using-external-scripts
title: Accessing CodeIgniter Session Data using External Scripts
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2010-02-20-accessing-codeigniter-session-data-using-external-scripts

content:
PHP Sourcecode so that a non CodeIgniter PHP script can access data stored in the users CodeIgniter session.


guid: 2010-03-04-read-get-url-variables-using-javascript
title: Read GET URL variables using JavaScript
author: Thomas Hunter II
url: https://thomashunter.name/posts/2010-03-04-read-get-url-variables-using-javascript
feedurl: https://thomashunter.name/feed.rss
content:
Function which will either build an associative array of the GET URL variables on the current page, or return the value of a specified GET variable in JavaScript.


guid: 2010-04-25-dynamic-form-actions-using-different-buttons
title: Dynamic Form Actions using Different Buttons
author: Thomas Hunter II
url: https://thomashunter.name/posts/2010-04-25-dynamic-form-actions-using-different-buttons
feedurl: https://thomashunter.name/feed.rss
content:
A method for dynamically changing form action depending on which submit button is clicked using JavaScript. Does not require knowledge of the forms name.


guid: 2010-06-13-time-differences-between-php-and-mysql
title: Time Differences between PHP and MySQL
author: Thomas Hunter II
url: https://thomashunter.name/posts/2010-06-13-time-differences-between-php-and-mysql
feedurl: https://thomashunter.name/feed.rss
content:
The difference between reported times of MySQL and PHP compared on both a Windows and a Linux server.


guid: 2011-06-06-how-to-login-via-ssh-without-a-password
title: How to login via SSH Without a Password
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-06-06-how-to-login-via-ssh-without-a-password
feedurl: https://thomashunter.name/feed.rss
content:
SSH is the defacto way for working on remote servers. But, isn't it annoying to always enter your password? Follow this tutorial to generate and enable a certificate.


guid: 2011-07-31-scheduling-a-daily-cron-task-in-debian-to-check-external-ip
title: Scheduling a daily Cron task in Debian to check external IP
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
content: Build a cron task in debian to check your public IP. Great for hosting local websites.
url:
https://thomashunter.name/posts/2011-07-31-scheduling-a-daily-cron-task-in-debian-to-check-external-ip


guid: 2011-08-24-using-swipes-in-macvim-in-os-x
title: Using Swipes in MacVIM in OS X
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-08-24-using-swipes-in-macvim-in-os-x
feedurl: https://thomashunter.name/feed.rss
content:
This tutorial will allow you to use the OS X "Swipe" (three finger drag) features in your MacVIM. In this example, I use it to move between buffers.


guid: 2011-09-14-git-colored-output-shortcut-commands-autocompletion-and-bash-prompt
title: GIT: Colored Output, Shortcut Commands, Autocompletion, and Bash Prompt
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2011-09-14-git-colored-output-shortcut-commands-autocompletion-and-bash-prompt

content:
Follow this tutorial and you'll get colored git output, some productivity-boosting shortcuts, command / branch name autocompletion, and the current branch name in your bash prompt.


guid: 2011-09-26-really-simple-ssh-proxy-socks5
title: Really simple SSH proxy (SOCKS5)
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-09-26-really-simple-ssh-proxy-socks5
feedurl: https://thomashunter.name/feed.rss
content:
SOCKS5 is a simple, eloquent method for getting yourself a proxified connection to the internet. All you need to get a proxy connection working is to run an SSH server somewhere, run a single command locally, and configure your software (or OS) to use this proxy.


guid: 2011-11-20-tumblr-like-template-syntax-php-library
title: Tumblr-like Template Syntax PHP Library
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-11-20-tumblr-like-template-syntax-php-library
feedurl: https://thomashunter.name/feed.rss
content:
This is an explanation of a PHP templating language I'm working on, which will have a similar syntax with the language used by Tumblr.


guid: 2011-11-23-how-to-generate-a-self-signed-ssl-certificate-for-lighttpd
title: How to generate a Self Signed SSL Certificate for lighttpd
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2011-11-23-how-to-generate-a-self-signed-ssl-certificate-for-lighttpd

content:
Follow this tutorial to setup your Linux lighttpd server with a self signed SSL certificate. The server we use in this example is Debian based, but it should work with most distro's.


guid: 2011-11-25-force-or-prevent-www-subdomain-with-lighttpd
title: Force or Prevent WWW subdomain with lighttpd
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-11-25-force-or-prevent-www-subdomain-with-lighttpd
feedurl: https://thomashunter.name/feed.rss
content:
Contains instructions for configuring your lighttpd server to either force or prevent the www subdomain from working.


guid: 2011-11-26-tech-startup-entrepreneur-books
title: Tech Startup Entrepreneur Books
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-11-26-tech-startup-entrepreneur-books
feedurl: https://thomashunter.name/feed.rss
content:
These are my recommendations for two great books if you are an entrepreneur in the tech startup space.


guid: 2011-12-26-xpath-expressions-explained
title: Xpath Expressions Explained
author: Thomas Hunter II
url: https://thomashunter.name/posts/2011-12-26-xpath-expressions-explained
feedurl: https://thomashunter.name/feed.rss
content:
This quick tutorial shows a fairly complex Xpath selector and breaks down each segment. For those who work with CSS this will be easier to understand.


guid: 2011-12-29-how-to-use-javascript-event-bindings-to-make-callbacks-cleaner
title: How to use javascript event bindings to make callbacks cleaner
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2011-12-29-how-to-use-javascript-event-bindings-to-make-callbacks-cleaner

content:
It's common practice to start writing a bunch of nested functions in JavaScript. This tutorial describes a way to keep your code clean while doing so.


guid: 2012-02-09-why-node-js-is-awesome-a-short-history-of-web-applications
title: Why Node.js is awesome: A short history of web applications
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2012-02-09-why-node-js-is-awesome-a-short-history-of-web-applications


guid: 2012-04-15-redirect-all-pages-from-old-domain-to-new-domain-using-htaccess
title: Redirect all Pages from Old Domain to New Domain Using .htaccess
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2012-04-15-redirect-all-pages-from-old-domain-to-new-domain-using-htaccess


guid: 2012-08-28-spaghetti-architecture-and-the-importance-of-starting-over
title: Spaghetti Architecture and the Importance of Starting Over
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2012-08-28-spaghetti-architecture-and-the-importance-of-starting-over


guid: 2012-09-22-book-review-couchdb-and-php-web-development-beginners-guide
title: CouchDB and PHP Web Development
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2012-09-22-book-review-couchdb-and-php-web-development-beginners-guide


guid: 2012-09-27-mongodb-node-js-driver-collection-select-doesnt-require-callback
title: MongoDB Node.js driver collection select doesn't require callback
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2012-09-27-mongodb-node-js-driver-collection-select-doesnt-require-callback


guid: 2013-02-26-lansing-startup-weekend-february-2013-geomonsters-presentation
title: Lansing Startup Weekend February 2013: GeoMonsters Presentation
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2013-02-26-lansing-startup-weekend-february-2013-geomonsters-presentation


guid: 2013-07-27-i-wrote-a-book-instant-backbonejs-application-development
title: Instant Backbone.js Application Development
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2013-07-27-i-wrote-a-book-instant-backbonejs-application-development


guid: 2013-12-02-see-which-user-accounts-hackers-try-to-access-on-your-server-the-most
title: See which user accounts hackers try to access on your server the most
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2013-12-02-see-which-user-accounts-hackers-try-to-access-on-your-server-the-most


guid: 2013-12-03-letting-php-know-that-its-connection-behind-nginx-is-over-https
title: Letting PHP know that its connection behind NGINX is over HTTPS
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2013-12-03-letting-php-know-that-its-connection-behind-nginx-is-over-https


guid: 2013-12-29-book-review-debian-7-system-administration-best-practices
title: Book Review: Debian 7: System Administration Best Practices
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2013-12-29-book-review-debian-7-system-administration-best-practices


guid: 2014-07-03-linux-kernel-3-15-3-on-linux-mint-17-or-other-debian-based-distributions
title: Linux Kernel 3.15.3 on Linux Mint 17 (or other Debian-Based Distributions)
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2014-07-03-linux-kernel-3-15-3-on-linux-mint-17-or-other-debian-based-distributions


guid: 2014-10-05-robot-onslaught-multiplayer-twin-stick-2d-shooter-using-pubnub
title: Robot Onslaught: Multiplayer twin-stick 2D shooter using PubNub
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2014-10-05-robot-onslaught-multiplayer-twin-stick-2d-shooter-using-pubnub


guid: 2015-09-28-what-i-learned-while-publishing-an-html5-mobile-app-part-15-development-and-debugging
title: HTML5 Mobile Apps, Part 1: Development and Debugging
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
http://codeplanet.io/what-i-learned-while-publishing-an-html5-mobile-app-part-15-development-and-debugging/


title: HTML5 Mobile Apps, Part 3: Distribution
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
guid:
2015-09-30-what-i-learned-while-publishing-an-html5-mobile-app-part-35-distribution-and-marketplaces

url:
https://codeplanet.io/what-i-learned-while-publishing-an-html5-mobile-app-part-35-distribution-and-marketplaces/


guid: 2015-10-02-what-i-learned-while-publishing-an-html5-mobile-app-part-55-ecosystem-opinions
title: HTML5 Mobile Apps, Part 5: Ecosystem Opinions
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://codeplanet.io/what-i-learned-while-publishing-an-html5-mobile-app-part-55-ecosystem-opinions/


guid: 2018-04-02-creating-and-integrating-a-node-lambda-function-with-api-gateway
title: Node.js and Lambda Three Part Series
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2018-04-02-creating-and-integrating-a-node-lambda-function-with-api-gateway


guid: 2018-08-16-javascript-object-property-descriptors-proxies-and-preventing-extension
title: JavaScript Object Property Descriptors, Proxies, and Preventing Extension
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://medium.com/intrinsic/javascript-object-property-descriptors-proxies-and-preventing-extension-1e1907aa9d10


guid: 2018-12-12-books-im-reading-to-become-a-better-writer
title: Books I'm Reading to Become a Better Writer
author: Thomas Hunter II
url: https://thomashunter.name/posts/2018-12-12-books-im-reading-to-become-a-better-writer
feedurl: https://thomashunter.name/feed.rss
content:
I recently picked up several books on writing. I'd like to share these books with you in the hopes that you might also find them interesting.


guid: 2019-01-23-the-increasing-need-for-regulation-of-the-software-industry
title: The Increasing Need for Regulation of the Software Industry
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://thomashunter.name/posts/2019-01-23-the-increasing-need-for-regulation-of-the-software-industry


guid: 2019-03-28-why-should-i-use-a-reverse-proxy-if-node-js-is-production-ready
title: Why should I use a Reverse Proxy if Node.js is Production-Ready?
author: Thomas Hunter II
feedurl: https://thomashunter.name/feed.rss
url:
https://medium.com/intrinsic/why-should-i-use-a-reverse-proxy-if-node-js-is-production-ready-5a079408b2ca


guid: 5ea7f304b8b71d036558c316
title: 3D Printing: My DIY Formlabs Enclosure Project
author: Valerian Saliou
url: https://valeriansaliou.name/blog/3d-printing-formlabs-enclosure-project/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/05/enclosure-banner.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-banner.jpg" alt="3D Printing: My DIY Formlabs Enclosure Project"><p><em>When the <a href="https://formlabs.com/3d-printers/form-3/">Formlabs Form 3</a> was released back in late 2019, I jumped on to ordering it and trying it out, with a future hardware project in mind. I was new to SLA (<a href="https://en.wikipedia.org/wiki/Stereolithography">Stereolithography</a>), coming from the world of FFF/FDM (<a href="https://en.wikipedia.org/wiki/Fused_filament_fabrication">Fused Filament Fabrication</a>) with my MakerBot Method. Unfortunately, I did not realise the amount of smell and vapor that such a printing equipment produced in comparison to FDM printers. That&apos;s where I figured out that I needed to build myself an enclosure &#x2014; to ensure I could safely use a SLA printer in my flat.</em></p><p>Before starting venturing on to building an enclosure, I looked for existing 3D printer enclosure products on the market. Unfortunately, all enclosures that I could find were designed to host a single machine that did not fit the Form 3 required opening area (they were all not high enough to give the printer lid enough room to fully open). Adding to that, as my Form 3 came with a Form Wash and a Form Cure, the enclosured would need to be nearly 3 times the width that of a regular printer enclosure to host all 3 Formlabs machines. And, last, given the amount of potentially harmful resin smell coming from the printer itself, as well as highly flammable IPA (<a href="https://en.wikipedia.org/wiki/Isopropyl_alcohol">Isopropyl Alcohol</a>) fumes coming from the Form Wash &#x2014; well &#x2014; I needed the enclosure air to be vented outside and renewed.</p><p>That was about time to build my own custom enclosure!</p><h1 id="enclosure-design">Enclosure Design</h1><p><strong><u>Before I start showing the enclosure itself, here were my requirements:</u></strong></p><ul><li><strong>No heavy metal / wood / plastic work should be involved</strong> in building this enclosure. At best, I should buy a piece of furniture on the market and tune it with the right parts to transform it into an air-tight enclosure.</li><li>With practicality in mind, <strong>the enclosure would have to host the machines themselves</strong>, but also <strong>all printing tools, resin tanks and resin cartridges</strong>.</li><li>The enclosure should be able to <strong>vent spoiled machine area air outside and renew it</strong>, without leaking much smell outside (I would need to create a negative pressure area where the machines sit, thus positive pressure in the air extractor duct).</li><li>Given the flammability of Isopropyl Alcohol, <strong>the enclosure should let me safely store IPA tanks away from light and heat</strong> (and... sparks), with a chemical leaks recovery tank underneath. This also counts for spoiled IPA that should be stored in another tank before going for chemical waste or recycling.</li><li>The enclosure should have <strong>some space where I can work on post-processing prints</strong> and doing messy stuff, like cleaning resin spills or removing failed prints from resin tanks.</li><li>As machine area air would need to be renewed and vented outside on a permanent basis, I would not like to leave the window open at all times, or dig a hole in the window. <strong>The existing window air inlet should be removed and fitted with the air extraction system</strong>. Still, the window should be kept openable at any time, without requiring disassembling the air extractor connection. Ie. someone who&apos;s not aware of the air extractor, who would open the window, would not permanently damage the air extractor duct.</li></ul><p><strong><u>And now, here is the result based on the above requirements (teaser):</u></strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-overview.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="2000" height="1500" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-overview.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/05/enclosure-overview.jpg 1000w, https://valeriansaliou.name/blog/content/images/size/w1600/2021/05/enclosure-overview.jpg 1600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-overview.jpg 2000w" sizes="(min-width: 720px) 720px"><figcaption>An overview of the assembled enclosure. Printing area on the left, post-processing area on the right.</figcaption></figure><h1 id="overview-video">Overview Video</h1><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-overview-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-overview-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-overview-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-overview-video-track.webm">Play the overview video.</a>
</video><!--kg-card-end: html--><h1 id="shopping-list">Shopping List</h1><p>The beauty of this design lies in the very few components required to assemble it. The most expensive and largest parts are made of pieces of IKEA furniture, and custom-cut acrylic panels.</p><p><em>Note that some links shared there may refer to websites selling in France only. Please fallback to your local seller, eg. local IKEA with the same product reference. As those links refer to products that are due to be deprecated at some point in time, those links may not be valid anymore at the time you are reading this article.</em></p><p><strong>Here is the full shopping list (ie. list of required materials), which does not include the tooling (eg. screwdriver et al):</strong></p><ol><li><strong>2x <a href="https://www.ikea.com/fr/fr/p/bror-etabli-noir-contreplaque-de-pin-30333286/">IKEA BROR</a> large version 110x55cm</strong> (used for the printing area; we need to stack two of them in order to create the metal frame that will hold the acrylic panels; the second BROR workbench will only be partly used, you could recycle spare parts eg. its wooden surface to build a separate table);</li><li><strong>1x <a href="https://www.ikea.com/fr/fr/p/bror-desserte-noir-contreplaque-de-pin-60333850/">IKEA BROR</a> small version 85x55cm</strong> (used for the post-processing area);</li><li><strong>Acrylic panels</strong>, custom-cut, transparent with a thickness of 4mm, and the following exact sizes and counts: <em>1x [108.4cm x 74.2cm] + 2x [53.4cm x 74.2cm] + 2x [50.6cm + 74.2cm]</em> (I ordered mine online at the French website &quot;<a href="https://plexiglasssurmesure.fr">Plexiglass sur mesure</a>&quot;, which manufactured my panels and shipped them under 2 days);</li><li><strong>6x door hinges</strong> (no specific model, please refer to my pictures to find a lookalike version that fits nicely, from your local tool shop);</li><li><strong>2x door knobs</strong> (no specific model, please refer to my pictures as well; they need to be screwed with a small screw to the epoxy panel, from the rear of the door);</li><li><strong>1x 10m of adhesive seal tape</strong> to seal acrylic panels with the metal frame, as well as acrylic doors with their frame (the rubber band should be approximately 6mm high, 1cm thick, and be very smooth on pressure &#x2014; preferably black-colored);</li><li><strong>1x air extractor fan</strong>, with a 10cm diameter duct outlet;</li><li><strong>1x passive air inlet grid</strong>, with at least 10cm of diameter (this helps exchange air between the room air, the vented area, and the outside air, via the negative pressure created by the air extractor fan);</li><li><strong>Air duct parts, flat rectangle variant</strong>, made of: <a href="https://www.domomat.com/6659-coude-mixte-vertical-cmv-pour-conduit-rectangulaire-pvc-rigide-55-x-110-x-o100mm-atlantic-460004.html">2x 10cm diameter inlet/outlet connectors</a> + <a href="https://www.domomat.com/6658-coude-horizontal-90-ch-pour-conduit-rectangulaire-pvc-rigide-55-x-220mm-atlantic-460010.html">1x angle connector</a> + <a href="https://www.domomat.com/11044-conduit-pvc-rigide-rectangulaire-tr-plat-55-x-110mm-longueur-15m-atlantic-460021.html">2x long rectangle duct &quot;pipes&quot;</a> (you can find a look-alike version in your local shops, you may refer to my links there, and pictures below, to find a similar product, mine are from the brand &quot;Atlantic&quot;);</li><li><strong>M5 screw set</strong> with their <strong>bolts</strong> and a large set of <strong>screw spacers</strong>;</li></ol><p><strong>You may also need to buy some specific tools in order to proceed some operations:</strong></p><ol><li><strong>1x Diamond-coated drill bit</strong>, with a 10cm diameter (this is mounted on your drill, and helps cut a circular hole for the air extractor fan and air inlet grid, just make sure to drill at a low speed to avoid cracking the acrylic panel);</li></ol><p><strong>That above, was solely to build the piece of furniture itself. Which would not be handy without its own set of accessories:</strong></p><ol><li><strong>1x <a href="https://www.ikea.com/fr/fr/p/sysselsaetta-planche-a-decouper-30359016/">IKEA SYSSELS&#xC4;TTA</a></strong> (plastic kitchen cut plate, used to protect the wood frame from resin and IPA spills while post-processing your prints);</li><li><strong>2x <a href="https://www.ikea.com/fr/fr/p/ikea-365-boite-de-conservation-rectangulaire-plastique-60393066/">IKEA 365+</a> high rectangle boxes</strong> (used to store accessories and polymerized resin trash);</li><li><strong>2x <a href="https://www.ikea.com/fr/fr/p/ikea-365-boite-de-conservation-grand-rectangulaire-plastique-10393064/">IKEA 365+</a> large rectangle boxes with its <a href="https://www.ikea.com/fr/fr/p/ikea-365-couvercle-grand-rectangulaire-plastique-30393063/">IKEA 365+</a> sealed top lid</strong> (used to hermetically store smelly and messy trashes, eg. non-polymerized resin and IPA-spilled towels)</li><li><strong>2x <a href="https://www.ikea.com/fr/fr/p/kuggis-boite-avec-couvercle-blanc-20280207/">IKEA KUGGIS</a> small version</strong> (used to store post-processing hand tools + nitrile gloves);</li><li><strong>3x <a href="https://www.ikea.com/fr/fr/p/kuggis-boite-avec-couvercle-blanc-50282304/">IKEA KUGGIS</a> large version</strong> (used to store all maintenance tools, safety equipment and resin cartridges);</li><li><strong>1x <a href="https://www.ikea.com/fr/fr/p/kolon-protection-pour-sol-44881100/">IKEA KOLON</a></strong> (used to protect the enclosure wood panel from IPA spills or resin spills, you will need to cut it in order to match the wood panel size);</li><li><strong>1x large generic tool storage box</strong> (used to store fresh IPA tanks and contain any leak, mine is red on the picture, and can contain liquids);</li><li><strong>A set of <a href="https://www.amazon.fr/gp/product/B06ZYX6KL2">2.5L HDPE jerrycans</a></strong> to dilute and store your fresh IPA (I personnally dilute my 99.9% IPA down to 95% with water, and store it in small quantities in those 2.5L jerrycans &#x2014; for safety reasons);</li><li><strong>1x <a href="https://www.amazon.fr/gp/product/B07Y1QWFBV">Levoit Vista 200</a> air purifier</strong> (stored resin tanks and cartridge can be a little bit smelly, running this air purifier constantly at its lowest fan speed helps eliminate smells, thanks to its activated carbon filter and HEPA filter &#x2014; I&apos;m hiding it in the storage area under the printing machines);</li><li><strong>1x <a href="https://www.amazon.fr/gp/product/B082TGFYFH">EATON 3S 700 Off-line UPS</a></strong> (protects your expensive printing equipment against power surges, and provides backup power to your Form 3 for at least half an hour in case of loss of power; this one fits the maximum theoretical power all 3 machines could consume when running at the same time, as it is capable of delivering 420 Watts of peak power);</li><li><strong>1x <a href="https://www.amazon.fr/gp/product/B07K11GF7C">Schneider Electric 5 outlets power-strip</a> with an ON/OFF switch</strong> (it is connected directly on the EATON UPS battery-protected outlet, letting you shutdown or power the Form 3, Form Wash, Form Cure and air extractor at once, as all those do not have an ON/OFF button);</li></ol><h1 id="3d-printed-parts">3D-Printed Parts</h1><h2 id="window-air-extractor">Window Air Extractor</h2><p>The only part that cannot not be bought on the market is the window air extractor adapter. <strong>This part is screwed to the window, and latches with the 10cm diameter duct outlet that blows air out.</strong></p><p>As all setups are different, and each window has a size of its own, <strong>this part should be custom-fitted using a CAD software like Fusion 360, and then 3D-printed</strong>. As it&apos;s rather large (mine is around 40cm long), it should be printed in multiple parts, which are then screwed together and sealed against each other using neoprene glue. Screws used are M4 screws, with a bolt attached on the other side. There is a total of 6 screws and 6 bolts. Use short screws, eg. around 3cm long (that is, for cosmetic purposes). Double-check that all joints are airtight before mounting on the window, as any leak at this section will result in spoiled air to be recirculated back in the room. Also, make sure to place a double-sided adhesive tape on the plastic section that touches the window, to seal it.</p><p><strong>The three large parts on the scheme below are to be printed on an FDM printer</strong> (using PLA as a material is fine, but ABS can also be used for more durability). Those parts have been designed to be printable without any support whatsoever, thus you should disable them in your slicer (I would still recommend you keep the adhesion bed). The red and yellow parts should be printed placed with the longest edge on the Z axis (over the part height), while the green part should lie on the bed with the square edges on the X and Y axis with the duct outlet facing up towards the extruders. Cathedral-like supports have been included in the model, and should not be broken away once printed. I have done 2 prints with different parameters, the one that came out perfect on a MakerBot Method was the following: <em>white MakerBot PLA spool + high-quality presets + 200 micron layer thickness + 25% infill + 0.6mm roof &amp; floor thickness + purge tower disabled</em>.</p><p><strong>An o-ring gasket ensures the latch between the window air extractor and the duct exhaust is hermetically-sealed. This o-ring gasket should be printed in a flexible material</strong> &#x2014; I&apos;ve printed mine on the Form 3 using Formlab&apos;s Flexible V2 resin, with success. This print was configured with the gasket laying on the X-Y axis, with minimal height on the Z axis, and a resolution of 50 microns (100 microns is also okay).</p><p><strong><u>Window Air Extractor model:</u></strong></p><p>The &quot;Window Air Extractor&quot; 3D model <a href="https://a360.co/2VNBOoT"><strong>can be viewed online</strong></a>. I&apos;ve also prepared a Zip file containing the F3D and STEP CAD sources (<a href="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-window-air-extractor-files.zip"><strong>download it there</strong></a>), as well as STL exports of all 4 parts. You may rework my CAD model to ensure it fits with your window and duct setup.</p><p><em>If you are looking for a picture of the window air extractor printed and mounted on a window, I&apos;ve included two below, in the picture section.</em></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-window-air-extractor-model.png" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>The Window Air Extractor 3D model designed in Fusion 360. Made of 4 bodies.</figcaption></figure><h1 id="close-up-pictures">Close-Up Pictures</h1><h2 id="printing-area">Printing Area</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-19.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-19.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-19.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>Printing section with doors closed (left to right: Form 3, Form Wash and Form Cure).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-13.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-13.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-13.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>Printing section with doors opened (notice the book on the Form Wash: more on that a few pictures right below).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-8-1.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-8-1.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-8-1.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>There are 3 door hinges on each side, screwed to already-existing screw fits on the metal frame.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-3.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-3.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-3.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>Each door has its own handle for easy opening (screwed through a hole in the acrylic sheet).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-16.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-16.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-16.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>Acrylic sheets are screwed from the inside of the metal structure, so that they are airtight.</figcaption></figure><h2 id="air-extraction-system">Air Extraction System</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-2.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="1200" height="900" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-2.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/05/enclosure-closeups-2.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-2.jpg 1200w" sizes="(min-width: 720px) 720px"><figcaption>A bathroom air extractor fan sucks air out of the enclosed printing area into the duct. Ensure it is compatible with flammable vapors (IPA vapors from the Form Wash).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-11.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="900" height="1200" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-11.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-11.jpg 900w" sizes="(min-width: 720px) 720px"><figcaption>A passive air inlet sits on the opposite side of the fan, that lets air in thanks to negative pressure in the enclosure (in order to replace expelled air).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-18.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="1200" height="900" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-18.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/05/enclosure-closeups-18.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-18.jpg 1200w" sizes="(min-width: 720px) 720px"><figcaption>A flat rectangle air duct connects to the extractor fan and brings expelled air near the room window.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-7.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="1200" height="900" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-7.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/05/enclosure-closeups-7.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-7.jpg 1200w" sizes="(min-width: 720px) 720px"><figcaption>A 3D printed adapter that connects the air duct outlet to the window air inlet hole.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-14.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy" width="1200" height="900" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/05/enclosure-closeups-14.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/05/enclosure-closeups-14.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/05/enclosure-closeups-14.jpg 1200w" sizes="(min-width: 720px) 720px"><figcaption>The 3D printed adapter (PLA plastic; FDM &#x2014; white part) latches against the duct end and prevents positive-pressure air from escaping back to the room with a 3D printed o-ring gasket (Flexible resin; SLA &#x2014; black part). This latch system allows easy window opening and closing.</figcaption></figure><h2 id="storage-space">Storage Space</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-6.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Below the airtight section where the printer sits, there is enough room to store resin tanks, resin cartridges, and nitrile gloves. Note that I&apos;ve labeled all resin tanks and storage boxes.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-5.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Below the printer area, there is also a trash section, where a polymerised resin trash (right container) is separated from non-polymerised resin or smelly trashes (left container). On the back, a Levoit air purifier removes any slight smell of resin that may come out of the stored cartridges or tanks.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-9.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Below the printing storage space, lies another boxed storage space for smaller items, like post-processing tools (right), protective equipment (middle) and printer maintenance tools (left).</figcaption></figure><h2 id="post-processing-area">Post-Processing Area</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-1.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>This is where I post process prints (support removal), or clean resin spills. A thick plastic frame (surrounded with slots to contain spills) protects the wood frame on which it sits.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-4.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>The Formlabs Finish Kit has been repurposed to store pure water (left) and small IPA buckets to wash spills and tools (right). It is hermetic &#x2014; to prevent smells.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-17.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Below the post-processing area, is a storage space with doors, where I can safely store all fresh IPA provisions in 2.5L jerrycan (right), as well as spoiled IPA that should go to waste (left). When doors are closed, the stored IPA is kept away from light and any source of ignition. Note the red bucket in which all 2.5L tanks sit, which would help contain IPA spills from a leaking tank.</figcaption></figure><h2 id="miscellaneous">Miscellaneous</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-10.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>On the floor, hidden below the enclosure, lays a small Eaton UPS, that protects the costly Formlabs machines against electrical power surges, and keeps a printing Form 3 online for at least 30 minutes in case of a power outage.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-15.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Gaskets have been added to the Form Cure to prevent IPA fumes from leaking in small quantities. When closed and not in use, I usually place a shower cap on the top of the Form Cure, to seal it completely and prevent evaporation and IPA odors, as gaskets are far from perfect.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2020/04/enclosure-closeups-12.jpg" class="kg-image" alt="3D Printing: My DIY Formlabs Enclosure Project" loading="lazy"><figcaption>Formlabs-provided Form Wash and Form Cure recommended Wash and Cure times tables are attached to the printing area inner acrylic rear wall, right above the Formlabs machines. This saves me precious time when I need to wash and cure a resin that I don&apos;t use often.</figcaption></figure><h1 id="closing-notes">Closing Notes</h1><p>I would have paid a lot to buy an existing 3D-printing workstation furniture, which would have included the enclosure itself, the post-processing area, storage areas, as well as all the integrated backup power equipment. Sadly, that&apos;s not something we can already find on the market as of 2020. This was the main motivation behind my decision of writing this article and sharing my enclosure design with you.</p><p><em>If you were to build yours, and choose my design, I would love you to share pictures of the finished enclosure that you built with me <a href="mailto:valerian@valeriansaliou.name">over email</a>. As well, if you have questions or any issue in replicating it, drop me an email over the same email address. I&apos;d be happy to help!</em></p><hr><h3 id="they-replicated-it">They replicated it</h3><p>Some people have contacted me to ask me specific questions about my design, as they were building it themselves! I am glad to share pictures of the enclosures that have been built by other people (with their permission).</p><figure class="kg-card kg-gallery-card kg-width-wide kg-card-hascaption"><div class="kg-gallery-container"><div class="kg-gallery-row"><div class="kg-gallery-image"><img src="https://valeriansaliou.name/blog/content/images/2024/02/enclosure-3rd-party-1.jpg" width="1200" height="801" loading="lazy" alt="3D Printing: My DIY Formlabs Enclosure Project" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2024/02/enclosure-3rd-party-1.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2024/02/enclosure-3rd-party-1.jpg 1000w, https://valeriansaliou.name/blog/content/images/2024/02/enclosure-3rd-party-1.jpg 1200w" sizes="(min-width: 1200px) 1200px"></div></div></div><figcaption>1. Mauro (from Italy)</figcaption></figure><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 6029589a02a50e03b2178c0b
title: MakAir Series: The Inception of an Open-Source Ventilator Project
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Inception.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Inception.jpg" alt="MakAir Series: The Inception of an Open-Source Ventilator Project"><p><em>This is the first article of a series on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the Covid-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article lays out some generalities on the MakAir ventilator &amp; overall project, providing context and a chronology of the events, as well as a quick technical overview and a list of important project members that can be reached anytime for questions.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/MakAir-V3.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>The current MakAir production-grade ventilator (revision 3).</figcaption></figure><hr><h2 id="presentation-video">Presentation Video</h2><p>If you reading this article to get a quick overview of the MakAir ventilator, <strong>you may first watch the following video</strong>, with which you will understand the basics of how a MakAir works.</p><!--kg-card-begin: html--><video controls poster="https://valeriansaliou.name/blog/content/images/2021/02/makair-presentation-overview-video-poster.jpg" preload="none" style="width: 100%;">
  <source src="https://valeriansaliou.name/blog/content/images/2021/02/makair-presentation-overview-video-track.webm" type="video/webm; codecs=vp9,opus">
  <source src="https://valeriansaliou.name/blog/content/images/2021/02/makair-presentation-overview-video-track.mp4" type="video/mp4; codecs=hvc1">
  <a href="https://valeriansaliou.name/blog/content/images/2021/02/makair-presentation-overview-video-track.webm">Play the presentation video.</a>
</video><!--kg-card-end: html--><hr><h2 id="event-timeline">Event Timeline</h2><p>Back in March 2020, just 1 day before the Covid lockdown was to begin in France, I received a message from a friend of mine: &#x201C;<strong><em>Hey! It looks like the whole world will very soon lack ventilators. You bought a 3D printer at your company a few months ago and we are all full-time software developers, what do you think about building our own open-source ventilator?</em></strong>&#x201D;. I thought that this was crazy, but hey, let&#x2019;s at least try, so I said yes.</p><p>The day after, we confined into Crisp&#x2019;s offices (myself, <a href="https://jamin.me/">Baptiste Jamin</a> and <a href="https://www.waxzce.org/">Quentin Adam</a>), and started scrambling on the foundation of what later became a much bigger and mature project, that is the MakAir open-source mechanical ventilator.</p><p><strong>At first, it was just a <a href="https://github.com/makers-for-life/makair">simple repository on GitHub</a> where we uploaded all our source code and CAD designs</strong>, a Telegram room where we coordinated, and a <a href="https://www.youtube.com/channel/UCcf_3KXjeJSMz39J6gsyTSg">YouTube channel</a> where we posted video updates. <em>No strings attached.</em></p><p><strong>The MakAir project gathered at its peak a whopping team of 300+ volunteers, 30 of which were on-site</strong>, working together physically, while the rest of them were providing help on anything through Slack (from <a href="https://www.safran-group.com">Safran</a> engineers helping with the blower fan, to designers helping build a visual identity and a <a href="https://makair.life/">website</a> for the project).</p><p><strong>The French army quickly responded to our funding request</strong>, literally days after the project began, and backed the project as part of its quick-response effort to help innovative projects fighting Covid-19, through its investment vehicle <a href="https://www.defense.gouv.fr/aid">Agence Innovation D&#xE9;fense</a> (<a href="https://www.defense.gouv.fr/fre/actualites/articles/le-ministere-des-armees-soutient-le-projet-makair-un-respirateur-artificiel-simplifie-dedie-au-traitement-du-covid-19">read their announcement</a>, in French). Then, the <a href="https://en.wikipedia.org/wiki/French_Alternative_Energies_and_Atomic_Energy_Commission">French Atomic Energy Commission</a> of <a href="https://en.wikipedia.org/wiki/Grenoble">Grenoble</a> stepped in and welcomed us in their high-tech facilities to further grow the project. And later, <a href="https://en.wikipedia.org/wiki/Groupe_SEB">Groupe SEB</a> joined forces by becoming our industrial manufacturer (<a href="https://www.groupeseb.com/en/news/together-face-covid-19">read their announcement</a>).</p><p>Once we had something we were confident of, we collaborated with <a href="https://www.chu-nantes.fr/">CHU of Nantes</a> and <a href="https://www.chu-brest.fr/">CHU of Brest</a>, which are 2 large public hospitals, in order to start clinical trials on human patients. This required that we entered in a certification process with <a href="https://en.wikipedia.org/wiki/Agence_nationale_de_s&#xE9;curit&#xE9;_du_m&#xE9;dicament_et_des_produits_de_sant&#xE9;">ANSM</a> (the French <a href="https://en.wikipedia.org/wiki/Food_and_Drug_Administration">FDA</a>). Clinical trials soon started, though they are <em>still</em> in progress as of today. This is a super slow process, unfortunately, as you cannot shortcut medical safety even in times of crisis.</p><p>We did quite a lot of press as well, though we only pulled the trigger only when we felt confident that press coverage would not slow down the engineering side of the project. We got coverage in online press (<a href="https://www.lemonde.fr/campus/article/2020/04/20/entre-nantes-et-grenoble-une-course-contre-la-montre-pour-creer-un-respirateur-artificiel-a-bas-cout_6037140_4401467.html">Le Monde on 20th April 2020</a> or <a href="https://www.lepoint.fr/sciences-nature/a-brest-on-teste-le-respirateur-concu-par-des-geeks-06-05-2020-2374359_1924.php">Le Point on 6th May 2020</a>), as well as on TV news channels (<a href="https://www.youtube.com/watch?v=2X157RLbRIA">TF1 on 26th April 2020</a> and <a href="https://www.youtube.com/watch?v=M3QLCvUyIII">France 3 on 21st April 2020</a>).</p><p><strong>All those events happened in 1 month</strong>, between March and April 2020, which is still blowing my mind today. I never thought that in time of crisis, people who are not supposed to work together could collaborate in such an efficient and unstructured way, solving a hard problem in just a few weeks. This often bring me to compare it with huge Cold War projects, like the making of the Apollo program. <strong>Put the brightest human minds in a locked underground room, with an impossible deadline and a sense of urgency; they&#x2019;ll achieve incredible feats!</strong> (n.b. we were literally working 20/24h in a windowless underground room).</p><p>The project is still going on as of today, though at a slower pace. As everything is open-source (software, electronics and mechanical parts), anyone can basically reproduce, contribute and submit improvements.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Team.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy" width="1000" height="1000" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/02/MakAir-Team.jpg 600w, https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Team.jpg 1000w" sizes="(min-width: 720px) 720px"><figcaption>The team of volunteers working on the MakAir project, at its peak. This picture dates back from our time at CEA Grenoble.</figcaption></figure><hr><h2 id="technical-overview">Technical Overview</h2><p><em>This section aims at providing readers with a quick technical overview of the MakAir, with a bit of history on the design iterations that were made, as this can help understand decisions we took on the final design revision. That way readers can understand where we come from, where we are at, and where we&#x2019;re going. In-depth articles on each building block of the MakAir will be written throughout the series.</em></p><h3 id="first-design-iterations-deprecated-ones-">First design iterations (deprecated ones)</h3><p><strong>&#x26A0;&#xFE0F; <em><u>Disclaimer</u>: the following designs are deprecated and do not reflect the current inner workings of the MakAir ventilator. They are only provided for context on the chronology of the project.</em></strong></p><p>When we started the project, we took a first principles approach to what a ventilator is: basically <strong>a system that pushes air in and out of a patient&#x2019;s lungs</strong> (respectively for the inspiratory cycle and the expiratory cycle). Knowing that, we designed a simple blower-based system pushing air through a circuit of multiple 3D-printed 2-states &#x201C;1 bit&#x201D; Y-shaped valves. Think of it as an air routing circuit, creating paths. Turns out that, while this worked in our experiments, we got things wrong and had to start all over. There is a <a href="https://www.youtube.com/watch?v=GLX5o_y0Vx0">presentation video</a> of this system, in case you are interested. By the way, this design was super efficient at <a href="https://www.youtube.com/watch?v=wsjibAw78Ys">inflating condoms</a>, but that&#x2019;s about it. Huge leaks, high <a href="https://en.wikipedia.org/wiki/Dead_space_(physiology)">dead space</a> in the air circuit and lack of fine control all lead to a dead-end.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Air-Transistor-Experiment-V4.gif" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>The most advanced so-called &#x201C;air transistor&#x201D; design we came up with (iteration number 4). The red part turned at a target angle by a servomotor (2 position swaps per respiratory cycle).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Blower-Experiment-V7.gif" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>We designed our own blower turbines as well, due to a shortage of turbines on the global supply chain. As everything was closed, we could not source any, so we made our own design, scavenging DC motors from a previous drone project.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/O2-Mixer-Experiment-V5.gif" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>As oxygen needs to be mixed with air from the blower, we designed our own <a href="https://en.wikipedia.org/wiki/Venturi_effect">Venturi effect</a> &#x201C;oxygen mixer&#x201D;. We had engineers compute air flow simulations for us in order to find the perfect shape.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Hardware-Prototype.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>The first prototype of our control board, running a primitive MakAir firmware revision on an Arduino system.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Wooden-MakAir.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>Our first assembly, made of wooden frames. Note that we already moved away from our &#x201C;air transistor&#x201D; design at this point, as we were trying out a new blow-out valve system (which did not last long).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/3D-Printers-Array.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>An array of 3D printers we have been heavily relying on during the whole project prototyping phase. Those are <a href="https://www.makerbot.com/">MakerBot</a> Method printers, provided by CEA (10 in total).</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-blower-cure.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy" width="1400" height="900" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/03/makair-blower-cure.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/03/makair-blower-cure.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/03/makair-blower-cure.jpg 1400w" sizes="(min-width: 720px) 720px"><figcaption>Blades from one of our first 3D printed blower design being &quot;cooked&quot; (ie. cured) in a Form Cure from <a href="https://formlabs.com/">Formlabs</a>.</figcaption></figure><hr><h3 id="going-back-to-the-drawing-board">Going back to the drawing board</h3><p>Our first design, although flawed, <strong>led to important design decisions</strong> regarding the overall architecture of the ventilator.</p><p><strong><u>Thinking with first principles, our ventilator needed to:</u></strong></p><ul><li><strong>Produce air</strong>: using a blower turbine;</li><li><strong>Control air</strong>: a way to precisely control the air flow going in and out of the ventilator (to/from patient lungs);</li><li><strong>Filter air</strong>: components to filter the air going in and out of the ventilator (you do not want to blow Covid out of the ventilator);</li><li><strong>Measure air</strong>: sensors to measure the air pressure in the circuit (too high a pressure could blow the patient&#x2019;s <a href="https://en.wikipedia.org/wiki/Pulmonary_alveolus">pulmonary alveolus</a>);</li><li><strong>Provide power</strong>: batteries for backup power, and an AC-DC converter;</li><li><strong>Orchestrate it all</strong>: a board running our software and ventilation control algorithms, connected to hardware buttons and a small display for a user to adjust ventilation settings and check for the measured pressures;</li></ul><p>Luckily, we quickly figured out that <strong>we had to stop trying to 3D-print our own components</strong>, and instead rely on manufactured parts. This led us to completely reconsider our ventilator design around the 29th of March 2020. <strong>That is, 11 days after we started the project.</strong></p><hr><h3 id="final-design-current-one-">Final design (current one)</h3><p>As the project quickly got more serious, qualified engineers joined the project thereafter, which brought in some new fresh ideas in fields where we lacked professional knowledge (namely, electronics and mechanical design). In the long run, you really cannot rely on what hobbyists produce.</p><p>We kept our blower-based design, but <strong>removed the 3D-printed valve-based circuit</strong> and replaced it with <strong>flexible pipes</strong>, made of a medical-grade plastic (<a href="https://en.wikipedia.org/wiki/Biocompatibility">biocompatible</a>). To control the air that was allowed to go towards the lungs, or outwards from the lungs, we designed a pinch valve system, driven by a <a href="https://www.faulhaber.com/">Faulhaber</a> Brushless DC servomotor, <strong>capable of handling a lifetime of 1M+ respiratory cycles</strong> (if not way more; to give you an idea of what is required, a Covid patient must be ventilated for approximately 2 weeks, that&#x2019;s about 500k breathing cycles).</p><p><strong>This new pinch valve system has the simple task of pressing (pinching) the flexible air pipe</strong> to a certain degree. As we can measure the resulting air pressure instantly, we can feed this back to the control loop and pinch a bit more or a bit less by rotating the motor very precisely to a new target angle, until we stabilize to the desired air pressure in the pipe. <strong>We use a <a href="https://en.wikipedia.org/wiki/PID_controller">PID control</a> algorithm</strong> for that, which is also what your car uses under the hood whenever you enable cruise mode. More engineering details on that will be provided in the next articles of this series.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Pinch-Valve.gif" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>The latest design of our Faulhaber-driven pinch valve.</figcaption></figure><p>As we were building a <a href="https://en.wikipedia.org/wiki/Pressure_control">pressure-controlled</a> ventilator (also, more background on pressure and volume control will be provided in the next articles of the series), <strong>we needed pressure sensors that would feed instant pressure values</strong> measured in <a href="https://en.wikipedia.org/wiki/Centimetre_or_millimetre_of_water">mmH20/cmH20</a> to our algorithm. Those pressure sensors are mounted on the custom motherboard PCB that we designed for the MakAir ventilator. <strong>That motherboard is including a <a href="https://en.wikipedia.org/wiki/STM32">STM32</a> chip</strong>, which is running our firmware, as well as a <a href="https://en.wikipedia.org/wiki/LoRa">LoRa</a> chip which can be used to transmit instant ventilation metrics wirelessly, to a central node eg. a monitoring room in the hospital (unused as of today).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Motherboard-V3.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>Our V3 motherboard PCB. We have come a long way, as we are no longer using an handmade board plugged to a sloppy Arduino.</figcaption></figure><p>Our cornerstone work having been laid out on the <a href="https://github.com/makers-for-life/makair-firmware/">firmware</a> side with our ventilation algorithms, now we needed to provide a way for doctors and nurses to monitor the ventilation metrics in real-time on large modern display. That is when we introduced <strong>a second computing component to the ventilator: a control UI with a touchscreen, running on an ARM-powered board with a Linux system</strong>. We built a <a href="https://github.com/makers-for-life/makair-control-ui/">user interface</a> software that connects to the motherboard using a serial connection, and speaks a common <a href="https://github.com/makers-for-life/makair-telemetry/">telemetry protocol</a> to pull realtime ventilation data.</p><p>That way, the super-reliable yet simple embedded system that runs the sensitive ventilation algorithms (ie. the firmware), is completely isolated from the (probably) less-reliable ARM-based complex computer that runs the touch interface on Linux. Tesla did it in a similar way in their cars, by separating the drive computer from the multimedia computer (the large center display), which is a neat design choice regarding reliability &amp; safety.</p><p>Note that, while this touchscreen is still working in pair with the hardware buttons as of today, we plan to migrate all user controls to be touch-only in the future. This makes internationalization of the user interface easier, as it becomes all software.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Control-UI.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>The MakAir Control UI in action on its touchscreen, mounted on the ventilator. The ventilator was stopped on the picture, hence the message shown. The UI displays two graphs: a pressure graph and a flow graph, which are moving live as the patient gets ventilated. Important metrics are shown on the right and bottom. Controls and anomaly alarms are available on the top.</figcaption></figure><p>All those design changes &amp; improvements led to a simpler and more reliable, mass-producible, ventilator. All components were fitted inside a large but strong plastic casing, that was extruded where eg. a touchscreen, buttons or air connectors needed to be mounted. This feels a bit like 90s computers to me, which is enjoyable to work with and repair as everything is reachable with bare hands.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/Inside-MakAir.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy"><figcaption>Looking inside a MakAir ventilator (current revision). Electronics on the left, air circuit with blower and pinch valves on the bottom, lead-acid batteries and AC-DC converter at the top. Most of the complexity of the ventilator is concentrated inside the software that we developed, which is ran by the two independent computer chips.</figcaption></figure><hr><h3 id="full-version-history">Full version history</h3><p>In order for the reader to understand how far we&apos;ve came from during the project, <strong>I&apos;ve compiled ordered pictures of each MakAir version</strong>: prototypes, production models and experiments (ie. the MakAir Mini), that our team has built during the project&apos;s 2 months.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Models.jpg" class="kg-image" alt="MakAir Series: The Inception of an Open-Source Ventilator Project" loading="lazy" width="1600" height="1000" srcset="https://valeriansaliou.name/blog/content/images/size/w600/2021/02/MakAir-Models.jpg 600w, https://valeriansaliou.name/blog/content/images/size/w1000/2021/02/MakAir-Models.jpg 1000w, https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Models.jpg 1600w" sizes="(min-width: 720px) 720px"><figcaption>All MakAir models: our first cardboard prototype (Prototype 1), the wooden prototype that flew to CEA for a demo (Prototype 2), a quick casing we built for the first animal test (Prototype 3), our CEA-built model (Model 1) and our current SEB-built production model (Model 2). Our latest MakAir Mini experiment is shown as well, though it was built much later on.</figcaption></figure><hr><h2 id="project-milestones">Project Milestones</h2><p>For the purpose of keeping a record of important dates and milestones throughout the project, <strong>I made a list of each important milestone</strong>, with dates.</p><p>This can also help you capture how fast things have been going, as well as easily locate an hardware revision in time, based on each milestone that we achieved.</p><ul><li><strong>18th March 2020</strong>: the MakAir project begins in the offices of <a href="https://crisp.chat/">Crisp</a> in <a href="https://en.wikipedia.org/wiki/Nantes">Nantes, France</a>. First prototype of the MakAir ventilator done there (&#x201C;air transistor&#x201D; valves-based);</li><li><strong>22nd March 2020</strong>: the team of 3 moves from Crisp offices to a larger room in <a href="https://lepalace.work">Le Palace</a> in Nantes, France. The team quickly grew larger in the upcoming days (to around 10 engineers on-site). Second prototypes of the MakAir ventilator done there (pinch valves-based). Prof. <a href="https://www.univ-nantes.fr/pierre-antoine-gourraud">Pierre-Antoine Gourraud</a> joins and brings academic/medical credentials to the project;</li><li><strong>23rd March 2020</strong>: universities, hobbyist and local companies like General Electric lend us an array of 3D printers so that we can speed up prototyping (mostly older MakerBot machines);</li><li><strong>25th March 2020</strong>: we are getting officially supported by the French Army, which has been super-quick to respond to our needs and the needs of Covid-relief projects during the crisis;</li><li><strong>28th March 2020</strong>: our collaboration with CEA Grenoble officially starts, with the delivery of 10 brand-new <a href="https://www.makerbot.com/3d-printers/method/">MakerBot Method</a> printers to speed up prototyping and iterate faster (picked up from MakerBot warehouses in The Netherlands and delivered to France in record time). At the same time, we ship wooden prototype in a plane from Nantes to Grenoble, so that the CEA team can have a look at it;</li><li><strong>31st March 2020</strong>: our lab is moved from Nantes to Grenoble in France, the whole team (20+ people) jumps on a 9 hours overnight bus trip from one side of France to the other. We get access to better prototyping and manufacturing facilities to work on the next steps of the project. This is also when we order all required parts (in the thousands!) to plan for the mass-manufacturing of our ventilator, using the colossal funding resources of CEA (which is state-backed);</li><li><strong>3rd April 2020</strong>: first in-lab animal testing of our ventilator prototype for a 12h session, which was a success (on a pig which was dosed <a href="https://en.wikipedia.org/wiki/Curare">Curare</a> in order to put it to sleep and deactivate natural lung breathing). The team was incredibly tense during the test, as this was the first time that a MakAir ventilator could either safe a life, or kill the animal. All went well, and the pig woke up after the test in a healthy condition;</li><li><strong>6th April 2020</strong>: CEA&#x2019;s plastic specialist partner delivers its new MakAir casing. We now have a proper, professional-looking casing in which our components can be fitted. Note that we are not using this casing design anymore, though it looks similar in shape to what we have in the current version;</li><li><strong>8th April 2020</strong>: the manufacturing of 50 production units has started in the <a href="https://en.wikipedia.org/wiki/Cleanroom">cleanroom</a> of CEA Grenoble;</li><li><strong>10th April 2020</strong>: 3 members of the MakAir team hop on another 9 hours overnight road trip from Grenoble to Brest, France, to bring 3 MakAir V1 ventilators to <a href="https://fr.linkedin.com/in/erwan-l-her-9617036a">Erwan L&#x2019;Her</a>, a ventilation specialist in CHU Brest (a public hospital in Brest), with whom we are still collaborating and getting feedback from. He provided medical guidance from there on to the engineering team, in order for us to improve our ventilation algorithms. Note that <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7538074/">an academic paper</a> has also been redacted on how well the MakAir performs in comparison to other ventilators (in French);</li><li><strong>15th April 2020</strong>: second live animal test, on the same pig, using the latest ventilator hardware and software. Success as well, we were less tense this time;</li><li><strong>17th April 2020</strong>: job done at CEA Grenoble, the team is heading back to Nantes by bus, for yet another 9 hours trip. Setting the lab back in our initial location at Le Palace Nantes;</li><li><strong>21st April 2020</strong>: our press coverage program started, in order to bring public awareness about the project. The team is juggling between journalists coming in, and improving MakAir hardware and software on the engineering side;</li><li><strong>29th April 2020</strong>: first live prototype of our improved MakAir hardware with the touchscreen UI, displaying live ventilation graphs. The MakAir ventilator feels more alive than never before;</li><li><strong>Beginning of May 2020</strong>: the project scales down a bit as some volunteer engineer&#x2019;s companies are re-opening, the core team keeps iterating on details of software and hardware, improving the touchscreen interface as well as ventilation algorithms. I am also working on a new casing design on my end, which was later canceled (CAD model <a href="https://a360.co/2BoKPgj">available online</a>);</li><li><strong>20th May 2020</strong>: our team receives the third motherboard revision, which is the one we are still using as of today, featuring a STM32 SoC. This is around that time that Groupe SEB starts manufacturing the MakAir, using our latest hardware;</li><li><strong>21st July 2020</strong>: clinical trials start in Nantes, on human patients. Unfortunately, they are using a deprecated MakAir revision, as there is always a huge lag between the clinical trial authorizations we receive from authorities, and our current state of the art in terms of engineering. The way it works, is that we submit an approval request which would certify a given hardware and software, frozen in time, which then only gets approved around a month later;</li><li><strong>August 2020</strong>: summer happens, everyone is exhausted, the project is taken to a pause on the engineering side, though Groupe SEB continues iterating on their manufacturing processes to mass-produce MakAir ventilators;</li><li><strong>October 2020</strong>: working on a new MakAir casing on my end, which I named MakAir Mini, with the goal of fitting all our hardware components inside a 20x20x20cm casing. The goal is to reduce cost and size, as this casing can be plastic-molded for around 5 EUR of plastic. That was quite a challenge, though I managed to achieve it after 2 assembly tries. The MakAir Mini casing CAD model can be <a href="https://a360.co/2RHpseW">viewed online</a>, or <a href="https://www.youtube.com/watch?v=MN_E_gaBmio">in video</a>;</li><li><strong>October to December 2020</strong>: with the second lockdown in France, the core team is getting back to work. Huge improvements were made on the firmware and touchscreen interface, with a new volume-controlled ventilation mode (adding to the existing pressure controlled mode), improved safety and a lot of new settings. Those improvements all bring MakAir toes to toes with existing professional ventilators on the market. Also, a <a href="https://github.com/makers-for-life/makair-simulator">ventilator simulator</a> was designed, which allows the simulation of ventilation scenarios and physical behavior of the ventilator components, in a pure-digital environment (no atoms, just bits). Clinical trials are still ongoing at this point, in Nantes;</li><li><strong>February 2021</strong>: 2 MakAir ventilators with the latest hardware and software built in late 2020 are being shipped to Madagascar for use on patients (as part of a larger humanitarian effort);</li></ul><hr><h2 id="project-team-contacts">Project Team &amp; Contacts</h2><p>I thought that it was important to <strong>keep a record of all most-active contributors</strong> throughout the project, as they can be contacted at a later point whenever a technical or medical question is raised on a topic.</p><p>As MakAir is not an incorporated organization, but rather a fuzzy decentralized collective (like a <a href="https://en.wikipedia.org/wiki/Decentralized_autonomous_organization">DAO</a> on the blockchain), there is no central contact point. You should rather direct your questions to one of the individual contributors listed below, which are known to be the most active ones.</p><p><strong><u>I&#x2019;ve compiled a list of those contributors, categorized:</u></strong></p><p><strong>Overall project management</strong>:</p><ul><li><strong><a href="https://github.com/waxzce">Quentin Adam</a></strong>, CEO at <a href="https://www.clever-cloud.com/">Clever Cloud</a>. He helped grow the project from a team of 3 to 300+ contributors, through his vast personal/professional network;</li><li><strong><a href="https://github.com/baptistejamin">Baptiste Jamin</a></strong>, CEO at <a href="https://crisp.chat/">Crisp</a>. He&#x2019;s my co-founder at Crisp, and handled day-to-day problems, coordinating the engineering team and helping validate the ventilator on test equipment;</li></ul><p><strong>Firmware &amp; ventilation algorithms</strong>:</p><ul><li><strong><a href="https://fr.linkedin.com/in/gabriel-moneyron">Gabriel Moneyron</a></strong>, CEO at <a href="https://cooprint.fr">Cooprint</a>. He rebuilt the firmware nearly from scratch, re-implementing all the ventilation algorithms (pressure and volume controlled);</li><li><strong><a href="https://github.com/dsferruzza">David Sferruzza</a></strong>, works at <a href="https://www.startup-palace.com">Startup Palace</a>. He designed the serial telemetry protocol that connects the firmware to the touchscreen software, and built the initial firmware source code;</li></ul><p><strong>User-interface &amp; touch-screen</strong>:</p><ul><li><strong><a href="https://github.com/valeriansaliou">Valerian Saliou</a></strong>, CTO at <a href="https://crisp.chat/">Crisp</a>. That&#x2019;s me. I&#x2019;ve built the control UI software, used to display realtime ventilation graphs and control ventilation settings;</li></ul><p><strong>Mechanical parts &amp; manufacturing</strong>:</p><ul><li><strong><a href="https://fr.linkedin.com/in/cl&#xE9;ment-niclot-b53b4646">Cl&#xE9;ment Niclot</a></strong>, mechanical engineer at <a href="https://rtsys.eu/">RTSYS</a>. He designed various mechanical components in the MakAir, like the currently-used pinch valve part;</li><li><a href="https://github.com/eliottvincent"><strong>Eliott Vincent</strong></a>, software engineer at <a href="https://crisp.chat/">Crisp</a>. He helped coordinate with all our suppliers &amp; manufacturers, and build each one of our prototypes in the lab;</li></ul><p><strong>Motherboard, PCB design &amp; electronics</strong>:</p><ul><li><strong><a href="https://fr.linkedin.com/in/vincent-le-cunff-343327a5">Vincent Le Cunff</a></strong>, electronics engineer at <a href="https://www.tronico-alcen.com/">Tronico</a>. He designed our current motherboard design, running the firmware on a STM32 SoC;</li><li><strong><a href="https://github.com/pi-r-p">Pierre Papin</a></strong>, software engineer at <a href="https://senx.io/">SenX</a>. He handled day-to-day electronic needs, designed some PCB designs, tested power components, batteries, etc.;</li></ul><p><strong>Medical questions &amp; clinical trials</strong>:</p><ul><li><strong><a href="https://fr.linkedin.com/in/pierre-antoine-gourraud-35779b6">Pierre-Antoine Gourraud</a></strong>, professor &amp; researcher at <a href="https://www.univ-nantes.fr/">University of Nantes</a>. He provided all the medical background and connections to the medical world (CHU Nantes);</li></ul><hr><h2 id="closing-notes">Closing Notes</h2><p>If you have been reading this article to the end, I&#x2019;d like to thank you for that. This article was the first of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>Worldwide pandemics due to respiratory virus infections might, <a href="https://www.medpagetoday.com/infectiousdisease/publichealth/72867">according to earlier predictions</a>, <strong><em>become more common in the future</em></strong>. If another respiratory pathogen like <a href="https://en.wikipedia.org/wiki/Severe_acute_respiratory_syndrome_coronavirus">SARS-CoV</a> happens again in the future, causing tension on traditional ventilator manufacturing &amp; supply chains, those articles should help teams worldwide kickstart their crisis ventilator project even faster than we did.</p><p>I&#x2019;d be glad if those articles help someone in the future avoid the mistakes we made while we learnt how to design ventilators. We were among the first to design open-source ventilators, starting from nothing, as all existing commercial ventilator designs are closed-source/proprietary at the time. This would help response teams save precious time, ultimately leading to preserving life.</p><p><em><strong>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</strong></em></p><hr><p>&#x1F1EB;&#x1F1F7; <em>Written from Nantes, France.</em></p>


guid: 6038c1339a28aa03ac1f80b8
title: MakAir Series: How to Ventilate a Human Patient 101
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-how-to-ventilate-a-human-patient-101/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Medical-101.png
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/02/MakAir-Medical-101.png" alt="MakAir Series: How to Ventilate a Human Patient 101"><p><em>This is the second article <a href="https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/">of a series</a> on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the COVID-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article contains the medical knowledge that&#x2019;s required to understand how mechanical ventilation works. It takes a problem-first approach, as we want to avoid any complex theory there, which would be useless to the engineers we are. This article will thus explain in a simple way how to (medically) ventilate a patient, as well as teach some important medical vocabulary. This vocabulary will be re-used in further articles when explaining how things work on the engineering side. <em>Let&#x2019;s dive into it!</em></p><p>&#x1F449;&#x1F3FB; <u><strong>Disclaimer</strong></u><strong>: </strong><em><strong>I am not a doctor, nor do I have any background in the medical field. I am an engineer by training. This content should be considered for engineering purposes only, as it might be sometimes imprecise, though sufficient for building a ventilator.</strong></em></p><hr><h2 id="how-human-lungs-work">How Human Lungs Work</h2><h3 id="volume-and-pressure-basics">Volume and pressure basics</h3><p>A typical human <strong>breathes 12 times per minute</strong>, and <strong>circulates around 6L of air</strong> in and out of their lungs, at rest. That&#x2019;s about <strong>500mL of air per inspiration that goes in and out of the lungs</strong>, from which gas exchanges occur (O2 enters the system, CO2 leaves it).</p><p>That&#x2019;s about volume. Speaking about the pressure of air inside the lungs during a respiratory cycle, and showing it as a graph as a function of time, we get:</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/pressure-graph.png" class="kg-image" alt="MakAir Series: How to Ventilate a Human Patient 101" loading="lazy"><figcaption>Normal pressure time curve in the lungs (schematized).</figcaption></figure><p>As you can see on the graph above, the pressure in the lungs can be split in 3 phases for a full respiratory cycle. That&#x2019;s peak pressure: <code>P(peak)</code>, plateau pressure: <code>P(plateau)</code> and positive end-expiratory pressure: <code>PEEP</code>, also called <code>P(expiratory)</code>.</p><p><strong>The lung respiratory cycle can be summed up as such, in terms of pressure:</strong></p><ol><li><strong>The lungs inspire air in</strong>: peak pressure occurs;</li><li><strong>The lung resistance blows a bit of air out</strong>: the pressure stabilizes, coming to a plateau;</li><li><strong>The diaphragm relaxes and the lungs expire air out</strong>: pressure decrease, though retaining a minimum air pressure inside (the positive end-expiratory pressure).</li></ol><p>Note that this pressure cycle is super important to understand, in order to continue with the aspects of mechanical ventilation.</p><p>Forcing a given pressure for each stage of the graph above ensures <strong>a certain volume of air gets in the lungs</strong>. This volume of air contains a certain amount of O2, hence why <strong>the pressure needs to be tuned in order for the required amount of O2 to be provided</strong>.</p><p>As well, <strong>the lungs can only accept so much pressure</strong>. During a natural ventilation cycle, pressure is under control by natural processes and the lungs are safe from any over-pressure event. During mechanical ventilation, over-pressure is a thing and should be closely monitored, as the ventilator could push too much air in the lungs, creating too much pressure inside of them.</p><p>The <strong>lungs alveoli are very fragile</strong>: while they can allow for short pressure spikes, sustaining an elevated pressure might permanently alter some of them, leaving long-term sequels on the patient&#x2019;s respiratory capacity (ie. amount of gas exchange that can occur per breath).</p><hr><h3 id="important-ventilation-concepts">Important ventilation concepts</h3><p>At each breath, the lungs inflate with a given amount of air. This is the <strong>Tidal Volume</strong>, which can be described as the normal volume of air displaced between inhalation and exhalation, without applying extra effort. In a regular healthy person, <strong>tidal volume is around 500 mL per breath</strong>. It can also be expressed as a function of body mass, with 12 mL per kilogram of <a href="https://www.mdcalc.com/ideal-body-weight-adjusted-body-weight">lean body weight</a> at 12 cycles per minute. Calculating the tidal volume from body mass is handy when it comes to ventilating a patient, as each patient physiology has different oxygenation needs.</p><p>The air contains a stable percentage of oxygen at any altitude, that is 21%. The air pressure may vary, thus the total actual amount of O2 molecules vary as well, though the percentage of O2 in the air stays stable at all elevations. Whenever inspiring such natural air, the <strong>Fraction of Inspired Oxygen</strong>, a.k.a. <strong>FiO2</strong>, is <code>0.21</code> (21% O2). While ventilating a patient which experience difficulty breathing, <strong>the FiO2 value can be adjusted by inserting oxygen into the system</strong>, giving out a FiO2 between <code>0.21</code> and <code>0.5</code>. The FiO2 value could even go up to <code>1.0</code> for some conditions, that is, 100% of oxygen in the air. Note that when we refer to FiO2 values, the precision is always imperfect. When ventilating a patient, 100% O2 could be in reality 80% O2 due to physic constraints in the air flow. This would still be sufficient though to maintain the patient in a stable condition.</p><p>When lungs are inflating, an important metric that tells how easily air can get inside of them is the <strong>Lung Compliance</strong>, or pulmonary compliance. It is a measure of the lung&#x2019;s ability to stretch and expand. We will mostly be referring to <strong>static lung compliance</strong> through this series of articles. Static lung compliance is the change of volume (in the lungs), for a given applied pressure (of air to the lungs). In other words, it tells how easily the lungs inflate when given input air.</p><p>At any given time, <strong>Oxygen Saturation</strong> in the blood can be measured. Oxygen saturation tells how well the lungs perform at their function, that is taking O2 in and giving back CO2 out. Oxygen saturation is the fraction of oxygen-saturated hemoglobin relative to total hemoglobin (unsaturated + saturated hemoglobin). Normal <em>arterial</em> <strong>blood oxygen saturation level is between 95% and 100%</strong>. Whenever this level goes below 90%, the body can get into a lot of trouble, ie. organs can fail and cardiac arrest can happen. Blood oxygen saturation is an important metric for hospitals to measure when ventilating a patient, using eg. <a href="https://en.wikipedia.org/wiki/Pulse_oximetry">pulse oximetry</a> (SpO2). This tells about how efficient the ventilator is &#x2014; and its configured settings &#x2014; at keeping the patient well and alive.</p><hr><h2 id="what-happens-for-covid-19-ards-patients">What Happens For COVID-19 &amp; ARDS Patients</h2><p>Patients who develop <strong>Acute respiratory distress syndrome</strong> (ARDS) due to COVID-19, fail to maintain proper natural ventilation (oxygenation), due to a rapidly-evolving inflammation in the lungs. ARDS impairs the lungs ability to exchange O2 and CO2, which in turn pose a life-threatening risk on organs and heart.</p><p>People that once experienced ARDS exhibit long-term damage to their lungs function, for which respiratory capacity is dramatically decreased post-remission, affecting their quality of life.</p><p>While the underlying causes of ARDS are being treated, an ARDS patient might need to be kept stable under mechanical ventilation, depending on the situation. Mechanical ventilation settings are chosen based on the medical condition of the patient. Light conditions could be treated with <a href="https://en.wikipedia.org/wiki/Non-invasive_ventilation">non-invasive ventilation</a>, using a simple face mask (the patient would be kept conscious). In worst cases, the patient would be put asleep, natural lung function would be deactivated, the patient would be <a href="https://en.wikipedia.org/wiki/Tracheal_intubation">intubated</a> and ventilated in a (positive) pressure controlled mode.</p><p>Given that <strong>COVID-19 patients under invasive ventilation may stay intubated for 2 weeks</strong>, specific ventilators capable of handling prolongated ventilation need to be used. Not all ventilators are fit for this, as the most occurring ones are support ventilators used for ventilation aid, or emergency purposes (eg. in emergency vehicles). Only the bulkiest and most expensive ICU ventilators are capable of handling COVID-19 patients, <strong>for which the price can range from 25,000 USD to 50,000 USD</strong> (!!). <a href="https://www.hamilton-medical.com/">Hamilton Medical</a> and <a href="https://www.draeger.com/">Dr&#xE4;ger</a> manufacture such ventilators.</p><hr><h2 id="how-to-ventilate-a-patient">How To Ventilate a Patient</h2><h3 id="ventilation-modes">Ventilation modes</h3><p>Depending on the condition of the patient, the way they get ventilated can be chosen amongst a known list of ventilation modes.</p><p>Ventilation modes are named using acronyms, eg. <code>PC-CMV</code>, where the first part of the acronym contains the mode (eg. <code>PC</code>), and the second part tells about the mode behavior (eg. <code>CMV</code>).</p><p>The most popular ventilation modes are <strong>Pressure Control</strong> (PC) and <strong>Volume Control</strong> (VC). Those are two schools of thought in ventilation, that are quite equivalent in terms of treatment outcomes. Medical personnel are usually more familiar with one or the other, which explains why they chose pressure over volume or volume over pressure.</p><p>For each ventilation mode, a specific behavior can be chosen. The <strong>respiratory cycles can be either forced by the ventilator, or requested by the patient</strong>. In the latter case, the ventilator would detect an inspiration trigger by the patient by sensing a drop in pressure, as the lungs would try to inflate. Some middle grounds exist, where the chosen ventilation mode would force a given number of respiratory cycles, while the patient could still ask spontaneously for a cycle between two forced cycles. What matters in that case is that the measured amount of cycles in the current minute approximates the configured cycles target.</p><p>As of end 2020, <strong>the MakAir ventilator implements 5 combinations of ventilation modes</strong> (3 pressure modes, 2 volume modes). Those are the most common ventilation modes, which can also be found on traditional ventilators.</p><p>Other more advanced, exotic or experimental ventilation modes exist, but we chose not to implement them. For instance, <a href="https://en.wikipedia.org/wiki/Dual-control_modes_of_ventilation">dual-control modes of ventilation</a> exist, where the ventilator would try to reach both a target pressure and volume at the same time. For instance, with <code>PRVC</code> or Pressure Regulated Volume Control, a tidal volume target is added to a Pressure Assist Control mode (<code>PC-AC</code>).</p><p><strong>Supported ventilation modes on the MakAir are:</strong></p><ul><li><code>Pressure CMV</code> or Pressure Continuous Mandatory Ventilation: respiratory rate is forced and a target pressure is set to be matched (<em>note that this mode was the only one we supported on MakAir Model 1 in April 2020</em>) (<strong>forced mode</strong>, <strong>invasive</strong>);</li><li><code>Pressure AC</code> or Pressure Assist Control: a target respiratory rate and pressure are set, though the patient can trigger a cycle by inspiring air (<strong>forced mode with trigger</strong>, <strong>invasive</strong>);</li><li><code>Pressure VSAI/BIPAP</code> or Bilevel Positive Airway Pressure: an inspiratory airway pressure is delivered (<code>IPAP</code>), that is higher to the expiratory airway pressure (<code>EPAP</code>), while the patient is kept fully responsive of triggering cycles. This mode is common in the treatment of sleep apnea (<em>note that <code>VSAI</code> is a French term that we use, for which <code>BIPAP</code> is the English equivalent</em>) (<strong>spontaneous mode</strong>, <strong>non-invasive</strong>);</li><li><code>Volume CMV</code> or Volume Continuous Mandatory Ventilation: respiratory rate is forced, a target tidal volume is set to be matched (resulting in a target minute volume in the lungs, in L/min) (<strong>forced mode</strong>, <strong>invasive</strong>);</li><li><code>Volume AC</code> or Volume Assist Control: a target respiratory rate and tidal volume are set, though the patient can trigger a cycle by inspiring air (<strong>forced mode with trigger</strong>, <strong>invasive</strong>);</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/makair-control-ui.png" class="kg-image" alt="MakAir Series: How to Ventilate a Human Patient 101" loading="lazy"><figcaption>The MakAir Control UI running in Pressure AC mode, on which ventilation measurements can be seen. Note the difference between target (15) and measured (39) cycles per minute, due to the ventilation algorithm still converging to the set value.</figcaption></figure><hr><h3 id="units-in-use">Units in use</h3><p>In the world of mechanical ventilation, some unusual units are used to express physical values such as pressure.</p><p>Most units in use on ventilators are time-based units, as well as <a href="https://en.wikipedia.org/wiki/Fluid_dynamics">fluid</a> characterization units (pressure, volume, flow).</p><p><strong>Units that are used in mechanical ventilation:</strong></p><ul><li><code>cmH2O</code> &amp; <code>mmH2O</code>: <a href="https://en.wikipedia.org/wiki/Centimetre_or_millimetre_of_water">centimetre &amp; millimetre of water</a>, unit of pressure used to measure air pressure;</li><li><code>ms</code> &amp; <code>s</code>: unit of time used eg. to calculate the inspiration-expiration ratio;</li><li><code>L</code> &amp; <code>mL</code>: unit of volume, used eg. to express the volume of air in the lungs (the <a href="https://en.wikipedia.org/wiki/Tidal_volume">tidal volume</a>);</li><li><code>L/min</code>: unit of flow, used eg. to express the amount of air going in and out of the lungs every unit of time (the <a href="https://en.wikipedia.org/wiki/Minute_ventilation">minute volume</a>);</li><li><code>cycles/minute</code>: number of respiratory cycles per minute;</li></ul><hr><h3 id="ventilation-settings">Ventilation settings</h3><p>Aside from configuring the ventilation mode, which defines the behavior of algorithms with measurements, ventilation target values can be configured as well. The ventilator will attempt to reach those target values at any time during ventilation (or ring an alert if it fails to do so).</p><p>Settings should be configured using educated doctor decisions &amp; calculations, as every patient condition require specific tunings before they can be connected to the ventilator.</p><p>For instance, <strong>severe ARDS patients tend to have very resistive lungs that would collapse on themselves</strong> if high pressure could not be sustained in the lungs anytime, making it very hard to inflate again. This is known as the fibrotic stage of ARDS. In such cases, a doctor would make the decision to set an higher-than-usual <code>PEEP</code> target.</p><p><strong>Some settings are commonly used in mechanical ventilation:</strong></p><ul><li><code>P(peak)</code> in cmH2O: target peak pressure during inspiration (<em>automatically calculated</em>);</li><li><code>P(plateau)</code> in cmH2O: target plateau pressure during inspiration;</li><li><code>PEEP</code> or <code>P(expiratory)</code> in cmH2O: target positive end-expiratory pressure, which is the minimum pressure held in the lungs after expiration;</li><li><code>Inspiratory duration</code> in ms: target inspiratory duration;</li><li><code>Inspiration-expiration ratio</code> as (1:x): target inspiratory-to-expiratory ratio;</li><li><code>Tidal volume</code> in mL: volume of air pushed to the lungs at each cycle;</li><li><code>Cycles per minute</code> in cycles/minute: number of breaths per minute (full respiratory cycle);</li><li><code>Minute volume</code> in L/min: volume of air pushed to the lungs every minute (in other words: <code>minute volume = (tidal volume x cycles per minute)</code>);</li></ul><p><strong>More advanced settings are available:</strong></p><ul><li><code>Trigger offset</code> in cmH2O: the drop in pressure that would trigger an inspiration (typically as low as a pressure offset of 2-5 cmH2O);</li><li><code>Plateau duration</code> in ms: the duration of the plateau stage on the pressure curve (please refer to the typical pressure graph shown at the beginning of the article to visualize when plateau occurs);</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/makair-settings.png" class="kg-image" alt="MakAir Series: How to Ventilate a Human Patient 101" loading="lazy"><figcaption>Ventilation settings can be edited through the MakAir Control UI. Some settings are available for certain ventilation modes only. Medical personnel can configure ventilation settings before connecting the patient, or modify settings in-flight while the patient is connected.</figcaption></figure><hr><h3 id="safety-alarms">Safety &amp; alarms</h3><p>For patient safety and regulatory compliance reasons, <strong>any mechanical ventilator must alert the medical personnel whenever an anomaly occurs</strong>.</p><p><strong>Alarms must be loud and audible</strong>, and emit a specific pitch. The screen must also show the list of ongoing alarms, with different priorities, from low to high.</p><p><strong>Most alarms are predefined</strong> and thus cannot be customized. For instance, the ventilator must be able to detect whenever the patient is disconnected from the ventilator. Some alarms can also be customized, depending on the current ventilation mode.</p><p><strong>Custom alarms can be set up in order to alert medical teams when a value goes over or under a set threshold:</strong></p><ul><li><code>Inspiratory minute volume</code> in L/min: minimum and maximum allowed air flow per minute during inspiratory cycles;</li><li><code>Expiratory minute volume</code> in L/min: minimum and maximum allowed air flow per minute during expiratory cycles;</li><li><code>Respiratory rate</code> in cycles/minute: breathing cycles per minute that must be achieved;</li><li><code>Tidal volume</code> in mL: volume that goes in and out of the lungs for each respiratory cycle (this one is important, as when multiplied with the number of cycles per minute, it tells how much O2 the patient takes in per minute);</li><li><code>Leak volume</code> in mL/min: maximum volume of air per minute that is allowed to leak out of the circuit (any ventilation circuit has some level of leak, which can be due to the breathing system and/or the patient&#x2019;s physiology);</li><li><code>Peak pressure</code> in cmH2O: maximum allowed pressure during the peak stage of the pressure curve;</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/02/makair-alarms-severe.png" class="kg-image" alt="MakAir Series: How to Ventilate a Human Patient 101" loading="lazy"><figcaption>A MakAir Control UI exhibiting a range of high-severity alarms. Note that since at least one high-severity alarm is being raised, the ventilator emits a high-pitched noise from its buzzer every few seconds, and stops when no high-severity alarm remains.</figcaption></figure><hr><h2 id="closing-notes">Closing Notes</h2><p>This article was the second of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>Being an engineer and having no medical background, I would like to re-emphasis that <strong>this article is aimed at engineers and might contain medical imprecisions</strong>. If you are an engineer looking to understand how mechanical ventilation works by taking a first principle approach, this article might have helped you (please let me know!). If you are a doctor and you spotted errors, please leave a comment or <a href="https://valeriansaliou.name">contact me</a> so that I can fix them!</p><p>All the medical knowledge contained in this article has been learned from online resources and volunteering doctors coming on-site during the time of the project to teach us medical concepts on a drawing board (literally!). We came from having zero knowledge on how the lungs work to what got us to design a working ventilator. We are still learning though, thanks to clinical trial results and further research related to adding new features and improving ventilation algorithms.</p><p><strong><em>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</em></strong></p><hr><p>&#x1F1EE;&#x1F1F9; <em>Written from Venice, Italy.</em></p>


guid: 603d0cc29a28aa03ac1f80f9
title: MakAir Series: The Building Blocks of a Mechanical Ventilator
author: Valerian Saliou
url: https://valeriansaliou.name/blog/makair-series-the-building-blocks-of-a-mechanical-ventilator/
feedurl: https://journal.valeriansaliou.name/rss/
enclosure_url: https://valeriansaliou.name/blog/content/images/2021/03/MakAir-Building-Blocks.jpg
content_mime_type: text/html
content:
<img src="https://valeriansaliou.name/blog/content/images/2021/03/MakAir-Building-Blocks.jpg" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator"><p><em>This is the third article <a href="https://valeriansaliou.name/blog/makair-series-the-inception-of-an-open-source-ventilator-project/">of a series</a> on the <a href="https://github.com/makers-for-life">MakAir open-source ventilator</a>, a project that I founded with friends, which was born at the beginning of the COVID-19 pandemic in early 2020. Think of this series of articles as a ledger of all that happened and all that was created. These articles could serve as the foundation of a project reboot in future years, in the event a similar pandemic was to emerge again.</em></p><p>This article describes the building blocks of a mechanical ventilator, in a high-level way. That is, it does not go too deep into the engineering concepts, which will be covered in later articles. The goal here is to describe how a ventilator works, and how all components interface together. Also, as regulatory &amp; compliance requirements constrain how the machine gets designed, they will be listed and explained.</p><p><em>Looking up the definition of <a href="https://en.wikipedia.org/wiki/Mechanical_ventilation">Mechanical Ventilation</a>:</em></p><blockquote>Mechanical ventilation, assisted ventilation or intermittent mandatory ventilation, is the medical term for artificial ventilation where mechanical means are used to assist or replace spontaneous breathing.</blockquote><hr><h2 id="how-a-ventilator-works">How a Ventilator Works</h2><p>A ventilator is a machine that <strong>produces a target air volume at a given pressure</strong>. It regulates the air flow and pressure in real-time, in order to match the ventilation requirements of the patient, without compromising on safety.</p><p>In order to produce air, multiple methods exist. While a ventilator could use a <a href="https://en.wikipedia.org/wiki/Piston">piston</a> system, the most compact and flexible way to produce air is by using a <a href="https://en.wikipedia.org/wiki/Turbine">turbine</a>. Turbines, while noisy, are handy as they can be very precisely ramped up or down. While keeping a small form-factor, <strong>they are capable of producing 200 L/min+ of air</strong>.</p><p>Now that air is produced, the respiratory cycle needs to be forced. This means that <strong>the ventilator would need to regulate how much air can come in or out</strong> (of the lungs) at any given time. In other words, in needs to increase or decrease the resistance in the inlet or outlet circuits. During an inspiratory cycle, the lungs inflate, thus the inlet circuit must be open, while the outlet circuit must be kept closed. In the hold phase (at the end of the inspiration), the inlet and outlet circuits must be closed, in order to retain air in the lungs. Then, during the expiratory cycle, the outlet must be opened while the inlet must stay close, that way the air gets out of the lungs.</p><p>Note that when &#x201C;opened&#x201D; or &#x201C;closed&#x201D; are mentioned, you should read &#x201C;<strong><em>opened or closed to a certain degree</em></strong>&#x201D;. Depending on the current state of the patient, all of the available air volume might not be blowed in or out of the lungs, hence why the ventilator needs to have a precise control of its pneumatics aperture.</p><p>In order to perform all that is listed above, <strong>the ventilator needs to be able to sense what is happening</strong>. To do so, it needs to measure in real-time the pressure in the inspiratory system (in cmH2O), as well as the air flow in the inspiratory and expiratory branches (in L/min). These measurements are fed to a control algorithm that regulates the aperture of the pneumatics (in degrees) and the blower turbine speed (in RPM).</p><p>All those components would not work in tandem if not controlled by a central brain, that is software (eg. <a href="https://github.com/makers-for-life/makair-firmware">MakAir firmware</a>). Software needs to be ran on a CPU on a motherboard (eg. <a href="https://github.com/makers-for-life/makair-motherboard">MakAir motherboard</a>), to which all sensors and actuators are connected.</p><p>As well, some components can be quite power hungry, like the blower turbine. Stable power must be provided at any time by a power unit, composed of an AC-DC transformer, and backup power with batteries. Given the cyclic nature of some components, there might be power surges into the system, which the power components need to handle properly.</p><p>In order to get feedback on the current state of the ventilation of a patient, <strong>a user interface should be exposed to the end-user</strong>, which live information on important measured values, as well as easy-access buttons to adjust settings, as the ventilator is running.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/Pneumatic-Circuit.png" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A schematic view of the pneumatic circuit, that was designed in the making of MakAir Model 1. The only change we have made to this design since, is adding a second flow sensor on the expiratory branch. Please zoom to read details.</figcaption></figure><p>That is about as simple as the basic functions of a ventilator are! We will dive into the details next.</p><hr><h2 id="internal-components">Internal Components</h2><p>Given the constraints listed above, we could easily make a list of components that would need to be fitted together to do the job.</p><p><strong>Following the air circuit, we would have in order:</strong></p><ol><li>A <strong>blower turbine</strong> to inject pressurized air into the system;</li><li>A <strong>pinch valve system</strong>, able to apply more or less mechanical pressure on a flexible pipe, effectively increasing or decreasing the pipe diameter and thus adjusting the air flow, at constant blower speed (a valve per branch, thus one for inspiratory and one for expiratory);</li><li>A range of <strong>air pressure and air flow sensors</strong>, providing instant measurements to the software layer;</li><li>A <strong>main motherboard PCB</strong>, housing a SoC (<a href="https://en.wikipedia.org/wiki/System_on_a_chip">System on a Chip</a>), running ventilation software. All sensors and actuators (blower, valves) are to be connected to the motherboard;</li><li>An <strong>AC-DC transformer and a battery pack</strong>, providing a stable DC power supply to all electronics;</li></ol><p>Note that the component list above is not exhaustive. Some more useful components might be added in between two required components to fulfill a smaller task, eg. mix pressurized air with pure oxygen.</p><p>As well, in recent versions of the MakAir ventilator, we have added a 7&#x201D; touchscreen interface, which is not 100% required for the ventilator to perform its functions, though it lets the user <strong>configure more advanced settings</strong> as well as <strong>select another ventilation mode</strong>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-internals.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A view of MakAir Model 2 ventilator internals. It does not take much physical parts to build a ventilator. Most of the complexity lies in the software.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/Casing-Layout-Pneumatics.png" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A drawing of the pneumatics stage of MakAir Model 1, which has not changed much in Model 2. Note that the two flow sensors are missing, as we added them since. Design by <a href="https://lasmafedotova.lv/">L&#x101;sma Fedotova</a>.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/Casing-Layout-Electronics.png" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A drawing of the electronics stage of MakAir Model 1, which is the same in Model 2. Design by <a href="https://lasmafedotova.lv/">L&#x101;sma Fedotova</a>.</figcaption></figure><hr><h2 id="casing-design">Casing Design</h2><p>The <strong>ventilator casing holds all components together</strong>, firmly and solidly. As the casing is the only part of the machine that end users (ie. doctors) see, it must inspire robustness (ie. confidence).</p><h3 id="casing-sturdiness-to-weight-sweet-spot">Casing sturdiness-to-weight sweet spot</h3><p>The MakAir being an emergency ventilator at first, there is a high chance that some of them will be shipped in remote areas using eg. military transportation, in large quantities. This use-case means that <strong>the ventilator casing must handle <a href="https://en.wikipedia.org/wiki/Shock_(mechanics)">shocks</a> in multiples of <em>g</em></strong> (the acceleration of gravity), at worst.</p><p>In real-world use on a patient, the ventilator may sometimes lay on a slippery surface for weeks. Hence, <strong>the ventilator casing should have a rubber feet system providing adherence</strong> to any kind of surface. Rubber feet added to a some weight both make the ventilator slip-safe. For instance, MakAir Standard models (the large ones) weigh 22 kg, while MakAir Mini weighs around 3 kg (the difference being due to the casing plastic density and lead-acid batteries in the standard models). Having some weight is desirable, though the ventilator should be as light as possible for obvious practical reasons. It is important to point out that raw casing weight contributes <strong><em>a lot</em></strong> to the final product weight.</p><hr><h3 id="casing-insulation-to-fluids">Casing insulation to fluids</h3><p>The ventilator needs to have <strong>some level of insulation against fluids</strong>, eg. water or solvants like IPA, used to disinfect its surfaces before and after use. It is a common practice in hospitals to disinfect medical device exteriors using liquids.</p><p>As the front side would usually be washed with more attention than other parts due to the front panel being often touched by doctors, insulation is especially important there. This is unfortunately the side in which <strong>liquids could enter more easily due to the amount of assembled parts</strong> and may cause damage to nearby electronics.</p><hr><h3 id="casing-internationalization">Casing internationalization</h3><p>To ease with internationalization, <strong>printed text on the casing should be limited to a minimum</strong>, eg. only to the rear where labels are shown on the AC, Ethernet and USB ports. The rest of localized texts should be handled in a <strong>pure-software way</strong>, on the touch screen. We found out that it was quite tedious to manage, for instance, front button label prints for each country. Thus, we decided to handle all controls in a pure-software way. This makes ventilator manufacturing quite convenient, as one ventilator design is able to target all countries at once. Shipping it to say, Mexico, means simply changing the default language flag in the firmware&#x2019;s memory chip from English to Spanish (instead of the whole front panel surface).</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-model-1-front.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>The front panel of an old MakAir Model 1, in which we had to produce localized front panels, here in English. This would not be very practical if we were to mass-manufacture this model, and store them in warehouses without knowing in advance which country they would be departing for. Better let software handle this!</figcaption></figure><hr><h3 id="casing-manufacturing-cost">Casing manufacturing &amp; cost</h3><p>In order to mass-produce a product, its components should be manufactured with a high velocity. Considering the ventilator casing, each casing should take a few seconds to a minute to produce. That way, costs are kept to a minimum while manufacturing output is high.</p><p><strong>Plastic molds should be built as soon as possible during the project</strong>, whenever the ventilator is ready for manufacturing. A mold has, unfortunately, a large upfront one-time cost (around 15,000 EUR, in our case), but can be made profitable quickly on large production volumes, as it lowers the cost to produce each unit (<strong>a MakAir Mini casing raw cost is estimated to be 5 EUR using <a href="https://en.wikipedia.org/wiki/Injection_moulding">plastic injection</a></strong>).</p><p>This compares to the MakAir Standard casings, which are based on extruded electrical boxes that cost 300 EUR each &#x2014; <em>plus labor cost to get it ready for final assembly</em>. That could easily add up to a total cost of 400 EUR.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-landfill.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A &#x201C;landfill&#x201D; of old MakAir prototype casings. We had to extrude (cut through) each one using sharp blades in order to create holes to fit the front panel et al. All those casings are very heavy and impractical for mass production, but were great for prototyping purposes.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-standard-vs-mini-comparison.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A MakAir Mini prototype, sitting on the top of a MakAir Model 2 production ventilator. For the size (and weight) of 1 MakAir Standard, we could ship 6 MakAir Mini.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-mini-raw-casing.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A 3D printed MakAir Mini prototype casing, before assembly. This designs aims at being entirely moldable in plastic for a low cost and high repeatability.</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-mini-finished-casing.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A 3D printed MakAir Mini prototype casing, after assembly. Note that the casing is bare plastic, all buttons are shown on the top screen. Also note that the edges and corners of this prototype are too sharp, we would need to round them up in later revisions for safety reasons.</figcaption></figure><hr><h3 id="considerations-for-prototyping">Considerations for prototyping</h3><p>While developing the original iPhone, Apple engineers <a href="https://www.theverge.com/2019/3/19/18263844/apple-iphone-prototype-m68-original-development-board-red">were working on a large development board</a>, which was much easier to iterate on using bare hands. Miniaturization came afterwards, when manufacturing the final product. We did exactly the same thing with the MakAir.</p><p><strong>It is always a good idea to prototype on a scaled-up version of your product, letting engineers swap components and try new things with their bare hands</strong>. Our original MakAir Model 1 and Model 2 were designed to ease prototyping and maintenance. We then built a much smaller MakAir Mini using the same components.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-model-1-prototyping.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>Two engineers assembling a now-deprecated MakAir Model 1 prototype, with their bare hands. Note that there is so much space that multiple people can work on the same machine at the same time.</figcaption></figure><hr><h2 id="regulatory-safety-requirements">Regulatory &amp; Safety Requirements</h2><p>As all electronics on the market, any MakAir device produced must comply with certain safety standards.</p><p>As well, due to the MakAir being a medical device used on humans, <strong>it must implement several important regulatory points, and then get certified by each target country</strong> regulatory authority (eg. FDA for the US).</p><p><strong>I have compiled a non-exhaustive list of the most important regulatory and safety requirements</strong> that must be considered when designing and manufacturing a mechanical ventilator such as the MakAir.</p><h3 id="regulatory-requirements">Regulatory requirements</h3><p>Those points are absolutely necessary in order for the ventilator to pass medical certification:</p><ul><li>Electronics within the ventilator emit <strong><a href="https://en.wikipedia.org/wiki/Electromagnetic_radiation">Electromagnetic Radiations</a></strong>, that must be kept within limits as it may interfere with nearby medical equipments in the hospital. This is especially true for the AC-DC transformer, which is the most likely emitter;</li><li>The ventilator must <strong>emit audible and visible alarms</strong> when doctors are to be notified of an anomaly during patient ventilation. Sounds must be emitted at a specific pitch and level from a buzzer, which must be sufficiently loud to be heard from other rooms in an hospital. Alarm codes, severity and associated buzzer pitches <a href="https://github.com/makers-for-life/makair-telemetry/blob/master/src/alarm.rs">are normalized</a>;</li><li>Whenever the patient needs to be moved rooms, or the hospital loses AC power, <strong>the ventilator must move to backup power on batteries</strong>. It must guarantee an autonomy of 30 minutes minimum, though the MakAir has been tested to last up to 4 hours on batteries;</li><li>All materials that come into contact with air that get into the patient lungs (mostly plastics), <strong>must be proven to be <a href="https://en.wikipedia.org/wiki/Biocompatibility">biocompatible</a></strong> through a laboratory test. They must not release harmful chemicals over time. If they do so, it must be within safety limits;</li><li>Input and output air should <strong>always be filtered using</strong> (at least) <a href="https://en.wikipedia.org/wiki/HEPA">HEPA-14 filters</a>, which blocks most particles with a diameter of 0.1 microns (the COVID-19 virus is 0.125 microns). This means that the patient doesn&#x2019;t get cross-contaminated by germs from the hospital room air, and that virus particles don&#x2019;t get blown out of the ventilator;</li><li>As the ventilator might be shipped in cargo airplanes in large quantities, the battery that&#x2019;s inside should conform to airline <a href="https://www.faa.gov/hazmat/resources/lithium_batteries/">FAA regulations</a>. <strong>Lithium batteries can be problematic</strong> (given their power density), which is why <strong>lead-acid batteries might be preferred</strong>, though bulkier and heavier. Batteries could as well be swapped out, and installed upon delivery of the machine;</li><li>Before and after the ventilator is used, medical personnel must be able to <strong>disinfect the air circuit and casing</strong> with eg. an <a href="https://en.wikipedia.org/wiki/Isopropyl_alcohol">Isopropyl alcohol</a> solution;</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-internals-breakdown.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>MakAir Mini during assembly. So many electronics are packed together, emitting electromagnetic radiations, which must be kept in check.</figcaption></figure><hr><h3 id="safety-standards-ease-of-use">Safety standards &amp; ease of use</h3><p>Those safety &amp; usability points are must-have in a mechanical ventilator if is to be accepted by hospitals and doctors:</p><ul><li>The <strong>expiratory branch should be easily cleanable and replaceable</strong>, <em>unless a brand new ventilator is to be used for each patient</em>. As this branch uses negative pressure relative to the lungs, patient air always goes out from there. The inspiratory branch is not affected, as it uses <a href="https://en.wikipedia.org/wiki/Positive_pressure">positive pressure</a>. The pipes and the air flow sensor should be entirely replaceable on the expiratory branch, after each use.</li><li>The ventilator <strong>must sometimes be used up to 2 weeks on the same patient</strong>, continuously. It has to be able to sustain this (mechanically), and allow for medical personnel to intervene and wash pipes whenever the patient cough and clogs the tubes (ie. the one that connect the ventilator to the patient, brought by the hospital);</li><li>For usage safety reasons, <strong>the ventilator plastic casing must not exhibit sharp edges</strong>. All edges and corners should be rounded, which also facilitates disinfection of the ventilator exterior;</li><li>As pure oxygen (100% O2) might be injected and circulated in the ventilator pneumatic circuit, <strong>any spark could start a fire</strong>. Consideration must be taken to make things safe under pure oxygen;</li><li>The casing should be <strong>molded using a high-quality color-neutral <a href="https://en.wikipedia.org/wiki/Acrylonitrile_butadiene_styrene">ABS plastic</a></strong>. Color pigments might release chemicals, hence why most medical devices have that specific white-yellowish color as their ABS plastic does not contain any color pigment;</li><li>In terms of usability, anything shown on the control screen, especially fonts, <strong>must be visible from a certain distance</strong> (meaning: from the far side of the bed). Not all people have a perfect vision, especially when working late during the night where eyes become tired and things get blurry;</li><li>The <strong>control screen should lay at a specific orientation</strong>, from 90 degrees (ie. vertical), to 45 degrees. Medical personnel should not have to bend over the ventilator to use the screen (<em>as we did in the MakAir Mini prototype</em>, on which we plan to let the screen be deployed at a custom angle using an articulated joint &#x2014; you can <a href="https://a360.co/2RHpseW">see it in 3D there</a>);</li><li>As the ventilator targets an international &#x201C;audience&#x201D;, its interface and documentation <strong>should be available in multiple languages</strong>. That is to avoid human error due to misunderstandings. Language could be pre-configured during manufacturing, and adjusted later through the ventilator interface. It is important that it persists across machine reboots;</li><li>Pressure sensors measure the air pressure relative to ambiant air. As the ambiant air pressure may vary in certain situations, the ventilator should <strong>auto-calibrate its air pressure offset</strong> regularly, when the ventilator is not in use (eg. when the machine is manufactured, and later every time it boots up);</li><li>The ventilation control software (ie. the firmware) must be proven to be 100% reliable, as <strong>it has the full responsibility of maintaining the patient alive</strong> in the case of <a href="https://www.sciencedirect.com/topics/medicine-and-dentistry/invasive-ventilation">invasive ventilation</a>. 4 minutes without oxygen induces <a href="https://medlineplus.gov/ency/article/001435.htm">permanent brain damage</a>, <em>this point must not be treated lightly</em>. As well, in case of a mechanical failure of a sensor or actuator, the firmware must detect it and start buzzing alarms so that medical personnel can move the patient to a second ventilator;</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://valeriansaliou.name/blog/content/images/2021/03/makair-filter-box-sls.jpg" class="kg-image" alt="MakAir Series: The Building Blocks of a Mechanical Ventilator" loading="lazy"><figcaption>A MakAir filter box, in which an HEPA-14 filter is fitted. It is 3D printed using <a href="https://en.wikipedia.org/wiki/Selective_laser_sintering">SLS</a> in Medical Grade <a href="https://en.wikipedia.org/wiki/Nylon">Nylon</a>, which can be used as a biocompatible material (if post-processed &amp; washed accordingly).</figcaption></figure><hr><h2 id="closing-notes">Closing Notes</h2><p>This article was the third of a series on the MakAir story, providing insider details on the project, medical knowledge on the state of the art of mechanical ventilation, as well as in-depth engineering explanations on how we designed our ventilator algorithms and electronics. <strong>The overall goal of this series of articles is to build the legacy of the MakAir project.</strong></p><p>As <strong>commercial ventilator designs are patented and thus not freely available on the Internet</strong>, getting together the building blocks of a ventilator could take someone weeks or months. This is literally what the MakAir team went through, by iterating on designs until we stuck to the right one. Being given a bird&apos;s-eye view of existing ventilators would have helped us save tremendous time when we began, which is the goal of this article. I hope you will find solace in it.</p><p><strong><em>Also, I&#x2019;m always available to answer questions, please ask them in the comments section of this article!</em></strong></p><hr><p>&#x1F1EE;&#x1F1F9; <em>Written from Rome, Italy.</em></p>


