# rss_item

guid: https://www.neilhenning.dev/posts/fpscev-inst-simplify/
title: Using Floating-Point Scalar Evolution to Simplify Instructions
url: https://www.neilhenning.dev/posts/fpscev-inst-simplify/
feedurl: https://www.duskborn.com/index.xml
content:
<p>In this post, we&rsquo;ll use floating-point scalar evolution (fpscev) to simplify instructions based on any range data we&rsquo;ve managed to deduce.</p>
<p>This post is the latest in a series about my experimentations with floating-point scalar evolution, you really want to read the them in order:</p>
<ul>
<li><a href="https://www.neilhenning.dev/posts/fpscev/">An Experimental Floating-Point Scalar Evolution</a></li>
<li><a href="https://www.neilhenning.dev/posts/fpscev-fast-math-propagation/">Using Floating-Point Scalar Evolution to Propagate Fast-Math Flags</a></li>
</ul>
<p>The LLVM optimization pass discussed in this post is available <a href="https://github.com/sheredom/fpscev/commit/efce1ce5f2649ddf447b3a6fb76bdd457284f775">on github here</a>.</p>
<h1 id="instruction-simplification">Instruction Simplification</h1>
<p>LLVM is missing a bunch of instruction simplification surrounding floating-point - precisely because it is missing the fpscev analysis.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> tail call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.sqrt.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f)
</span></span></code></pre></div><p>In the above example, lets say we have deduced that <code>%f</code> is always negative. The <a href="https://en.cppreference.com/w/c/numeric/math/sqrt">rules for sqrt</a> state that if the input is negative, the output is NaN. LLVM cannot spot this code because it cannot deduce the range of <code>%f</code>.</p>
<p>Another example is with the minnum/maxnum/minimum/maximum intrinsics:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> tail call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.minnum.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span></code></pre></div><p>If <code>%f2</code> was strictly less than <code>%f</code>, then we could replace this call with <code>%f2</code> <em>since it is always less than</em>. Again LLVM cannot spot this.</p>
<p>LLVM <em>does</em> have some pseudo floating-point evaluations surrounding the fcmp and all the various round-to-integer intrinsics. For instance the following fcmp:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#f92672">%</span>b <span style="color:#f92672">=</span> fcmp oeq <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#f92672">%</span>f2
</span></span></code></pre></div><p>Will be simplified to the constant false if the compiler has locally worked out that <code>%f</code> can never be equal to <code>%f2</code>.</p>
<p>This is great that LLVM has some insight into these values - but I&rsquo;d still argue having a single mechanism based around the fpscev analysis would be more fruitful in the long term for optimizations.</p>
<h1 id="overview-of-the-pass">Overview of the Pass</h1>
<p>The new pass I&rsquo;ve added is <code>FPInstSimplifyPass</code> (see <a href="https://github.com/sheredom/fpscev/blob/master/fpscev.cpp#L1585">https://github.com/sheredom/fpscev/blob/master/fpscev.cpp#L1585</a>) - it runs over a function using an <code>InstVisitor</code>, and uses the fpscev analysis to simplify instructions and intrinsics.</p>
<p>The main purpose of this pass is to try and remove instructions or replace instructions with more simple variants where knowing the range of the inputs lets us deduce a more simple form.</p>
<p>For example, lets say we have a simple code snippet of the form:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_oeq_false</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">16.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>b <span style="color:#f92672">=</span> fcmp oeq <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3, <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  ret i1 <span style="color:#f92672">%</span>b
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Because <code>%f3</code> and <code>%f</code> cannot overlap (and neither can be NaN), their values can never compare equal. LLVM already manages to optimize this to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_oeq_false</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  ret i1 false
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>It does this in a roundabout way though - it turns the floating-point operations above into integer operations (in the float-to-int pass), and then relies on scalar evolution of integers to optimize the code.</p>
<p>Another case, this time with something that fails to optimize with LLVM:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_ole_true</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">15.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>b <span style="color:#f92672">=</span> fcmp ole <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>  ret i1 <span style="color:#f92672">%</span>b
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>LLVM turns this into:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_ole_true</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span><span style="color:#ae81ff">1</span> <span style="color:#f92672">=</span> zext i4 <span style="color:#f92672">%</span>i to i32
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span><span style="color:#ae81ff">2</span> <span style="color:#f92672">=</span> zext i4 <span style="color:#f92672">%</span>i2 to i32
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f31 <span style="color:#f92672">=</span> add nuw nsw i32 <span style="color:#f92672">%</span><span style="color:#ae81ff">2</span>, <span style="color:#ae81ff">15</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>b2 <span style="color:#f92672">=</span> icmp uge i32 <span style="color:#f92672">%</span>f31, <span style="color:#f92672">%</span><span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span>  ret i1 <span style="color:#f92672">%</span>b2
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>A little weird - but it basically canonicalizes the ordered-less-than-equal into an ordered-greater-than-equal (and swaps the operands), and then converts this into integer operations like it did above. For some reason LLVM won&rsquo;t optimize this further (even though I think integer scalar-evolution should be able to?).</p>
<p>I personally think for both the above cases LLVM shouldn&rsquo;t be converting them to use integers, and I suspect it does this only so that scalar evolution can help with optimization. Instead, my fpscev analysis can be used to optimize <em>both</em> the cases above. Lets take a look!</p>
<h2 id="floating-point-comparison">Floating-Point Comparison</h2>
<p>Lets start our floating-point comparison optimization by looking at the example that LLVM doesn&rsquo;t do a fantastic job of above:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_ole_true</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">15.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>b <span style="color:#f92672">=</span> fcmp ole <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>  ret i1 <span style="color:#f92672">%</span>b
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In this example <code>%f</code> is always ordered-less-than-equal to <code>%f3</code> because <code>%f3</code> is in the range [15.0..30.0] and <code>%f</code> is in the range [0.0..15.0].</p>
<p>Using my fpscev analysis we can use the range of these values to fold this fcmp away:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span>FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>    fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(inst.getOperand(<span style="color:#ae81ff">0</span>))<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> FPSCEV yFpscev <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>    fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(inst.getOperand(<span style="color:#ae81ff">1</span>))<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>CmpInst<span style="color:#f92672">::</span>Predicate predicate <span style="color:#f92672">=</span> CmpInst<span style="color:#f92672">::</span>BAD_FCMP_PREDICATE;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">switch</span> (inst.getPredicate()) {
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">default</span><span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">return</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">case</span> CmpInst<span style="color:#f92672">::</span>FCMP_OLE:
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (<span style="color:#f92672">!</span>xFpscev.isNaN() <span style="color:#f92672">&amp;&amp;</span> <span style="color:#f92672">!</span>yFpscev.isNaN()) {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> (xFpscev.isLessThanEqual(yFpscev)) {
</span></span><span style="display:flex;"><span>      predicate <span style="color:#f92672">=</span> CmpInst<span style="color:#f92672">::</span>FCMP_TRUE;
</span></span><span style="display:flex;"><span>    } <span style="color:#66d9ef">else</span> <span style="color:#a6e22e">if</span> (xFpscev.isGreaterThan(yFpscev)) {
</span></span><span style="display:flex;"><span>      predicate <span style="color:#f92672">=</span> CmpInst<span style="color:#f92672">::</span>FCMP_FALSE;
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">break</span>;
</span></span><span style="display:flex;"><span><span style="color:#75715e">// all the other cases...
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (predicate <span style="color:#f92672">!=</span> CmpInst<span style="color:#f92672">::</span>BAD_FCMP_PREDICATE) {
</span></span><span style="display:flex;"><span>  inst.setPredicate(predicate);
</span></span><span style="display:flex;"><span>  modified <span style="color:#f92672">=</span> true;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Since LLVM already has logic to handle folding FCMP_TRUE and FCMP_FALSE away, I just modify the predicate of the existing fcmp operation and will let LLVM handle the rest.</p>
<p>The resulting code after applying this optimization is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define i1 <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fcmp_ole_true</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">1.500000e+01</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>b <span style="color:#f92672">=</span> fcmp true <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>  ret i1 <span style="color:#f92672">%</span>b
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>As can be seen the fcmp ole has become an fcmp true - the optimization has correctly noted that the compare would always return true. Note that because this is an ordered compare we had to be careful with NaNs. You&rsquo;ll notice that I had a check on both the fpscev&rsquo;s to the fcmp to see whether either of them were NaN - and not apply the optimization if so. This is another case where my fpscev analysis, precisely because it is very good at detecting no-NaN situations, allows us to perform a greater set of optimizations across a broader set of input values.</p>
<h2 id="intrinsics">Intrinsics</h2>
<p>The other case where I try and use fpscev analysis to simplify instructions is with intrinsics. LLVM is pretty poor at folding floating-point min/max operations for instance, and fpscev allows us to perform similar optimizations as with fcmp on these.</p>
<p>One thing that sucks with LLVM&rsquo;s InstVisitor is that it doesn&rsquo;t have a nice way to iterate through intrinsics. The best method I&rsquo;ve found is to do something like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">void</span> <span style="color:#a6e22e">visitIntrinsicInst</span>(IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">switch</span> (inst.getIntrinsicID()) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">default</span><span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">break</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">case</span> Intrinsic<span style="color:#f92672">::</span>sqrt:
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// do something!
</span></span></span></code></pre></div><p>But this means you end up with uber-functions containing a ton of implementation detail on each intrinsic. Also the fact that everything inhabits a big switch statement means that it is easy to fall foul of C++&rsquo;s insane switch fallthrough default.</p>
<p>To get around this I&rsquo;ve used some <em>very basic</em> templates to let us have each intrinsic optimization in its own function, and using a macro to reduce duplicating the intrinsic id:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsicInst(IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">switch</span> (inst.getIntrinsicID()) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">default</span><span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span>;
</span></span><span style="display:flex;"><span><span style="color:#75715e">#define INTRINSIC_VISIT(x)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  case x:                                                                      \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    return visitIntrinsic&lt;x&gt;(inst)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>sqrt);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>pow);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>fabs);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>minnum);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>maxnum);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>minimum);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>maximum);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>copysign);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>floor);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>ceil);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>trunc);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>rint);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>nearbyint);
</span></span><span style="display:flex;"><span>    INTRINSIC_VISIT(Intrinsic<span style="color:#f92672">::</span>round);
</span></span><span style="display:flex;"><span><span style="color:#75715e">#undef INTRINSIC_VISIT
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>This just lets me have all the intrinsics above inhabit their own function which makes debugging much easier (at least for me). Note - I use this pattern pseudo-regularly in C++ and I always recommend that you do not provide a default implementation of <code>visitIntrinsic</code>. Doing this means you&rsquo;ll get a linker error if you forget to explicitly add the visit method for that intrinsic.</p>
<h2 id="square-root">Square Root</h2>
<p>For the sqrt intrinsic the main thing we can do is with negative inputs. A negative input to a sqrt always results in a NaN.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>sqrt<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the input is all negative, the result is NaN.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (xFpscev.isAllNegative()) {
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// If the instruction cannot return NaNs, replace it with undef.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#66d9ef">if</span> (inst.hasNoNaNs()) {
</span></span><span style="display:flex;"><span>      inst.replaceAllUsesWith(UndefValue<span style="color:#f92672">::</span>get(inst.getType()));
</span></span><span style="display:flex;"><span>    } <span style="color:#66d9ef">else</span> {
</span></span><span style="display:flex;"><span>      inst.replaceAllUsesWith(ConstantFP<span style="color:#f92672">::</span>getNaN(inst.getType()));
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>So the code above gets the fpscev for the input, checks if all the values of it are negative, and then removes the sqrt entirely if so.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">sqrt_all_negative</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">-</span><span style="color:#ae81ff">1.0</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.sqrt.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In the above example we can see that the input to sqrt is always negative, and so my optimization will turn the code into:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">sqrt_all_negative</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">-</span><span style="color:#ae81ff">1.000000e+00</span>)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#ae81ff">0x7FF8000000000000</span>
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>EG. always returning a NaN.</p>
<p>There is another case we have to be wary of - what if the sqrt intrinsic has a no-NaNs fast-math flag specified?</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">sqrt_all_negative_nnan</span>(i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call nnan <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.sqrt.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In this case - the user has specifically said that sqrt <em>cannot</em> produce a NaN, but we know 100% that a NaN is being produced. This is called a <a href="https://www.llvm.org/docs/LangRef.html#poison-values">poison value</a> in LLVM. LLVM does not have a way to specifically create a poison value, so in the above case I replace the sqrt with an undef instead:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">sqrt_all_negative_nnan</span>(i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> undef
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="x-to-the-power-of-y">X to the Power of Y</h2>
<p>The pow intrinsic is something that I couldn&rsquo;t get a great handle on its range for fpscev due to the complexity of the function. One thing I can do though is deduce the range of Y - the value we are promoting X to the power of. LLVM has a companion intrinsic powi - X to the power of integer Y. I&rsquo;ve implemented the pow/powi functions for OpenCL-level ulp requirements in a past life, and I know one of the complications of pow is that if you have a fractional component to the value you are raising to the power of, this increases the complexity of the calculation. Knowning that the power you are raising to is entirely integer (as is the case with powi) lets math library writers do some fun optimizations (I vaguely recall having at least 20% faster calls to powi versus pow when I worked on this many years ago!).</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>pow<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> y <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV yFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(y)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If y is an integer
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (yFpscev.isInteger) {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> fltSemantics <span style="color:#f92672">&amp;</span>semantics <span style="color:#f92672">=</span> xFpscev.min.getSemantics();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// 2^24 is the last integer number that we can fully represent in both
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#75715e">// floating-point and i32.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#66d9ef">const</span> APInt <span style="color:#a6e22e">minInt</span>(<span style="color:#ae81ff">32</span>, <span style="color:#f92672">-</span>(<span style="color:#ae81ff">1</span> <span style="color:#f92672">&lt;&lt;</span> <span style="color:#ae81ff">24</span>), true);
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> APInt <span style="color:#a6e22e">maxInt</span>(<span style="color:#ae81ff">32</span>, <span style="color:#ae81ff">1</span> <span style="color:#f92672">&lt;&lt;</span> <span style="color:#ae81ff">24</span>, true);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// If y is between min/max int, we can use powi instead of pow!
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#66d9ef">if</span> (yFpscev.isGreaterThanEqual(getFromInt(semantics, minInt, true)) <span style="color:#f92672">&amp;&amp;</span>
</span></span><span style="display:flex;"><span>        yFpscev.isLessThanEqual(getFromInt(semantics, maxInt, true))) {
</span></span><span style="display:flex;"><span>      IRBuilder<span style="color:#f92672">&lt;&gt;</span> irb(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>      StringRef <span style="color:#a6e22e">yName</span>(y<span style="color:#f92672">-&gt;</span>getName());
</span></span><span style="display:flex;"><span>      Twine <span style="color:#a6e22e">name</span>(yName, <span style="color:#e6db74">&#34;.i32cast&#34;</span>);
</span></span><span style="display:flex;"><span>      Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> yCast <span style="color:#f92672">=</span> irb.CreateFPToSI(y, irb.getInt32Ty(), name);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      Instruction <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> powi <span style="color:#f92672">=</span> irb.CreateIntrinsic(
</span></span><span style="display:flex;"><span>          Intrinsic<span style="color:#f92672">::</span>powi, inst.getType(), {x, yCast}, <span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>      powi<span style="color:#f92672">-&gt;</span>takeName(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      <span style="color:#75715e">// Copy the fpscev information from the original instruction onto the new.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>      fpse<span style="color:#f92672">-&gt;</span>copyFPSCEVFromValue(powi, <span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      inst.replaceAllUsesWith(powi);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The two conditions we need to pass if we can convert pow to powi is that Y <strong>must</strong> be an integer-hiding in a floating-point (which fpscev tracks), and that this integer <strong>must</strong> be less than 2^24. 2^24 is an interesting value because it just so happens to be the last integer value that can be represented without conversion loss in a 32-bit floating-point.</p>
<p>Once these two conditions are met, we can convert Y to a signed integer, and then call the powi intrinsic instead.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">pow</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, i25 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> sitofp i25 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.pow.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The above becomes:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">pow</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, i25 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> sitofp i25 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2.i32cast <span style="color:#f92672">=</span> fptosi <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2 to i32
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.powi.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, i32 <span style="color:#f92672">%</span>f2.i32cast)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>After this optimization is applied.</p>
<h2 id="floating-point-absolute-value">Floating-Point Absolute Value</h2>
<p>Abosolute value can be simplified if the value it was performing the absolute of was already positive.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>fabs<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the input is all not negative, we can just use the input.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (xFpscev.isAllNonNegative()) {
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(x);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>We just check if the input x was all not negative, and if so replace the fabs intrinsic with the value being fed to fabs. This means the following example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fabs_already_positive</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#ae81ff">1.0</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.fabs.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Can be folded to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">fabs_already_positive</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#ae81ff">1.0</span>)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><h2 id="minimums-and-maximums">Minimum&rsquo;s and Maximum&rsquo;s</h2>
<p>For both minnum/maxnum/minimum/maximum I perform a similar optimization - so I&rsquo;ll just look at maxnum here for the sake of brevity.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>maxnum<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> y <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV yFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(y)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If one of the inputs is always less than the other, we fold away the
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#75715e">// intrinsic.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (xFpscev.isGreaterThanEqual(yFpscev)) {
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(x);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> <span style="color:#a6e22e">if</span> (yFpscev.isGreaterThanEqual(xFpscev)) {
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(y);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>For maxnum, if we know that either of the inputs are always greater than or equal to the other value (assuming that neither of them are NaNs), we can replace the call to maxnum with the input that is guaranteed to dominate the other.</p>
<p>This means that the following code:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">maxnum_x</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">16.0</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.maxnum.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Is folded to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">maxnum_x</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fadd <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#ae81ff">1.600000e+01</span>
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Removing the call to maxnum entirely! This same optimization technique is applied to all four of the min/max intrinsics as I previously stated.</p>
<h2 id="copy-sign">Copy Sign</h2>
<p>The intrinsic copysign has a few interesting optimization avenues.</p>
<ul>
<li>If the input and the output have the same sign, then the intrinsic is a no-op.</li>
<li>If the input and output always have opposite signs, then we are really just negating the input X.</li>
<li>If the input Y whom we are copying the sign from is always positive, then we are really just calling fabs on the input X.</li>
</ul>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>copysign<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> y <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV yFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(y)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> ((xFpscev.isAllNonNegative() <span style="color:#f92672">&amp;&amp;</span> yFpscev.isAllNonNegative()) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>      (xFpscev.isAllNegative() <span style="color:#f92672">&amp;&amp;</span> yFpscev.isAllNegative())) {
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(x);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> <span style="color:#a6e22e">if</span> ((xFpscev.isAllNonNegative() <span style="color:#f92672">&amp;&amp;</span> yFpscev.isAllNegative()) <span style="color:#f92672">||</span>
</span></span><span style="display:flex;"><span>             (xFpscev.isAllNegative() <span style="color:#f92672">&amp;&amp;</span> yFpscev.isAllNonNegative())) {
</span></span><span style="display:flex;"><span>    IRBuilder<span style="color:#f92672">&lt;&gt;</span> irb(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> value <span style="color:#f92672">=</span> irb.CreateFNeg(x);
</span></span><span style="display:flex;"><span>    value<span style="color:#f92672">-&gt;</span>takeName(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// Copy the fpscev information from the original instruction onto the new.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    fpse<span style="color:#f92672">-&gt;</span>copyFPSCEVFromValue(value, <span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> (Instruction <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> otherInst <span style="color:#f92672">=</span> dyn_cast<span style="color:#f92672">&lt;</span>Instruction<span style="color:#f92672">&gt;</span>(value)) {
</span></span><span style="display:flex;"><span>      otherInst<span style="color:#f92672">-&gt;</span>setFastMathFlags(inst.getFastMathFlags());
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(value);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> <span style="color:#a6e22e">if</span> (yFpscev.isAllNonNegative()) {
</span></span><span style="display:flex;"><span>    IRBuilder<span style="color:#f92672">&lt;&gt;</span> irb(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    Instruction <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> fabs <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>        irb.CreateUnaryIntrinsic(Intrinsic<span style="color:#f92672">::</span>fabs, x, <span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    fabs<span style="color:#f92672">-&gt;</span>takeName(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// Copy the fpscev information from the original instruction onto the new.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    fpse<span style="color:#f92672">-&gt;</span>copyFPSCEVFromValue(fabs, <span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(fabs);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Above we can see the three cases of optimizations we can apply - which matches to five variants of copysign we could see in the wild. The following example has all five of these cases:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_positive</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_negative</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f5 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f5
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_opposite_0</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_opposite_1</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_y_positive</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.copysign.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And after my optimization is applied:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_positive</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_negative</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_opposite_0</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> fsub <span style="color:#66d9ef">float</span> <span style="color:#f92672">-</span><span style="color:#ae81ff">0.000000e+00</span>, <span style="color:#f92672">%</span>f2
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_both_opposite_1</span>(i4 <span style="color:#f92672">%</span>i, i4 <span style="color:#f92672">%</span>i2) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i2 to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> fneg <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f4 <span style="color:#f92672">=</span> fsub <span style="color:#66d9ef">float</span> <span style="color:#f92672">-</span><span style="color:#ae81ff">0.000000e+00</span>, <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f4
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">copysign_y_positive</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f, i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f3 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.fabs.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f3
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Note: I don&rsquo;t <em>really understand</em> why LLVM has produced fsub from -0 instead of fneg when I used the IRBuilder here - but it should effectively be the same thing.</p>
<h2 id="floating-point-rounding">Floating-Point Rounding</h2>
<p>The rounding intrinsics floor/ceil/trunc/rint/nearbyint/round can all be folded to their input if the input is already an integer (because if the input is a whole number, rounding will do nothing). Since the optimization for each intrinsic is the same, lets just look at round:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPInstSimplifyPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>round<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>  Value <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> x <span style="color:#f92672">=</span> inst.getOperand(<span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span> fpse<span style="color:#f92672">-&gt;</span>getFPSCEV(x)<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the input is already an integer, floor is a no-op.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (xFpscev.isInteger) {
</span></span><span style="display:flex;"><span>    inst.replaceAllUsesWith(x);
</span></span><span style="display:flex;"><span>    toRemoves.push_back(<span style="color:#f92672">&amp;</span>inst);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The code is simple - if the input is an integer, replace the intrinsic with the input. This will fold code like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">round_already_integer</span>(i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f2 <span style="color:#f92672">=</span> call <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span>llvm.round.<span style="color:#a6e22e">f32</span>(<span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f)
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f2
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>To:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>define <span style="color:#66d9ef">float</span> <span style="color:#960050;background-color:#1e0010">@</span><span style="color:#a6e22e">round_already_integer</span>(i4 <span style="color:#f92672">%</span>i) {
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">%</span>f <span style="color:#f92672">=</span> uitofp i4 <span style="color:#f92672">%</span>i to <span style="color:#66d9ef">float</span>
</span></span><span style="display:flex;"><span>  ret <span style="color:#66d9ef">float</span> <span style="color:#f92672">%</span>f
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Note these are optimizations that LLVM will do in very simple cases anyway (like the above example) - but LLVM will fall apart once you start to hit more complex variants. As I&rsquo;ve said before - removing these piecemeal optimizations in LLVM and unifying them around the more correct fpscev analysis seems like a better approach for the long term.</p>
<h1 id="conclusion">Conclusion</h1>
<p>With this pass we&rsquo;ve started to really get into the fun with fpscev - instruction simplification. Being able to fold away complex code for more simpler variants is what I got into the compiler business for in the first place, and with fpscev we have a great extra tool to do that.</p>
<p>In the next post I&rsquo;ll look at refining the range of some of the more complex fpscev intrinsics by using some advice from the rather fantastic <a href="https://twitter.com/marc_b_reynolds/status/1137820453777657858">Marc B. Reynolds</a> to reduce the calculated ranges. Look forward to seeing you there!</p>


guid: https://www.neilhenning.dev/posts/last-day-amd/
title: Farewell AMD / Hello Unity
url: https://www.neilhenning.dev/posts/last-day-amd/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Announcement time - today is my last day at AMD. I&rsquo;ll be taking a few weeks off, before starting a new gig at Unity mid-July.</p>
<p>It is also the end of my five year involvement in Vulkan &amp; SPIR-V - having contributed to both specifications since 2014.</p>
<p>I&rsquo;ve very much enjoyed my time in this slice of the industry, but it was time for a new challenge.</p>
<h1 id="farewell-amd">Farewell AMD</h1>
<p>I&rsquo;ve been at AMD a little over a year now - and it has been an interesting challenge. I was brought in to improve the performance of the LLVM compiler stack (LLPC) as compared to the legacy Shader Compiler (SC). All-in during my time there I improved the LLVM stack by just over 2x performance across a raft of key shaders, which I am pretty proud of.</p>
<p>I also implemented a bunch of fun shader extensions in the stack which was a lot of fun (VK_EXT_buffer_device_address, VK_KHR_vulkan_memory_model, and quite few secret-sauce ones still to come!).</p>
<p>I think the stack is in a good state going forward.</p>
<p>As always - there is <em>always</em> a few things about any employment that aren&rsquo;t as great. AMD is an <em>old</em> company, 50 years old this year. Some of the internal systems feel about that old too. Doing expenses is a pain - the system is archaic and frustrating, getting new hardware wasn&rsquo;t always as easy as it could be, and I wasn&rsquo;t quite prepared for cross-team politics that a big company like AMD is going to have.</p>
<p>The other thing I <em>really wish</em> I had asked about before I start was how code is actually written. Perforce was the main version control system I used while at AMD, and I find perforce to be an unmitigated disaster for software development. I genuinely think I was at about 10% of my peak productivity.</p>
<p>I think coming from a relatively small company like Codeplay (about 85 people when I left) who had much leaner systems and which individuals could easily move the needle left me a little ill-prepared for the reality of a big companies. You live and learn - I am now much wiser.</p>
<p>That being said - I still did some good and enjoyable work, made some positive changes, and made some good friends. AMD has a bunch of really talented staff that move mountains to make good changes.</p>
<h1 id="farewell-khronos">Farewell Khronos</h1>
<p>I&rsquo;ve been working on the Vulkan &amp; SPIR-V specifications for about five years now, and I was instrumental in getting a bunch of really important features (most notably the subgroup operations in Vulkan 1.1 / SPIR-V 1.3) through. People often mock design-by-committee for arguing over what they see as death-by-over-design - but I found the whole standards process to be a really amazing collaboration between what are effectively enemy companies. They fact you have so many eyes on the same set of specifications means that you get a much better and more rounded result.</p>
<p>I&rsquo;ll miss my weekly phone calls discussing the ins and outs of how shaders execute on all the fun hardware we make them work on. There is some good features that I worked on in Khronos to come to developers in future - I look forward to people getting to use some of my final specification work!</p>
<h1 id="im-bursting-to-tell-you-what-is-next">I&rsquo;m Bursting to Tell You What Is Next</h1>
<p>Even though I&rsquo;ve looked about 70 years old since I was 15 - I&rsquo;m not retiring or anything like that!</p>
<p>I&rsquo;ve decided to join <a href="https://unity.com">Unity</a> to work on the <a href="https://unity.com/dots#burst-compiler">Burst Compiler</a>. The Burst Compiler is something that is <em>incredibly</em> interesting to me as a compiler engineer - lets take a language and add all the constraints we require to reduce it to a performance-orientated subset, and make a compiler that can make this code run <em>fast</em>. After some initial discussions with <a href="https://twitter.com/lucasmeijer">Lucas Meijer</a>, and many conversations with <a href="https://twitter.com/deplinenoise">Andreas Fredriksson</a> and <a href="https://twitter.com/xoofx">Alexandre Mutel</a> I was 100% convinced. The chance to make such a big difference to so many developers lives is such a huge appeal to me personally - I always want my code to make a positive impact to as many people as I can. I also get to focus much more heavily on LLVM which is a technology I love to work with too - super excited!</p>
<p>A couple of weeks to relax (I have some extensive gardening still to do) and then I&rsquo;ll be onboarding at Unity and getting to grips with all the future fun there is to have. I&rsquo;ll keep y&rsquo;all posted!</p>


guid: https://www.neilhenning.dev/posts/fpscev-improved-range/
title: FPSCEV - Improving the Range Analysis
url: https://www.neilhenning.dev/posts/fpscev-improved-range/
feedurl: https://www.duskborn.com/index.xml
content:
<p>In this post, we&rsquo;ll improve our floating-point scalar evolution (fpscev) to do a better job at evaluating the range of some of the more complex intrinsics we analyze.</p>
<p>This post is the latest in a series about my experimentations with floating-point scalar evolution, you really want to read them in order:</p>
<ul>
<li><a href="https://www.neilhenning.dev/posts/fpscev/">An Experimental Floating-Point Scalar Evolution</a></li>
<li><a href="https://www.neilhenning.dev/posts/fpscev-fast-math-propagation/">Using Floating-Point Scalar Evolution to Propagate Fast-Math Flags</a></li>
<li><a href="https://www.neilhenning.dev/posts/fpscev-inst-simplify/">Using Floating-Point Scalar Evolution to Simplify Instructions</a></li>
</ul>
<p>The fpscev pass changes discussed in this post is available <a href="https://github.com/sheredom/fpscev/commit/e32e04ce9f229ca27ef40213c4ec695efc1a7cc2">on github here</a>.</p>
<h1 id="improving-the-range">Improving the Range</h1>
<p>When I wrote the first draft of the fpscev analysis, any functionality that LLVM&rsquo;s <a href="https://llvm.org/doxygen/classllvm_1_1APFloat.html#a9fccce07c15282e9fd964254f21d81c7">APFloat</a> did not contain I simply assumed the worst case for the range of the input (-NaN..NaN unless fast-math flags were specified). For instance, the code used to evaluate the range of exp2 was:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span>FPSCEV <span style="color:#a6e22e">fpscev</span>(inst.getType());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// Exp always has a positive result.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">const</span> APFloat zero <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getZero(fpscev.min.getSemantics(), false);
</span></span><span style="display:flex;"><span>fpscev.min <span style="color:#f92672">=</span> zero;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> FastMathFlags flags <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>fpscev.max <span style="color:#f92672">=</span> applyFastMathFlags(fpscev.max, flags);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>fpse.map[<span style="color:#f92672">&amp;</span>inst] <span style="color:#f92672">=</span> fpscev;
</span></span></code></pre></div><p>I could do no better because APFloat did not contain an exp2 call that I could make use of to evaluate the range. After having some twitter discussions with the rather awesome <a href="https://twitter.com/marc_b_reynolds/status/1137820453777657858">Marc B. Reynolds</a>, I realised I should revisit my assumption on the ranges of some of these intrinsics to see if I couldn&rsquo;t make a better guess than what was there.</p>
<h1 id="exp2">Exp2</h1>
<p>For exponentials with a base of 2 as I&rsquo;ve already stated there isn&rsquo;t an LLVM APFloat function that I can just call into. There is something sort of similar that I can use though to estimate the range such that we can return something better than 0..NaN! LLVM APFloat does contain a <a href="https://llvm.org/doxygen/classllvm_1_1APFloat.html#ab6f3e137b277266516df5fd9eb692719">scalbn</a> definition for APFloat. Scalbn allows you to efficiently multiply a value by 2-to-the-power-of another value - <code>x * 2^y</code>. If <code>x</code> <em>just happened</em> to be 1.0 - then we are effectively calculating <code>2^y</code>. This just so happens to be exp2! The one downside is that we can only specify a <code>y</code> that is an integer - so we will lose some precision on the range by doing this.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span>FPSCEV <span style="color:#a6e22e">exp2Bounds</span>(FPSCEV fpscev) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> fltSemantics <span style="color:#f92672">&amp;</span>semantics <span style="color:#f92672">=</span> fpscev.getSemantics();
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> APFloat one <span style="color:#f92672">=</span> getOne(semantics);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">bool</span> losesInfo;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (fpscev.min.isFinite()) {
</span></span><span style="display:flex;"><span>    fpscev.min.roundToIntegral(APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>    fpscev.min.convert(APFloat<span style="color:#f92672">::</span>IEEEdouble(), APFloat<span style="color:#f92672">::</span>rmTowardNegative,
</span></span><span style="display:flex;"><span>                       <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">int</span> power <span style="color:#f92672">=</span> <span style="color:#66d9ef">static_cast</span><span style="color:#f92672">&lt;</span><span style="color:#66d9ef">int</span><span style="color:#f92672">&gt;</span>(fpscev.min.convertToDouble());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    fpscev.min <span style="color:#f92672">=</span> scalbn(one, power, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> {
</span></span><span style="display:flex;"><span>    fpscev.min <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getInf(semantics, true);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (fpscev.max.isFinite()) {
</span></span><span style="display:flex;"><span>    fpscev.max.roundToIntegral(APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>    fpscev.max.convert(APFloat<span style="color:#f92672">::</span>IEEEdouble(), APFloat<span style="color:#f92672">::</span>rmTowardPositive,
</span></span><span style="display:flex;"><span>                       <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">int</span> power <span style="color:#f92672">=</span> <span style="color:#66d9ef">static_cast</span><span style="color:#f92672">&lt;</span><span style="color:#66d9ef">int</span><span style="color:#f92672">&gt;</span>(fpscev.max.convertToDouble());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    fpscev.max <span style="color:#f92672">=</span> scalbn(one, power, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> {
</span></span><span style="display:flex;"><span>    fpscev.max <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getInf(semantics, false);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">return</span> fpscev;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The approach I take is:</p>
<ol>
<li>Round the minimum range value down to the next whole integer, the maximum range up to the next whole integer.</li>
<li>Convert them to doubles rounding the values away from zero again (this only really matters in the case where we are evaluating a quad-precision number that isn&rsquo;t fully representable in 64-bits of precision).</li>
<li>Convert these doubles to integer (as scalbn takes an int <code>y</code>). I rely on the fact that compilers generally round an overflowing double to INT_MAX/INT_MIN which will easily create an infinity/-infinity when we raise that value by the power of 2.</li>
<li>Then use scalbn like I stated above to create the bounds.</li>
<li>Note: I only attempt this approach at all if the min/max value were finite in the first place.</li>
</ol>
<p>Lets have a look at the range of this exp2 value plotted across some key values to understand how accurate our evaluation is:</p>

<img src="/images/fpscev-range/exp2.png"  class="center"  style="border-radius: 8px;"    />


<p>The chart above firstly shows that the <em>actual</em> value of exp2 if we could calculate it correctly lies inclusively within the calculated min/max ranges we&rsquo;ve established using scalbn. So the result is always <strong>correct</strong>. You can also see that the higher the input value to the exp2 function, the more imprecise our range analysis becomes (as exp2 gets more, well, <em>exponential</em> the higher the input value this is to be expected).</p>
<p>All in all - this is pretty good! Much better than always returning a range of 0..NaN anyway.</p>
<h1 id="log2">Log2</h1>
<p>For log2 I explored a similar idea to the solution I used for exp2 above. While LLVM APFloat doesn&rsquo;t have a suitable log2 equivalent function, it does have <a href="https://en.cppreference.com/w/cpp/numeric/math/ilogb">ilogb</a> though. The ilogb function effectively extracts the exponent from the floating-point number without fudging it like which happens with frexp (frexp always returns a number between 0.5..1, and so has to fudge the exponent it returns to make sure of that). Because ilogb loses the fractional part of our floating-point number, we can use ilogb of the minimum-range to get the lower bound of our result. For the upper bound, we use ilogb and then add one onto it - this rounds up the estimation to the next highest whole number.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span>FPSCEV <span style="color:#a6e22e">log2Bounds</span>(FPSCEV fpscev) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> fltSemantics <span style="color:#f92672">&amp;</span>semantics <span style="color:#f92672">=</span> fpscev.getSemantics();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the full range is negative, always returns a NaN.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (fpscev.isAllNegative()) {
</span></span><span style="display:flex;"><span>    APFloat nan <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getNaN(semantics, false);
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> FPSCEV(nan, nan, false);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> APInt ilogbMin(<span style="color:#ae81ff">32</span>, ilogb(fpscev.min));
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// We do +1 here so we round the range up.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">const</span> APInt ilogbMax(<span style="color:#ae81ff">32</span>, ilogb(fpscev.max) <span style="color:#f92672">+</span> <span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  APFloat values[<span style="color:#ae81ff">2</span>] <span style="color:#f92672">=</span> {getFromInt(semantics, ilogbMin, true),
</span></span><span style="display:flex;"><span>                       getFromInt(semantics, ilogbMax, true)};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the input is entirely greater than zero, we can use ilogb to get a much
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#75715e">// firmer estimate on the log result.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (fpscev.isGreaterThan(APFloat<span style="color:#f92672">::</span>getZero(semantics))) {
</span></span><span style="display:flex;"><span>    fpscev.min <span style="color:#f92672">=</span> getMinimum(values);
</span></span><span style="display:flex;"><span>    fpscev.max <span style="color:#f92672">=</span> getMaximum(values);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> {
</span></span><span style="display:flex;"><span>    fpscev.min <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getInf(semantics, true);
</span></span><span style="display:flex;"><span>    fpscev.max <span style="color:#f92672">=</span> values[<span style="color:#ae81ff">1</span>];
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// Can&#39;t be sure about this because we&#39;re getting very vague bounds, so wipe
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#75715e">// it.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  fpscev.isInteger <span style="color:#f92672">=</span> false;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">return</span> fpscev;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Lets have a look at the chart for this to show how good our bounds are:</p>

<img src="/images/fpscev-range/log2.png"  class="center"  style="border-radius: 8px;"    />


<p>As can be seen - the actual result is entirely within our calculated min/max range, and because logarithms grow <em>so</em> slowly the range isn&rsquo;t so bad (as compared to our exponential prediction from before).</p>
<h1 id="log--log10">Log &amp; Log10</h1>
<p>Logarithms have some very fun rules - one of which is the change of base formula. For those unfamiliar, any log can be transfered into any other log calculation by log<!-- raw HTML omitted -->a<!-- raw HTML omitted -->(x) = log<!-- raw HTML omitted -->b<!-- raw HTML omitted -->(x) / log<!-- raw HTML omitted -->b<!-- raw HTML omitted -->(a). This means we can use the calculation of the bounds we already discovered for log2 to calulate a bound for any other log! I use this same trick for log and log10 - so I&rsquo;ll just cover log here to keep this succinct.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPScalarEvolutionPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>log<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  FPSCEV fpscev <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>      fpse.getFPSCEV(inst.getOperand(<span style="color:#ae81ff">0</span>))<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscev <span style="color:#f92672">=</span> log2Bounds(fpscev);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// We are calculating log(x), but we only have ilogb (which gives us an
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#75715e">// effective lower bound of log2(x)). To convert this bound into log(x) by
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#75715e">// multiplying it by log(2).
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  APFloat <span style="color:#a6e22e">converter</span>(
</span></span><span style="display:flex;"><span>      <span style="color:#ae81ff">0.693147180559945309417232121458176568075500134360255254120</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">bool</span> losesInfo;
</span></span><span style="display:flex;"><span>  converter.convert(fpscev.getSemantics(), APFloat<span style="color:#f92672">::</span>rmNearestTiesToEven,
</span></span><span style="display:flex;"><span>                    <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscev.min.multiply(converter, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>  fpscev.max.multiply(converter, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpse.map[<span style="color:#f92672">&amp;</span>inst] <span style="color:#f92672">=</span> fpscev.cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>I basically calculate the log2 range of the input, and then use the change of base formula for logarithms to turn this into the bounds for log instead. Lets look at the chart:</p>

<img src="/images/fpscev-range/log.png"  class="center"  style="border-radius: 8px;"    />


<p>Pretty good! The bounds follow a similar trajectory to the previous log2 calculation, but in this newer range of log.</p>
<h1 id="pow">Pow</h1>
<p>Anyone who has tried to implement <code>pow(x, y)</code> will know that there is a common hack that gets touted as some miracle solution to this calculation - turning <code>pow(x, y)</code> into <code>exp2(y * log2(x))</code>. There are numerous reasons why this <em>isn&rsquo;t generally a good idea</em> (the precision loss is terrible when you do the <code>y * log2(x)</code>, you have to handle a negative <code>x</code> to an integer <code>y</code> that might be odd/even and this affects the sign of the result, etc) - but since we have no alterative and we are ok to lose a bit of precision as long as we lose the precision correctly (EG. as long as we always grow the range through precision loss rather than accidentally shrink it).</p>
<p>Pow has a lot of chances to go awry (fractional bases, fractional powers, negatives) - so I play it very safe:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span><span style="color:#75715e">// For pow we use exp2(y * log2(x)) to get the best bounds we can hope for.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">template</span> <span style="color:#f92672">&lt;&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span> FPScalarEvolutionPass<span style="color:#f92672">::</span>visitIntrinsic<span style="color:#f92672">&lt;</span>Intrinsic<span style="color:#f92672">::</span>pow<span style="color:#f92672">&gt;</span>(
</span></span><span style="display:flex;"><span>    IntrinsicInst <span style="color:#f92672">&amp;</span>inst) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FastMathFlags fmf <span style="color:#f92672">=</span> inst.getFastMathFlags();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV xFpscev <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>      fpse.getFPSCEV(inst.getOperand(<span style="color:#ae81ff">0</span>))<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> FPSCEV yFpscev <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>      fpse.getFPSCEV(inst.getOperand(<span style="color:#ae81ff">1</span>))<span style="color:#f92672">-&gt;</span>cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  FPSCEV log2Fpscev <span style="color:#f92672">=</span> log2Bounds(xFpscev);
</span></span><span style="display:flex;"><span>  FPSCEV fpscevs[<span style="color:#ae81ff">4</span>] <span style="color:#f92672">=</span> {log2Fpscev, log2Fpscev, log2Fpscev, log2Fpscev};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">0</span>].min.multiply(yFpscev.min, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">0</span>].max.multiply(yFpscev.min, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">1</span>].min.multiply(yFpscev.max, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">1</span>].max.multiply(yFpscev.max, APFloat<span style="color:#f92672">::</span>rmTowardNegative);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">2</span>].min.multiply(yFpscev.min, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">2</span>].max.multiply(yFpscev.min, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">3</span>].min.multiply(yFpscev.max, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">3</span>].max.multiply(yFpscev.max, APFloat<span style="color:#f92672">::</span>rmTowardPositive);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">0</span>] <span style="color:#f92672">=</span> exp2Bounds(fpscevs[<span style="color:#ae81ff">0</span>]);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">1</span>] <span style="color:#f92672">=</span> exp2Bounds(fpscevs[<span style="color:#ae81ff">1</span>]);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">2</span>] <span style="color:#f92672">=</span> exp2Bounds(fpscevs[<span style="color:#ae81ff">2</span>]);
</span></span><span style="display:flex;"><span>  fpscevs[<span style="color:#ae81ff">3</span>] <span style="color:#f92672">=</span> exp2Bounds(fpscevs[<span style="color:#ae81ff">3</span>]);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  FPSCEV result;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  result.min <span style="color:#f92672">=</span> getMinimum({fpscevs[<span style="color:#ae81ff">0</span>].min, fpscevs[<span style="color:#ae81ff">0</span>].max, fpscevs[<span style="color:#ae81ff">1</span>].min,
</span></span><span style="display:flex;"><span>                           fpscevs[<span style="color:#ae81ff">1</span>].max, fpscevs[<span style="color:#ae81ff">2</span>].min, fpscevs[<span style="color:#ae81ff">2</span>].max,
</span></span><span style="display:flex;"><span>                           fpscevs[<span style="color:#ae81ff">3</span>].min, fpscevs[<span style="color:#ae81ff">3</span>].max});
</span></span><span style="display:flex;"><span>  result.max <span style="color:#f92672">=</span> getMaximum({fpscevs[<span style="color:#ae81ff">0</span>].min, fpscevs[<span style="color:#ae81ff">0</span>].max, fpscevs[<span style="color:#ae81ff">1</span>].min,
</span></span><span style="display:flex;"><span>                           fpscevs[<span style="color:#ae81ff">1</span>].max, fpscevs[<span style="color:#ae81ff">2</span>].min, fpscevs[<span style="color:#ae81ff">2</span>].max,
</span></span><span style="display:flex;"><span>                           fpscevs[<span style="color:#ae81ff">3</span>].min, fpscevs[<span style="color:#ae81ff">3</span>].max});
</span></span><span style="display:flex;"><span>  result.isInteger <span style="color:#f92672">=</span> false;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpse.map[<span style="color:#f92672">&amp;</span>inst] <span style="color:#f92672">=</span> result.cloneWithFastMathFlags(fmf);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The approach is basically:</p>
<ol>
<li>Get the log2 bounds of the input <code>x</code>.</li>
<li>Multiply this by the bounds for <code>y</code> resulting in 8 multiplications for the whole space of the pairings.</li>
<li>Feed these multiplications into the exp2 bounds.</li>
<li>Then work out the minimum/maximum of all the resulting values.</li>
</ol>
<p>This gets us a very imprecise result, as the following chart shows:</p>

<img src="/images/fpscev-range/pow.png"  class="center"  style="border-radius: 8px;"    />


<p>It&rsquo;s really very <em>terribly</em> imprecise - we are losing so much precision by the fact our exp and log are so very imprecise too. While this isn&rsquo;t great - it is <em>still</em> better than having pow always return -NaN..NaN - at least we have some semblance of a range!</p>
<h1 id="sqrt">Sqrt</h1>
<p>Sqrt is one of the few functions that the C stdlib version is required to have 0.5 ulp (EG. the same precision as basic mathematical operators like + or -). This means, as long as we carefully round the min/max ranges as we translate from the APFloat to double, we can use the C stdlib sqrt implementation for our range calculation - huzzah!</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-cpp" data-lang="cpp"><span style="display:flex;"><span>FPSCEV <span style="color:#a6e22e">sqrtBounds</span>(FPSCEV fpscev) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> fltSemantics <span style="color:#f92672">&amp;</span>semantics <span style="color:#f92672">=</span> fpscev.getSemantics();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the full range is negative, always returns a NaN.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (fpscev.isAllNegative()) {
</span></span><span style="display:flex;"><span>    APFloat nan <span style="color:#f92672">=</span> APFloat<span style="color:#f92672">::</span>getNaN(semantics, false);
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> FPSCEV(nan, nan, false);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">bool</span> losesInfo;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (fpscev.isAllNonNegative()) {
</span></span><span style="display:flex;"><span>    fpscev.min.convert(APFloat<span style="color:#f92672">::</span>IEEEdouble(), APFloat<span style="color:#f92672">::</span>rmTowardNegative,
</span></span><span style="display:flex;"><span>                       <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// Get the next number less than the current (unless we are zero).
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#66d9ef">if</span> (<span style="color:#f92672">!</span>fpscev.min.isZero()) {
</span></span><span style="display:flex;"><span>      fpscev.min.next(true);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>    fpscev.min <span style="color:#f92672">=</span> APFloat(std<span style="color:#f92672">::</span>sqrt(fpscev.min.convertToDouble()));
</span></span><span style="display:flex;"><span>    fpscev.min.convert(semantics, APFloat<span style="color:#f92672">::</span>rmTowardNegative, <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// If the whole fpscev is not negative, it means max must be positive.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">if</span> (<span style="color:#f92672">!</span>fpscev.isAllNegative()) {
</span></span><span style="display:flex;"><span>    fpscev.max.convert(APFloat<span style="color:#f92672">::</span>IEEEdouble(), APFloat<span style="color:#f92672">::</span>rmTowardPositive,
</span></span><span style="display:flex;"><span>                       <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>    fpscev.max.next(false);
</span></span><span style="display:flex;"><span>    fpscev.max <span style="color:#f92672">=</span> APFloat(std<span style="color:#f92672">::</span>sqrt(fpscev.max.convertToDouble()));
</span></span><span style="display:flex;"><span>    fpscev.max.convert(semantics, APFloat<span style="color:#f92672">::</span>rmTowardPositive, <span style="color:#f92672">&amp;</span>losesInfo);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fpscev.isInteger <span style="color:#f92672">=</span> false;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">return</span> fpscev;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The basic approach is that we:</p>
<ol>
<li>Check if the entire fpscev is not negative (meaning that min is definitely going to produce a non-NaN result).</li>
<li>If so - convert the minimum value to APFloat&rsquo;s double representation. This just allows us to convert to double using the <code>convertToDouble</code> method.</li>
<li>If the converted value is not 0, then we move it over to the next smallest floating-point value using the <code>next</code> method (akin to <code>nextafter</code> in C).</li>
<li>Convert to double, use std::sqrt, then convert back, rounding towards negative as we convert back to the original semantics (ensuring we grow the min range on ties).</li>
<li>Check if the fpscev is not all negative (meaning that max is definitely going to produce a non-NaN result).</li>
<li>Do the same approach as for min, but use <code>next</code> to move the value up, and convert with rounding towards positive as we convert back (to grow the range).</li>
</ol>
<p>This means we might get a <em>slightly</em> off result, but the result will be pretty strongly bound to what sqrt would produce.</p>
<p>For instance, for an input between 0..511, we produce an output range of:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>min: <span style="color:#ae81ff">0</span>
</span></span><span style="display:flex;"><span>max: <span style="color:#ae81ff">22.6053104</span>
</span></span><span style="display:flex;"><span>isInteger: <span style="color:#ae81ff">0</span>
</span></span></code></pre></div><p>Which is much better than the 0..NaN we have previously.</p>
<h1 id="conclusion">Conclusion</h1>
<p>With this improvement I&rsquo;ve managed to really tighten up some of the functions I couldn&rsquo;t get a good handle on previously.</p>
<p>In the next post I&rsquo;m going to look at if it is possible to make our fpscev&rsquo;s aware of where they are used, and see if we can&rsquo;t constrain the range even further. Watch this space!</p>


guid: https://www.neilhenning.dev/utest_h/
title: utest.h
url: https://www.neilhenning.dev/utest_h/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Anyone who has used the <a href="https://github.com/google/googletest">googletest</a>
library with a complex C++ codebase will know how powerful this form of unit
testing can be for productivity. While the library has a ton of features and is
seriously powerful, there are a number of technical deficits that come with it:</p>
<ul>
<li>It only works with modern C++.</li>
<li>It has a complex build system integration, with multiple source files and
headers.</li>
<li><a href="https://www.neilhenning.dev/posts/utest-h-performance/">The startup time of using googletest is terrible</a>.</li>
</ul>
<p>So with all this in mind I wrote my own library -
<a href="https://github.com/sheredom/utest.h">utest.h</a>. The library is:</p>
<ul>
<li>9x faster than googletest.</li>
<li>Works with C and C++.</li>
<li>Works with mixing C <em>and</em> C++ in the same testing application (very useful if
you want to test a C++ and C API to your code).</li>
<li>Has test fixtures.</li>
<li>Allows filtering of test cases.</li>
<li>Compiles with <code>-Wall -Werror -Weverything</code>.</li>
</ul>
<p>Lets look at an example application:</p>
<p>In a single file (C or C++, it doesn&rsquo;t matter) you need to need to call the
macro <code>UTEST_MAIN()</code> to create the main method for the application.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;utest.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UTEST_MAIN</span>()
</span></span></code></pre></div><p>Note this doesn&rsquo;t <em>need</em> to be in a separate C/C++ file to the actual tests!</p>
<p>Then we&rsquo;ll have two files - a C and a C++ file just to show that it works.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;utest.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UTEST</span>(from, c) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">ASSERT_TRUE</span>(<span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;utest.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UTEST</span>(from, cpp) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">ASSERT_EQ</span>(true, false);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Now we&rsquo;ll compile this all together with clang and show the output.</p>

  <figure class="center" >
    <img src="/images/utest_h.gif"  alt="Terminal example of using utest.h"   style="border-radius: 8px;"  />
    
      <figcaption class="center"  style="color: white;" >Terminal example of using utest.h</figcaption>
    
  </figure>


<p>Anyone who is familiar with googletest will notice that I&rsquo;ve intentionally kept
the output similar (even colour highlighted) to stop any mental gymnastics that
could be involved.</p>
<p>Lastly the license - the code is as close to public domain as I could get in
a license, using the <a href="https://unlicense.org">unlicense</a> to let any of you do
basically anything you want with it.</p>


guid: https://www.neilhenning.dev/posts/library-updates/
title: Some Miscellaneous Library Improvements
url: https://www.neilhenning.dev/posts/library-updates/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;ve been working on my various C/C++ single header libraries some over the
Christmas break, and I thought I&rsquo;d just jot down the improvements I&rsquo;ve made in a
post.</p>
<h2 id="processh---subprocessh">process.h -&gt; subprocess.h</h2>
<p>I&rsquo;ve had a long standing complaint that the name <code>process.h</code> conflicted with a
Windows system header. I&rsquo;ve decided (after a good three years of indecision mind
you) to just
<a href="https://github.com/sheredom/subprocess.h/pull/13">rename the header and the entry points</a>
to <code>subprocess.h</code> instead.</p>
<p>This isn&rsquo;t a perfect solution, but it avoids me having to namespace my headers
using my own name (which as a Scotsman I just can&rsquo;t quite stomach for some
reason).</p>
<h2 id="utesth">utest.h</h2>
<p>I added the ability to assert/expect on pointers types, which was a longstanding
annoyance. So the following now works:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">int</span><span style="color:#f92672">*</span> a, <span style="color:#f92672">*</span>b;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(a, b);
</span></span></code></pre></div><p>I&rsquo;ve also made it so that on GCC/Clang with C, or with C++11 using auto, you can
perform checks on void-pointers, and things of array type:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">static</span> <span style="color:#66d9ef">int</span> data[<span style="color:#ae81ff">42</span>];
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">void</span><span style="color:#f92672">*</span> a;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(a, data);
</span></span></code></pre></div><p>I also made the testing of the <code>--list-tests</code> option that was contributed by
<a href="https://github.com/ldrumm">ldrumm</a> (which required spawning a
separate process) work on Windows and macOS too by using my
<a href="https://github.com/sheredom/subprocess.h">subprocess.h</a>.</p>
<p>I also added a product page for <a href="https://www.neilhenning.dev/utest_h/">utest.h</a> to
help me try and advertise it more greatly for people than before.</p>
<h2 id="utf8h">utf8.h</h2>
<p>I fixed <a href="https://github.com/sheredom/utf8.h/issues/60">a bug</a> with <code>utf8casecmp</code>
in my UTF-8 parsing library and added some extra tests to stop the regression
happening again.</p>
<p>The &lsquo;bug&rsquo; wasn&rsquo;t <em>really a bug</em> in that the function behaved in a similar
fashion to the specification of the GNU extension <code>strcasecmp</code> that it is based
on, but it made sense to make <code>utf8casecmp</code> match what <code>strcasecmp</code> does.</p>


guid: https://www.neilhenning.dev/posts/json-h-updates/
title: C89 & Single Header json.h
url: https://www.neilhenning.dev/posts/json-h-updates/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;ve been working on my various C/C++ single header libraries some over the
Christmas break, and while I already did an
<a href="https://www.neilhenning.dev/posts/library-updates/">update on my work</a>, I decided
to leave <a href="https://github.com/sheredom/json.h">json.h</a> for its own post.</p>
<h2 id="c89-support">C89 Support</h2>
<p>One requested thing over the years has been C89 support, and I was reluctant to
do this mostly out of sheer laziness. I&rsquo;d have to convert a bunch of comments
over and remove my use of long long. This was my <em>entire</em> reasons for not doing
it, which was dumb in hindsight. So I&rsquo;ve
<a href="https://github.com/sheredom/json.h/commit/14a36a8424273556f88e799a7a7f534d09089c59">got it working now</a> for the people that really really care.</p>
<h2 id="single-header-support">Single Header Support</h2>
<p>I always maintained that having one header and one source file was the best
design decision for the library. It is a big implementation in the source file,
and I mostly have a distaste for having a header file where you have to do some
preprocessor define in a single source file.</p>
<p>But single headers are so super useful for users to just copy the one file and
ship it.</p>
<p>So <a href="https://github.com/sheredom/json.h/commit/22dead46037551c9a29a1a6229ea2f38dc01e1fb">I&rsquo;ve done made the change to bring it into a separate header</a>.</p>
<p>This will definitely result in a slower compile time when bringing in the
header, but I realised that in reality you will probably only include json.h in
a single file when you are interacting with the JSON. So it seemed fine overall
for this as a compromise.</p>


guid: https://www.neilhenning.dev/posts/json_h_readme_md/
title: Extracting the json.h README.md Code Samples For Compiling
url: https://www.neilhenning.dev/posts/json_h_readme_md/
feedurl: https://www.duskborn.com/index.xml
content:
<p>One thing that has always worried me with writing C samples in documentation for
my single-header libraries is that you can&rsquo;t be 100% sure that they will compile
successfully. You can always extract them and run them, but you might change
them and forget to re-test. Having this be automatic is so powerful. Rust has
<a href="https://doc.rust-lang.org/rustdoc/documentation-tests.html">this feature</a> built
in to the ecosystem so that all code samples are testing automagically.</p>
<p>So I wondered - is there anyway to do something similar for my single-header
libraries? I decided to try with <a href="https://github.com/sheredom/json.h">json.h</a>.</p>
<h2 id="the-code-samples">The Code Samples</h2>
<p>I want to have more and more code samples to show how easy the library is to
use, but the big one there currently is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span> <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> json[] <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;{</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">a</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : true, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">b</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : [false, null, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">foo</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">]}&#34;</span>;
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> root <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_parse</span>(json, <span style="color:#a6e22e">strlen</span>(json));
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(root<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_object);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_object_s<span style="color:#f92672">*</span> object <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_object_s<span style="color:#f92672">*</span>)root<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(object<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">2</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> a <span style="color:#f92672">=</span> object<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> a_name <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(a_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(a_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> a_value <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(a_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_true);
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(a_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> b <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> b_name <span style="color:#f92672">=</span> b<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(b_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_value <span style="color:#f92672">=</span> b<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_array);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_array_s<span style="color:#f92672">*</span> array <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_array_s<span style="color:#f92672">*</span>)b_value<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(array<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_1st <span style="color:#f92672">=</span> array<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_1st_value <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_1st_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_false);
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_1st_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_2nd <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_2nd_value <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_2nd_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_null);
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_2nd_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_3rd <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_3rd<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_3rd_value <span style="color:#f92672">=</span> b_3rd<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(b_3rd_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_string);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> string <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span>)b_3rd_value<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(string<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">assert</span>(string<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span> <span style="color:#75715e">/* Don&#39;t forget to free the one allocation! */</span>
</span></span><span style="display:flex;"><span> <span style="color:#a6e22e">free</span>(root);
</span></span></code></pre></div><p>I want to be able to parse the
<a href="https://github.com/sheredom/json.h/blob/master/README.md">README.md</a>, extract
the code samples, turn them into a test for use with my utest.h library will
run. I already use CMake for building just the unit-tests, and so given I
already use it and I&rsquo;m pretty familiar with it (despite its glaring flaws), I
wondered if I could use it to do the extraction.</p>
<h2 id="cmake-of-horrors">CMake of Horrors</h2>
<p>So CMake has
<a href="https://cmake.org/cmake/help/latest/command/string.html#regular-expressions">built-in regex string support</a>
so I thought could I use that to do the extraction? The one big issue is that
CMake only supports greedy matching of regex - meaning that I have to be super
careful when searching for start/end tokens with which to match.</p>
<p>First of all we need to read the whole file into a CMake variable:</p>
<pre tabindex="0"><code>file(READ ${CMAKE_CURRENT_SOURCE_DIR}/../README.md readme_md)
</code></pre><p>CMake has this wonderfully messed up method for differentiating between strings
and lists - where a list in CMake terminology is just a string that has
semi-colons within it. The problem is that code samples in languages like C use
semi-colons as end of statement terminators - which will cause us issues. The
best way I&rsquo;ve found around this is to change the semi-colons to some symbol that
wouldn&rsquo;t appear in the original source. I used the &lsquo;@&rsquo; symbol for this since
there isn&rsquo;t an operator in C for it:</p>
<pre tabindex="0"><code>string(REPLACE &#34;;&#34; &#34;@&#34; readme_md &#34;${readme_md}&#34;)
</code></pre><p>Ok now we have the string as a real string (non-list) we can extract the code
samples themselves. You&rsquo;ll notice that in the README.md all code samples begin
with &ldquo;```c&rdquo; and end with &ldquo;```&rdquo;. So we can use this to look for our code.</p>
<p>As I said earlier CMake is <em>greedy</em> when it comes to regex, which means if we
used the more natural &ldquo;.*&rdquo; we&rsquo;d match from the very first code sample to the
very last in the file. Not ideal. Instead we need to use the more constrained
search of &ldquo;```c[^`]*```&rdquo; - search for the start pattern, and then all
symbols except a &ldquo;`&rdquo; until we get to our end. This stores each match as a list
entry into the variable <code>snippets</code> - meaning we have introdued some semi-colons
into the string too:</p>
<pre tabindex="0"><code>string(REGEX MATCHALL &#34;```c[^`]*```&#34; snippets &#34;${readme_md}&#34;)
</code></pre><p>Now to help us be able to test the examples we want to be able to compile each
of the code snippets in isolation from each other. I first attempted to create
a <code>UTEST(foo, bar)</code> wrapper around each snippet, but I could not figure out how
to create these wrappers such that they would be unique. What I mean is that the
first snippet would be <code>UTEST(generated, snippet0)</code> the next
<code>UTEST(generated, snippet1)</code>, etc.  For the life of me I couldn&rsquo;t work out how
this was possible. So instead I just wrapped each snippet into its own braced
region which guaranteed their isolation.</p>
<pre tabindex="0"><code>string(REPLACE &#34;```c&#34; &#34;{&#34; snippets ${snippets})
string(REPLACE &#34;```&#34; &#34;}\n\n&#34; snippets ${snippets})
</code></pre><p>Now all we need to do is remove the semi-colons that were added for the lists,
and then turn all &ldquo;@&rdquo; symbols we introduced before back into semi-colons:</p>
<pre tabindex="0"><code>string(REPLACE &#34;;&#34; &#34;&#34; snippets &#34;${snippets}&#34;)
string(REPLACE &#34;@&#34; &#34;;&#34; snippets &#34;${snippets}&#34;)
</code></pre><p>And then we just need to write out the file into some location for inclusion:</p>
<pre tabindex="0"><code>file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/generated.h &#34;${snippets}&#34;)
</code></pre><h2 id="sample-generatedh">Sample generated.h</h2>
<p>For the current master json.h, the <code>generated.h</code> file is:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s <span style="color:#f92672">*</span><span style="color:#a6e22e">json_parse</span>(
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span>src,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">size_t</span> src_size);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span>payload;
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">size_t</span> type;
</span></span><span style="display:flex;"><span>};
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s <span style="color:#f92672">*</span><span style="color:#a6e22e">json_parse_ex</span>(
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span>src,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">size_t</span> src_size,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">size_t</span> flags_bitset,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">void</span><span style="color:#f92672">*</span>(<span style="color:#f92672">*</span>alloc_func_ptr)(<span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span>, <span style="color:#66d9ef">size_t</span>),
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span>user_data,
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">struct</span> json_parse_result_s <span style="color:#f92672">*</span>result);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">enum</span> json_parse_flags_e {
</span></span><span style="display:flex;"><span>  json_parse_flags_default <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_trailing_comma <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x1</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_unquoted_keys <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x2</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_global_object <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x4</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_equals_in_object <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x8</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_no_commas <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x10</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_c_style_comments <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x20</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_deprecated <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x40</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_location_information <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x80</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_single_quoted_strings <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x100</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_hexadecimal_numbers <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x200</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_leading_plus_sign <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x400</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_leading_or_trailing_decimal_point <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x800</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_inf_and_nan <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x1000</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_multi_line_strings <span style="color:#f92672">=</span> <span style="color:#ae81ff">0x2000</span>,
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_simplified_json <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>      (json_parse_flags_allow_trailing_comma <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_unquoted_keys <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_global_object <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_equals_in_object <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_no_commas),
</span></span><span style="display:flex;"><span>  json_parse_flags_allow_json5 <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>      (json_parse_flags_allow_trailing_comma <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_unquoted_keys <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_c_style_comments <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_single_quoted_strings <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_hexadecimal_numbers <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_leading_plus_sign <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_leading_or_trailing_decimal_point <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_inf_and_nan <span style="color:#f92672">|</span>
</span></span><span style="display:flex;"><span>       json_parse_flags_allow_multi_line_strings)
</span></span><span style="display:flex;"><span>};
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> json[] <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;{</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">a</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : true, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">b</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : [false, null, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">foo</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">]}&#34;</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> root <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_parse</span>(json, <span style="color:#a6e22e">strlen</span>(json));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(root<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_object);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_s<span style="color:#f92672">*</span> object <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_object_s<span style="color:#f92672">*</span>)root<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(object<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">2</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> a <span style="color:#f92672">=</span> object<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> a_name <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(a_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(a_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> a_value <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(a_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_true);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(a_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> b <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> b_name <span style="color:#f92672">=</span> b<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(b_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_value <span style="color:#f92672">=</span> b<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_array);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_s<span style="color:#f92672">*</span> array <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_array_s<span style="color:#f92672">*</span>)b_value<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(array<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_1st <span style="color:#f92672">=</span> array<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_1st_value <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_1st_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_false);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_1st_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_2nd <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_2nd_value <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_2nd_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_null);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_2nd_value<span style="color:#f92672">-&gt;</span>payload <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_3rd <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_3rd<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_3rd_value <span style="color:#f92672">=</span> b_3rd<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_3rd_value<span style="color:#f92672">-&gt;</span>type <span style="color:#f92672">==</span> json_type_string);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> string <span style="color:#f92672">=</span> (<span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span>)b_3rd_value<span style="color:#f92672">-&gt;</span>payload;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(string<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(string<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">/* Don&#39;t forget to free the one allocation! */</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">free</span>(root);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> json[] <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;{</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">a</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : true, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">b</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74"> : [false, null, </span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">foo</span><span style="color:#ae81ff">\&#34;</span><span style="color:#e6db74">]}&#34;</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> root <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_parse</span>(json, <span style="color:#a6e22e">strlen</span>(json));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_s<span style="color:#f92672">*</span> object <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_value_as_object</span>(root);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(object <span style="color:#f92672">!=</span> NULL);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(object<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">2</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> a <span style="color:#f92672">=</span> object<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> a_name <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(a_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(a_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;a&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> a_value <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#a6e22e">json_value_is_true</span>(a_value));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_object_element_s<span style="color:#f92672">*</span> b <span style="color:#f92672">=</span> a<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> b_name <span style="color:#f92672">=</span> b<span style="color:#f92672">-&gt;</span>name;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(b_name<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_name<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;b&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_s<span style="color:#f92672">*</span> array <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_value_as_array</span>(b<span style="color:#f92672">-&gt;</span>value);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(array<span style="color:#f92672">-&gt;</span>length <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_1st <span style="color:#f92672">=</span> array<span style="color:#f92672">-&gt;</span>start;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_1st_value <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#a6e22e">json_value_is_false</span>(b_1st_value));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_2nd <span style="color:#f92672">=</span> b_1st<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_value_s<span style="color:#f92672">*</span> b_2nd_value <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>value;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#a6e22e">json_value_is_null</span>(b_2nd_value));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_array_element_s<span style="color:#f92672">*</span> b_3rd <span style="color:#f92672">=</span> b_2nd<span style="color:#f92672">-&gt;</span>next;
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(b_3rd<span style="color:#f92672">-&gt;</span>next <span style="color:#f92672">==</span> NULL);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> json_string_s<span style="color:#f92672">*</span> string <span style="color:#f92672">=</span> <span style="color:#a6e22e">json_value_as_string</span>(b_3rd<span style="color:#f92672">-&gt;</span>value);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(string <span style="color:#f92672">!=</span> NULL);
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(<span style="color:#ae81ff">0</span> <span style="color:#f92672">==</span> <span style="color:#a6e22e">strcmp</span>(string<span style="color:#f92672">-&gt;</span>string, <span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">assert</span>(string<span style="color:#f92672">-&gt;</span>string_size <span style="color:#f92672">==</span> <span style="color:#a6e22e">strlen</span>(<span style="color:#e6db74">&#34;foo&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">/* Don&#39;t forget to free the one allocation! */</span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">free</span>(root);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>I wanted to keep normal <code>assert.h</code> assert&rsquo;s in the sample source, but I also
want these to be turned into my utest.h <code>ASSERT_TRUE</code> macros, so I just use the
preprocessor to define these over, and include the source into the test:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define assert(x) ASSERT_TRUE(x)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UTEST</span>(generated, readme) {
</span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;generated.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>}
</span></span></code></pre></div><p>And the output when I run?</p>
<pre tabindex="0"><code>[ RUN      ] generated.readme
[       OK ] generated.readme (9715ns)
</code></pre><p>A pass!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Ok - it is not as nice as what Rust has built-in, but it works! I can now modify
the README.md and be sure that the code compiles correctly. I even found a bug
in the sample in the process, so well worth the work. Just a shame I had to
invest in <a href="https://github.com/sheredom/json.h/pull/68">proper demonology</a> to
support this within the C eco-system.</p>


guid: https://www.neilhenning.dev/posts/lessons_of_a_remote_worker/
title: Lessons of a Remote Worker
url: https://www.neilhenning.dev/posts/lessons_of_a_remote_worker/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Jumping into cold water is always a shocking experience. Being able to calm yourself, acclimatise to the temperature of the water - before you attempt to swim is how you survive.</p>
<p>A ton of people worldwide have just plunged into the icy water of working from home, and since I&rsquo;ve been doing this for six years I have some advice from the good and the bad of what I learned over the time.</p>
<p>In this post I&rsquo;ll attempt to cover some of these lessons to try and help y&rsquo;all who&rsquo;ve had the cold water treatment learn how to swim.</p>
<h2 id="dont-work-in-your-pyjamas">Don&rsquo;t Work In Your Pyjamas</h2>
<p>The number one mistake people make when working from home is to pour yourself out of bed and immediately sit at your new desk and begin work. <strong>Don&rsquo;t do this!</strong></p>
<p>I personally like to get a sweat on every day either out running with my dog or if I&rsquo;m housebound then on my exercise bike. So my routine is that I get up, change into my workout gear, take the dog outside for his morning duties, and make a coffee. This routine takes me around twenty minutes or so and gives me some distance from sleeping Neil to working Neil.</p>
<p>If you aren&rsquo;t an exercise-aholic like myself then I recommend you shower and get changed into &lsquo;work gear&rsquo; before sitting at your new desk.</p>
<h2 id="take-lots-of-regular-breaks">Take Lots of Regular Breaks</h2>
<p>The number one mistake I made when I first started working remotely was to not take enough breaks. If you think back to any office environment you have worked in, I&rsquo;d take a guess that around twenty-five minutes out of every hour has some non-screen interaction. People coming and going as they get a drink / use the toilet / talk to random people about random things. Even if you are one of those introverted headphone wearing office dwellers (like I was) you are still generally getting a ton of visual distractions as people move around you that give you a break from the screen.</p>
<p>When people switch from this to working remotely there is two things that bite - a fear of missing out on interactions that have now moved to chat apps or calls, and a worry that people are convinced you aren&rsquo;t actually working if you don&rsquo;t respond instantly to their direct interactions with you. These two things combined can let you almost become magnetized to your chair.</p>
<p><strong>Don&rsquo;t do this!</strong> Think about the office environment - how many times have you looked for a colleague at their desk and they aren&rsquo;t there. You don&rsquo;t immediately think <em>&lsquo;This bampot is skiving!&rsquo;</em>. The same is true for remote working, so give yourself a break.</p>
<h2 id="trust-your-staff">Trust Your Staff</h2>
<p>I&rsquo;ve already seen a bunch of leads worried that they won&rsquo;t know what their staff are up to, worried they won&rsquo;t work on the right things, worried they will go off into the weeds when the grass just needs cut. I&rsquo;ve seen pitches from extrovert control freaks that all staff should be forced into a continuously running group call for the entire working day to simulate the working environment. What this all boils down to is that you don&rsquo;t trust the people working for you - which honestly <strong>is a much deeper problem than them being at home</strong>. This screams to me of a failure of hiring the wrong person or a failure of not training the person.</p>
<p>When I was at AMD I learned that having an asynchronous way for people to know what you are working on is key. The entire game engineering part of the org I was part of would keep weekly logs on confluence of what we were doing, which let people across the org and actually across the company see what we were doing. This is a great way to keep people updated on what you are doing and also letting you develop side interactions about your work.</p>
<p>When I joined Unity I brought this with me, and keep a running yearly log on all the things I&rsquo;ve been up to. It provides a great way for remote people to feel connected to what other people are doing and have side conversations on the work.</p>
<h2 id="stop-working">Stop Working</h2>
<p>The thing I still struggle with the most isn&rsquo;t a lack of working, it is working too much. When you were in the office and had to leave at five to have dinner with the family, that forced a physical break between you and your ability to work. When the computer is in the same dwelling as you the temptation to say <em>&lsquo;I&rsquo;ll just finish this one other thing that has been annoying me&rsquo;</em> is a fight. Especially when it is a bug you have caused. Just this week at Unity I found two bugs in some complication optimizations I&rsquo;ve been working on, and I found myself itching to keep working on them late into the night because I felt guilty that I was holding up a release of Burst.</p>
<p><strong>Don&rsquo;t do this!</strong> I need to learn this lesson fully myself, but since we all know crunching is bad, crunching in the home can&rsquo;t be any better.</p>
<h2 id="dont-conference-call-during-your-lunch-break">Don&rsquo;t Conference Call During Your Lunch Break</h2>
<p>One thing I definitely advocate is to take your lunch break away from your desk or computer. Get some fresh air, do something else round the house, socialize with your equally quarantined partner / cat / Rocket League friends.</p>
<p>One thing I think you should never never never do is have some form of <em>&lsquo;Let&rsquo;s all eat lunch together on a conference call to scratch the social itch.&rsquo;</em>. It is totally ok to have a social-only set of interactions with your co-workers, but that should be on the companies time. Think back to when you are in the office - how much time in a day do you spend during company time interacting with your co-workers about non-company things? Think of all the watercooler or coffee-making moments where you have a five minute stop and chat. You don&rsquo;t clock out to do this, so why should you sacrifice your contractually allowed lunch break to recreate the same just because you are remote?</p>
<p>Like I said before, I use my lunch break to get out of the house. I walk the dog, go for a run, exercise on my bike. Hell I&rsquo;ve even taken a bath in my lunch hour. Do <em>anything</em> you can to get away from the desk for an hour.</p>

  <figure class="center" >
    <img src="/images/BenjiAndMeRunning.jpg"  alt="Picture of Benji and I out running"   style="border-radius: 8px;"  />
    
      <figcaption class="center"  style="color: white;" >Out in the fresh air with Benji!</figcaption>
    
  </figure>


<h2 id="lean-into-your-new-super-power">Lean Into Your New Super Power</h2>
<p>I found working from home to be utterly liberating. I&rsquo;m a highly socially capable introvert - anyone who has met me will know I am life-and-soul of any party, but I naturally want to be away in the wilds with my dog away from people. I need a huge period of time to recharge from social interactions - a week away for work would need two months for me to recover.</p>
<p>For me the ability to go full focus when I started working from home was so empowering. I could go into a zen like zone that I&rsquo;d never been able to in the office. I honestly reckon that I was operating about 10% of my peak performance when I was in the office. My productivity and happiness at how I&rsquo;m getting to really think through the problems I get to work on has been super charged since I&rsquo;ve been remote. Remote working <em>is my super power</em> - it&rsquo;s how I get to be as good at what I do as I am.</p>
<h2 id="dont-let-real-life-invade-your-work">Don&rsquo;t Let Real Life Invade Your Work</h2>
<p>It&rsquo;s so easy for real life to invade your work when you are at home. This was a hard adjustment for me and my partner. <em>&lsquo;Oh you are at home so you can do X around the house!&rsquo;</em> - no, no you can&rsquo;t. It&rsquo;s very easy to see that because the person is at home they can do the dishes, clean the home, look after the kids, whatever. It&rsquo;s super key that you resist this urge otherwise you won&rsquo;t succeed at home. It is better to just have a blanket ban on doing <em>anything</em> around the house during work hours that you wouldn&rsquo;t have been able to do in the office, at least to start with. Have a seperation of work-you and home-you - even though they are sharing the same location!</p>
<h2 id="take-advantage-of-the-lack-of-commute">Take Advantage of the Lack of Commute</h2>
<p>If you are like the average office worker you&rsquo;ll have somewhere around an hour commute built into your day. Now you&rsquo;ve just got that time back to do what you want with. Read books, learn a language, play Witcher III some more, build Lego with the kids - whatever you always wanted to do but didn&rsquo;t have time? Now is the time to do it. So liberating to get that dead time back!</p>
<h2 id="summary">Summary</h2>
<p>This isn&rsquo;t an exhaustive list of all the things that&rsquo;ll make remote working work for you - just a meandering list of some of the things I&rsquo;ve learned along the way. For me working from home has been a game changer in my life. I&rsquo;m happier, healthier, I got a dog that I get to spend every minute of the day with, I bought a house in the stunning Isle of Skye with views to die for, and live in a close knit community. I&rsquo;ve built social interactions outwith my work colleagues into my life which let me broaden my view points.</p>
<p>If you are in an enforced Coronavirus-derived work from home then I encourage you to lean into it and see if its for you. I realise that some extroverted people will miss the chaos of the office, but if you&rsquo;ve been one of those introverts that has been forced to live in the extroverted world <em>this is the time</em> to see if working from home would allow you to be happier, healthier, and honestly a better worker. It worked for me at least!</p>


guid: https://www.neilhenning.dev/posts/fixed-up-rss/
title: Fixed Up RSS
url: https://www.neilhenning.dev/posts/fixed-up-rss/
feedurl: https://www.duskborn.com/index.xml
content:
<p>One thing I&rsquo;ve unfortunately neglected for a little while now on my blog was the RSS feed - mostly because I didn&rsquo;t have a clue how to make Hugo do it as I&rsquo;d like. There was a couple of issues:</p>
<ul>
<li>Only the summaries of my posts were fed via RSS - not the full contents. I <em>do not</em> do any sort of content tracking on my site because I don&rsquo;t want to foist vaporware like Google Analytics on my readers, so it really doesn&rsquo;t matter to me whether people read my content from the RSS feed or from the main site - that they read it is enough!</li>
<li>The links from the RSS feed were broken.</li>
<li>I didn&rsquo;t have an explicit link to signify that my site <em>even</em> supported RSS on the main page.</li>
</ul>
<p>Let&rsquo;s look at these in-order.</p>
<h2 id="full-contents-please">Full Contents Please!</h2>
<p>So I wanted full content rather than summaries of my posts on the RSS feed. Luckily Hugo provides a way to intercept and replace the default RSS template with your own custom one. I followed <a href="https://www.godo.dev/tutorials/hugo-full-text-rss/">this guide</a> to try and get it working - but in my RSS reader of choice while the full contents were now being included in the final RSS feed, the HTML tags were being displayed wrongly. I scratched my head for a long while over this before I realised that with RSS 2.0 there are two ways to encode HTML data within an RSS description - and my reader only liked the <code>CDATA</code> approach.</p>
<p>What I ended up with was the following:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#75715e">&lt;!--https://www.godo.dev/tutorials/hugo-full-text-rss/--&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">&lt;!--https://raw.githubusercontent.com/gohugoio/hugo/master/tpl/tplimpl/embedded/templates/_default/rss.xml--&gt;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>{{- $pctx := . -}}
</span></span><span style="display:flex;"><span>{{- if .IsHome -}}{{ $pctx = .Site }}{{- end -}}
</span></span><span style="display:flex;"><span>{{- $pages := slice -}}
</span></span><span style="display:flex;"><span>{{- if or $.IsHome $.IsSection -}}
</span></span><span style="display:flex;"><span>{{- $pages = $pctx.RegularPages -}}
</span></span><span style="display:flex;"><span>{{- else -}}
</span></span><span style="display:flex;"><span>{{- $pages = $pctx.Pages -}}
</span></span><span style="display:flex;"><span>{{- end -}}
</span></span><span style="display:flex;"><span>{{- $limit := .Site.Config.Services.RSS.Limit -}}
</span></span><span style="display:flex;"><span>{{- if ge $limit 1 -}}
</span></span><span style="display:flex;"><span>{{- $pages = $pages | first $limit -}}
</span></span><span style="display:flex;"><span>{{- end -}}
</span></span><span style="display:flex;"><span><span style="color:#f92672">&lt;rss</span> <span style="color:#a6e22e">version=</span><span style="color:#e6db74">&#34;2.0&#34;</span> <span style="color:#a6e22e">xmlns:atom=</span><span style="color:#e6db74">&#34;http://www.w3.org/2005/Atom&#34;</span><span style="color:#f92672">&gt;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&lt;channel&gt;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;title&gt;</span>{{ if eq  .Title  .Site.Title }}{{ .Site.Title }}{{ else }}{{ with .Title }}{{.}} on {{ end }}{{ .Site.Title }}{{ end }}<span style="color:#f92672">&lt;/title&gt;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;link&gt;</span>{{ .Permalink }}<span style="color:#f92672">&lt;/link&gt;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;description&gt;</span>Recent content {{ if ne  .Title  .Site.Title }}{{ with .Title }}in {{.}} {{ end }}{{ end }}on {{ .Site.Title }}<span style="color:#f92672">&lt;/description&gt;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;generator&gt;</span>Hugo -- gohugo.io<span style="color:#f92672">&lt;/generator&gt;</span>{{ with .Site.LanguageCode }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;language&gt;</span>{{.}}<span style="color:#f92672">&lt;/language&gt;</span>{{end}}{{ with .Site.Author.email }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;managingEditor&gt;</span>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}<span style="color:#f92672">&lt;/managingEditor&gt;</span>{{end}}{{ with .Site.Author.email }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;webMaster&gt;</span>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}<span style="color:#f92672">&lt;/webMaster&gt;</span>{{end}}{{ with .Site.Copyright }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;copyright&gt;</span>{{.}}<span style="color:#f92672">&lt;/copyright&gt;</span>{{end}}{{ if not .Date.IsZero }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;lastBuildDate&gt;</span>{{ .Date.Format &#34;Mon, 02 Jan 2006 15:04:05 -0700&#34; | safeHTML }}<span style="color:#f92672">&lt;/lastBuildDate&gt;</span>{{ end }}
</span></span><span style="display:flex;"><span>    {{ with .OutputFormats.Get &#34;RSS&#34; }}
</span></span><span style="display:flex;"><span>      {{ printf &#34;<span style="color:#f92672">&lt;atom:link</span> <span style="color:#a6e22e">href=</span><span style="color:#e6db74">%q</span> <span style="color:#a6e22e">rel=</span><span style="color:#e6db74">\&#34;self\&#34;</span> <span style="color:#a6e22e">type=</span><span style="color:#e6db74">%q</span> <span style="color:#f92672">/&gt;</span>&#34; .Permalink .MediaType | safeHTML }}
</span></span><span style="display:flex;"><span>    {{ end }}
</span></span><span style="display:flex;"><span>    {{ range $pages }}
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;item&gt;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&lt;title&gt;</span>{{ .Title }}<span style="color:#f92672">&lt;/title&gt;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&lt;link&gt;</span>{{ .Permalink }}<span style="color:#f92672">&lt;/link&gt;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&lt;pubDate&gt;</span>{{ .Date.Format &#34;Mon, 02 Jan 2006 15:04:05 -0700&#34; | safeHTML }}<span style="color:#f92672">&lt;/pubDate&gt;</span>
</span></span><span style="display:flex;"><span>      {{ with .Site.Author.email }}<span style="color:#f92672">&lt;author&gt;</span>{{.}}{{ with $.Site.Author.name }} ({{.}}){{end}}<span style="color:#f92672">&lt;/author&gt;</span>{{end}}
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&lt;guid&gt;</span>{{ .Permalink }}<span style="color:#f92672">&lt;/guid&gt;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#f92672">&lt;description&gt;</span>{{ safeHTML &#34;<span style="color:#75715e">&lt;![CDATA[&#34; }}{{- .Content | safeHTML -}}{{ safeHTML &#34;]]&gt;</span>&#34; }}<span style="color:#f92672">&lt;/description&gt;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&lt;/item&gt;</span>
</span></span><span style="display:flex;"><span>    {{ end }}
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">&lt;/channel&gt;</span>
</span></span><span style="display:flex;"><span><span style="color:#f92672">&lt;/rss&gt;</span>
</span></span></code></pre></div><p>With the only thing I changed was the line:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-xml" data-lang="xml"><span style="display:flex;"><span><span style="color:#f92672">&lt;description&gt;</span>{{ safeHTML &#34;<span style="color:#75715e">&lt;![CDATA[&#34; }}{{- .Content | safeHTML -}}{{ safeHTML &#34;]]&gt;</span>&#34; }}<span style="color:#f92672">&lt;/description&gt;</span>
</span></span></code></pre></div><p>What this does is dump the original HTML content as is (that&rsquo;s what the <code>safeHTML</code> command does) and wrap it in the <code>&lt;![CDATA[</code> and <code>]]&gt;</code> tags. This meant that my content was being displayed correctly.</p>
<p>I then just had to place the above in <code>&lt;root&gt;/layouts/_default/rss.xml</code> et voila, it worked!</p>
<h2 id="links-should-really-work">Links Should Really Work</h2>
<p>All my links in the RSS were broken - the links to my main site and also the links to image content on my main blog. I don&rsquo;t <em>quite</em> know how I forgot to do this (or how my site actually worked for as long as it did), but I had forgotten to specify the <code>baseurl</code> in the <code>config.toml</code>. Just pointing this at my site fixed the issue.</p>
<h2 id="advertising-rss-support">Advertising RSS Support</h2>
<p>Although my site always (poorly) supported RSS, I never advertised it. To solve this I&rsquo;ve just added an RSS badge to the footer of the page to help people find the feed that want it. Nothing fancy, just an image and a link.</p>
<h2 id="wrapping-it-all-up">Wrapping It All Up</h2>
<p>Now I have fully working RSS - and thanks to a very helpful push from an avid reader (thanks Augustin!) I&rsquo;ve also enabled SSL/https on my site - and made it the primary way to view my content. <a href="https://www.gandi.net">My hosting provider Gandi</a> has offered simple SSL integration for a while now, but I had to upgrade my small instance to small+SSL, and I just hadn&rsquo;t gotten round to it. But no more - I now serve secure content by default!</p>
<p>Sorry for any RSS fans that&rsquo;ve been maligned for a while now, but hopefully this post will get my RSS subscriptions to rise now that my content is displayed in all its glory.</p>
<p>You can find the RSS feed at this link <a href="https://www.neilhenning.dev/index.xml">https://www.neilhenning.dev/index.xml</a>.</p>
<p>Tioraidh a chàirdean!</p>


guid: https://www.neilhenning.dev/posts/rust-lit/
title: How to Test Rust Command Line Applications
url: https://www.neilhenning.dev/posts/rust-lit/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Testing command line applications in Rust is something that I found was a bit of a grey area. The main documentation for Rust has a section <a href="https://rust-cli.github.io/book/tutorial/testing.html#testing-cli-applications-by-running-them">Testing CLI applications by running them</a> which explains how they suggest you test command lines applications:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-rust" data-lang="rust"><span style="display:flex;"><span><span style="color:#66d9ef">use</span> std::process::Command;  <span style="color:#75715e">// Run programs
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">use</span> assert_cmd::prelude::<span style="color:#f92672">*</span>; <span style="color:#75715e">// Add methods on commands
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">use</span> predicates::prelude::<span style="color:#f92672">*</span>; <span style="color:#75715e">// Used for writing assertions
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[test]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">file_doesnt_exist</span>() -&gt; Result<span style="color:#f92672">&lt;</span>(), Box<span style="color:#f92672">&lt;</span>std::error::Error<span style="color:#f92672">&gt;&gt;</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">let</span> <span style="color:#66d9ef">mut</span> cmd <span style="color:#f92672">=</span> Command::cargo_bin(<span style="color:#e6db74">&#34;grrs&#34;</span>)<span style="color:#f92672">?</span>;
</span></span><span style="display:flex;"><span>    cmd.arg(<span style="color:#e6db74">&#34;foobar&#34;</span>)
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#e6db74">&#34;test/file/doesnt/exist&#34;</span>);
</span></span><span style="display:flex;"><span>    cmd.assert()
</span></span><span style="display:flex;"><span>        .failure()
</span></span><span style="display:flex;"><span>        .stderr(predicate::<span style="color:#66d9ef">str</span>::contains(<span style="color:#e6db74">&#34;No such file or directory&#34;</span>));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Ok(())
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Basically you use a <code>#[test]</code> inside an integration test (in the <code>tests</code> folder) which uses a <code>std::process</code> to spawn the command line tool you&rsquo;ve developed. This works, but it has two failings in my opinion:</p>
<ol>
<li>If you want to test the same command line tool with multiple inputs (maybe you want to test that it fails in a specific way?) you&rsquo;re probably going to add a bunch of tests and include all the boiler plate code to call them.</li>
<li>There isn&rsquo;t a really clean way to check that complex output from your application appears as expected.</li>
</ol>
<p>I&rsquo;m coming from the background of writing compilers, where the state of the art for testing is <a href="https://llvm.org/docs/CommandGuide/lit.html">LLVM&rsquo;s Lit</a> - a tester that combined with the LLVM tool <code>FileCheck</code> lets you do awesome things like:</p>
<pre tabindex="0"><code>; RUN: opt &lt; %s -constprop -S | FileCheck %s

; CHECK-LABEL: shift_undef_64
define void @shift_undef_64(i64* %p) {
  %r1 = lshr i64 -1, 4294967296 ; 2^32
  ; CHECK: store i64 undef
  store i64 %r1, i64* %p

  %r2 = ashr i64 -1, 4294967297 ; 2^32 + 1
  ; CHECK: store i64 undef
  store i64 %r2, i64* %p

  %r3 = shl i64 -1, 4294967298 ; 2^32 + 2
  ; CHECK: store i64 undef
  store i64 %r3, i64* %p

  ret void
}
</code></pre><p>Basically it is:</p>
<ul>
<li>Running a program of your choosing with the options that you want.</li>
<li>The current file is fed as input to the program.</li>
<li>Then the output is parsed through <code>FileCheck</code> which also takes the same input.</li>
<li>And it uses special <code>CHECK:</code> statements hidden in comments to assert that the output matches some of the stuff that we expect.</li>
</ul>
<p>This lets us perform some pretty impressive checks that the output matches what we want, and crucially lets us <strong>not</strong> check some things, which is pretty useful for the things we don&rsquo;t care about.</p>
<p>I&rsquo;m currently working on a compiler framework written in Rust for the craic. I&rsquo;m not yet at the point where I want to link y&rsquo;all to the bodgey code and it is in no way usable in any way, shape, or form yet. But one thing I wanted was to be able to unit test the compiler framework by using the command line tools similar to how LLVM did it above. This will let me generate a huge number of tests in a really easy to extend format - I know this because I&rsquo;ve used LLVM&rsquo;s tooling for years and can attest it is very powerful.</p>
<h2 id="attempt-1---use-llvms-lit">Attempt 1 - Use LLVM&rsquo;s Lit?</h2>
<p>So my first though was just to use LLVM&rsquo;s lit tool directly. This quickly fell apart as an idea for a number of reasons:</p>
<ul>
<li>The actual lit tooling is a bunch of python scripts, and I don&rsquo;t want to require anything else beyond Rust.</li>
<li>The tool that <em>actually</em> does the heavy lifting in lit is the <code>FileCheck</code> tool, which is an executable shipped with LLVM.</li>
<li>The <code>FileCheck</code> executable is only included in an LLVM you build yourself - you can&rsquo;t just get it with an LLVM install.</li>
</ul>
<p>So all in all this didn&rsquo;t seem like a good way to integrate this sort of testing with Rust. So I was about to try and roll my own when I noticed a really awesome crate <a href="https://crates.io/crates/lit">lit</a>, so I decided to investigate using it.</p>
<h2 id="the-lit-crate">The lit crate</h2>
<p>So first of all, we need to use the crate. And since we <em>only</em> require the crate for testing we do:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-toml" data-lang="toml"><span style="display:flex;"><span>[<span style="color:#a6e22e">dev-dependencies</span>]
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">lit</span> = <span style="color:#e6db74">&#34;0.3.4&#34;</span>
</span></span></code></pre></div><p>Next up we want to add an integration test that uses this. An integration test can depend on the entire main crate you are developing. In my case I&rsquo;ve got a library (<code>yair-io</code>) and two executables (<code>yair-as</code> and <code>yair-dis</code>) that use it, and I want to drive these from the integration test. The way you do it is like so:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-rust" data-lang="rust"><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> <span style="color:#66d9ef">crate</span> lit;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[cfg(test)]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">mod</span> tests {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">use</span> std::env;
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">use</span> std::env::consts;
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">use</span> std::path::PathBuf;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">bin_dir</span>() -&gt; <span style="color:#a6e22e">PathBuf</span> {
</span></span><span style="display:flex;"><span>        env::current_exe()
</span></span><span style="display:flex;"><span>            .ok()
</span></span><span style="display:flex;"><span>            .map(<span style="color:#f92672">|</span><span style="color:#66d9ef">mut</span> path<span style="color:#f92672">|</span> {
</span></span><span style="display:flex;"><span>                path.pop();
</span></span><span style="display:flex;"><span>                path.pop();
</span></span><span style="display:flex;"><span>                path
</span></span><span style="display:flex;"><span>            })
</span></span><span style="display:flex;"><span>            .unwrap()
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">yair_as_exe</span>() -&gt; String {
</span></span><span style="display:flex;"><span>        bin_dir()
</span></span><span style="display:flex;"><span>            .join(format!(<span style="color:#e6db74">&#34;yair-as</span><span style="color:#e6db74">{}</span><span style="color:#e6db74">&#34;</span>, env::consts::<span style="color:#66d9ef">EXE_SUFFIX</span>))
</span></span><span style="display:flex;"><span>            .to_str()
</span></span><span style="display:flex;"><span>            .unwrap()
</span></span><span style="display:flex;"><span>            .to_string()
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">yair_dis_exe</span>() -&gt; String {
</span></span><span style="display:flex;"><span>        bin_dir()
</span></span><span style="display:flex;"><span>            .join(format!(<span style="color:#e6db74">&#34;yair-dis</span><span style="color:#e6db74">{}</span><span style="color:#e6db74">&#34;</span>, env::consts::<span style="color:#66d9ef">EXE_SUFFIX</span>))
</span></span><span style="display:flex;"><span>            .to_str()
</span></span><span style="display:flex;"><span>            .unwrap()
</span></span><span style="display:flex;"><span>            .to_string()
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">#[test]</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">lit</span>() {
</span></span><span style="display:flex;"><span>        lit::run::tests(<span style="color:#f92672">|</span>config<span style="color:#f92672">|</span> {
</span></span><span style="display:flex;"><span>            config.add_search_path(<span style="color:#e6db74">&#34;tests/lit&#34;</span>);
</span></span><span style="display:flex;"><span>            config.add_extension(<span style="color:#e6db74">&#34;ya&#34;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>            config.constants.insert(<span style="color:#e6db74">&#34;yair_as&#34;</span>.to_owned(), yair_as_exe());
</span></span><span style="display:flex;"><span>            config.constants.insert(<span style="color:#e6db74">&#34;yair_dis&#34;</span>.to_owned(), yair_dis_exe());
</span></span><span style="display:flex;"><span>        })
</span></span><span style="display:flex;"><span>        .expect(<span style="color:#e6db74">&#34;Lit tests failed&#34;</span>);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>So lets break this down:</p>
<ul>
<li>We use the same <code>#[cfg(test)]</code> that we&rsquo;d expect from a Rust test.</li>
<li>And we have a single <code>#[test]</code> tagged function that&rsquo;ll run our lit tests.</li>
<li>The <code>lit()</code> function calls into the lit crate to run the tests (<code>lit::run::tests</code>).
<ul>
<li>We need to tell it where our test cases live by using <code>add_search_path</code> to point it at the folder containing your text files to run.</li>
<li>And also what file extensions we want to consider ok for testing by using <code>add_extension</code>. In my case my human-readable IR files are like <code>foo.ya</code> so the extension is just <code>ya</code>.</li>
<li>And then we need to create some variables that map to where the executables built by rust are located. I add two constants for <code>yair_as</code> and <code>yair_dis</code> to map to where the actual executables exist on disk. To get these locations I use a helper function <code>bin_dir</code> to work out where these executables will be placed on the file-system based on where the integration test is on the file-system.</li>
</ul>
</li>
</ul>
<p>Note you can add any other constants you want here too for use in the lit scripts. So if you wanted to get the OS being used in the scripts you could add <code>config.constants.insert(&quot;os&quot;.to_owned(), consts::OS.to_owned());</code> for instance.</p>
<p>Now we&rsquo;ve got this you can run <code>cargo test</code> and it&rsquo;ll have a new integration test that&rsquo;ll run your lit tests! But since you don&rsquo;t have a single actual test yet, nothing will actually run yet.</p>
<p>So let&rsquo;s look at an example of how to write a test case for lit that&rsquo;ll be picked up by the above integration test.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// RUN: @yair_as @file | @yair_dis -
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// CHECK: mod &#34;😀&#34; {
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>mod <span style="color:#e6db74">&#34;😀&#34;</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// CHECK: fn foo(a : u8, b : &lt;u8, 2&gt;) : &lt;u8, 2&gt; {
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  fn <span style="color:#a6e22e">foo</span>(a: u8, b: <span style="color:#f92672">&lt;</span>u8, <span style="color:#ae81ff">2</span><span style="color:#f92672">&gt;</span>) <span style="color:#f92672">:</span> <span style="color:#f92672">&lt;</span>u8, <span style="color:#ae81ff">2</span><span style="color:#f92672">&gt;</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// CHECK: b0(v2 : u8, v3 : &lt;u8, 2&gt;):
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#a6e22e">bar</span>(a: u8, b: <span style="color:#f92672">&lt;</span>u8, <span style="color:#ae81ff">2</span><span style="color:#f92672">&gt;</span>)<span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>      <span style="color:#75715e">// CHECK: [[name:\w+]] = insert v3, v2, 1
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>      c <span style="color:#f92672">=</span> insert b, a, <span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span>      <span style="color:#75715e">// CHECK: ret $$name
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>      ret c
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>So let&rsquo;s break down what I&rsquo;ve done piece by piece:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// RUN: @yair_as @file | @yair_dis -
</span></span></span></code></pre></div><p>The first line tells the lit crate how to run this test case by using the <code>RUN:</code> statement. To access constants in lit you use the <code>@foo</code> syntax, so in our case to access the <code>yair-as</code> tool that I setup previously to map to the constant <code>yair_as</code> I use <code>@yair_as</code> to run it. There is a special constant that represents the path to the current file - <code>@file</code>. My <code>yair-as</code> assembler tool takes a positional argument for the input file, so what we are doing here is assembling the current file using my tool, and spitting out the output to the default which is stdout. This produces the binary representation of my intermediate representation, which is kinda hard to test! So what I&rsquo;m doing is roundtripping it back out via the dissassembler <code>yair-dis</code> (mapped to the <code>@yair_dis</code> constant) and then we can check that the output of the disassembler, which is human readable, matches what the input was.</p>
<p>You&rsquo;ll note that there is a pipe operator between the two tools, passing the stdout from one into the stdin of the other. Also, the stdout from the total run is what is fed into the lit checks to follow. Also - we <strong>do not</strong> have a separate <code>FileCheck</code> tool like LLVM uses - which is awesome! It just works, which I really appreciate.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// CHECK: mod &#34;😀&#34; {
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>mod <span style="color:#e6db74">&#34;😀&#34;</span> {
</span></span></code></pre></div><p>Next up I have a module 😀 being declared, and I make sure that this is output correctly. You&rsquo;ll notice that the rest of the file follows a similar pattern - some statements in the original IR matched with some <code>CHECK:</code>&rsquo;s that ensure that the output was correct. You don&rsquo;t <em>have</em> to check every line, you could check no lines if you wanted to - but this sparse matching of the output to ensure that some parts of it match what you want is so powerful.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// CHECK: [[name:\w+]] = insert v3, v2, 1
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>c <span style="color:#f92672">=</span> insert b, a, <span style="color:#ae81ff">1</span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// CHECK: ret $$name
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>ret c
</span></span></code></pre></div><p>You can also use some more powerful features of checks - variables. Basically you can check that something you&rsquo;ve matched on earlier appears later in the file. So in the above example I am storing the name of the result from the <code>insert</code> instruction into the variable <code>name</code>, and then getting that name in the subsequent check using <code>$$name</code>. This means that if I changed the naming convention at a later date everything would keep working with the test case - nice!</p>
<p>Lastly - the output from <code>cargo test</code> of lit is as follows:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>running <span style="color:#ae81ff">1</span> test
</span></span><span style="display:flex;"><span>PASS :: tests/lit/var/basic.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/var/array.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/var/struct.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/var/vector.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/var/export.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/fn/one_arg.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/fn/bad_export.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/fn/one_arg_body.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/fn/export.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/fn/no_args.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/multiple_modules.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/bad_fn_or_var.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/simple.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/missing_closing_brace.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/bad_quoted.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/unicode.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/empty.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/missing_opening_brace.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/missing_module_name.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/mod/bad_simple.ya
</span></span><span style="display:flex;"><span>PASS :: tests/lit/insts/insert.ya
</span></span><span style="display:flex;"><span>test tests::lit ... ok
</span></span></code></pre></div><p>You can see that it appears like a single running test (because like I said previously we are adding a single integration test that powers the lit runs), but the output from that test is a bunch of runs telling you what passes and what fails. If I manually make a test case fail, the output is like so:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-bash" data-lang="bash"><span style="display:flex;"><span>FAIL :: tests/lit/insts/insert.ya
</span></span><span style="display:flex;"><span>tests/lit/insts/insert.ya:15: could not find match: <span style="color:#e6db74">&#39;something not there&#39;</span>
</span></span><span style="display:flex;"><span>next line: <span style="color:#e6db74">&#39;}&#39;</span>
</span></span></code></pre></div><p>Telling me the line with the <code>CHECK</code> that it could not find.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I really love this crate - it is super useful for testing these kind of applications. Full kudos to <a href="https://github.com/dylanmckay">Dylan McKay</a> for writing such an awesome crate, and I hope I&rsquo;ve convinced a few more people to try it out.</p>


guid: https://www.neilhenning.dev/posts/utest-h-supports-wpo/
title: utest.h Now Supports Whole Program Optimization
url: https://www.neilhenning.dev/posts/utest-h-supports-wpo/
feedurl: https://www.duskborn.com/index.xml
content:
<p>A user of utest.h posted an <a href="https://github.com/sheredom/utest.h/issues/31">intruiging issue</a> that on an release build with Visual Studio my unit testing framework for C/C++, <a href="https://github.com/sheredom/utest.h">utest.h</a>, no tests were being ran. But on a debug build they were. Before I even got to the issue they worked out that whole program optimization was causing the issue and closed it themselves.</p>
<p>But this got me thinking - there is no reason why my library shouldn&rsquo;t work with whole program optimization enabled, so why was this behaving badly?</p>
<h1 id="visual-studios-whole-program-optimization">Visual Studio&rsquo;s Whole Program Optimization</h1>
<p>To enable whole program optimization in Visual Studio requires the <code>/GL</code> option on the <code>cl.exe</code> compiler, and <code>/LTCG</code> on the <code>link.exe</code>. What this does is effectively defer some of the optimizations and optimization ability to link time. At link time it can then perform optimizations across multiple individual compilation units to (in theory!) result in a more optimal resulting executable.</p>
<p>In CMake you can enable this for an executable like so:</p>
<pre tabindex="0"><code>if(&#34;${CMAKE_CXX_COMPILER_ID}&#34; STREQUAL &#34;MSVC&#34;)
  target_compile_options(utest_test_wpo PRIVATE &#34;$&lt;$&lt;CONFIG:RELEASE&gt;:/GL&gt;&#34;)
  target_link_options(utest_test_wpo PRIVATE &#34;$&lt;$&lt;CONFIG:RELEASE&gt;:/LTCG&gt;&#34;)
endif()
</code></pre><p>What the above does is turn on the option <em>only</em> for release builds, for the given target. The first thing I did when trying to work out this issue was to duplicate the existing tests I had for utest, and build a separate target <code>utest_test_wpo</code>. I used the above CMake to turn on the whole program optimization, configured CMake to use release and got the following output:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] Running <span style="color:#ae81ff">0</span> test cases.
</span></span><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] <span style="color:#ae81ff">0</span> test cases ran.
</span></span><span style="display:flex;"><span>[ PASSED ] <span style="color:#ae81ff">0</span> tests.
</span></span></code></pre></div><p>So this matched what the user found at least!</p>
<h1 id="the-problem">The Problem</h1>
<p>So the issue boils down to a clever little trick I use in utest.h - how it discovers test. Now for those that aren&rsquo;t aware my utest.h library looks for tests in a specific format:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#a6e22e">UTEST</span>(from, c) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">ASSERT_TRUE</span>(<span style="color:#ae81ff">1</span>);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And allows you to run them. This works <em>across</em> C or C++ files - if you have <code>T</code> tests spread across <code>F</code> files, when you run the produced executable it&rsquo;ll run all <code>T</code> tests - even though they started in separate files. I don&rsquo;t use any custom magic to make this work - so how can I cross compilation units to work out what to run?</p>
<p>I use an idiom that is <em>awful</em> and <em>awesome</em> in equal measures - global constructors. For those used to C++ you&rsquo;ll be aware that global variables can have constructors to initialize their state. It is general wisdom among those of us burned by the feature to either a) not use global variable constructors at all, or b) if you do make them dumb as possible. This is because the order that all global variables will be initialized is undefined - and thus any interdependency between these variables could cause problems.</p>
<p>What is less well known is that the three major C compilers have ways to add global constructors in C code.</p>
<p>Clang/GCC both support the following:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">static</span> <span style="color:#66d9ef">void</span> <span style="color:#a6e22e">f</span>(<span style="color:#66d9ef">void</span>) <span style="color:#a6e22e">__attribute__</span>((constructor));
</span></span></code></pre></div><p>You just tag a function as being a <code>constructor</code> using the attribute mechanism, and it&rsquo;ll be run during the globals initialization phase.</p>
<p>MSVC (Visual Studio) is a little more complicated, but you can achieve the same thing with:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">static</span> <span style="color:#66d9ef">void</span> <span style="color:#66d9ef">__cdecl</span> <span style="color:#a6e22e">f</span>(<span style="color:#66d9ef">void</span>);
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">__declspec</span>(<span style="color:#a6e22e">allocate</span>(<span style="color:#e6db74">&#34;.CRT$XCU&#34;</span>)) <span style="color:#66d9ef">void</span>(<span style="color:#66d9ef">__cdecl</span> <span style="color:#f92672">*</span> f_)(<span style="color:#66d9ef">void</span>) <span style="color:#f92672">=</span> f;
</span></span></code></pre></div><p>What this does is add a function pointer to a special section <code>.CRT$XCU</code> - where all the global initializers are placed. On startup the compiler will run through this list of function pointers to initialize all the global state. It&rsquo;s aesthetically uglier than the Clang/GCC approach, but it works!</p>
<p>With this key piece of technology we can then use global constructors for each of the <code>UTEST</code> tests to register themselves with a global registry of &rsquo;tests that can be run&rsquo;, effectively just appending themselves to a malloc&rsquo;ed region. And this is <strong>fast</strong> - on my 2017 MacBook Pro it takes around 50ms to initialize 502 tests for running, around 99us per test. As an aside, I did an <a href="https://www.neilhenning.dev/posts/utest-h-performance/">expose of the performance of utest.h</a> and TL;DR it&rsquo;s blindingly fast.</p>
<p>Back to the problem - remember the problem? MSVC with whole program optimization was not running any tests? So for all intents and purposes these global constructors, the function pointer <code>f_</code> above, are not actually used in my code. What I mean is that I don&rsquo;t ever manually call <code>f_</code> myself, it is just registered and shoved in the global initializer section. So under whole program optimization the compiler comes along and says &lsquo;Oh hey! All these variables are doing stuff and we don&rsquo;t use them? Get rid!&rsquo;, meaning that none of the global constructors are then ran, and as a result none of the tests will register themselves with the global test registry and so it looks like there is nothing to run.</p>
<p>Pretty smart of the compiler in some respects, but now we need to defeat it.</p>
<h1 id="the-solution">The Solution</h1>
<p>I was recently made aware of a funky feature of the MSVC compiler thanks to an investigation I did into using rpmalloc with LLVM to improve compile times on multithreaded workloads on Windows. <a href="https://twitter.com/maniccoder">@maniccoder</a>, who knows a thing or two about allocators, gave me <a href="https://twitter.com/maniccoder/status/1250775765672701954?s=20">this gist</a> on how to do it. The part that was new to me though was:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// Make sure symbols are not purged in linker
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#75715e">#pragma comment(linker, &#34;/include</span><span style="color:#75715e">:_rpmllinit_&#34;)</span><span style="color:#75715e">
</span></span></span></code></pre></div><p>So you can use a special MSVC specific pragma to pass options to the linker, and those options can include a special option to <strong>always</strong> preserve a symbol even if the compiler thinks it is dead. Seems relevant! Now since my <code>UTEST</code> test fixtures use the preprocessor, I cannot use the <code>#pragma</code> variant because it would be invoked at the wrong place. But I knew I could use the <code>__pragma()</code> variant to do the same thing within the preprocessed macro&rsquo;s body.</p>
<p>So my global test initializer before this change was:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_INITIALIZER(f)                                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void);                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  __declspec(allocate(&#34;.CRT$XCU&#34;)) void(__cdecl * f##_)(void) = f;             \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void)
</span></span></span></code></pre></div><p>And so I just added the <code>__pragma()</code> line to preserve the symbol:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_INITIALIZER(f)                                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void);                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  __pragma(comment(linker, &#34;/include</span><span style="color:#75715e">:&#34; #f &#34;_&#34;));                               \</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  __declspec(allocate(&#34;.CRT$XCU&#34;)) void(__cdecl * f##_)(void) = f;             \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void)
</span></span></span></code></pre></div><p>And it worked - kinda. So the above worked for all global initializers from tests in a C function compiled for Win64 - 64-bit. It failed for 32-bit executable builds, and for C++ code.</p>
<p>So starting with the C++ tests I realised that even though I&rsquo;ve compiled the <code>f_</code> function pointer with the <code>__cdecl</code> calling convention, I hadn&rsquo;t actually stopped the compiler mangling the variable name which would change the symbol name.</p>
<p>So to fix this I added another macro:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#if defined(__cplusplus)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_C_FUNC extern &#34;C&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#else
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_C_FUNC
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#endif
</span></span></span></code></pre></div><p>That uses <code>extern &quot;C&quot;</code> when C++ is being used, and then modified the initializer to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_INITIALIZER(f)                                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void);                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  __pragma(comment(linker, &#34;/include</span><span style="color:#75715e">:&#34; #f &#34;_&#34;));                               \</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  UTEST_C_FUNC __declspec(allocate(&#34;.CRT$XCU&#34;)) void(__cdecl * f##_)(void) = f;\
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void)
</span></span></span></code></pre></div><p>Note that it doesn&rsquo;t matter what the signature of the <code>f</code> function itself is, only that the function pointer global variable does not end up with a mangled name. This fixed the C++ specific link failure.</p>
<p>For the 32-bit windows it was a little funky - but I had a vague memory in my head of dealing with this in a previous life. Basically 32-bit symbols are prepended with an extra underscore <code>_</code> on the symbol name. I don&rsquo;t actually know why this is the case to be honest, but I&rsquo;ve hit this issue before. So to fix that I added:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#if defined(_WIN64)
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_SYMBOL_PREFIX
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#else
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_SYMBOL_PREFIX &#34;_&#34;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#endif
</span></span></span></code></pre></div><p>Which will have an additional underscore only on 32-bit builds using MSVC, and then changed the initializer to:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define UTEST_INITIALIZER(f)                                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void);                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  __pragma(comment(linker, &#34;/include</span><span style="color:#75715e">:&#34; UTEST_SYMBOL_PREFIX #f &#34;_&#34;));           \</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  UTEST_C_FUNC __declspec(allocate(&#34;.CRT$XCU&#34;)) void(__cdecl * f##_)(void) = f;\
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  static void __cdecl f(void)
</span></span></span></code></pre></div><p>Which fixed all my build problems. Now utest.h supports whole program optimization correctly.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] Running <span style="color:#ae81ff">502</span> test cases.
</span></span><span style="display:flex;"><span>[ RUN      ] utest_cmdline.filter_with_list
</span></span><span style="display:flex;"><span>[       OK ] utest_cmdline.<span style="color:#a6e22e">filter_with_list</span> (<span style="color:#ae81ff">4155002</span>ns)
</span></span><span style="display:flex;"><span>[ RUN      ] c.ASSERT_TRUE
</span></span><span style="display:flex;"><span>[       OK ] c.<span style="color:#a6e22e">ASSERT_TRUE</span> (<span style="color:#ae81ff">77</span>ns)
</span></span><span style="display:flex;"><span>[ RUN      ] c.ASSERT_FALSE
</span></span><span style="display:flex;"><span>[       OK ] c.<span style="color:#a6e22e">ASSERT_FALSE</span> (<span style="color:#ae81ff">53</span>ns)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>...
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] <span style="color:#ae81ff">502</span> test cases ran.
</span></span><span style="display:flex;"><span>[  PASSED  ] <span style="color:#ae81ff">502</span> tests.
</span></span></code></pre></div>


guid: https://www.neilhenning.dev/posts/simple-c-command-line-parser/
title: Simple C Command Line Parser
url: https://www.neilhenning.dev/posts/simple-c-command-line-parser/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;ve been feeling a little overwhelmed by the languages I generally code in (C#, C++, Rust) and wanted to get my teeth into something a bit simpler again (obviously C!). One thing I&rsquo;ve never been that happy with in C was the lack of a good command line argument parser. There is a great <a href="https://github.com/nothings/single_file_libs#argv">list of option parsers</a> that <a href="https://twitter.com/nothings">@nothings</a> has curated over the years, but most of them are C++, and the ones for C are just a bit underwhelming to me.</p>
<p>Since the thing I&rsquo;m working on requires a command line application I thought <em>&lsquo;Why not write my own?&rsquo;</em> and herein lies the fun little trip I took.</p>
<h1 id="the-requirements">The Requirements</h1>
<p>So I want something that was:</p>
<ul>
<li>Written in C entirely.</li>
<li>Does not dynamically allocate any memory to store the arguments.</li>
<li>Can automatically generate the <code>--help</code> output for the supported options.</li>
</ul>
<p>To keep things simple I decided that short form options and long form options would have a defined structure:</p>
<ul>
<li>They can take no args and are of the form of <code>-h</code> and <code>--help</code>.</li>
<li>Or if they take an argument they are structured like <code>-o &lt;arg&gt;</code> or <code>--output=&lt;arg&gt;</code>.</li>
<li>They can only be specified zero or one times.</li>
</ul>
<p>I wasn&rsquo;t aiming to make some library that I could parachute into any project, more that I just wanted something simple that I could own myself.</p>
<h1 id="abusing-include-files">Abusing Include Files</h1>
<p>I <em>really</em> dislike anything that causes me to copy and paste code around. My distaste stems from the fact that you regularly want to then update all N places where you copied something, and it becomes onerous to keep these in check. Concretely for doing a command line parser, I didn&rsquo;t want to copy and paste code any time I added a new command line option.</p>
<p>Instead I wanted some way I could define all my options in a standard format and the rest of the tooling would pick this up and turn it into what I wanted. The only real way to do this in C is with the preprocessor. Since I knew that I&rsquo;d be wanting to use the same list of options for multiple purposes (EG. I&rsquo;d want to use them to parser <code>argv</code> but also print out their descriptions for <code>--help</code>), I decided to use an include file - a <code>.inc</code>. In general the notion for <code>.inc</code> files is that they have some preprocessor macro in their body that the includer defines before including the <code>.inc</code> file. Then they can include the <code>.inc</code> file multiple times and change what the macro does. This means that unlike most includes, <code>.inc</code> files <strong>do not</strong> have include guards / pragma once to stop multiple includes of them - because including it multiple times is <em>exactly</em> what we want to do.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#ifndef OPTION
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#error OPTION(short-name, long-name, number-of-trailing-args, description) was not defined!
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#endif
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">OPTION</span>(o, output, <span style="color:#ae81ff">1</span>, <span style="color:#e6db74">&#34;the output file to write to&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">OPTION</span>(h, help, <span style="color:#ae81ff">0</span>, <span style="color:#e6db74">&#34;print all the options and exit&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#undef OPTION
</span></span></span></code></pre></div><p>Here is my really simple <code>options.inc</code> that I used. It firstly includes a little if check to ensure that the required <code>OPTION</code> macro was specified, with a description of how the macro is intended to be used. Then it contains the list of the options that I have. At present just two options - one to specify where the output of the tool will go, and the other to display the help message and exit. Note the minor quality of life thing whereby the <code>.inc</code> file undefined the <code>OPTION</code> macro - this just saves you either doing it yourself after each include of the <code>.inc</code> file, or having the compiler warn about multiply defines macros!</p>
<p>Now that I have this I can use it to actually implement the parsing.</p>
<h1 id="parsing-argv">Parsing Argv</h1>
<p>So the first thing I wanted to ensure was that my <code>OPTION</code> macros were not malformed - specifically that the &rsquo;number-of-trailing-args&rsquo; was either 0 or 1. Even though I&rsquo;m the only one working on this and I hopefully shouldn&rsquo;t be dumb, I&rsquo;ve found over the years that any notion that you won&rsquo;t be dumb in some way is generally disproven. We are all stupid meatbags at times and so while you are hyper aware of the constraints it is best to be as defensive a coder as possible. Your future self will thank you!</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  bootstrap_assert(n &lt;= 1,                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">                   &#34;option&#34; #s &#34;/&#34; #l &#34; must have 0 or 1 trailing arguments&#34;);
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span></code></pre></div><p>So you can see that I just <code>#define</code> the <code>OPTION</code> macro and in this instance make it call my custom assert. Then I just have to include the <code>.inc</code> file and it&rsquo;ll do this for every option I&rsquo;ve defined.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> arg_is_set <span style="color:#f92672">=</span> <span style="color:#e6db74">&#34;set&#34;</span>;
</span></span><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d) const char *arg_##s = 0;
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span></code></pre></div><p>Next I needed a location to store the results of each of my variable. Using the same pattern again I can define variables of the form <code>arg_h</code> for the help argument, <code>arg_o</code> for the output, etc. For arguments that do not have any arguments I added a single <code>arg_is_set</code> here too - this we will use later to check whether the argument was set or not.</p>
<p>Now to the parser. This one is a bit of a mouthful I&rsquo;ll just warn you up front - it actually shows one of the big downsides to code generation with the preprocessor in C/C++ in that code in preprocessors is very difficult to debug within both compiler errors and when using a debugger. So remember that I didn&rsquo;t want <em>any</em> runtime memory allocation. So the one place you generally would use this is for positional arguments. Think of things like the list of files you&rsquo;d pass to a compiler, you can have N of these and they don&rsquo;t require options to be passed in. To get this to work I&rsquo;m going to reuse <code>argv</code> to store the list of the positional arguments. Since each positional argument always arrived in <code>argv</code> in the first place, it stands to reason that they can definitely fit!</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>  <span style="color:#66d9ef">int</span> positional_arguments <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">for</span> (<span style="color:#66d9ef">int</span> i <span style="color:#f92672">=</span> <span style="color:#ae81ff">1</span>; i <span style="color:#f92672">&lt;</span> argc; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  {                                                                            \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    if (0 == strcmp(&#34;-&#34; #s, argv[i])) {                                        \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      bootstrap_assert(arg_##s == 0, &#34;option&#34; #s &#34;/&#34; #l                        \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">                                     &#34; cannot be specified multiple times&#34;);   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      if (n == 0) {                                                            \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        arg_##s = arg_is_set;                                                  \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      } else {                                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        bootstrap_assert((i + 1) &lt; argc, &#34;trailing argument for option&#34; #s     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">                                         &#34;/&#34; #l &#34; was not provided&#34;);          \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        arg_##s = argv[i + 1];                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        i++;                                                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        continue;                                                              \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      }                                                                        \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    } else if (0 == strncmp((n == 0) ? &#34;--&#34; #l : &#34;--&#34; #l &#34;=&#34;, argv[i],         \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">                            strlen(&#34;--&#34; #l &#34;=&#34;))) {                            \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      bootstrap_assert(arg_##s == 0, &#34;option&#34; #s &#34;/&#34; #l                        \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">                                     &#34; cannot be specified multiple times&#34;);   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      if (n == 0) {                                                            \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        arg_##s = arg_is_set;                                                  \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      } else {                                                                 \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        arg_##s = argv[i] + strlen(&#34;--&#34; #l &#34;=&#34;);                               \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">        continue;                                                              \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">      }                                                                        \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    }                                                                          \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  }
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">// Otherwise if we didn&#39;t find our option then it is a position argument!
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    argv[positional_arguments<span style="color:#f92672">++</span>] <span style="color:#f92672">=</span> argv[i];
</span></span><span style="display:flex;"><span>  }
</span></span></code></pre></div><p>So let&rsquo;s break down the above:</p>
<ul>
<li>First I do the check for the small option form <code>s</code>, by string comparing <code>&quot;-&quot; #s</code> against the argument.
<ul>
<li>I then check if the option has been specified before which I&rsquo;m not handling.</li>
<li>If the option has zero trailing arguments, then just set the global variable for <code>arg_##s</code> to the <code>arg_is_set</code> helper I defined before.</li>
<li>If the option has one trailing argument, then I check there is definitely been a provided argument (EG. someone didn&rsquo;t have <code>-o</code> at the end of the argument list!).</li>
<li>And store the value in <code>argv[i + 1]</code> into the variable for the option.</li>
<li>I need to bump <code>i</code> to skip the next argument because we&rsquo;ve consumed it as the trailing argument of this one.</li>
<li>And then <code>continue</code>, this just gives us a nice easy way to skip the backup code to store the argument as a positional one.</li>
</ul>
</li>
<li>Next I do the check for the long form <code>l</code> - which requires a slightly different string compare. Because long form options that take an argument come like <code>--output=&lt;arg&gt;</code> I need to check if the option had a trailing argument, and then use a <code>strncmp</code> for this.
<ul>
<li>The only other difference is when there is a trailing argument I need to <strong>not</strong> bump <code>i</code> this time - because the argument is included in the option. To get at the argument we just get the pointer to the first character after the <code>--output=</code> - by using <code>strlen</code>.</li>
</ul>
</li>
<li>And then the fallback case is to just store the argument into the start of <code>argv</code> as a positional argument.</li>
</ul>
<h1 id="using-the-parsed-options">Using the Parsed Options</h1>
<p>So the first obvious way to use the options is to print the help:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (arg_h) {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">printf</span>(<span style="color:#e6db74">&#34;usage bootstrap&#34;</span>);
</span></span><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  if (n == 0) {                                                                \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    printf(&#34; [ %s | %s ]&#34;, &#34;-&#34; #s, &#34;--&#34; #l);                                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  } else {                                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    printf(&#34; [ %s &lt;arg&gt; | %s=&lt;arg&gt; ]&#34;, &#34;-&#34; #s, &#34;--&#34; #l);                       \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  }
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>    <span style="color:#a6e22e">printf</span>(<span style="color:#e6db74">&#34; &lt;args&gt;</span><span style="color:#ae81ff">\n\n</span><span style="color:#e6db74">&#34;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  if (n == 0) {                                                                \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    printf(&#34; %s | %s %s\n&#34;, &#34;-&#34; #s, &#34;--&#34; #l, d);                               \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  } else {                                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">    printf(&#34; %s &lt;arg&gt; | %s=&lt;arg&gt; %s\n&#34;, &#34;-&#34; #s, &#34;--&#34; #l, d);                   \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  }
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">return</span> <span style="color:#ae81ff">0</span>;
</span></span><span style="display:flex;"><span>  }
</span></span></code></pre></div><p>Breaking this down I:</p>
<ul>
<li>Check that the <code>arg_h</code> variable is non-null - which means the user requested help.</li>
<li>Use an <code>OPTION</code> macro to print the short usage guide - something like <code>usage bootstrap [ -o &lt;arg&gt; | --output=&lt;arg&gt; ] [ -h | --help ] &lt;args&gt;</code>.</li>
<li>Then I use a second macro to print the longer usage guide with the descriptions.</li>
</ul>
<p>In the end this produces a help like:</p>
<pre tabindex="0"><code>usage bootstrap [ -o &lt;arg&gt; | --output=&lt;arg&gt; ] [ -h | --help ] &lt;args&gt;

 -o &lt;arg&gt; | --output=&lt;arg&gt; the output file to write to
 -h | --help print all the options and exit
</code></pre><p>The only thing that&rsquo;d make this nicer would be to align the descriptions. But for now its good enough for me.</p>
<p>Just to test that all the options were set and the positional arguments were stored like I expected, I added a second macro during debugging:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#define OPTION(s, l, n, d)                                                     \
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">  fprintf(stderr, &#34;%s = %s\n&#34;, #s, (arg_##s != 0) ? arg_##s : &#34;not set&#34;);
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;options.inc&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">fprintf</span>(stderr, <span style="color:#e6db74">&#34;Positional arguments:</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">for</span> (<span style="color:#66d9ef">int</span> i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; i <span style="color:#f92672">&lt;</span> positional_arguments; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">fprintf</span>(stderr, <span style="color:#e6db74">&#34;%s</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>, argv[i]);
</span></span><span style="display:flex;"><span>  }
</span></span></code></pre></div><p>All this does is print whether each option was set or not, and then the list of positional arguments we recovered from the command line.</p>
<p>In total this is around 100 lines of C and does what I want in a very simple and easily controlled manner. I&rsquo;m very happy to be able to go back to basics and write some fun C code like this from time to time!</p>


guid: https://www.neilhenning.dev/posts/introducing_ubench_h/
title: Introducing ubench.h
url: https://www.neilhenning.dev/posts/introducing_ubench_h/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Ever since I worked on my amazing little single-header C/C++ library <a href="https://github.com/sheredom/utest.h/">utest.h</a>, which mimics enough of Google&rsquo;s googletest framework but <a href="https://www.neilhenning.dev/posts/utest-h-performance/">in a significantly leaner footprint</a> (and a single header!), I&rsquo;ve wanted something similar for doing benchmarking. Google has their own benchmarking library <a href="https://github.com/google/benchmark">Google benchmark</a>, but it fails a couple of key tests that I have for libraries:</p>
<ul>
<li>It isn&rsquo;t a single header library.</li>
<li>It doesn&rsquo;t support C benchmarks.</li>
<li>It is <em>huge</em>.</li>
<li>It has a build process that results in a non trivial integration process.</li>
</ul>
<p>While none of these are a death knell for an otherwise incredibly useful library, it just didn&rsquo;t meet enough of the requirements for my own uses.</p>
<p>So I&rsquo;ve wrote my own - <a href="https://github.com/sheredom/ubench.h">ubench.h</a>.</p>
<h2 id="introducing-ubenchh">Introducing ubench.h</h2>
<p>So given all the infrastructure I had already worked out to get my utest.h library, I could re-use a significant amount of the code. I started by copying the code from utest.h and doing a rename of <code>UTEST</code> -&gt; <code>UBENCH</code> and <code>utest</code> -&gt; <code>ubench</code>. Next up, I removed all the code that wasn&rsquo;t required from testing - all the asserts and expects that make up a unit testing framework. I also removed <a href="https://github.com/sheredom/utest.h/#define-a-fixtured-testcase">fixture</a> and <a href="https://github.com/sheredom/utest.h/#define-an-indexed-testcase">indexed</a> testcases because I wanted to push for a minimally viable product to begin with.</p>
<h2 id="ubench">UBENCH</h2>
<p>To declare a benchmark you use the <code>UBENCH</code> macro:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;ubench.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UBENCH</span>(foo, bar) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">usleep</span>(<span style="color:#ae81ff">100</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1000</span>);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The macro takes two parameters - the set of the benchmark, and the name of the individual benchmark. In the body of the benchmark you specify the code you want to profile - in this case I&rsquo;m just sleeping for a certain amount of time as a good no-op example.</p>
<p>In one file you then need to specify the <code>UBENCH_MAIN()</code> - this macro will define the <code>main</code> function of your executable, define all the global state we require, and call into ubench.h to run the benchmarks:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#include</span> <span style="color:#75715e">&#34;ubench.h&#34;</span><span style="color:#75715e">
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">UBENCH_MAIN</span>()
</span></span></code></pre></div><p>You can alternatively call the <code>ubench_main</code> function yourself if you want to use your own <code>main</code> function instead. First you need to specify the state for ubench.h:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#a6e22e">UBENCH_STATE</span>();
</span></span></code></pre></div><p>And then when you are ready to call into the ubench.h framework do:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">int</span> <span style="color:#a6e22e">main</span>(<span style="color:#66d9ef">int</span> argc, <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> argv[]) {
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// do your own thing
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">return</span> <span style="color:#a6e22e">ubench_main</span>(argc, argv);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>Note - you need to specify <code>UBENCH_MAIN</code> or <code>UBENCH_STATE</code> in a single source file only.</p>
<h2 id="command-line-output">Command Line Output</h2>
<p>The command line output for the library is similar to utest.h and thus also similar to googletest&rsquo;s output:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] Running <span style="color:#ae81ff">1</span> benchmarks.
</span></span><span style="display:flex;"><span>[ RUN      ] foo.bar
</span></span><span style="display:flex;"><span>[       OK ] foo.<span style="color:#a6e22e">bar</span> (<span style="color:#ae81ff">102289906</span>ns <span style="color:#f92672">+-</span> <span style="color:#ae81ff">1.751264</span><span style="color:#f92672">%</span>)
</span></span><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] <span style="color:#ae81ff">1</span> benchmarks ran.
</span></span><span style="display:flex;"><span>[  PASSED  ] <span style="color:#ae81ff">1</span> benchmarks
</span></span></code></pre></div><p>The one notable difference is how tests pass/fail, and the additional of <code>+- 1.751264%</code> to the timing information. Benchmarks are only useful if the repeated runs of them result in similar numbers. You need to run a benchmark a number of times and then get an average of the result. The problem is that the average can be vastly affected by huge outliers in the data set (EG. was the test ran from a cold start with dirty caches?) which can drastically affect the reliability of your results. To combat this, the benchmarks are recorded multiple times in multiple passes. Only when the standard deviation of the test is lower than 2.5 percent (a good initial value for a reliable and reproducible result) can the benchmark be seen as being useful. The <code>+- 1.751264%</code> is the standard deviation that was recorded for that example. If after a number of runs the variance of the runs does not conform to a small enough deviation then the benchmark is regarded to have failed, and is reported as such.</p>
<h2 id="ubench_do_nothing">UBENCH_DO_NOTHING</h2>
<p>Benchmarks are best when they are simple enough to record a single simple thing, that gives good reproducible results. One thing that defeats this is that compilers are so handy now at removing dead code - optimizing away the very thing you want to measure! To get around this I&rsquo;ve included the <code>UBENCH_DO_NOTHING</code> macro, which takes a pointer argument and looks, to the compiler, like it will trash the memory within (read and write it). This way the compiler shouldn&rsquo;t optimize out the data you are using and thus the code that you are using to produce it:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#a6e22e">UBENCH</span>(<span style="color:#66d9ef">do</span>, nothing) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">static</span> <span style="color:#66d9ef">char</span> a[<span style="color:#ae81ff">8</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span>];
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">static</span> <span style="color:#66d9ef">char</span> b[<span style="color:#ae81ff">8</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span>];
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(a);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">memcpy</span>(b, a, <span style="color:#66d9ef">sizeof</span>(a));
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(b);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>The above makes the compiler think that <code>a</code> is getting some data placed within it, and then after the <code>memcpy</code> that something is being done with <code>b</code>. The end result is we can get a meaningful performance number for how expensive the <code>memcpy</code> is.</p>
<h2 id="summary">Summary</h2>
<p>The library is already simple yet useful for measuring some stuff - but I obviously want to do more. Fixtures are the obvious addition - you may want some state to be initialized once and then we profile the modifications of the data on its own rather than alongside the cost of initialization / destruction. I&rsquo;d also like to give the user more options for controlling how many times the benchmark is run to get a low enough standard deviation.</p>
<p>But I think as a programmer our number one job is to get something simple done and then use it in production. Production always helps shuffle out the actual missing things you need from the code rather than in the abstract up front trying to work out the full set of requirements. I hope the library proves useful to some of you out there, and you can keep track of the progress on GitHub at <a href="https://github.com/sheredom/ubench.h">https://github.com/sheredom/ubench.h</a>.</p>


guid: https://www.neilhenning.dev/posts/white_guilt_and_juneteenth/
title: White Guilt and Juneteenth
url: https://www.neilhenning.dev/posts/white_guilt_and_juneteenth/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;m currently sitting at home on an unexpected days holiday - for Juneteenth.
The high hiedyins of Unity wanted to give us some space to reflect as a company
on the racial inequality in the world.</p>
<p><strong>And I&rsquo;m sitting here feeling guilty as all heck.</strong> Why should I, as a peely
wally white guy, benefit <strong>yet again</strong> from black voices rightfully pointing out
the utterly unjust situation that we (white folks) have put them through for at
least four hundred years. Shamefully I didn&rsquo;t even <em>know</em> what Juneteenth was
until they announced it was a holiday, and I quickly scuttled over to
<a href="https://en.wikipedia.org/wiki/Juneteenth">wikipedia</a> to read what actually
happened on that day.</p>
<p>To try and shake just a sliver of my guilt I&rsquo;ve donated my wages for today to
the <a href="https://www.stephenlawrence.org.uk/">Stephen Lawrence Charitable Trust</a> to
in some way help towards all the good they do.</p>
<p>I also want to talk a bit about white guilt - why we <strong>should</strong> feel guilty,
and especially the role that Scotland played in and profited from the horrors of
the slave trade, and the ongoing problems with racism and discrimination that
still affect Scotland today.</p>
<h2 id="white-guilt">White Guilt</h2>
<blockquote>
<p>But the one thing the American dream fails to mention, Is I was many steps
ahead to begin with - White Privilege II by Macklemore &amp; Ryan Lewis</p>
</blockquote>
<p>The line above from the great <em>&ldquo;White Privilege II&rdquo;</em> song by Macklemore &amp; Ryan
Lewis was in reference to America but it applies equally to white men
everywhere. We&rsquo;ve had a head start on nearly ever situation we&rsquo;ve been involved
in and didn&rsquo;t realise because that&rsquo;s just how things went for us. We&rsquo;d been born
white and thus benefitted from a racist system. We&rsquo;d been born male which meant
we benefitted from a sexist system. We&rsquo;ve had job offers, higher wages,
promotions, lenient police intervention, and the list goes on.</p>
<p>If we are assertive and egotistical we&rsquo;re praised as leaders while the system
has actively punished black people and woman for showing the same traits that
are so valued in white men.</p>
<p>And the worst bit about all of this for me personally is that <em>because</em> this
discrimination didn&rsquo;t affect me personally in any way I was a passive ally to a
broken system.</p>
<p>So I&rsquo;ve got white guilt - specifically white <strong>man</strong> guilt - and it is right
that I do. I&rsquo;ve implicitly benefitted from a system that was rigged in my favour
and I was too wrapped up in my own ego to realise.</p>
<h2 id="scotland-and-slavery">Scotland and Slavery</h2>
<p>David Hayman made a fantastic program for BBC Scotland -
<a href="https://www.bbc.co.uk/programmes/b0bqvv10">Slavery: Scotland&rsquo;s Hidden Shame</a>.
In it, David goes into depth about all the horrible ways that Scotland directly
benefitted from slavery (lots of people here owned slaves as a
<em>&lsquo;good investment&rsquo;</em>), and also how Scots themselves disproportionately made up
the staff employed in managing the slave trade (from the physicians on the
boats to those doing the paperwork).</p>
<p>We Scots have to acknowledge the part our country played in this through
education and introspection. In high school history, the lessons I was taught
consisted of a walk through all the times we won against the English in war,
how France is the Auld Ally and our friend, and then it skipped forward to the
industrial revolution and the slum-like conditions the families lived in. The
period of failed colonisation (see the hilariously badly planned
<a href="https://en.wikipedia.org/wiki/Darien_scheme">Panama colony</a> - including taking
winter kilts to the tropics), the reasons we went into union with England
(Scotland was so poorly managed we went bankrupt), and the subsequent
involvement of Scots in the slave trade were all massive missing parts of my
education.</p>
<p>Another fact that people aren&rsquo;t taught here is that the racist and facist KKK
(who truly should be considered a terrorist organisation I don&rsquo;t even understand
how they can exist) was
<a href="https://www.theguardian.com/education/2002/jan/19/highereducation.historybooks1">founded by Scots</a>.</p>
<p>It&rsquo;s important that we get taught these lessons.</p>
<h2 id="scotland-today">Scotland Today</h2>
<p>Scotland today is a country that is trying to be an outward looking and
welcoming country. We talk about welcoming refugees and being open to people
from all backgrounds and faiths. And yet two days ago our First Minister had to
<a href="https://www.bbc.co.uk/news/uk-scotland-glasgow-west-53086488">condemn racist thugs</a>
on the streets of our biggest city Glasgow.</p>
<p>Scotland has a <em>long</em> history of discrimination. At least where I&rsquo;m
from south of Glasgow, there have been multiple waves of immigrants in the past
few hundred years that have caused discrimination to ramp up. My own family has
heavy Irish roots - having fled either during the potato famine or when the
republican movement flared up in Northern Ireland. There was a huge movement of
Italians post World War II that settled in Glasgow and the surrounding areas.
There are decent communities of incomers from Bangladesh, India, Pakistan, and
Turkey too.</p>
<p>And each of these groups faced persecution. The Irish, being predominantly
catholic incomers, met with hostility of the
<a href="https://en.wikipedia.org/wiki/Orange_Order">Orange Order</a> and had derogatory
phrases like <em>&ldquo;Fenian Scum!&rdquo;</em> hurled at them, the Italians were called
<em>&ldquo;Tallies&rdquo;</em> and faced an anti-Italian fervour, and anyone who looked even
<em>slightly</em> like they were from the Middle East or East Asia was called a
<em>&ldquo;Paki&rdquo;</em>.</p>
<p>I was from a catholic family in a <em>very</em> protestant town, and before the age of
16 I was:</p>
<ul>
<li>Petrol bombed.</li>
<li>Chased with Samurai swords.</li>
<li>Had bricks thrown at me.</li>
<li>Involved in many altercations of the fisticuff kind.</li>
</ul>
<p>Every walk home from school was a gauntlet of &lsquo;what would happen today&rsquo;. If I
wore the colour green on a day where the Orange Order were doing one of their
nonsense &lsquo;parades&rsquo; (they march and play flutes, followed by a rowdy crowd of
drunk racists looking for a fight) I&rsquo;d have to run - fast.</p>
<p>I have never <em>once</em> witnessed overt racism against black people here, but I&rsquo;ve
seen plenty of people who have had <em>&ldquo;Paki&rdquo;</em> thrown in their face in my time. I
thought things here had got better, or at least I had hoped. But when you hear
the things that the racist thugs above said it is clear we&rsquo;ve still got a ways
to go.</p>
<h2 id="what-can-white-people-do">What Can White People Do</h2>
<p>There are some things we white men can do to help correct the historial wrongs
we&rsquo;ve inadvertently took part in:</p>
<ul>
<li>Make space for black and other under-represented groups. We need to put our
elbows out and use our privilege as a way to pull others up.</li>
<li>Push for diversity in hiring. This requires having hard conversations with
your direct management, but also using company wide meetings that allow
questions of the high hiedyins as an opportunity to ask about diversity.</li>
<li>Try to seek out people who don&rsquo;t look like you and engage with them. My twitter and
social circles are predominantly white and male, and for too long I put this
down to <em>&ldquo;Well these are just the people in the industry I know so&hellip;&rdquo;</em>. This
was <em>wrong</em> of me and I need to do better.</li>
<li>Speak up when you see injustice in the world and your workplace. The white
man privilege we&rsquo;ve always had gives us a shield against most of the
repercussions that under-represented groups would face if they speak out.</li>
<li>Where possible donate to groups that help the black community.</li>
<li>Arrest Brett Hankison, Jonathan Mattingly, and Myles Cosgrove for the murder
of Breonna Taylor.</li>
</ul>
<p>Let us try and make a better world for everyone, together. We&rsquo;re all the better
when we&rsquo;re in it together.</p>


guid: https://www.neilhenning.dev/posts/switching_ubench_h_to_confidence_interval/
title: I'm now 99% Confident in ubench.h
url: https://www.neilhenning.dev/posts/switching_ubench_h_to_confidence_interval/
feedurl: https://www.duskborn.com/index.xml
content:
<p>So I recently created a new single header C/C++ library for benchmarking your
code - <a href="https://github.com/sheredom/ubench.h">ubench.h</a>.</p>
<p>It&rsquo;s funny how as soon as you push something public you realise all the flaws in
it - but that&rsquo;s fine! Software (like people) should grow and develop as it gets
older.</p>
<p>The way I was trying to gauge the accuracy of the result from multiple benchmark
runs was somewhat&hellip; broken. In
<a href="https://github.com/sheredom/ubench.h/pull/3">this PR</a> I&rsquo;ve switched to using a
confidence interval, and I&rsquo;ll now explain why.</p>
<h2 id="the-broken-approach">The Broken Approach</h2>
<p>So I&rsquo;ve never studied statistics in any capacity. I&rsquo;d always followed the
believe that statisticians were like demi-lawyers - good at manipulating the
same set of &lsquo;facts&rsquo; to tell whatever story they want.</p>
<p>But ever since I watched the rather awesome
<a href="https://twitter.com/despair">Elan Ruskin</a> talk about
<a href="https://www.gdcvault.com/play/1023323/Three-Statistical-Tests-Every-Game">statistical tests for games</a>
I realised I needed to level up.</p>
<p>So the first thing I learned about was standard deviation - well really I
had to re-learn this because I&rsquo;m pretty certain I got taught it in high school
and immediately forgot it&hellip; For those uninitiated the standard deviation gives
you a measure of how variable the average of a set of data is. Let&rsquo;s say you&rsquo;ve
got 10 recordings of something - the average is just the sum of all items
divided by the number of recordings, otherwise know as the mean.</p>
<p>You then use this mean, subtract it from each of the original values, square
them, divide by the number of iterations and then take the sqrt of that:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">ubench_int64_t</span> ns[iterations];
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ubench_int64_t</span> avg_ns <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; i <span style="color:#f92672">&lt;</span> iterations; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  avg_ns <span style="color:#f92672">+=</span> ns[i];
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>avg_ns <span style="color:#f92672">/=</span> iterations;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; i <span style="color:#f92672">&lt;</span> iterations; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">double</span> v <span style="color:#f92672">=</span> ns[i] <span style="color:#f92672">-</span> avg_ns;
</span></span><span style="display:flex;"><span>  deviation <span style="color:#f92672">+=</span> v <span style="color:#f92672">*</span> v;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>deviation <span style="color:#f92672">=</span> <span style="color:#a6e22e">sqrt</span>(deviation <span style="color:#f92672">/</span> iterations);
</span></span></code></pre></div><p>Once I had this I then used the standard deviation to measure how variable the
result was. A high standard deviation means that you don&rsquo;t have confidence that
the result you&rsquo;ve recorded is stable enough to be meaningful. Conversely a low
standard deviation makes you confident. Previously I was aiming for a standard
deviation below 2.5% as a gauge of how good the result was.</p>
<p>The problem is that getting a standard deviation below a low margin across many
different types of machines we test on (laptops, desktops, mobiles, consoles,
etc) can be quite difficult. Pre-emption of the running thread might cause a
temporary spike in a given tests result, or you might get a very fast run that
hits no speed bumps and thus runs really fast. These outlier results can
really affect the standard deviation of the result.</p>
<p>Since I didn&rsquo;t know much about stats I <em>wrongly</em> thought that the best way to
cut out the occassional high variance results was to just sort all the runs and
then slice off the top and bottom N items:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">ubench_int64_t</span> ns[iterations];
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">ubench_int64_t</span> avg_ns <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// Sort the results.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">qsort</span>(ns, iterations, <span style="color:#66d9ef">sizeof</span>(<span style="color:#66d9ef">ubench_int64_t</span>), ubench_sort_func);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">1</span>; i <span style="color:#f92672">&lt;</span> (iterations <span style="color:#f92672">-</span> <span style="color:#ae81ff">1</span>); i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  avg_ns <span style="color:#f92672">+=</span> ns[i];
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>avg_ns <span style="color:#f92672">/=</span> (iterations <span style="color:#f92672">-</span> <span style="color:#ae81ff">2</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">1</span>; i <span style="color:#f92672">&lt;</span> (iterations <span style="color:#f92672">-</span> <span style="color:#ae81ff">1</span>); i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">double</span> v <span style="color:#f92672">=</span> ns[i] <span style="color:#f92672">-</span> avg_ns;
</span></span><span style="display:flex;"><span>  deviation <span style="color:#f92672">+=</span> v <span style="color:#f92672">*</span> v;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>deviation <span style="color:#f92672">=</span> <span style="color:#a6e22e">sqrt</span>(deviation <span style="color:#f92672">/</span> (iterations <span style="color:#f92672">-</span> <span style="color:#ae81ff">2</span>));
</span></span></code></pre></div><p>This would cut off the top and bottom worse skewed results. The problem is that
this didn&rsquo;t really reduce the standard deviation that much in the first place,
and secondly it discounted valuable collected data.</p>
<p>Another problem with the standard deviation is that its not necessarily true
that collecting more data will reduce the deviation itself - just because you
are collecting more data doesn&rsquo;t meant that the data will be any less deviant.</p>
<p>So I needed another approach.</p>
<h2 id="im-99-confident-with-my-interval">I&rsquo;m 99% Confident with my Interval</h2>
<p>If we look at the typical spread of data for any measurement, the data <em>tends</em>
to centre around the average or mean value. This is called the normal
distribution or a bell curve, and generally looks like:</p>

<img src="/images/confidence-interval/bellmouth.jpg"  class="center"  style="border-radius: 8px;"    />


<p>In this chart, x = 0 is the the average, and y is the number of samples that
matched the given value of x. This is what data <em>tends</em> to look like from
benchmarks. We can see that most of the samples (larger y) occurs closer to the
mean, and there is less and less the further we go away. Given I want to cut out
any outliers large or small, what we really want is a way to say <em>&lsquo;Is most of
the data close enough to the average?&rsquo;</em>.</p>
<p>Luckily some people smarter than me have a way to do this - and it is called a
confidence interval. A confidence interval is a way to say <em>&lsquo;If we only care
about X% of the data, how close is that to the average?&rsquo;</em>. For my use case I&rsquo;m
using a 99% confidence interval - EG. I&rsquo;m asking the stats to give me a number
for how close the 99% of the samples that are closest to the average actually
are:</p>

<img src="/images/confidence-interval/with-bars.jpg"  class="center"  style="border-radius: 8px;"    />


<p>We can see in this chart the two extra lines - this is my confidence interval,
I&rsquo;m only wanting to know how close the X% of the values that are occuring within
these bounds are to the average.</p>
<p>Calculating a confidence interval is pretty easy too:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; i <span style="color:#f92672">&lt;</span> iterations; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  avg_ns <span style="color:#f92672">+=</span> ns[i];
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>avg_ns <span style="color:#f92672">/=</span> iterations;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">for</span> (i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>; i <span style="color:#f92672">&lt;</span> iterations; i<span style="color:#f92672">++</span>) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">double</span> v <span style="color:#f92672">=</span> ns[i] <span style="color:#f92672">-</span> avg_ns;
</span></span><span style="display:flex;"><span>  deviation <span style="color:#f92672">+=</span> v <span style="color:#f92672">*</span> v;
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>deviation <span style="color:#f92672">=</span> <span style="color:#a6e22e">sqrt</span>(deviation <span style="color:#f92672">/</span> iterations);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// Confidence is the 99% confidence index - whose magic value is 2.576.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>confidence <span style="color:#f92672">=</span> <span style="color:#ae81ff">2.576</span> <span style="color:#f92672">*</span> deviation <span style="color:#f92672">/</span> <span style="color:#a6e22e">sqrt</span>(iterations);
</span></span></code></pre></div><p>Now there is this magic constant <code>2.576</code> - I genuinely have no idea where it
comes from or why it is what it is. But
<a href="https://en.wikipedia.org/wiki/Confidence_interval_">some smart stats people</a>
has done some math stats stuff and&hellip; this is the value I need.</p>
<p>Now instead of aiming for a 2.5% range with my botched standard deviation, I
instead look for a 2.5% range with my 99% confidence interval instead. The
great thing about the confidence interval is the more data you throw at it the
lower it goes. I think this is because you cut out more outliers (the 1%) the
more samples you have, but whatever the reason it means that if I haven&rsquo;t
achieved the 2.5% range I want I can just re-run it with more samples and that
generally does what I want. Nice!</p>
<p>The output from ubench.h has now changed to display the confidence interval too,
and I added a request from <a href="https://twitter.com/zeuxcg">@zeuxcg</a> to change the
displayed timing in ms/s/us where appropriate - which makes it much more
legible:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[ RUN      ] cpp_my_fixture.strchr
</span></span><span style="display:flex;"><span>[       OK ] cpp_my_fixture.<span style="color:#a6e22e">strchr</span> (mean <span style="color:#ae81ff">4.115</span>ms, confidence interval <span style="color:#f92672">+-</span> <span style="color:#ae81ff">2.039480</span><span style="color:#f92672">%</span>)
</span></span><span style="display:flex;"><span>[ RUN      ] cpp_my_fixture.strrchr
</span></span><span style="display:flex;"><span>[       OK ] cpp_my_fixture.<span style="color:#a6e22e">strrchr</span> (mean <span style="color:#ae81ff">57.057</span>ms, confidence interval <span style="color:#f92672">+-</span> <span style="color:#ae81ff">0.445810</span><span style="color:#f92672">%</span>)
</span></span></code></pre></div><h2 id="conclusion">Conclusion</h2>
<p>So I&rsquo;ve made ubench.h more consistent at profiling and actually used real
mathematically sound statistics rather than my bodged attempt at it - nice! For
my users it&rsquo;ll mean more consistent data and more confidence (har har) that it
is doing the right thing.</p>


guid: https://www.neilhenning.dev/posts/adding_fixtures_ubench_h/
title: Adding fixtures to ubench.h
url: https://www.neilhenning.dev/posts/adding_fixtures_ubench_h/
feedurl: https://www.duskborn.com/index.xml
content:
<p>One thing that was missing from ubench.h (intentionally to keep the initial
code drop simple) was fixtures. In
<a href="https://github.com/sheredom/ubench.h/pull/4">this PR</a> I&rsquo;ve added them.</p>
<p>So what are fixtures and why should you use them?</p>
<h2 id="what-are-fixtures">What are fixtures?</h2>
<p>Fixtures are a way to setup and teardown state that doesn&rsquo;t contribute to the
timing of the actual benchmark itself:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">// First you declare a struct that contains the state you need.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#66d9ef">struct</span> foo {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">unsigned</span> size;
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">char</span><span style="color:#f92672">*</span> foo;
</span></span><span style="display:flex;"><span>};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// Then you setup the benchmark state - using the ubench_fixture implicit field.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">UBENCH_F_SETUP</span>(foo) {
</span></span><span style="display:flex;"><span>  ubench_fixture<span style="color:#f92672">-&gt;</span>size <span style="color:#f92672">=</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">128</span>;
</span></span><span style="display:flex;"><span>  ubench_fixture<span style="color:#f92672">-&gt;</span>foo <span style="color:#f92672">=</span> (<span style="color:#66d9ef">char</span><span style="color:#f92672">*</span>)<span style="color:#a6e22e">malloc</span>(ubench_fixture<span style="color:#f92672">-&gt;</span>size);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// In the teardown you can free up anything you allocated.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">UBENCH_F_TEARDOWN</span>(foo) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">free</span>(ubench_fixture<span style="color:#f92672">-&gt;</span>foo);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// And then declare the benchmarks that use the state.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">UBENCH_F</span>(foo, bar) {
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(ubench_fixture<span style="color:#f92672">-&gt;</span>foo);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">memset</span>(ubench_fixture<span style="color:#f92672">-&gt;</span>foo, <span style="color:#ae81ff">0</span>, ubench_fixture<span style="color:#f92672">-&gt;</span>size);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(ubench_fixture<span style="color:#f92672">-&gt;</span>foo);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// You can declare multiple benchmarks that use the same fixture too.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">UBENCH_F</span>(foo, haz) {
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>As you can see setting up and declaring a fixture isn&rsquo;t difficult.</p>
<h2 id="why-you-should-use-them">Why you should use them?</h2>
<p>Let&rsquo;s take the counter example to the fixture above - using the non-fixture
approach:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#a6e22e">UBENCH</span>(foo, bar) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">unsigned</span> size <span style="color:#f92672">=</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">1024</span> <span style="color:#f92672">*</span> <span style="color:#ae81ff">128</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">char</span><span style="color:#f92672">*</span> foo <span style="color:#f92672">=</span> (<span style="color:#66d9ef">char</span><span style="color:#f92672">*</span>)<span style="color:#a6e22e">malloc</span>(size);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(foo);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">memset</span>(foo, <span style="color:#ae81ff">0</span>, size);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">UBENCH_DO_NOTHING</span>(foo);
</span></span><span style="display:flex;"><span>  <span style="color:#a6e22e">free</span>(foo);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>And if we look at the results of both these tests we can see that the
non-fixture one runs like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] Running <span style="color:#ae81ff">1</span> benchmarks.
</span></span><span style="display:flex;"><span>[ RUN      ] foo.bar
</span></span><span style="display:flex;"><span>[       OK ] foo.<span style="color:#a6e22e">bar</span> (mean <span style="color:#ae81ff">15.506</span>ms, confidence interval <span style="color:#f92672">+-</span> <span style="color:#ae81ff">1.553863</span><span style="color:#f92672">%</span>)
</span></span><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] <span style="color:#ae81ff">1</span> benchmarks ran.
</span></span><span style="display:flex;"><span>[  PASSED  ] <span style="color:#ae81ff">1</span> benchmarks.
</span></span></code></pre></div><p>Whereas the fixture one runs like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] Running <span style="color:#ae81ff">1</span> benchmarks.
</span></span><span style="display:flex;"><span>[ RUN      ] foo.bar
</span></span><span style="display:flex;"><span>[       OK ] foo.<span style="color:#a6e22e">bar</span> (mean <span style="color:#ae81ff">4.125</span>ms, confidence interval <span style="color:#f92672">+-</span> <span style="color:#ae81ff">0.844092</span><span style="color:#f92672">%</span>)
</span></span><span style="display:flex;"><span>[<span style="color:#f92672">==========</span>] <span style="color:#ae81ff">1</span> benchmarks ran.
</span></span><span style="display:flex;"><span>[  PASSED  ] <span style="color:#ae81ff">1</span> benchmarks.
</span></span></code></pre></div><p>So we can see that the average time for the fixtured one is about 4x faster than
the non-fixtured one - because it hasn&rsquo;t timed the malloc/free cost per
benchmark. This means that if you really only care about what memset costs
you&rsquo;ve now got a much more accurate way to get that.</p>
<p>This is where fixtures are really great - hide the cost of setting up the data
for your benchmark from the thing you actually want to benchmark.</p>
<h2 id="one-last-thing">One last thing&hellip;</h2>
<p>One thing that <em>might</em> not be obvious is that fixtured tests can have a wall
clock time that is longer than the non-fixtured variants. For the examples above
the non-fixture one runs in:</p>
<pre tabindex="0"><code>real 0m0.592s
user 0m0.368s
sys  0m0.177s
</code></pre><p>Whereas the fixture benchmark runs in:</p>
<pre tabindex="0"><code>real 0m3.631s
user 0m3.508s
sys  0m0.071s
</code></pre><p>So why is this? In general the longer running a benchmark is the more accurate
the result will be - so because we&rsquo;re measuring less things in the actual
benchmark samples the framework is having to run more of them to get an accurate
result.</p>
<p>Just thought you might appreciate the heads up!</p>


guid: https://www.neilhenning.dev/posts/subprocess-h-async-reading/
title: Asynchronous Reading in subprocess.h
url: https://www.neilhenning.dev/posts/subprocess-h-async-reading/
feedurl: https://www.duskborn.com/index.xml
content:
<p>One hotly requested feature for my single header C/C++ process spawning and
management library <a href="https://github.com/sheredom/subprocess.h">subprocess.h</a> was
the ability to read from the standard output or error of a spawned process while
it was still executing. In
<a href="https://github.com/sheredom/subprocess.h/pull/24">this PR</a> I&rsquo;ve added support
for this, but it requires some change of behaviour in your code if you want to
use it.</p>
<h2 id="c-files-and-asynchronous-reading">C FILE&rsquo;s and Asynchronous Reading</h2>
<p>I use a really <em>clever hack</em> to give my users a standard C FILE handle with
which they can read the standard output and error, and write to the standard
input of any processes they spawn. This is immensely powerful because it means
anyone using my API can use an already familiar paradigm to interact with the
processes they spawn. Users could <code>fprintf</code> to the standard input of a spawned
process, or call <code>fgets</code> to read from the standard output or error. While I love
this feature - it causes a complication when trying to support asynchronous
reading.</p>
<p>For the normal process spawning on Windows I was using <code>CreatePipe</code> to map a
<code>HANDLE</code> on the parent process to <code>HANDLE</code> that&rsquo;ll be used for the standard
pipes on the spawned process. A <code>CreatePipe</code> is also known as an anonymous pipe
in Windows lingo - and it turns out you cannot sync from this pipe until the
spawned process has complete. But there is another mechanism called a named pipe
using the <code>CreateNamedPipe</code> call that can help us here.</p>
<p>Named pipes support an additional flag on creation <code>FILE_FLAG_OVERLAPPED</code> that
lets you get at the data from the pipe before the other end of the pipe has been
closed. I thought <em>&lsquo;Great! I&rsquo;ll use this et voila! Everything will work!&rsquo;</em>. It
didn&rsquo;t.</p>
<p>The problem is that if you use the overlapped creation flags to a named pipe you
cannot then do the <em>clever hack</em> anymore to get a C FILE that reads from this.
It&rsquo;ll just hang forever.</p>
<p>So instead I&rsquo;ve introduced some new helpers methods <code>subprocess_read_stdout</code> and
<code>subprocess_read_stderr</code> that use the correct APIs on each operating system to
allow for asynchronous reading from a pipe.</p>
<p>Note that while the main problems for this feature have been with Windows
support and the example below is Windows specific - the feature supports all
three major desktop platforms. Worry not!</p>
<h2 id="example">Example</h2>
<p>As an example of how we can use this I&rsquo;ve wrote up the following little example
that works on Windows (no reason why it couldn&rsquo;t work on other operating
systems, I just used Windows for the test):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">int</span> <span style="color:#a6e22e">main</span>() {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> commandLine[] <span style="color:#f92672">=</span> {<span style="color:#e6db74">&#34;ping&#34;</span>, <span style="color:#e6db74">&#34;-t&#34;</span>, <span style="color:#e6db74">&#34;www.bbc.net.uk&#34;</span>, <span style="color:#ae81ff">0</span>};
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">struct</span> subprocess_s process;
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">unsigned</span> bytes_read;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (<span style="color:#ae81ff">0</span> <span style="color:#f92672">!=</span> <span style="color:#a6e22e">subprocess_create</span>(
</span></span><span style="display:flex;"><span>    commandLine,
</span></span><span style="display:flex;"><span>    subprocess_option_inherit_environment <span style="color:#f92672">|</span> subprocess_option_enable_async,
</span></span><span style="display:flex;"><span>    <span style="color:#f92672">&amp;</span>process)) {
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">fprintf</span>(stderr, <span style="color:#e6db74">&#34;subprocess_create failed!&#34;</span>);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">do</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">char</span> buffer[<span style="color:#ae81ff">1024</span>] <span style="color:#f92672">=</span> {<span style="color:#ae81ff">0</span>};
</span></span><span style="display:flex;"><span>    bytes_read <span style="color:#f92672">=</span> <span style="color:#a6e22e">subprocess_read_stdout</span>(<span style="color:#f92672">&amp;</span>process, buffer, <span style="color:#66d9ef">sizeof</span>(buffer));
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">printf</span>(<span style="color:#e6db74">&#34;Read %u bytes - &#39;%s&#39;</span><span style="color:#ae81ff">\n</span><span style="color:#e6db74">&#34;</span>, bytes_read, buffer);
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">while</span> (bytes_read <span style="color:#f92672">!=</span> <span style="color:#ae81ff">0</span>);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>This will loop infinitely reading from the ping command with the <code>-t</code> option
specified. It&rsquo;ll read the output from ping as it is flushed and spit out what it
read and the number of bytes.</p>
<pre tabindex="0"><code>Read 42 bytes - &#39;
Pinging www.bbc.net.uk [212.58.237.253] &#39;
Read 24 bytes - &#39;with 32 bytes of data:
&#39;
Read 27 bytes - &#39;Reply from 212.58.237.253: &#39;
Read 27 bytes - &#39;bytes=32 time=22ms TTL=53
&#39;
Read 27 bytes - &#39;Reply from 212.58.237.253: &#39;
Read 27 bytes - &#39;bytes=32 time=22ms TTL=53
&#39;
</code></pre><p>A few things:</p>
<ul>
<li>Note the new <code>subprocess_create</code> option <code>subprocess_option_enable_async</code>. This
is needed to setup the pipe for being able to read from it asynchronously. If
you use this option you should refrain from using <code>subprocess_stdout</code> or
<code>subprocess_stderr</code> to read from the spawned process - they will <strong>not</strong> work.</li>
<li>Note the new function <code>subprocess_read_stdout</code> to read from the standard
output of the spawned process. It&rsquo;ll return non-zero while there is any data
still to be read. When the child process terminates then it&rsquo;ll return 0
signalling the end.</li>
</ul>
<p>One additional thing to bear in mind is that the <code>subprocess_read_stdout</code> and
<code>subprocess_read_stderr</code> functions are blocking. If you want to read from both
of these simultaneously then it is recommended that you either:</p>
<ul>
<li>Spawn a thread for one or both to read from it. It is thread safe to read from
both pipes of a spawned process at the same time.</li>
<li>Or use the <code>subprocess_option_combined_stdout_stderr</code> option if you just want
the full contents of both pipes both don&rsquo;t care to differentiate from either
pipes output.</li>
</ul>
<p>I hope you find this new functionality useful.</p>
<p>An additional gentle reminder that you can stay tuned for future updates by
using the <a href="https://www.neilhenning.dev/index.xml">RSS</a> feed.</p>


guid: https://www.neilhenning.dev/posts/introducing-hashmap-h/
title: Introducing hashmap.h
url: https://www.neilhenning.dev/posts/introducing-hashmap-h/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I asked on twitter the other day whether anyone had a hashmap that could work
with string slices - parts of a string that are <strong>not</strong> null-terminated and thus
have to have an explicit length to accompany the pointer.</p>
<p>I didn&rsquo;t get any responses on this so I commented with a follow-up that I had
grabbed some code written a few years back by the awesome
<a href="https://twitter.com/petewarden">Pete Warden</a> of Google fame, and morphed it
into what I required:</p>
<blockquote>
<p>Authors note: this part used to contain a tweet, But Hellish Tusk / Space Karen / Elon Musk butchered the platform so it is now gone.</p>
</blockquote>
<p>Much to my surprise Pete was happy not only for me to do these modifications,
but also since he was no longer maintaining the hashmap code he&rsquo;d happily
redirect users to any effort I put together:</p>
<blockquote>
<p>Authors note: this part used to contain a tweet, But Hellish Tusk / Space Karen / Elon Musk butchered the platform so it is now gone.</p>
</blockquote>
<p>So I&rsquo;ve done the work and I&rsquo;m now introducing my latest library, <strong>nearly
entirely not written by me</strong>,
<a href="https://github.com/sheredom/hashmap.h">hashmap.h</a>!</p>
<h2 id="null-terminated-to-slices">Null-Terminated to Slices</h2>
<p>So Pete&rsquo;s code was pretty solid as is. The main difference was that it relied on
null-terminated strings as the key, whereas I wanted to use string slices. My
first modification was to change the entry points that used a key to instead
take a key and a length. So <code>hashmap_put</code> went from:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#66d9ef">map_t</span> in, <span style="color:#66d9ef">char</span><span style="color:#f92672">*</span> key, <span style="color:#66d9ef">any_t</span> value);
</span></span></code></pre></div><p>To:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>HASHMAP_WEAK <span style="color:#66d9ef">int</span> <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#66d9ef">struct</span> hashmap_s <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> hashmap,
</span></span><span style="display:flex;"><span>                             <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">char</span> <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> key,
</span></span><span style="display:flex;"><span>                             <span style="color:#66d9ef">const</span> <span style="color:#66d9ef">unsigned</span> len,
</span></span><span style="display:flex;"><span>                             <span style="color:#66d9ef">void</span> <span style="color:#f92672">*</span><span style="color:#66d9ef">const</span> value);
</span></span></code></pre></div><p>You&rsquo;ll notice that I also went <code>const</code> mad (and again wished that <code>const</code> was
the default and <code>mutable</code> or <code>mut</code> was required on variables - sigh!), and
removed the typedef for <code>any_t</code>. The last point is just a general stylistic
thing I have for my libraries - I really dislike that APIs like <code>Windows.h</code>
abstract you so far away from the underlying types with all the SHOUTY CASE
<code>LDPWORD</code>&rsquo;s and such, that I generally try and have no typedef&rsquo;s if I can get
away with it.</p>
<h2 id="supports-utf-8-keys">Supports UTF-8 Keys</h2>
<p>Pete&rsquo;s code also used the <code>string.h</code> ASCII-string functions of C to compare
whether the key ever matched. Since I wanted to use this hashmap in conjunction
with UTF-8 strings (using my <a href="https://github.com/sheredom/utf8.h">utf8.h</a>
library) I instead used <code>memcmp</code>. Now that I have an explicit length for the
string slice this became possible.</p>
<h2 id="single-header">Single Header</h2>
<p>The last major change I made was to smush the hashmap.c and hashmap.h files
together into a single header. I am pretty obsessed with single headers as a way
to get round the botched nature of C and C++&rsquo;s package story (or lack thereof).
This meant leveraging some exists hacks to stop the compiler complaining about
multiple function definitions (by using weak function references instead).</p>
<h2 id="the-license">The License</h2>
<p>Pete&rsquo;s code was already marked explicitly public domain - do what you want with
it. I&rsquo;ve found over the years that while public domain is all fine and well,
having an explicit license like the <a href="https://unlicense.org/">unlicense</a> or the
<a href="https://creativecommons.org/share-your-work/public-domain/cc0/">CC0</a> can make
lawyers happy because there is at least some legal text to reference. It also
makes GitHub&rsquo;s license scraping happier because these licenses are already ones
that it knows about.</p>
<p>So I&rsquo;ve licensed this header under the unlicense - it matches what my existing
single-header projects use and is something that my users already favour.</p>
<h2 id="hashmap-all-the-things">Hashmap All The Things</h2>
<p>So I&rsquo;m pretty happy with the code I <em>mostly</em> did not write - and I hope that my
packaging and testing of the header allows it to be more widely useful to some
of you fine folks out there.</p>
<p>A big thanks again to Pete Warden for writing this code and being so gracious
about me making these changes. I hope this proves useful to some of you out
there too.</p>


guid: https://www.neilhenning.dev/posts/hourglass_apis_in_rust/
title: Exploring Hourglass APIs in Rust
url: https://www.neilhenning.dev/posts/hourglass_apis_in_rust/
feedurl: https://www.duskborn.com/index.xml
content:
<p>There are two talks on APIs that I think <em>every</em> programmer should watch, learn,
and study from:</p>
<ul>
<li><a href="https://caseymuratori.com/blog_0024">Designing and Evaluating Reusable Components</a>
from <a href="https://twitter.com/cmuratori">Casey Muratori</a>. This
<em>absolutely wonderful</em> talk is the fundamental source for how to design APIs.</li>
<li><a href="https://youtu.be/PVYdHDm0q6Y">Hourglass Interfaces for C++ APIs</a> from
<a href="https://twitter.com/stefanusdutoit">Stefanus Du Toit</a>. This talk discusses
having a rich API for users that is backed by a C API into proprietary code.</li>
</ul>
<p>These two talks give a really good and grounded lesson on fundamental API
design - the choices you make and the ramifications they can have. In
particular, the design challenge and solution of the hourglass API design is a
really nice approach to being able to ship a high functioning API that is backed
by some proprietary software that you don&rsquo;t want to ship. For example:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> MyDataOpaque;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> <span style="color:#e6db74">&#34;C&#34;</span> MyDataOpaque<span style="color:#f92672">*</span> <span style="color:#a6e22e">mydata_create</span>(<span style="color:#66d9ef">int</span>);
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> <span style="color:#e6db74">&#34;C&#34;</span> <span style="color:#66d9ef">void</span> <span style="color:#a6e22e">mydata_destroy</span>(MyDataOpaque<span style="color:#f92672">*</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> MyData final {
</span></span><span style="display:flex;"><span>  explicit <span style="color:#a6e22e">MyData</span>(<span style="color:#66d9ef">int</span> someState) <span style="color:#f92672">:</span> <span style="color:#a6e22e">opaque</span>(<span style="color:#a6e22e">mydata_create</span>(someState)) {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#f92672">~</span><span style="color:#a6e22e">MyData</span>() { <span style="color:#a6e22e">mydata_destroy</span>(opaque); }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>private:
</span></span><span style="display:flex;"><span>  MyDataOpaque<span style="color:#f92672">*</span> <span style="color:#66d9ef">const</span> opaque;
</span></span><span style="display:flex;"><span>};
</span></span></code></pre></div><p>This is a classic hourglass API in C++ - we&rsquo;ve got the C API that returns some
opaque data, and the hourglass <code>MyData</code> struct lets us expose this to C++ users
in a way they are familiar with. In Stefanus&rsquo; talk he went into detail about how
they also <em>implement</em> the C API in C++ - meaning that both sides of the thin C
API are written in the high level language they enjoy - C++.</p>
<p>This all got me thinking - would this approach be possible in some fashion with
Rust? First the caveats:</p>
<ul>
<li>I <em>suspect</em>, but am not well versed enough with Rust, that using different
versions of Rust to build either side of the hourglass API could cause
explosions. So I&rsquo;m going to assume that both sides are built with the same
version.</li>
<li>I really <em>hope</em> that it is safe to use memory allocating functions (like <code>Box</code>
or <code>Vec</code>) on both Rust sides of a C API.</li>
</ul>
<p>These pretty major caveats aside - is it possible?</p>
<h2 id="the-bottom-of-the-hourglass">The Bottom of the Hourglass</h2>
<p>First we&rsquo;ll implement the bottom of the hourglass - this would be the
proprietary code that you don&rsquo;t want to ship (all your secret sauce might be in
it!).</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-rust" data-lang="rust"><span style="display:flex;"><span><span style="color:#75715e">#[repr(C)]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">struct</span> <span style="color:#a6e22e">MyDataOpaque</span> {
</span></span><span style="display:flex;"><span>  some_state: <span style="color:#66d9ef">i32</span>,
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">impl</span> MyDataOpaque {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">new</span>(some_state: <span style="color:#66d9ef">i32</span>) -&gt; <span style="color:#a6e22e">MyDataOpaque</span> {
</span></span><span style="display:flex;"><span>    MyDataOpaque { some_state }
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[no_mangle]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">extern</span> <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">mydata_create</span>(some_state: <span style="color:#66d9ef">i32</span>) -&gt; <span style="color:#f92672">*</span><span style="color:#66d9ef">mut</span> MyDataOpaque {
</span></span><span style="display:flex;"><span>  Box::into_raw(Box::new(MyDataOpaque::new(some_state)))
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[no_mangle]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">extern</span> <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">mydata_destroy</span>(d : <span style="color:#f92672">*</span><span style="color:#66d9ef">mut</span> MyDataOpaque) {
</span></span><span style="display:flex;"><span>  <span style="color:#75715e">// This causes the Drop to be called on the box, freeing everything.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>  <span style="color:#66d9ef">let</span> _ <span style="color:#f92672">=</span> <span style="color:#66d9ef">unsafe</span>{ Box::from_raw(d) };
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>It&rsquo;s a quite simple bit of code (for our simple example), but the crux of it is
just some opaque struct, and some unmangled public extern symbols for the
exported C API.</p>
<h2 id="building-the-two-crate-solution">Building the Two Crate Solution</h2>
<p>I wanted a way to build a Rust crate for the above, then build with cargo on a
second crate that uses it. I <strong>did not want</strong> to have a single instantiation of
cargo because I wanted to be 100% sure that I was exercising the export and
import paths of Rust. To do this I used a <code>build.rs</code> like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-rust" data-lang="rust"><span style="display:flex;"><span><span style="color:#66d9ef">use</span> std::env;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">use</span> std::process::Command;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">main</span>() {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">let</span> out_dir <span style="color:#f92672">=</span> env::var_os(<span style="color:#e6db74">&#34;OUT_DIR&#34;</span>).unwrap();
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">let</span> working_dir <span style="color:#f92672">=</span> env::var_os(<span style="color:#e6db74">&#34;CARGO_MANIFEST_DIR&#34;</span>).unwrap();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    Command::new(<span style="color:#e6db74">&#34;cargo&#34;</span>)
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#e6db74">&#34;build&#34;</span>)
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#e6db74">&#34;--manifest-path&#34;</span>)
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#f92672">&amp;</span>format!(
</span></span><span style="display:flex;"><span>            <span style="color:#e6db74">&#34;</span><span style="color:#e6db74">{}</span><span style="color:#e6db74">/MyDataOpaque/Cargo.toml&#34;</span>,
</span></span><span style="display:flex;"><span>            working_dir.to_str().unwrap()
</span></span><span style="display:flex;"><span>        ))
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#e6db74">&#34;--target-dir&#34;</span>)
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#f92672">&amp;</span>format!(<span style="color:#e6db74">&#34;</span><span style="color:#e6db74">{}</span><span style="color:#e6db74">&#34;</span>, out_dir.to_str().unwrap()))
</span></span><span style="display:flex;"><span>        .arg(<span style="color:#e6db74">&#34;--release&#34;</span>)
</span></span><span style="display:flex;"><span>        .status()
</span></span><span style="display:flex;"><span>        .unwrap();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    println!(
</span></span><span style="display:flex;"><span>        <span style="color:#e6db74">&#34;cargo:rustc-link-search=native=</span><span style="color:#e6db74">{}</span><span style="color:#e6db74">/release&#34;</span>,
</span></span><span style="display:flex;"><span>        out_dir.to_str().unwrap()
</span></span><span style="display:flex;"><span>    );
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>This build script calls cargo in a sub-folder called <code>MyDataOpaque</code>, builds the
crate, and then adds the built library as a link requirement to our crate.</p>
<h2 id="building-the-top-of-the-hourglass">Building the top of the Hourglass</h2>
<p>So now we&rsquo;ve got a Rust library, exposed via a C API, that we want to write Rust
bindings for.</p>
<p>Just to <em>prove</em> it worked I first tried using <code>cbindgen</code> to generate a C API for
the bottom of the hourglass, and then <code>bindgen</code> to take this C API and generate
FFI bindings for it in the top of the hourglass, but these have a ton of
dependencies (like libclang) that I wasn&rsquo;t happy about. So I instead decided to
just manually write the C API for the top of the hourglass.</p>
<blockquote>
<p>Aside: assuming this approach was viable you could forsee an <code>hourglass_bindgen</code>
crate that did something similar to what <code>cbindgen</code> does, but just writes out
the Rust FFI to the C API instead.</p>
</blockquote>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-rust" data-lang="rust"><span style="display:flex;"><span><span style="color:#75715e">#[repr(C)]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">struct</span> <span style="color:#a6e22e">MyDataOpaque</span> {
</span></span><span style="display:flex;"><span>    _private: [<span style="color:#66d9ef">u8</span>; <span style="color:#ae81ff">0</span>],
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[link(name = </span><span style="color:#e6db74">&#34;MyDataOpaque&#34;</span><span style="color:#75715e">)]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> <span style="color:#e6db74">&#34;C&#34;</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">mydata_create</span>(some_state: <span style="color:#66d9ef">i32</span>) -&gt; <span style="color:#f92672">*</span><span style="color:#66d9ef">mut</span> MyDataOpaque;
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">mydata_destroy</span>(opaque: <span style="color:#f92672">*</span><span style="color:#66d9ef">mut</span> MyDataOpaque);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">struct</span> <span style="color:#a6e22e">MyData</span> {
</span></span><span style="display:flex;"><span>    opaque: <span style="color:#f92672">*</span><span style="color:#66d9ef">mut</span> MyDataOpaque,
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">impl</span> MyData {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">pub</span> <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">new</span>(some_state: <span style="color:#66d9ef">i32</span>) -&gt; <span style="color:#a6e22e">MyData</span> {
</span></span><span style="display:flex;"><span>        MyData {
</span></span><span style="display:flex;"><span>            opaque: <span style="color:#a6e22e">unsafe</span> { mydata_create(some_state) },
</span></span><span style="display:flex;"><span>        }
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">impl</span> Drop <span style="color:#66d9ef">for</span> MyData {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">drop</span>(<span style="color:#f92672">&amp;</span><span style="color:#66d9ef">mut</span> self) {
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">unsafe</span> {
</span></span><span style="display:flex;"><span>            mydata_destroy(self.opaque);
</span></span><span style="display:flex;"><span>        }
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">#[cfg(test)]</span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">mod</span> tests {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">use</span> <span style="color:#66d9ef">super</span>::<span style="color:#f92672">*</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#75715e">#[test]</span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">fn</span> <span style="color:#a6e22e">create_destroy</span>() {
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">let</span> _ <span style="color:#f92672">=</span> MyData::new();
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>As we can see this code is very similar to the C++ example I showed at the
beginning of the post - it wraps a C API in a Rust struct and then can use the
existing safe mechanisms of Rust (like the <code>Drop</code> trait) to ensure safety of the
data used.</p>
<p>And if I run it?</p>
<pre tabindex="0"><code>running 1 test
test tests::create_destroy ... ok
</code></pre><p>It works, nice!</p>
<h2 id="conclusion">Conclusion</h2>
<p>Hourglass APIs appear to work in Rust (minus the caveats above that I&rsquo;ll have to
explore further). I do think that maybe using <code>no_std</code> in the bottom of the
hourglass could mitigate even issues with multiple versions of Rust being used,
but again I&rsquo;ll have to verify it.</p>
<p>You could even forsee of an approach where in the <code>build.rs</code> instead of building
some other crate (whose source you don&rsquo;t want to ship), you could use the
<code>TARGET</code> to fetch the correctly pre-built version from a web service, or maybe
have a bunch of the library versions resident in the crate (obviously the total
crate size limit might become an issue though), and pick between them.</p>
<p>It&rsquo;s pretty awesome to me that this approach looks feasible though!</p>


guid: https://www.neilhenning.dev/posts/aoc2020/
title: Advent of Code 2020
url: https://www.neilhenning.dev/posts/aoc2020/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;ve always wanted to do the
<a href="https://adventofcode.com/2020">Advent of Code 2020</a> puzzles as a way to wind
down in the run up to Christmas, but never quite got my act in gear to do it.</p>
<p>This year I&rsquo;m changing that, and I&rsquo;ve just pushed my 1st day&rsquo;s solutions to my
<a href="https://github.com/sheredom/AOC2020">AOC 2020 repository on GitHub</a>. Solution
parameters aside:</p>
<ul>
<li>I&rsquo;m doing this in Rust. I started writing it out in Zig but honestly the docs
are nowhere near as clear as with Rust, and the tooling isn&rsquo;t there. There is
a more general point here about how much effort it takes for languages to
achieve that status (and how <em>much</em> we have Mozilla to thank for doing Rust by
themselves for so blooming long to get it amazing!).</li>
<li>I&rsquo;ve just created a simple command line app that you can do <code>cargo run</code> and
it&rsquo;ll print the solution to the command line.</li>
<li>I&rsquo;m pretty sure there will be a more optimal way to do what I&rsquo;ve done, I&rsquo;m
really using this as a way to polish up my Rust skills some more!</li>
</ul>
<p>You can check out the <a href="https://github.com/sheredom/AOC2020">https://github.com/sheredom/AOC2020</a> if you are interested
in my progress, check in tomorrow for those solutions. Tioraidh bye!</p>


guid: https://www.neilhenning.dev/posts/aoc2020_in_review/
title: Advent of Code 2020 - Review
url: https://www.neilhenning.dev/posts/aoc2020_in_review/
feedurl: https://www.duskborn.com/index.xml
content:
<p>I&rsquo;ve always wanted to do the
<a href="https://adventofcode.com/2020">Advent of Code 2020</a> puzzles as a way to wind
down in the run up to Christmas, but never quite got my act in gear to do it.</p>
<p>This year I&rsquo;ve changed that, and I&rsquo;ve pushed all my solutions to my
<a href="https://github.com/sheredom/AOC2020">AOC 2020 repository on GitHub</a>. I thought
that now that I&rsquo;ve completed the challenge, I&rsquo;d do a review mostly of how I used
Rust to complete the challenges and what I found.</p>
<h2 id="rust-lessons-learned">Rust Lessons Learned</h2>
<h3 id="iterators">Iterators</h3>
<p>I really got more comfortable with iterator&rsquo;s and began to reach for <code>map</code>,
<code>fold</code>, <code>filter</code>, etc. the more I went into the puzzles. To begin with I stuck
to what effectively was a &lsquo;safe&rsquo; C-like coding experience - using mutable
variables and loops. The more I went into the challenge I began to use more Rust
things.</p>
<p>The one major downside I did find to iterators is that if they go wrong, you
start to want to debug the body of them and then the closure can become
frustrating. You either introduce a scope into the closure and bung the stuff in
there, or have a seperate method instead of a closure to aid in debugging. Even
still, I did feel many times that having a dumb C-like loop with mutable
variables ended up being a nicer experience when things went wrong.</p>
<h3 id="shadowing-isnt-always-evil">Shadowing Isn&rsquo;t Always Evil</h3>
<p>I&rsquo;ve always felt variable shadowing was an evil that should be stopped. Lets
take C for example here:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">int</span> <span style="color:#a6e22e">foo</span>(<span style="color:#66d9ef">int</span> a) {
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> (a <span style="color:#f92672">&lt;</span> <span style="color:#ae81ff">42</span>) {
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">int</span> a <span style="color:#f92672">=</span> <span style="color:#ae81ff">13</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">return</span> a;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>I really dislike in C that it lets you shadow a local variable with a previously
accessible variable - it just introduces the kind of problems like you see above
where the user <em>probably</em> wanted to return 13 if <code>a &lt; 42</code>, but they declared a
new variable instead. In C-like languages I like to force shadowing to be an
error as a result.</p>
<p>I came into Rust with this same view - shadowing is clearly bad language design!
Except that Rust is uniquely placed to make shadowing <em>sometimes</em> nice:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>fn <span style="color:#a6e22e">foo</span>(a: i32) <span style="color:#f92672">-&gt;</span> i32 {
</span></span><span style="display:flex;"><span>  let mut a <span style="color:#f92672">=</span> a;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span> a <span style="color:#f92672">&lt;</span> <span style="color:#ae81ff">42</span> {
</span></span><span style="display:flex;"><span>    a <span style="color:#f92672">=</span> <span style="color:#ae81ff">13</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  let a <span style="color:#f92672">=</span> a;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  a
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>In this example we can declare a variable mutable, shadow it with another
variable, and then <strong>assign</strong> the original variable to the new variable. All in
a single step! This lets us temporarily make the <code>a</code> we are using mutable to do
things with it. What&rsquo;s even cooler is we can then do the same in reverse after
we have decided <code>a</code> should no longer be mutable.</p>
<p>Before the Rust pedants come out the woodwork, the above should <em>really</em> be
something more like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>fn <span style="color:#a6e22e">foo</span>(a: i32) <span style="color:#f92672">-&gt;</span> i32 {
</span></span><span style="display:flex;"><span>  let a <span style="color:#f92672">=</span> <span style="color:#66d9ef">if</span> a <span style="color:#f92672">&lt;</span> <span style="color:#ae81ff">42</span> {
</span></span><span style="display:flex;"><span>    <span style="color:#ae81ff">13</span>
</span></span><span style="display:flex;"><span>  } <span style="color:#66d9ef">else</span> {
</span></span><span style="display:flex;"><span>    a
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  a
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>So we can avoid making <code>a</code> mutable at all (but I couldn&rsquo;t evangelise my point
without the much uglier code before!).</p>
<h3 id="its-easy-to-use-crates-maybe-too-easy">It&rsquo;s Easy to Use Crates (maybe too easy&hellip;)</h3>
<p>Using crates is <strong>ridiculously</strong> easy. The number of times I added a crate just
to try solving a problem using it, realised I didn&rsquo;t need it so I then removed
it, and moved on with my solution - was insane. I think that coming from C/C++
languages where I am most fluent has made me supremely dependency adverse, but
with Rust it&rsquo;s so trivial to just try a crate that you get sucked in.</p>
<p>I still think its a useful life skill to always attempt to avoid a dependency
where you might not need one, but if you are to use dependencies then Rust is
definitely what you want to be using for sure.</p>
<h2 id="crates-used">Crates Used</h2>
<h3 id="the-exec_time-crate">The exec_time crate</h3>
<p>For Day 1: Report Repair, I started by using the
<a href="https://crates.io/crates/exec_time">exec_time</a> crate, which I use throughout my
solutions. The crate just prints the time taken to execute whatever method the
included macro is placed on:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#[exec_time]
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>fn <span style="color:#a6e22e">run_something</span>() {}
</span></span></code></pre></div><p>Which prints the following to the command line:</p>
<pre tabindex="0"><code>Time run_something: 0 mills
</code></pre><p>The crate was a useful enough way to just dump the time taken in the function,
but there is some bug with mutable arguments that causes the macro to fail. I
worked around this by never passing mutable arguments, but its not a viable
crate to use to be honest.</p>
<h3 id="the-colour-crate">The colour crate</h3>
<p>I also used <a href="https://crates.io/crates/colour">colour</a> crate in my first solution
and all the solutions thereafter. It just lets you print to the command line
with colours, with an easy API:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>red_ln<span style="color:#f92672">!</span>(<span style="color:#e6db74">&#34;RED!&#34;</span>);
</span></span><span style="display:flex;"><span>green_ln<span style="color:#f92672">!</span>(<span style="color:#e6db74">&#34;GREEN!&#34;</span>);
</span></span></code></pre></div><p>I did this <em>solely</em> for festive reasons - Christmas colours are green, red, and
white - this meant that I printed each part 1 solution in red, each part 2
solution in green, and the <code>exec_time</code> crate above prints its timing info in
white text interpersed throughout - very festive!</p>
<h3 id="the-intbits-crate">The intbits crate</h3>
<p>For Day 3: Toboggan Trajectory, I used the
<a href="https://crates.io/crates/intbits">intbits</a> as part of this puzzle solution. The
crate just lets you access bits in an integer with a nice API:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>let mut i <span style="color:#f92672">=</span> <span style="color:#ae81ff">0</span>;
</span></span><span style="display:flex;"><span>i.<span style="color:#a6e22e">set_bit</span>(<span style="color:#ae81ff">4</span>, true);
</span></span><span style="display:flex;"><span>assert<span style="color:#f92672">!</span>(i.<span style="color:#a6e22e">bit</span>(<span style="color:#ae81ff">4</span>));
</span></span></code></pre></div><h3 id="the-bit_set-crate">The bit_set crate</h3>
<p>For Day 5: Binary Boarding, I wanted to use an N-wide bitset for this challenge,
to keep data used to a minimum again, and I went with the
<a href="https://crates.io/crates/bit-set">bit_set</a> crate:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>let mut seats <span style="color:#f92672">=</span> BitSet<span style="color:#f92672">::</span><span style="color:#a6e22e">with_capacity</span>(<span style="color:#ae81ff">1000</span>);
</span></span><span style="display:flex;"><span>seats.<span style="color:#a6e22e">insert</span>(<span style="color:#ae81ff">4</span>);
</span></span><span style="display:flex;"><span>assert<span style="color:#f92672">!</span>(seats.<span style="color:#a6e22e">contains</span>(<span style="color:#ae81ff">4</span>));
</span></span></code></pre></div><h3 id="the-itertools-crate">The itertools crate</h3>
<p>For Day 7: Handy Haversacks, I used the
<a href="https://crates.io/crates/itertools">itertools</a> crate which has a really nice
way to get tuples out of an iter:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#a6e22e">let</span> (lhs, rhs) <span style="color:#f92672">=</span> line.<span style="color:#a6e22e">splitn</span>(<span style="color:#ae81ff">2</span>, <span style="color:#e6db74">&#34;some seperator&#34;</span>).<span style="color:#a6e22e">next_tuple</span>().<span style="color:#a6e22e">unwrap</span>();
</span></span></code></pre></div><p>I ended up using this crate <em>everywhere</em> - so that I could use the <code>next_tuple</code>
method on the result of <code>splitn</code> to get a known tuple from the result. It was
very much overkill for what I actually needed, but super useful.</p>
<h3 id="the-num-integer-crate">The num-integer crate</h3>
<p>For Day 13: Shuttle Search I used the
<a href="https://crates.io/crates/num-integer">num-integer</a> crate solely to get access
to the least common multiplier (lcm) method:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>assert_eq<span style="color:#f92672">!</span>(<span style="color:#a6e22e">lcm</span>(<span style="color:#ae81ff">3</span>, <span style="color:#ae81ff">5</span>), <span style="color:#ae81ff">15</span>);
</span></span></code></pre></div><p>This was <em>probably</em> overkill (bringing in the whole crate), but I wanted to use
Rust in all its glory as much as possible.</p>
<h3 id="the-regex-crate">The regex crate</h3>
<p>I&rsquo;ve been told Advent of Code wouldn&rsquo;t be complete without using some form of
regex, and I hadn&rsquo;t actually used regex at all in Rust before. I used the
<a href="https://crates.io/crates/regex">regex</a> crate to accomplish what I needed, and
while I&rsquo;m no further forward in being better with regex than I was before, I did
find the implementation of it using Rust to be pleasant to use (especially when
you need to extract multiple bits from some complicated string).</p>
<h3 id="the-bit_reverse-crate">The bit_reverse crate</h3>
<p>I used the <a href="https://crates.io/crates/bit_reverse">bit_reverse</a> crate for my 20th
days solution - this was the problem I hated solving more than any other in this
years Advent of Code, and I used the bit reverse functionality to quickly get
the bits in an integer in the reverse order. This was a <em>really</em> messy solution
to the problem and honestly I&rsquo;m not that proud of it, but it <strong>did the job</strong>, my
code ran pretty fast, and it served a purpose.</p>
<h2 id="conclusion">Conclusion</h2>
<p>My major takeaway from Advent of Code is that it&rsquo;s a really great way to level
up your usage of a language you already know a little of.</p>
<p>Originally I was going to attempt to use the
<a href="https://ziglang.org/">Zig language</a> to solve the problems - but I knew
<em>nothing</em> of Zig&rsquo;s syntax and tooling, I found the docs a little confusing to
figure out (I was trying to work out how to load a file into a buffer!), and I
quickly realised that I just <strong>wouldn&rsquo;t</strong> complete the Advent of Code challenge
while learning an entire new language and trying to work out the solutions.</p>
<p>With Rust I had been experimenting with things already and knew enough to be
dangerous, but not enough to feel comfortable with it. Advent of Code was
<strong>tremendously</strong> useful for reinforcing a thin level of knowledge of a language
with a more robust appreciation of its quirks.</p>
<p>With that - I&rsquo;m calling time on 2020, tha bliadhna ùr mo chàraidean, and I&rsquo;ll
see you in the new year!</p>


guid: https://www.neilhenning.dev/posts/how-i-structure-my-working-day/
title: How I Structure My Working Day
url: https://www.neilhenning.dev/posts/how-i-structure-my-working-day/
feedurl: https://www.duskborn.com/index.xml
content:
<p>The more I&rsquo;ve delved into the darkest corners of optimizing compilers, the more
that incessant urge to optimize things has leeched out into my day to day life.
I&rsquo;ve found myself ever more obsessed over what the most optimal method to do
things is. A few examples:</p>
<ul>
<li>While brushing my teeth twice a day I know I have a full two minutes where I
can make use of my spare hand to do things. Like I might pack the dishwasher,
lock the doors, turn off lights, put stuff into the recycling - anything that
means I&rsquo;ve double buffered another activity while I&rsquo;m stuck brushing.</li>
<li>I got really obsessed with the number of trips from the dishwasher to put
things away, versus the time taken to load up on extra dishes for a lower
number of trips. I kid you not I&rsquo;ve experimented and mostly realised that (if
optimizing for time taken as your primary metric) the number of trips is much
less important than the time taken to load/unload dishes in an individual
trip.</li>
</ul>
<p>One other area of my life where I&rsquo;ve adopted this philosophy is the structure of
my workday. I thought it might be interesting for people to see if there is any
tips in how I lay out my day that they could adopt to optimize their own
experiences also.</p>
<h2 id="booting-up">Booting Up</h2>
<p>The first step in any day is the process of booting up. I follow the same
routine each morning:</p>
<ul>
<li>I wake up.</li>
<li>I use the facilities.</li>
<li>I weigh myself.</li>
<li>I take my dog outside so he can relieve himself.</li>
<li>I put on my running gear.</li>
<li>I make a coffee. Caffeinated. I only have a single with-caffeine coffee each
day as I&rsquo;ve experimented and found it to be the optimal balance of getting
that &lsquo;wake up&rsquo; feeling versus the affect it can have on sleeping at the other
end of the day.</li>
</ul>
<p>Once I&rsquo;ve performed these most basic tasks in a mostly pre-coffee zombie state,
I&rsquo;m ready to load up on the problems of the day.</p>
<h2 id="dealing-with-the-urgents">Dealing with the Urgents</h2>
<p>Once I&rsquo;m at the computer I do a full parse of slack / work email, answering
any of the easy questions and dealing with the easier tasks.</p>
<p>Then I tend to filter through to see if there are any urgent bugs that I need to
take care of. These bugs are generally of the &lsquo;stop the world&rsquo; kind - EG. I&rsquo;d
only drop whatever main feature I&rsquo;m working on for a bug if it is truly some
sort of regression / ship stopper. My number one aim for any of these bugs is to
reproduce them (not always easy) and then give the user a temporary workaround.</p>
<p>Because of the natural cadence of software releases it means that even if I rush
to fix a users issue, there is still some time lag between me fixing it and them
being able to get access to that fix (while everything is tested for release).
Workarounds are a key part to unblock a user while the real fix is deployed.</p>
<h2 id="urgent-bug-or-feature">Urgent Bug or Feature</h2>
<p>Once I&rsquo;ve loaded up on a specific urgent bug to fix, or a feature I am
developing, I spend thirty minutes or so acclimatizing myself to the issue.</p>
<p>If it&rsquo;s a bug - can I reproduce it? Can I check our logs? Can I use a debug build
of our Burst compiler to give me more information? Generally at this stage I&rsquo;m
only trying to work out what is the <em>cause</em> not what the fix is.</p>
<p>If it&rsquo;s a feature - I&rsquo;m just paging back in what I was doing the day before and
trying to get started up again. I have long followed a philosophy of lacing
<code>#error</code>&rsquo;s throughout my code before I leave it for the night so that the next
day it won&rsquo;t compile but I&rsquo;ve left some breadcrumbs for where my thinking was
the night before. Again - all I&rsquo;m trying to do here is page back in all the
problems I was trying to solve at this stage, not do any actual work.</p>
<h2 id="run-run-run">Run Run Run</h2>
<p>Then I go for a run. I grab my dog (he runs with me every day!) grab my
headphones, and listen to any of my favourite podcasts while I&rsquo;m exercising. I
do at least a three mile run every day, and try to run at least thirty miles a
week. What is important at this stage is to get into the monotony of the
exercise, don&rsquo;t think about work, enjoy the podcast and get a sweat on.</p>

<img src="/images/BenjiAndMeRunning2.jpg"  alt="Benji and myself running"  class="center"  style="border-radius: 8px;"    />


<p>There is now a pretty hefty set of scientific evidence that states that the
<em>right kind</em> of distraction from a problem actually helps you solve it (for
instance <a href="https://www.bbc.com/worklife/article/20180605-why-being-easily-distracted-can-be-a-very-good-thing">here is a BBC article about just that</a>.
The TL;DR is that as long as you have some monotonous task like sorting things
(sorting lego blocks by colour, putting the dishes away, hanging out washing on
a washing line, etc) or more broadly where your brain can mostly power off from
any in-focus intensive task, your brain kicks off an asynchronous process in
your head to actually solve the problem while you are doing the thing. This is
why you hear <em>so many stories</em> of people solving problems in the shower, or
waking up with a solution - you are using the same technique.</p>
<p>I&rsquo;ve found myself that by loading up on the problems and then immediately
context switching to exercise, I generally solve them better and faster. I could
sit at the screen for hours and get no-where, or I could go for an hour run and
solve it in the thirty or so minutes after I sit back down. Nearly every time I
<em>engage the crunch like behaviour</em> of trying to power through to a solution it
sets me back on actually solving it - so this technique is something I&rsquo;d
definitely recommend trying.</p>
<h2 id="the-rest-of-my-day">The Rest of My Day</h2>
<p>I get about four or five good hours in a day of active problem solving coding. It
might be more or less for other people, but aslong as I can get some peace, have
my favourite music blaring, and can distract myself occassionally to make a
decaf coffee or throw the frisbee to Benji, I&rsquo;ll get a solid block of good work
done in an average day.</p>

<img src="/images/Benji.jpg"  alt="Benji with a frisbee in his mouth"  class="center"  style="border-radius: 8px;"    />


<p>Since the my team is split mostly between EU / US, I generally have most of my big
meetings in my afternoon. So by that time I&rsquo;m already pretty tired from the
solid work block, and ready to engage in the meeting instead.</p>
<p>When I&rsquo;m done for the night? I&rsquo;m done. I work my eight hours a day, and walk
away from the machine. I like to cook so I generally make dinner as the bridge
between work Neil and home Neil (even though they inhabit the same house!).</p>


guid: https://www.neilhenning.dev/posts/early-2021-mega-update/
title: Early 2021 Mega Update
url: https://www.neilhenning.dev/posts/early-2021-mega-update/
feedurl: https://www.duskborn.com/index.xml
content:
<p>The last four years of my coding life have been incredibly stressful. In this
time I&rsquo;ve:</p>
<ul>
<li>Started a job at AMD in the GPU compiler team.</li>
<li>Left a job at AMD for Unity in the Burst compiler team.</li>
<li>Wrapped up my involvement as a creator of Vulkan and SPIR-V.</li>
<li>Learnt the C# and Rust programming languages.</li>
</ul>
<p>You couple this with the absolute trainwreck that was 2020 (COVID yay) and now
it looks like 2021 might be mostly a write-off too, and the fact that my dog
Benji has progressively worsening epilepsy that is horrific to witness - it is
safe to say that I&rsquo;m probably at the <strong>highest level</strong> of stress I&rsquo;ve ever been
under.</p>
<p>I do realise that I&rsquo;m in a very fortunate position nonetheless though - I have a
good and stable job at Unity, I live on a remote island with a big garden and
lots of places I can run while maintaining an easy social distancing. I know a
lot of people have it much worse - very much a first-world-problem level of
stress.</p>
<p>But one thing that has been nagging me for, and I admit with a good dose of
shame, this entire time was that the bug count on my open source C libraries had
creeped up to a level of real annoyance.</p>
<p>These issues could be broke up into a few categories:</p>
<ul>
<li>Later Clang/GCC would add new warnings that my libraries would fail on (I like
to ensure that all the libraries can compile with the full shebang of
<code>-Werror -Wall -Wextra</code>).</li>
<li>I didn&rsquo;t support clang-cl.exe on Windows at all (my libraries assumed that you
were using any of the Visual Studio compilers from 2008 to 2019).</li>
<li>There were a raft of general bugs people had found.</li>
<li>And a few quality of life feature requests thrown in there too.</li>
</ul>
<p>So in this early part of 2021 I&rsquo;ve went on a pretty severe gutting of my
libraries to fix the bugs, implement the features, test the compilers that I
didn&rsquo;t previously test, and generally just get my issue list as close to zero as
possible.</p>
<p>The libraries that I put all the effort into are the following six:</p>
<ul>
<li>🧪 <a href="https://github.com/sheredom/utest.h">utest.h</a> - a single header unit
testing framework that supports C and C++.</li>
<li>⏱️ <a href="https://github.com/sheredom/ubench.h">ubench.h</a> - a single header unit
benchmarking framework that supports C and C++.</li>
<li>📚 <a href="https://github.com/sheredom/utf8.h">utf8.h</a> - a single header UTF-8 string
supporting header for C and C++.</li>
<li>🗄️ <a href="https://github.com/sheredom/json.h">json.h</a> - a single header JSON parsing
and writing header that supports JSON5 and simplified-JSON, and works with C and C++.</li>
<li>🐜 <a href="https://github.com/sheredom/subprocess.h">subprocess.h</a> - a single header
process spawning, joining, and interacting with library for C and C++.</li>
<li>#️⃣ <a href="https://github.com/sheredom/hashmap.h">hashmap.h</a> - a single header
hashmap implementation for C and C++.</li>
</ul>
<h2 id="github-actions">GitHub Actions</h2>
<p>To fix that I wasn&rsquo;t testing the latest Clang and GCC I decided to finally pull
the trigger on using GitHub Actions instead of my old Travis pipelines. Travis
was starting to have some integration issues (I don&rsquo;t know the details, they
just suddenly stopped appearing on the PRs when I pushed changes), and since I
knew I wanted to end up on GitHub Actions anyway I used this as the reason to
just do it.</p>
<p>GitHub Actions was a little daunting at first, but once you get into it the yml
format is very powerful. For example here is the
<a href="https://github.com/sheredom/subprocess.h/blob/master/.github/workflows/cmake.yml">workflow of my subprocess.h library</a>.</p>
<p>The most interesting parts are how I upgraded to the latest clang/gcc:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sh" data-lang="sh"><span style="display:flex;"><span>- name: Setup dependencies
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span>: startsWith<span style="color:#f92672">(</span>matrix.os, <span style="color:#e6db74">&#39;ubuntu&#39;</span><span style="color:#f92672">)</span>
</span></span><span style="display:flex;"><span>  run: sudo apt-get install -y gcc-10 g++-10 clang-10
</span></span></code></pre></div><p>And then how I used these configurations:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sh" data-lang="sh"><span style="display:flex;"><span>- name: Configure CMake with GCC
</span></span><span style="display:flex;"><span>  shell: bash
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span>: matrix.compiler <span style="color:#f92672">==</span> <span style="color:#e6db74">&#39;gcc&#39;</span>
</span></span><span style="display:flex;"><span>  working-directory: <span style="color:#e6db74">${</span>{github.workspace<span style="color:#e6db74">}</span><span style="color:#f92672">}</span>/build
</span></span><span style="display:flex;"><span>  run: cmake $GITHUB_WORKSPACE/test -DCMAKE_BUILD_TYPE<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>{ matrix.type <span style="color:#e6db74">}</span><span style="color:#f92672">}</span> -DCMAKE_C_COMPILER<span style="color:#f92672">=</span>gcc-10 -DCMAKE_CXX_COMPILER<span style="color:#f92672">=</span>g++-10
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>- name: Configure CMake with Clang <span style="color:#f92672">(</span>Ubuntu<span style="color:#f92672">)</span>
</span></span><span style="display:flex;"><span>  shell: bash
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span>: <span style="color:#f92672">(</span>matrix.compiler <span style="color:#f92672">==</span> <span style="color:#e6db74">&#39;clang&#39;</span><span style="color:#f92672">)</span> <span style="color:#f92672">&amp;&amp;</span> startsWith<span style="color:#f92672">(</span>matrix.os, <span style="color:#e6db74">&#39;ubuntu&#39;</span><span style="color:#f92672">)</span>
</span></span><span style="display:flex;"><span>  working-directory: <span style="color:#e6db74">${</span>{github.workspace<span style="color:#e6db74">}</span><span style="color:#f92672">}</span>/build
</span></span><span style="display:flex;"><span>  run: cmake $GITHUB_WORKSPACE/test -DCMAKE_BUILD_TYPE<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>{ matrix.type <span style="color:#e6db74">}</span><span style="color:#f92672">}</span> -DCMAKE_C_COMPILER<span style="color:#f92672">=</span>clang-10 -DCMAKE_CXX_COMPILER<span style="color:#f92672">=</span>clang++-10
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>- name: Configure CMake with Clang <span style="color:#f92672">(</span>Windows<span style="color:#f92672">)</span>
</span></span><span style="display:flex;"><span>  shell: bash
</span></span><span style="display:flex;"><span>  <span style="color:#66d9ef">if</span>: <span style="color:#f92672">(</span>matrix.compiler <span style="color:#f92672">==</span> <span style="color:#e6db74">&#39;clang&#39;</span><span style="color:#f92672">)</span> <span style="color:#f92672">&amp;&amp;</span> startsWith<span style="color:#f92672">(</span>matrix.os, <span style="color:#e6db74">&#39;windows&#39;</span><span style="color:#f92672">)</span>
</span></span><span style="display:flex;"><span>  working-directory: <span style="color:#e6db74">${</span>{github.workspace<span style="color:#e6db74">}</span><span style="color:#f92672">}</span>/build
</span></span><span style="display:flex;"><span>  run: cmake $GITHUB_WORKSPACE/test -DCMAKE_BUILD_TYPE<span style="color:#f92672">=</span><span style="color:#e6db74">${</span>{ matrix.type <span style="color:#e6db74">}</span><span style="color:#f92672">}</span> -T ClangCL
</span></span></code></pre></div><p>My build matrix is pretty extensive too:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sh" data-lang="sh"><span style="display:flex;"><span>strategy:
</span></span><span style="display:flex;"><span>  matrix:
</span></span><span style="display:flex;"><span>    os: <span style="color:#f92672">[</span>ubuntu-latest, windows-latest, macOS-latest<span style="color:#f92672">]</span>
</span></span><span style="display:flex;"><span>    type: <span style="color:#f92672">[</span>Debug, RelWithDebInfo, MinSizeRel, Release<span style="color:#f92672">]</span>
</span></span><span style="display:flex;"><span>    compiler: <span style="color:#f92672">[</span>default, clang, gcc<span style="color:#f92672">]</span>
</span></span><span style="display:flex;"><span>    exclude:
</span></span><span style="display:flex;"><span>      - <span style="color:#f92672">{</span>os: <span style="color:#e6db74">&#34;macOS-latest&#34;</span>, compiler: <span style="color:#e6db74">&#34;clang&#34;</span><span style="color:#f92672">}</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#f92672">{</span>os: <span style="color:#e6db74">&#34;windows-latest&#34;</span>, compiler: <span style="color:#e6db74">&#34;gcc&#34;</span><span style="color:#f92672">}</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#f92672">{</span>os: <span style="color:#e6db74">&#34;macOS-latest&#34;</span>, compiler: <span style="color:#e6db74">&#34;gcc&#34;</span><span style="color:#f92672">}</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#f92672">{</span>os: <span style="color:#e6db74">&#34;ubuntu-latest&#34;</span>, compiler: <span style="color:#e6db74">&#34;default&#34;</span><span style="color:#f92672">}</span>
</span></span><span style="display:flex;"><span>      - <span style="color:#f92672">{</span>os: <span style="color:#e6db74">&#34;ubuntu-latest&#34;</span>, compiler: <span style="color:#e6db74">&#34;default&#34;</span><span style="color:#f92672">}</span>
</span></span><span style="display:flex;"><span>runs-on: <span style="color:#e6db74">${</span>{ matrix.os <span style="color:#e6db74">}</span><span style="color:#f92672">}</span>
</span></span></code></pre></div><p>And all in all this lets me build for the three desktop platforms, using five
different compiler toolchains.</p>
<p>Even though this let me ditch Travis I couldn&rsquo;t ditch appveyor in the process. I
would have liked to have a single CI integration as that&rsquo;d have made maintaining
everything easier, but GitHub Actions does not allow you to test with older
Visual Studio installations.</p>
<p>Quite a few of my users love the older Visual Studio&rsquo;s and won&rsquo;t upgrade under
any circumstances, and so I&rsquo;ve
<a href="https://github.com/sheredom/subprocess.h/blob/master/appveyor.yml">kept appveyor around</a>
to ensure this path is tested.</p>
<h2 id="utesth">utest.h</h2>
<p>In <a href="https://github.com/sheredom/utest.h">utest.h</a> I removed the
<code>#include &lt;winbase.h&gt;</code> entirely. This meant that the huge headers that were
pulled in as part of including utest.h on windows were no longer required, which
sped up compile time on the Windows platform.</p>
<h2 id="utf8h">utf8.h</h2>
<p>In <a href="https://github.com/sheredom/utf8.h">utf8.h</a> I added some new helper
functions <code>utf8makevalid</code> and <code>utf8rcodepoint</code> - allowing you to sanitize a
potentially invalid UTF-8 string by replacing malformed codepoints with an ASCII
replacement, and allowing you to reverse iterate through the string.</p>
<p>I also fixed a long standing bug whereby the Greek Capital Theta symbol &lsquo;ϴ&rsquo; does
not have a uniform mapping from upper to lower case, or vice-versa. This means
that if you did the dance of doing upper -&gt; lower -&gt; upper you <em>could</em> have a
different string than you started with (but the fix means I correctly handle
that letter&rsquo;s encoding).</p>
<h2 id="jsonh">json.h</h2>
<p>In <a href="https://github.com/sheredom/json.h">json.h</a> I added all the fun JSON tests
from the <a href="https://github.com/nst/JSONTestSuite">JSONTestSuite</a> project, and
fixed any and all the corner case bugs this uncovered. Mostly I was incorrectly
finding an end-of-file and assuming the JSON to be valid (without checking that
strings/objects/arrays were actually closed correctly). I also took the
opportunity to add clang sanitizer tests as since json.h uses a single
allocation to handle the entire JSON DOM, there was a greater chance of trashing
problems. Luckily there are zero issues with the library under its current
testing load - nice!</p>
<p>I also added a new function <code>json_extract_value</code> which lets you pull out a value
from a JSON DOM into a new allocation. This is useful if you want to get a bit
of a DOM and do something with just that (you could free the original allocation
for instance).</p>
<h2 id="subprocessh">subprocess.h</h2>
<p>In <a href="https://github.com/sheredom/subprocess.h">subprocess.h</a> I fixed two fun
issues:</p>
<ul>
<li>On Linux if you did not specify the <code>subprocess_option_inherit_environment</code>
option then you had to have a qualified path to the executable (either a full
path with the starting &lsquo;/&rsquo;, or a relative path like &lsquo;./&rsquo;).</li>
<li>I also added a new entry-point <code>subprocess_create_ex</code> that lets users specify
a custom environment to spawn the subprocess with.</li>
</ul>
<h2 id="hashmaph">hashmap.h</h2>
<p>In <a href="https://github.com/sheredom/hashmap.h">hashmap.h</a> I fixed a bug where if you
had three entries that mapped to the same hash, and therefore used the linear
probe part of hashing to find a location. If you removed the middle entry, and
then tried to reinsert the third, it&rsquo;d wrongly make two copies of that entry in
the hash table. The test for this shows the problem nicely:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> hashmap_s hashmap;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">int</span> x <span style="color:#f92672">=</span> <span style="color:#ae81ff">42</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">int</span> y <span style="color:#f92672">=</span> <span style="color:#ae81ff">13</span>;
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">int</span> z <span style="color:#f92672">=</span> <span style="color:#f92672">-</span><span style="color:#ae81ff">53</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_create</span>(<span style="color:#ae81ff">4</span>, <span style="color:#f92672">&amp;</span>hashmap));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// These all hash to the same value.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#f92672">&amp;</span>hashmap, <span style="color:#e6db74">&#34;000&#34;</span>, <span style="color:#ae81ff">3</span>, <span style="color:#f92672">&amp;</span>x));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#f92672">&amp;</span>hashmap, <span style="color:#e6db74">&#34;002&#34;</span>, <span style="color:#ae81ff">3</span>, <span style="color:#f92672">&amp;</span>y));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#f92672">&amp;</span>hashmap, <span style="color:#e6db74">&#34;007&#34;</span>, <span style="color:#ae81ff">3</span>, <span style="color:#f92672">&amp;</span>z));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">3u</span>, <span style="color:#a6e22e">hashmap_num_entries</span>(<span style="color:#f92672">&amp;</span>hashmap));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// Now we remove the middle value.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_remove</span>(<span style="color:#f92672">&amp;</span>hashmap, <span style="color:#e6db74">&#34;002&#34;</span>, <span style="color:#ae81ff">3</span>));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">2u</span>, <span style="color:#a6e22e">hashmap_num_entries</span>(<span style="color:#f92672">&amp;</span>hashmap));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#75715e">// And now attempt to insert the last value again. There was a bug where this
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// would insert a new entry incorrectly instead of resolving to the previous
</span></span></span><span style="display:flex;"><span><span style="color:#75715e">// entry.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">0</span>, <span style="color:#a6e22e">hashmap_put</span>(<span style="color:#f92672">&amp;</span>hashmap, <span style="color:#e6db74">&#34;007&#34;</span>, <span style="color:#ae81ff">3</span>, <span style="color:#f92672">&amp;</span>z));
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">ASSERT_EQ</span>(<span style="color:#ae81ff">2u</span>, <span style="color:#a6e22e">hashmap_num_entries</span>(<span style="color:#f92672">&amp;</span>hashmap));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">hashmap_destroy</span>(<span style="color:#f92672">&amp;</span>hashmap);
</span></span></code></pre></div><p>I also added a new function <code>hashmap_remove_and_return_key</code> which lets you
remove and item and also return the key. Remember that hashmap.h doesn&rsquo;t copy
the memory backing the key, so if you had to allocate memory for the key and
wanted to get the key back so that you could free this memory - this is the
function for you!</p>
<h2 id="github-sponsorship">GitHub Sponsorship</h2>
<p>First off - I <em>don&rsquo;t really need the money from GitHub sponsorship</em>. I&rsquo;ve got a
good job at Unity and my personal projects in C/C++ and Rust are simple labours
of love.</p>
<p>But a few people have asked if I would add a sponsorship option to my GitHub
anyway, because they like my libraries and use them.</p>
<p>So I&rsquo;ve <a href="https://github.com/sponsors/sheredom">added a sponsorship option</a> to my
GitHub for anyone that wants to help me justify the time to keep my libraries
updated, bug free, and great.</p>


guid: https://www.neilhenning.dev/posts/introducing-envcache/
title: Introducing envcache
url: https://www.neilhenning.dev/posts/introducing-envcache/
feedurl: https://www.duskborn.com/index.xml
content:
<p>One thing that has always bugged me with Rust is when you are interacting with
complex non-Rust software. Yes LLVM, I&rsquo;m looking at you! LLVM nearly always
requires you to either a) have <code>llvm-config</code> on the path already, or b) find an
LLVM install in some random place on the filesystem. I generally like to have a
version of LLVM built myself (using <code>RelWithDebInfo</code> and forcing asserts on)
that will naturally live at some arbitrary point in the filesystem - and then I
want to <em>somehow</em> point my Rust code that is using LLVM at that folder.</p>
<p>The general way to do this in Rust is to use an environment variable, and then
in a <code>build.rs</code> you can grab all the paths to the LLVM libraries for linking in.</p>
<p>For instance the <a href="https://crates.io/crates/llvm-sys">llvm-sys</a> crate has an
optional feature <code>no-llvm-linking</code> that lets you control where LLVM is sourced
from. I use this in a project I&rsquo;m cooking up to control how LLVM is brought into
my project. At present my workflow is something like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-sh" data-lang="sh"><span style="display:flex;"><span>LLVM_INSTALL_DIR<span style="color:#f92672">=</span>llvm-install cargo test
</span></span><span style="display:flex;"><span>cargo clippy
</span></span></code></pre></div><p>If you can see the problem already - when I run <em>other</em> commands on auto-pilot
like running <code>cargo clippy</code>, and I <strong>do not</strong> specify the <code>LLVM_INSTALL_DIR</code>,
the command fails to run because it requires <code>LLVM_INSTALL_DIR</code> to be specified.</p>
<p>Coming from a primarily CMake based workflow with my day-to-day job working with
LLVM, this completely does my head in. CMake has a way to cache variables and
then subsequent commands re-use the cached value. So I got thinking - why can&rsquo;t
I do something similar in Rust?</p>
<p>I&rsquo;d like to introduce a simple, dumb, but useful little crate -
<a href="https://crates.io/crates/envcache">envcache</a>. This crate lets you cache any
environment variables you want so they can be used across cargo commands. For
example, in a <code>build.rs</code>:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">extern</span> crate envcache;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>use envcache<span style="color:#f92672">::</span>EnvCache;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>fn <span style="color:#a6e22e">main</span>() {
</span></span><span style="display:flex;"><span>    let mut envcache <span style="color:#f92672">=</span> EnvCache<span style="color:#f92672">::</span><span style="color:#a6e22e">new</span>();
</span></span><span style="display:flex;"><span>    envcache.<span style="color:#a6e22e">cache</span>(<span style="color:#e6db74">&#34;LLVM_INSTALL_DIR&#34;</span>);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>By doing this simple thing above, it&rsquo;ll let me specify <code>LLVM_INSTALL_DIR</code> once,
and then have this cached value used thereafter for subsequent commands.</p>
<p>This works by saving a file in the <code>OUT_DIR</code> that you can write to with
<code>build.rs</code> - <code>envcache.config</code>. Crucially this means that if you run
<code>cargo clean</code> it&rsquo;ll wipe out the cache, and you&rsquo;ll need to re-specify the
environment variable. But it does mean you have to specify the environment
variable much less in general development which I find to be much nicer.</p>
<p>The source code is available <a href="https://github.com/sheredom/envcache">here</a>, and
I hope this is useful to some of you.</p>


guid: https://www.neilhenning.dev/posts/llvm-new-pass-manager/
title: Porting Burst to the New LLVM Pass Manager
url: https://www.neilhenning.dev/posts/llvm-new-pass-manager/
feedurl: https://www.duskborn.com/index.xml
content:
<p>Last week the Burst team shook up our day-to-day schedules with the opposite of a Unity classic - the hackweek. Unity has a strong culture around taking a week or two a year (once with the whole company, maybe once with your team), to hack on something that you think might be an interesting avenue for the future of Unity. These projects have often been spun out into products or parts of products in their own right.</p>
<p>But what about the opposite of hacking - what about spending some time consolidating what is currently there and working on the things that you really think will benefit the product over all, but haven&rsquo;t had the time to work on. We did exactly that last week.</p>
<p>One thing that has been bugging me since LLVM 12 landed was that we were still using the <em>legacy pass manager</em>, which was the default optimization pipeline for LLVM for every release prior to LLVM 12. But in LLVM 12 they finally, after about 7 years since I first heard about the new pass manager, pulled the trigger and switched the default pipeline over. Since the legacy pass manager is still in the codebase (it is still actually being used at present to power the backend infrastructure), we didn&rsquo;t have to use the new one with Burst. But the writing is clearly on the wall for the legacy pass manager, and I&rsquo;ve heard and read claims that the new pass manager does some things better than the legacy one, maybe meaning faster compilation or better codegen.</p>
<p>So TL;DR I&rsquo;ve added the new pass manager support to Burst, and here is a brain dump of how I went about it incase there is anyone else wanting to attempt the feat - hopefully you can avoid a few of the landmines I trod on!</p>
<h2 id="porting-over-a-pass">Porting Over a Pass</h2>
<p>The first thing is - how do you port a pass over from the legacy pass manager to the new one. Keep in mind that our code with Burst has to support LLVM 8..12 at present (for various platform reasons), so all my passes <strong>must</strong> work with the legacy pass manager and the new one concurrently, in the same codebase. I&rsquo;d argue that you should do this approach anyway even if you want to abandon the legacy pass manager because:</p>
<ul>
<li>While porting both the legacy and the new pass manager code should work in the same codebase, letting you A/B test the changes for any regressions.</li>
<li>Also lets you do performance profiling between the pass managers too.</li>
<li>Even after landing you might yet find some corner case in the new pass manager that is broken and need to switch back to the old approach.</li>
</ul>
<p>So let&rsquo;s take one of our custom passes in Burst - a pass that does a much more extensive job of removing dead loops from the code. We need this pass because we found a bunch of cases where vectorization was blocked by a dead inner loop. The pass structure looked like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> DeadLoopRemovalPass final : public llvm<span style="color:#f92672">::</span>FunctionPass
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">DeadLoopRemovalPass</span>() <span style="color:#f92672">:</span> llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">FunctionPass</span>(ID) {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">bool</span> <span style="color:#a6e22e">runOnFunction</span>(llvm<span style="color:#f92672">::</span>Function<span style="color:#f92672">&amp;</span> function) override;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">void</span> <span style="color:#a6e22e">getAnalysisUsage</span>(llvm<span style="color:#f92672">::</span>AnalysisUsage<span style="color:#f92672">&amp;</span> analysisUsage) <span style="color:#66d9ef">const</span> override;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    llvm<span style="color:#f92672">::</span>StringRef <span style="color:#a6e22e">getPassName</span>() <span style="color:#66d9ef">const</span> override { <span style="color:#66d9ef">return</span> <span style="color:#e6db74">&#34;Burst Dead Loop Removal&#34;</span>; }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">static</span> <span style="color:#66d9ef">char</span> ID;
</span></span><span style="display:flex;"><span>};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">char</span> DeadLoopRemovalPass<span style="color:#f92672">::</span>ID;
</span></span></code></pre></div><p>So I first looked at how LLVM did the dual support for legacy/new pass managers. Their general overall approach is one of two ways:</p>
<ul>
<li>Either they have a wrapper around the new pass manager structure, providing the same code to the legacy pass manager.</li>
<li>Or they have a shared class that both the legacy/new pass managers will use.</li>
</ul>
<p>I decided to use the latter approach, for no reason other than it seemed easier to port over to.</p>
<h3 id="the-implementation-struct">The Implementation Struct</h3>
<p>To do this I first removed the dependency of the struct above on the legacy <code>llvm::FunctionPass</code>, and made it a standalone struct:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> DeadLoopRemoval final
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    explicit <span style="color:#a6e22e">DeadLoopRemoval</span>() {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">bool</span> <span style="color:#a6e22e">run</span>(
</span></span><span style="display:flex;"><span>        llvm<span style="color:#f92672">::</span>Function<span style="color:#f92672">&amp;</span> function,
</span></span><span style="display:flex;"><span>        std<span style="color:#f92672">::</span>function<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopInfo<span style="color:#f92672">&amp;</span> ()<span style="color:#f92672">&gt;</span>,
</span></span><span style="display:flex;"><span>        std<span style="color:#f92672">::</span>function<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>DominatorTree<span style="color:#f92672">*</span> ()<span style="color:#f92672">&gt;</span>,
</span></span><span style="display:flex;"><span>        std<span style="color:#f92672">::</span>function<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>PostDominatorTree<span style="color:#f92672">*</span> ()<span style="color:#f92672">&gt;</span>) <span style="color:#66d9ef">const</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">static</span> llvm<span style="color:#f92672">::</span>StringRef <span style="color:#a6e22e">getPassName</span>() { <span style="color:#66d9ef">return</span> <span style="color:#e6db74">&#34;Burst Dead Loop Removal&#34;</span>; }
</span></span><span style="display:flex;"><span>};
</span></span></code></pre></div><p>Which entailed:</p>
<ul>
<li>Removing the inherit from <code>llvm::FunctionPass</code>.</li>
<li>Removing the <code>static char ID</code>.</li>
<li>Morphing <code>runOnFunction</code> -&gt; <code>run</code>, and passing in callbacks to retrieve the analysis information I required to implement the pass. I changed the name from <code>runOnFunction</code> -&gt; <code>run</code> just so that it gave me compile errors at the callsite as I was porting so I could find everything. Also note the <code>override</code> disappears from the function because we have no abstract virtual function to overload.</li>
<li>Changed <code>getPassName()</code> to be a static function and removed the <code>override</code>. The new pass manager uses a static pass name retriever whereas the old used a virtual member function - so this allows us to support both.</li>
</ul>
<h3 id="the-legacy-wrapping-pass">The Legacy Wrapping Pass</h3>
<p>Then before I even touched the new pass manager, I implemented the original pass for the legacy pass manager again, but making it just wrap the logic of the pass from the new <code>DeadLoopRemoval</code> struct:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> DeadLoopRemovalLegacyPass final : public llvm<span style="color:#f92672">::</span>FunctionPass
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">DeadLoopRemovalLegacyPass</span>() <span style="color:#f92672">:</span> llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">FunctionPass</span>(ID) {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">bool</span> <span style="color:#a6e22e">runOnFunction</span>(llvm<span style="color:#f92672">::</span>Function<span style="color:#f92672">&amp;</span> function) override
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">if</span> (<span style="color:#a6e22e">skipFunction</span>(function))
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> false;
</span></span><span style="display:flex;"><span>        }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getLI <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>LoopInfo<span style="color:#f92672">&amp;</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> getAnalysis<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopInfoWrapperPass<span style="color:#f92672">&gt;</span>().<span style="color:#a6e22e">getLoopInfo</span>();
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getDT <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>DominatorTree<span style="color:#f92672">*</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            llvm<span style="color:#f92672">::</span>DominatorTreeWrapperPass<span style="color:#f92672">*</span> <span style="color:#66d9ef">const</span> dominatorTreeWrapperPass <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>                getAnalysisIfAvailable<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>DominatorTreeWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> dominatorTreeWrapperPass <span style="color:#f92672">?</span>
</span></span><span style="display:flex;"><span>                <span style="color:#f92672">&amp;</span>dominatorTreeWrapperPass<span style="color:#f92672">-&gt;</span><span style="color:#a6e22e">getDomTree</span>() <span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>                nullptr;
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getPDT <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>PostDominatorTree<span style="color:#f92672">*</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            llvm<span style="color:#f92672">::</span>PostDominatorTreeWrapperPass<span style="color:#f92672">*</span> <span style="color:#66d9ef">const</span> postDominatorTreeWrapperPass <span style="color:#f92672">=</span>
</span></span><span style="display:flex;"><span>                getAnalysisIfAvailable<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>PostDominatorTreeWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> postDominatorTreeWrapperPass <span style="color:#f92672">?</span>
</span></span><span style="display:flex;"><span>                <span style="color:#f92672">&amp;</span>postDominatorTreeWrapperPass<span style="color:#f92672">-&gt;</span><span style="color:#a6e22e">getPostDomTree</span>() <span style="color:#f92672">:</span>
</span></span><span style="display:flex;"><span>                nullptr;
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">return</span> pass.<span style="color:#a6e22e">run</span>(function, getLI, getDT, getPDT);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">void</span> <span style="color:#a6e22e">getAnalysisUsage</span>(llvm<span style="color:#f92672">::</span>AnalysisUsage<span style="color:#f92672">&amp;</span> analysisUsage) <span style="color:#66d9ef">const</span> override
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>        <span style="color:#75715e">// We need loop analysis to replace our intrinsics.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        analysisUsage.addRequired<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopInfoWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>        analysisUsage.addPreserved<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopInfoWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#75715e">// We will use the dominator tree, and preserve it during this pass. We
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#75715e">// do this because recalculating the dominator tree after this pass was
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        <span style="color:#75715e">// much more expensive than just preserving it.
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>        analysisUsage.addPreserved<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>DominatorTreeWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>        analysisUsage.addPreserved<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>PostDominatorTreeWrapperPass<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    llvm<span style="color:#f92672">::</span>StringRef <span style="color:#a6e22e">getPassName</span>() <span style="color:#66d9ef">const</span> override { <span style="color:#66d9ef">return</span> DeadLoopRemoval<span style="color:#f92672">::</span><span style="color:#a6e22e">getPassName</span>(); }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">static</span> <span style="color:#66d9ef">char</span> ID;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>private:
</span></span><span style="display:flex;"><span>    DeadLoopRemoval pass;
</span></span><span style="display:flex;"><span>};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">char</span> DeadLoopRemovalLegacyPass<span style="color:#f92672">::</span>ID;
</span></span></code></pre></div><p>So this code is pretty straight-forward, I started with a copy-paste of the original pass and then started hacking it to use the new struct underneath:</p>
<ul>
<li>I first added the <code>DeadLoopRemoval pass</code> member to the struct.</li>
<li>Then I forwarded the <code>getPassName()</code> member virtual function to the static method in the implementing struct.</li>
<li>For the <code>runOnFunction</code> body I had to query the analysis passes I required in the lambdas, and pass these to the <code>pass.run</code> method.</li>
</ul>
<p>At this point I can test the ported pass with our existing test suite and the legacy pass manager, to ensure that worked before doing anything else. And it did! So next up, lets port it to the new pass manager.</p>
<h3 id="the-new-wrapping-pass">The New Wrapping Pass</h3>
<p>The new pass manager uses a mixin approach with the Curiously Recurring Template Pattern (CRTP, we&rsquo;ll come back to this) to implement the structure. This means no virtuals, no overrides, but there is still a specific function you are expected to provide. For a function pass you implement <code>llvm::PreservedAnalyses run(llvm::Function&amp;, llvm::FunctionAnalysisManager&amp;)</code> for instance. Our new pass manager wrapping pass thus looks like:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#66d9ef">struct</span> DeadLoopRemovalPass final : public llvm<span style="color:#f92672">::</span>PassInfoMixin<span style="color:#f92672">&lt;</span>DeadLoopRemovalPass<span style="color:#f92672">&gt;</span>
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    explicit <span style="color:#a6e22e">DeadLoopRemovalPass</span>() <span style="color:#f92672">:</span> <span style="color:#a6e22e">pass</span>() {}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    llvm<span style="color:#f92672">::</span>PreservedAnalyses <span style="color:#a6e22e">run</span>(llvm<span style="color:#f92672">::</span>Function<span style="color:#f92672">&amp;</span> function, llvm<span style="color:#f92672">::</span>FunctionAnalysisManager<span style="color:#f92672">&amp;</span> analysisManager)
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getLI <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>LoopInfo<span style="color:#f92672">&amp;</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> analysisManager.getResult<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopAnalysis<span style="color:#f92672">&gt;</span>(function);
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getDT <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>DominatorTree<span style="color:#f92672">*</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> analysisManager.getCachedResult<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>DominatorTreeAnalysis<span style="color:#f92672">&gt;</span>(function);
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">auto</span> getPDT <span style="color:#f92672">=</span> [<span style="color:#f92672">&amp;</span>]() <span style="color:#f92672">-&gt;</span> llvm<span style="color:#f92672">::</span>PostDominatorTree<span style="color:#f92672">*</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> analysisManager.getCachedResult<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>PostDominatorTreeAnalysis<span style="color:#f92672">&gt;</span>(function);
</span></span><span style="display:flex;"><span>        };
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">if</span> (pass.<span style="color:#a6e22e">run</span>(function, getLI, getDT, getPDT))
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            llvm<span style="color:#f92672">::</span>PreservedAnalyses preserved;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>            preserved.preserve<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>LoopAnalysis<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>            preserved.preserve<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>DominatorTreeAnalysis<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>            preserved.preserve<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>PostDominatorTreeAnalysis<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> preserved;
</span></span><span style="display:flex;"><span>        }
</span></span><span style="display:flex;"><span>        <span style="color:#66d9ef">else</span>
</span></span><span style="display:flex;"><span>        {
</span></span><span style="display:flex;"><span>            <span style="color:#66d9ef">return</span> llvm<span style="color:#f92672">::</span>PreservedAnalyses<span style="color:#f92672">::</span><span style="color:#a6e22e">all</span>();
</span></span><span style="display:flex;"><span>        }
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">static</span> llvm<span style="color:#f92672">::</span>StringRef <span style="color:#a6e22e">name</span>() { <span style="color:#66d9ef">return</span> DeadLoopRemoval<span style="color:#f92672">::</span><span style="color:#a6e22e">getPassName</span>(); }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>private:
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">const</span> DeadLoopRemoval pass;
</span></span><span style="display:flex;"><span>};
</span></span></code></pre></div><p>Digging into this:</p>
<ul>
<li>Our pass derives from <code>llvm::PassInfoMixin&lt;DeadLoopRemovalPass&gt;</code>, taking itself as the template argument (the CRTP paradigm).</li>
<li>We have an implementation of the <code>run</code> method like I stated above.</li>
<li>We use the explicit <code>analysisManager</code> to get the analysis passes we require, rather than using the <code>getAnalysis</code> that the old pass manager used to get at these.</li>
<li>We do not return a true/false value from our <code>run</code> method like the legacy pass manager <code>runOnFunction</code>. True meant <em>&lsquo;something changed, analysis passes might be invalid!&rsquo;</em>. For the new pass manager we explicitly return an <code>llvm::PreservedAnalyses</code> which we add what passes we preserved into.</li>
<li>Also note the <code>static llvm::StringRef name()</code> method - this is how you set the pass name of the pass.</li>
</ul>
<p>And that&rsquo;s it! We now have our pass working both with the new and legacy pass managers. I had to do this same approach with all 20 of our custom LLVM passes, quite a lot of typing, but meant I could be sure at each step it was doing the right thing.</p>
<h2 id="porting-our-custom-pass-pipeline">Porting Our Custom Pass Pipeline</h2>
<p>We long since abandoned the default LLVM pass pipeline for a custom one.</p>
<blockquote>
<p>As an aside my COVID cancelled LLVM talk was going to explain how we cut the cost of using <code>llvm::PassManagerBuilder</code>&rsquo;s default optimization pass structure by 40% by cutting out the cruft from it while still using the builder, but even that wasn&rsquo;t enough control for us to get the abosolute best compile time possible.</p>
</blockquote>
<p>So our legacy pass manager usage involved us calling each of the LLVM passes (a mix of the standard ones and our custom ones, hand crafted for best codegen and compile time):</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>mpm.<span style="color:#a6e22e">add</span>(<span style="color:#a6e22e">burst_CreateEarlyMemOptsLegacyPass</span>());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">&gt;=</span> <span style="color:#ae81ff">1</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">add</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createCFGSimplificationPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">add</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createSROAPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">add</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createEarlyCSEPass</span>());
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">&gt;=</span> <span style="color:#ae81ff">3</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">add</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createIPSCCPPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">add</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createGlobalOptimizerPass</span>());
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>This is just a sample from the pass pipeline, but it gives you an idea I hope.</p>
<p>For the new pass pipeline I did the obvious (<strong>and wrong</strong>) thing of copying that and changing it to the new pass manager:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#a6e22e">burst_AddEarlyMemOptsPass</span>(mpm);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">&gt;=</span> <span style="color:#ae81ff">1</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">SimplifyCFGPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">SROA</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">EarlyCSEPass</span>());
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">IPSCCPPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">GlobalOptPass</span>());
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>I did this for our entire pass pipeline, Visual Studio showed no errors! I hit compile, and explosions. It took me longer than I&rsquo;d care to admit to work out what the issue was - the new pass manager requires you to add a pass to the <strong>correct</strong> pass manager type.</p>
<p>With the legacy pass manager everything derived from <code>llvm::Pass</code>, including <code>llvm::ModulePass</code> and <code>llvm::FunctionPass</code>. Behind the scenes the pass manager would work out that you were using a function or module pass, and behind your back create a manager of the correct type (so if you used a function pass, it&rsquo;d make a function pass manager for you). This is a bad design, and so with the new pass managers they changed it.</p>
<p>Remember the CRTP thing I mentioned above? This is where it really hurts developers - Visual Studio didn&rsquo;t provide me with compile errors until I actually did a full compile, because the CRTP is basically an interface that requires all the templates to be instantiated with their implementing types, before it can work out you&rsquo;ve done the wrong thing. This shows up two bad bits of C++ - no interfaces (C# interfaces are so very nice for exactly this kind of issue), and templates nearly always result in horrific error messages. Very user hostile!</p>
<p>Because its been an easy 6 years since I last looked at the new pass manager (I honestly figured it would just die after such a long stint in the codebase but not being used by Clang), I took a look at the <code>llvm::PassBuilder</code> which helped me work out my mistake:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    llvm<span style="color:#f92672">::</span>FunctionPassManager fpm;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">burst_AddEarlyMemOptsPass</span>(fpm);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">&gt;=</span> <span style="color:#ae81ff">1</span>)
</span></span><span style="display:flex;"><span>    {
</span></span><span style="display:flex;"><span>        fpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">SimplifyCFGPass</span>());
</span></span><span style="display:flex;"><span>        fpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">SROA</span>());
</span></span><span style="display:flex;"><span>        fpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">EarlyCSEPass</span>());
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">createModuleToFunctionPassAdaptor</span>(std<span style="color:#f92672">::</span><span style="color:#a6e22e">move</span>(fpm)));
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">IPSCCPPass</span>());
</span></span><span style="display:flex;"><span>    mpm.<span style="color:#a6e22e">addPass</span>(llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">GlobalOptPass</span>());
</span></span><span style="display:flex;"><span>}
</span></span></code></pre></div><p>So you can see I had to explicitly make an <code>llvm::FunctionPassManager</code>, add the function passes to that, and then call <code>llvm::createModuleToFunctionPassAdaptor</code> to provide this function pass to the surrounding module pass manager. A bit verbose, and horrific template mess aside, a <strong>much much better</strong> design overall. This makes it really obvious now when you are mixing function pass managers with module passes in a poor way, because you have to re-create the function pass manager each time.</p>
<p>More explicit is always a win in my book.</p>
<p>Another thing that got me was that you have to manually register all the analysis passes you need with the analysis managers explicitly:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>PassBuilder<span style="color:#f92672">&amp;</span> pb; <span style="color:#75715e">// an already created pass builder
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>LoopAnalysisManager lam;
</span></span><span style="display:flex;"><span>pb.<span style="color:#a6e22e">registerLoopAnalyses</span>(lam);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>FunctionAnalysisManager fam;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>fam.<span style="color:#a6e22e">registerPass</span>([<span style="color:#f92672">&amp;</span>] { <span style="color:#66d9ef">return</span> llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">TargetIRAnalysis</span>(targetMachine<span style="color:#f92672">-&gt;</span>LLVM<span style="color:#f92672">-&gt;</span><span style="color:#a6e22e">getTargetIRAnalysis</span>()); });
</span></span><span style="display:flex;"><span>fam.<span style="color:#a6e22e">registerPass</span>([<span style="color:#f92672">&amp;</span>] { <span style="color:#66d9ef">return</span> llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">TargetLibraryAnalysis</span>(targetLibraryInfo); });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>AAManager aam;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>aam.registerFunctionAnalysis<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>BasicAA<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">==</span> <span style="color:#ae81ff">3</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    aam.registerFunctionAnalysis<span style="color:#f92672">&lt;</span>llvm<span style="color:#f92672">::</span>ScopedNoAliasAA<span style="color:#f92672">&gt;</span>();
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#66d9ef">if</span> (optLevel <span style="color:#f92672">&gt;=</span> <span style="color:#ae81ff">2</span>)
</span></span><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>    <span style="color:#a6e22e">burst_AddExtraAliasAnalysisPass</span>(fam, aam);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>fam.<span style="color:#a6e22e">registerPass</span>([aam] { <span style="color:#66d9ef">return</span> std<span style="color:#f92672">::</span><span style="color:#a6e22e">move</span>(aam); });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>pb.<span style="color:#a6e22e">registerFunctionAnalyses</span>(fam);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>CGSCCAnalysisManager cgam;
</span></span><span style="display:flex;"><span>pb.<span style="color:#a6e22e">registerCGSCCAnalyses</span>(cgam);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>ModuleAnalysisManager mam;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>pb.<span style="color:#a6e22e">registerModuleAnalyses</span>(mam);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>pb.<span style="color:#a6e22e">crossRegisterProxies</span>(lam, fam, cgam, mam);
</span></span></code></pre></div><p>Here I created an <code>llvm::PassBuilder</code> not to actually build the passes, but to build the analysis managers so that the analysis passes I required are available in the managers. There are some quirks:</p>
<ul>
<li>To provide an analysis to the analysis manager you pass in a callback that creates the pass. Note sure why this is needed (maybe have to delay the initialization?), but it confused me at the beginning!</li>
<li>To register our custom alias analysis we had to add that to a <code>llvm::AAManager</code>, and then register that with our function analysis manager.</li>
<li>You need to call <code>crossRegisterProxies</code> on the pass builder to make sure all the analysis managers know about each each (in the case that, say, a function requires a module-level analysis).</li>
</ul>
<h2 id="the-curious-case-of-the-optimized-optnone-functions">The Curious Case of the Optimized optnone Functions</h2>
<p>LLVM IR has a function level attribute <code>optnone</code> that disables optimization for a specific function. This is really useful for developers when they suspect there is a bug in a specific function, so they can just disable the optimizations there and work out what is going wrong more easily. When I ran our tests with the new pass manager, I noticed that our optimize none tests were&hellip; being optimized?! The functions have <code>optnone</code> on them, how could the pass manager be allowing them to be optimized?</p>
<p>In the legacy pass manager each pass has a <code>skipFunction</code> call on it that tells you if you should skip running on that function. If <code>optnone</code> was set, you&rsquo;d skip that function. The new pass manager of course doesn&rsquo;t use this mechanism and does something entirely different. Obviously.</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>PassInstrumentationCallbacks passInstrumentationCallbacks;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>OptNoneInstrumentation <span style="color:#a6e22e">optNoneInstrumentation</span>(false);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>optNoneInstrumentation.<span style="color:#a6e22e">registerCallbacks</span>(passInstrumentationCallbacks);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>llvm<span style="color:#f92672">::</span>PassBuilder <span style="color:#a6e22e">pb</span>(targetMachine<span style="color:#f92672">-&gt;</span>LLVM, llvm<span style="color:#f92672">::</span><span style="color:#a6e22e">PipelineTuningOptions</span>(), llvm<span style="color:#f92672">::</span>None, <span style="color:#f92672">&amp;</span>passInstrumentationCallbacks);
</span></span></code></pre></div><p>You have to register an <code>llvm::OptNoneInstrumentation</code> with a <code>llvm::PassInstrumentationCallbacks</code>, which is then registered in the <code>llvm::PassBuilder</code> constructor. I need to lie down. This then ensures that any function with <code>optnone</code> won&rsquo;t be optimized through some arcane process I don&rsquo;t really understand.</p>
<h2 id="the-performance">The Performance</h2>
<p>I&rsquo;ve long been told that the new pass manager will result in better performance because of how analysis passes are preserved better. So I was excited to run our tests for the first time.</p>
<p>First up the compile time performance:</p>

<img src="/images/new-pass-manager/CompileTimePerformance.png"  class="center"  style="border-radius: 8px;"    />


<p>I&rsquo;ve got to admit - after first running this I was a <em>little</em> disappointed - the new pass manager was meant to magically make everything faster! Then after I calmed down a bit I realised - <em>this is fine</em>. Since we&rsquo;ve already got our own custom curated set of passes that are ordered specifically to not trash analysis passes unnecessarily, it stands to reason that we wouldn&rsquo;t see any magical uplift with the new pass manager. The fact we are seeing numbers that are effectively within the signal-to-noise of each other means that when we do switch to the new pass manager by default, things won&rsquo;t instantly go slower as a result. That&rsquo;s a win.</p>
<p>Now for the binary size:</p>

<img src="/images/new-pass-manager/BinarySize.png"  class="center"  style="border-radius: 8px;"    />


<p>Nothing really to write home about here - slightly higher with -O0 and -O1, and slightly lower with -O2 and -O3. I did a diff of the assembly and its just the usual slight reordering of passes - mostly the same instructions just sometimes in a different order. Again, this is good though - no regressions means we can safely make the switch.</p>
<h2 id="one-last-pain-point">One Last Pain Point</h2>
<p>One thing I generally like to do on Windows is use a <code>RelWithDebInfo</code> compiled LLVM with a <code>Debug</code> or <code>Release</code> compiled our code. This gets me the best of both worlds in terms of some debuggability while not running dreadfully slow. For some reason while this has always worked with the legacy pass manager, someone in the LLVM community has added a bunch of:</p>
<div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;"><code class="language-c" data-lang="c"><span style="display:flex;"><span><span style="color:#75715e">#ifndef NDEBUG
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#75715e">// Some extra struct state!
</span></span></span><span style="display:flex;"><span><span style="color:#75715e"></span><span style="color:#75715e">#endif
</span></span></span></code></pre></div><p>To some of the new pass manager structs.</p>
<p>They are doing this so that on debug builds they can carry around some extra state in the structs to aid debuggability, but this means that the struct layout is now <strong>different</strong> between <code>Debug</code> and <code>Release</code> builds. This is absolutely awful, and means my <code>RelWithDebInfo</code> compiled approach no longer works. This took me 4 hours to narrow down - I just assumed some of the template mess in the new pass manager was somehow going wrong, it was only when I started printing the size of structs that my head hit the desk. I hope someone can remove these - I&rsquo;d rather carry around the extra state always and not use it, than have structs change size depending on the optimization level.</p>
<h2 id="conclusion">Conclusion</h2>
<p>This work took me a week to do for our reasonably customized LLVM pass pipeline and custom passes. It is a <strong>lot</strong> of boiler plate for something that effectively does the same as before, but it does set us up more cleanly for the day that LLVM actually removes the legacy pass manager from the codebase entirely, meaning that when that day comes we aren&rsquo;t floundering. The new pass manager has some nice features, it is just a shame that C++ is so broken as a language that it takes walking through a wall of template errors to work out what went wrong.</p>
<p>And before anyone @&rsquo;s me - I know concepts will in theory fix this, just colour me highly skeptical on that count.</p>
<p>Until next time!</p>


