# rss_item

guid: 66f2a6410c2ad20001fc1ede
title: How Mobile Networks Work
author: Mathew Duggan
url: https://matduggan.com/how-mobile-networks-work/
feedurl: https://matduggan.com/rss/
enclosure_url: https://matduggan.com/content/images/2024/09/cna.png
content_mime_type: text/html
content:
<img src="https://matduggan.com/content/images/2024/09/cna.png" alt="How Mobile Networks Work"><p>I&apos;ve spent a fair amount of time around networking. I&apos;ve worked for a small ISP, helped to set up campus and office networks and even done a fair amount of work with BGP and assisting with ISP failover and route work. However in my current role I&apos;ve been doing a lot of mobile network diagnostics and troubleshooting which made me realize <em>I actually don&apos;t know anything about how mobile networks operate</em>. So I figured it was a good idea for me to learn more and write up what I find. </p><p>It&apos;s interesting that without a doubt cellular internet is either going to become or has become the default Internet for most humans alive, but almost no developers I know have any idea how it works (including myself until recently). As I hope that I demonstrate below, it is untold amounts of amazing work that has been applied to this problem over decades that has really produced incredible results. As it turns out the network engineers working with cellular were doing nuclear physics while I was hot-gluing stuff together. </p><p><strong>I am not an expert. </strong>I will update this as I get better information, but use this as a reference for stuff to look up, not a bible. It is my hope, over many revisions, to turn this into a easier to read PDF that folks can download. However I want to get it out in front of people to help find mistakes. </p><p><strong>TL/DR: </strong>There is a shocking, eye-watering amount of complexity when it comes to cellular data as compared to a home or datacenter network connection. I could spend the next six months of my life reading about this and feel like I barely scratched the surface. However I&apos;m hoping that I have provided some basic-level information about how this magic all works. </p><p><strong>Corrections/Requests: </strong><a href="https://c.im/@matdevdug">https://c.im/@matdevdug</a>. I know I didn&apos;t get it all right, I promise I won&apos;t be offended. </p><h3 id="basics">Basics</h3><p>A modern cellular network at the core is comprised of three basic elements:</p><ul><li>the RAN (radio access network)</li><li>CN (core network)</li><li>Services network</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-4.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="1724" height="792" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-4.png 600w, https://matduggan.com/content/images/size/w1000/2024/09/image-4.png 1000w, https://matduggan.com/content/images/size/w1600/2024/09/image-4.png 1600w, https://matduggan.com/content/images/2024/09/image-4.png 1724w" sizes="(min-width: 720px) 720px"></figure><h3 id="ran"><strong>RAN</strong></h3><p>The RAN contains the base stations that allow for the communication with the phones using radio signals. When we think of a cell tower we are thinking of a RAN. When we are thinking of what a cellular network provides in terms of services, a lot of that is actually contained within the CN. That&apos;s where the stuff like user authorization, services turned on or off for the user and all the background stuff for the transfer and hand-off of user traffic. Think SMS and phone calls for most users today. </p><p><strong>Key Components of the RAN:</strong></p><ol><li><strong>Base Transceiver Station (BTS):</strong>&#xA0;The BTS is a radio transmitter/receiver that communicates with your phone over the air interface.</li><li><strong>Node B (or Evolved Node B for 4G or gNodeB for 5G):</strong>&#xA0;In modern cellular networks, Node B refers to a base station that&apos;s managed by multiple cell sites. It aggregates data from these cell sites and forwards it to the RAN controller.</li><li><strong>Radio Network Controller (RNC):</strong>&#xA0;The RNC is responsible for managing the radio link between your phone and the BTS/Node B.</li><li><strong>Base Station Subsystem (BSS):</strong>&#xA0;The BSS is a term used in older cellular networks, referring to the combination of the BTS and RNC.</li></ol><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/09/image-15.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="1936" height="660" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-15.png 600w, https://matduggan.com/content/images/size/w1000/2024/09/image-15.png 1000w, https://matduggan.com/content/images/size/w1600/2024/09/image-15.png 1600w, https://matduggan.com/content/images/2024/09/image-15.png 1936w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Link: </span><a href="https://www.cisco.com/c/en/us/products/collateral/wireless/nb-06-radio-access-networks-cte-en.html"><span style="white-space: pre-wrap;">https://www.cisco.com/c/en/us/products/collateral/wireless/nb-06-radio-access-networks-cte-en.html</span></a></figcaption></figure><h3 id="startup"><strong>Startup</strong></h3><ol><li><strong>Cell Search and Network Acquisition. </strong>The device powers on and begins searching for available cells by scanning the frequencies of surrounding base stations (e.g., eNodeB for LTE, gNodeB for 5G).</li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;             &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;             &#x2502;   Mobile     &#x2502;
&#x2502;              &#x2502;             &#x2502;   Device     &#x2502;
&#x2502;   Broadcast  &#x2502;             &#x2502;              &#x2502;
&#x2502;  &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; Search for  &#x2502; &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  &#x2502;
&#x2502;              &#x2502; Sync Signals&#x2502; Synchronizes &#x2502;
&#x2502;              &#x2502;             &#x2502;              &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;             &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- Device listens for synchronization signals.
- Identifies the best base station for connection.</code></pre><ol start="2"><li><strong>Random Access. </strong>After identifying the cell to connect to, the device sends a random access request to establish initial communication with the base station.This is often called RACH. If you want to read about it I found an incredible amount of detail here: <a href="https://www.sharetechnote.com/html/RACH_LTE.html">https://www.sharetechnote.com/html/RACH_LTE.html</a></li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;             &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;             &#x2502;   Mobile     &#x2502;
&#x2502;              &#x2502;             &#x2502;   Device     &#x2502;
&#x2502;  Random Access Response    &#x2502;              &#x2502;
&#x2502;  &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500; &#x2502; &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; Random Access&#x2502;
&#x2502;              &#x2502;             &#x2502; Request      &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;             &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- Device sends a Random Access Preamble.
- Base station responds with timing and resource allocation.</code></pre><ol start="3"><li><strong>Dedicated Radio Connection Setup (RRC Setup). </strong>The base station allocates resources for the device to establish a dedicated radio connection using the Radio Resource Control (RRC) protocol.</li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;             &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;             &#x2502;   Mobile     &#x2502;
&#x2502;              &#x2502;             &#x2502;   Device     &#x2502;
&#x2502;  RRC Setup   &#x2502;             &#x2502;              &#x2502;
&#x2502;  &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; Send RRC    &#x2502;              &#x2502;
&#x2502;              &#x2502; Request     &#x2502; &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  &#x2502;
&#x2502;              &#x2502;             &#x2502; RRC Response &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;             &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- Device requests RRC connection.
- Base station assigns resources and confirms.</code></pre><ol start="4"><li><strong>Device-to-Core Network Communication (Authentication, Security, etc.). </strong>Once the RRC connection is established, the device communicates with the core network (e.g., EPC in LTE, 5GC in 5G) for authentication, security setup, and session establishment.</li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;               &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;               &#x2502;   Mobile     &#x2502;
&#x2502;  &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; Forward       &#x2502;              &#x2502;
&#x2502;              &#x2502; Authentication Data          &#x2502;
&#x2502;              &#x2502; &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;   &#x2502;Authentication&#x2502;
&#x2502;              &#x2502;               &#x2502; Request      &#x2502;
&#x2502;              &#x2502;               &#x2502;              &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;               &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- Device exchanges authentication and security data with the core network.
- Secure communication is established.</code></pre><ol start="5"><li><strong>Data Transfer (Downlink and Uplink). </strong>After setup, the device starts sending (uplink) and receiving (downlink) data using the established radio connection.</li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;             &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;             &#x2502;   Mobile     &#x2502;
&#x2502;  &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; Data        &#x2502;              &#x2502;
&#x2502;  Downlink    &#x2502;             &#x2502;  &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  &#x2502;
&#x2502;  &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500; &#x2502; Data Uplink &#x2502; &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt;  &#x2502;
&#x2502;              &#x2502;             &#x2502;              &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;             &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- Data is transmitted between the base station and the device.
- Downlink (BS to Device) and Uplink (Device to BS) transmissions.</code></pre><ol start="6"><li><strong>Handover. </strong>If the device moves out of range of the current base station, a handover is initiated to transfer the connection to a new base station without interrupting the service.</li></ol><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-6.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="668" height="131" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-6.png 600w, https://matduggan.com/content/images/2024/09/image-6.png 668w"></figure><h3 id="signaling">Signaling</h3><p>As shown in the diagram above, there are a lot of references to something called &quot;signaling&quot;. Signaling seems to be a shorthand for handling a lot of configuration and hand-off between tower and device and the core network. As far as I can tell they can be broken into 3 types. </p><ol><li>Access Stratum Signaling<ol><li>Set of protocols to manage the radio link between your phone and cellular network. </li><li>Handles authentication and encryption</li><li>Radio bearer establishment (setting up a dedicated channel for data transfer)</li><li>Mobility management (handovers, etc)</li><li>Quality of Service control. </li></ol></li><li>Non-Access Stratum (NAS) Signaling <ol><li>Set of protocols used to manage the interaction between your phone and the cellular network&apos;s core infrastructure. </li><li>It handles tasks such as authentication, billing, and location services.</li><li>Authentication with the Home Location Register (HLR)</li><li>Roaming management</li><li>Charging and billing</li><li>IMSI Attach/ Detach procedure</li></ol></li><li>Lower Layer Signaling on the Air Interface<ol><li>This refers to the control signaling that occurs between your phone and the cellular network&apos;s base station at the physical or data link layer.</li><li>It ensures reliable communication over the air interface, error detection and correction, and efficient use of resources (e.g., allocating radio bandwidth).</li><li>Modulation and demodulation control</li><li>Error detection and correction using CRCs (Cyclic Redundancy Checks)</li></ol></li></ol><p><strong>High Level Overview of Signaling</strong></p><ol><li>You turn on your phone (AS signaling starts).</li><li>Your phone sends an Initial Direct Transfer (IDT) message to establish a radio connection with the base station (lower layer signaling takes over).</li><li>The base station authenticates your phone using NAS signaling, contacting the HLR for authentication.</li><li>Once authenticated, lower layer signaling continues to manage data transfer between your phone and the base station.</li></ol><p><strong>What is HLR?</strong></p><p>Home Location Register contains the subscriber data for a network. Their IMSI, phone number, service information and is what negotiates where in the world the user physically is. </p><h3 id="duplexing">Duplexing</h3><p>You have a lot of devices and you have a few towers. You need to do many uplinks and downlinks to many devices. </p><p>It is important that any cellular communications system you can send and receive in both directions at the same time. This enables conversations to be made, with either end being able to talk and listen as required. In order to be able to transmit in both directions, a device (UE) and base station must have a duplex scheme. There are a lot of them including Frequency Division Duplex (FDD), Time Division Duplex (TDD), Semi-static TDD and Dynamic TDD.</p><p><strong>Duplexing Types:</strong></p><ol><li><strong>Frequency Division Duplex (FDD):</strong>&#xA0;Uses separate frequency bands for downlink and uplink signals.<ol><li>Downlink: The mobile device receives data from the base station on a specific frequency (F1).</li><li>Uplink: The mobile device sends data to the base station on a different frequency (F2).</li><li>Key Principle: Separate frequencies for uplink and downlink enable simultaneous transmission and reception.</li></ol></li></ol><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;              &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;  Base Station&#x2502;              &#x2502;   Mobile     &#x2502;
&#x2502;              &#x2502;              &#x2502;   Device     &#x2502;
&#x2502;  &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt; &#x2502; F1 (Downlink)&#x2502; &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  &#x2502;
&#x2502;              &#x2502;              &#x2502;              &#x2502;
&#x2502;  &lt;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500; &#x2502; F2 (Uplink)  &#x2502; &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&gt;  &#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;              &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

Separate frequency bands (F1 and F2)</code></pre><ol start="2"><li><strong>Time Division Duplex (TDD):</strong>&#xA0;Alternates between downlink and uplink signals over the same frequency band.<ol><li>Downlink: The base station sends data to the mobile device in a time slot.</li><li>Uplink: The mobile device sends data to the base station in a different time slot using the same frequency.</li><li>Key Principle: The same frequency is used for both uplink and downlink, but at different times.</li></ol></li></ol><pre><code> &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;                     &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
 &#x2502;  Base Station&#x2502;                     &#x2502;  Mobile Phone&#x2502;
 &#x2502; (eNodeB/gNB) &#x2502;                     &#x2502;              &#x2502;
 &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;                     &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 1 (Downlink)
                 (Base station sends data)

     &#x25C4;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  Time Slot 2 (Uplink)
                 (Mobile sends data)
     
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 3 (Downlink)
                 (Base station sends data)
                 
     &#x25C4;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  Time Slot 4 (Uplink)
                 (Mobile sends data)
 
    - The same frequency is used for both directions.
    - Communication alternates between downlink and uplink in predefined time slots.</code></pre><p>Frame design</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-8.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="684" height="146" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-8.png 600w, https://matduggan.com/content/images/2024/09/image-8.png 684w"></figure><ol start="3"><ol><li>Downlink/Uplink: There are predetermined time slots for uplink and downlink, but they can be changed periodically (e.g., minutes, hours).</li><li>Key Principle: Time slots are allocated statically for longer durations but can be switched based on network traffic patterns (e.g., heavier downlink traffic during peak hours).</li><li>A frame typically lasts 10 ms and is divided into time slots for downlink (DL) and uplink (UL).</li><li>&quot;Guard&quot; time slots are used to allow switching between transmission and reception.</li></ol></ol><p><br>4.<strong> Dynamic Time Division Duplex (Dynamic TDD):</strong> </p><ol><ol><li>Downlink/Uplink: Time slots for uplink and downlink are dynamically adjusted in real time based on instantaneous traffic demands.</li><li>Key Principle: Uplink and downlink time slots are flexible and can vary dynamically to optimize the usage of the available spectrum in real-time, depending on the traffic load.</li><li>See second diagram for what &quot;guard periods&quot; are. Basically windows to ensure there are gaps and the signal doesn&apos;t overlap. </li></ol></ol><pre><code> &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;                     &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
 &#x2502;  Base Station&#x2502;                     &#x2502;  Mobile Phone&#x2502;
 &#x2502; (eNodeB/gNB) &#x2502;                     &#x2502;              &#x2502;
 &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;                     &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 1 (Downlink)
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 2 (Downlink)
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 3 (Downlink)
     &#x25C4;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  Time Slot 4 (Uplink)
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 5 (Downlink)
     
     &#x25C4;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  Time Slot 6 (Uplink)
     
    - More slots for downlink in scenarios with high download traffic (e.g., streaming video).
    - Dynamic slot assignment can change depending on the real-time demand.</code></pre><pre><code> &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;                     &#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
 &#x2502;  Base Station&#x2502;                     &#x2502;  Mobile Phone&#x2502;
 &#x2502; (eNodeB/gNB) &#x2502;                     &#x2502;              &#x2502;
 &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;                     &#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 1 (Downlink)
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 2 (Downlink)
     [Guard Period]                          (Switch from downlink to uplink)
     &#x25C4;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;  Time Slot 3 (Uplink)
     [Guard Period]                          (Switch from uplink to downlink)
     &#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x25BA;  Time Slot 4 (Downlink)
     
    - Guard periods allow safe switching from one direction to another.
    - Guard periods prevent signals from overlapping and causing interference.</code></pre><h3 id="core">Core</h3><p>So I&apos;ve written a lot about what the RAN does. But we haven&apos;t really touched on what the core network concept does. Basically once the device registers with the base station using the random access procedure discussed above, the device is enabled and allows the core network to do a bunch of stuff that we typically associate with &quot;having a cellular plan&quot;. </p><p>For modern devices when we say authentication we mean &quot;mutual authentication&quot;, which means the device authenticates the network and the network authenticates the device. This is typically something like a subscriber-specific secret key and a random number to generate a response to the request sent by the device. Then the network sends an authentication token and the device compares this token with the expected token to authenticate the network. It looks like the following:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-9.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="590" height="259"></figure><pre><code>&#x250C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2510;
&#x2502;    Encryption &amp;       &#x2502;
&#x2502;  Integrity Algorithms &#x2502;
&#x251C;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2524;
&#x2502;  - AES (Encryption)   &#x2502;
&#x2502;  - SNOW 3G (Encryption&#x2502;
&#x2502;  - ZUC (Encryption)   &#x2502;
&#x2502;  - SHA-256 (Integrity)&#x2502;
&#x2514;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2500;&#x2518;

- AES: Strong encryption algorithm commonly used in LTE/5G.
- SNOW 3G: Stream cipher used for encryption in mobile communications.
- ZUC: Encryption algorithm used in 5G.
- SHA-256: Integrity algorithm ensuring data integrity.</code></pre><p>The steps of the core network are as follows:</p><ul><li>Registration (also called attach procedure): The device connects to the core network (e.g., EPC in LTE or 5GC in 5G) to register and declare its presence. This involves the device identifying itself and the network confirming its identity.</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-10.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="551" height="231"></figure><ul><li>Mutual Authentication: The network and device authenticate each other to ensure a secure connection. The device verifies the network&#x2019;s authenticity, and the network confirms the device&#x2019;s identity.</li><li>Security Activation: After successful authentication, the network and the device establish a secure channel using encryption and integrity protection to ensure data confidentiality and integrity.</li><li>Session Setup and IP Address Allocation: The device establishes a data session with the core network, which includes setting up bearers (logical paths for data) and assigning an IP address to enable internet connectivity.</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-11.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="562" height="216"></figure><p></p><h3 id="how-data-gets-to-phone">How Data Gets To Phone</h3><p>Alright we&apos;ve talked about how the phone finds a tower to talk to, how the tower knows who the phone is and all the millions of steps involved in getting the mobile phone an actual honest-to-god IP address. How is data actually getting to the phone itself?</p><ol><li>Configuration for Downlink Measurement: Before downlink data transmission can occur, the mobile device (UE) must be configured to perform downlink measurements. This helps the network optimize transmission based on the channel conditions. Configuration messages are sent from the base station (eNodeB in LTE or gNB in 5G) to instruct the UE to measure certain DL reference signals.</li><li>Reference Signal (Downlink Measurements): The mobile device receives reference signals from the network. These reference signals are used by the UE to estimate DL channel conditions. In LTE, Cell-specific Reference Signals (CRS) are used, and in 5G, Channel State Information-Reference Signals (CSI-RS) are used.</li><li>DL Channel Conditions (CQI, PMI, RI): The mobile device processes the reference signals to assess the downlink channel conditions and generates reports such as CQI (Channel Quality Indicator), PMI (Precoding Matrix Indicator), and RI (Rank Indicator). These reports are sent back to the base station.</li><li>DL Resource Allocation and Packet Transmission: Based on the UE&#x2019;s channel reports (CQI, PMI, RI), the base station allocates appropriate downlink resources. It determines the modulation scheme, coding rate, MIMO layers, and frequency resources (PRBs) and sends a DL scheduling grant to the UE. The data packets are then transmitted over the downlink.</li><li>Positive/Negative Acknowledgement (HARQ Feedback): After the UE receives the downlink data, it checks the integrity of the packets using CRC (Cyclic Redundancy Check). If the CRC passes, the UE sends a positive acknowledgement (ACK) back to the network. If the CRC fails, a negative acknowledgement (NACK) is sent, indicating that retransmission is needed.</li><li>New Transmission or Retransmission (HARQ Process): If the network receives a NACK, it retransmits the packet using the HARQ process. The retransmission is often incremental (IR-HARQ), meaning the device combines the new transmission with previously received data to improve decoding.</li></ol><p>Uplink is a little different but is basically the device asking for a timeslot to upload, getting a grant, sending the data up and then getting an ack that it is sent. </p><h3 id="gs">Gs</h3><p>So as everyone knows cellular networks have gone through a series of revisions over the years around the world. I&apos;m going to talk about them and just try to walk through how they are different and what they mean. </p><h3 id="1g">1G</h3><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-12.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="284" height="177"></figure><ul><li>Starts in Japan, moves to Europe and then the US and UK. </li><li>Speeds up to 2.4kbps and operated in the frequency band of 150 KHz. </li><li>Didn&apos;t work between countries, had low capacity, unreliable handoff and no security. Basically any receiver can listen to a conversation. </li></ul><h3 id="2g">2G</h3><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-14.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="400" height="269"></figure><ul><li>Launched in 1991 in Finland</li><li>Allows for text messages, picture messages and MMS.</li><li>Speeds up to 14.4kbps between 900MHz and 1800MHz bands</li><li>Actual security between sender and receiver with messages digitally encrypted. </li></ul><p><strong>Wait, are text messages encrypted?</strong></p><p>So this was completely new to me but I guess my old Nokia brick had some encryption on it. Here&apos;s how that process worked:</p><ol><li>Mobile device stores a secret key in the SIM card and the network generates a random challenge and sends it to the mobile device. </li><li>The A3 algorithm is used to compute a Signed Response (SRES) using the secret key and the random value. </li><li>Then the A8 algorithm is used with secret and the random value to generate a session encryption key Kc (64-bit key). This key will be used for encrypting data, including SMS.</li><li>After the authentication process and key generation, encryption of SMS messages begins. GSM uses a stream cipher to encrypt both voice and data traffic, including text messages. The encryption algorithm used for SMS is either A5/1 or A5/2, depending on the region and network configuration.<ol><li>A5/1: A stronger encryption algorithm used in Europe and other regions.</li><li>A5/2: A weaker variant used in some regions, but deprecated due to its vulnerabilities.</li><li>The A5 algorithm generates a keystream that is XORed with the plaintext message (SMS) to produce the ciphertext, ensuring the confidentiality of the message.</li></ol></li></ol><p>So basically text messages from the phone to the base station were encrypted and then exposed there. However I honestly didn&apos;t even know that was happening. </p><p><strong>TSMA and CDMA</strong></p><p>I remember a lot of conversations about GSM vs CDMA when you were talking about cellular networks but at the time all I really knew was &quot;GSM is European and CDMA is US&quot;. </p><ul><li>TSMA is GSM and uses time slots</li><li>CDMA allocates each user a special code to communicate over multiple physical channels</li><li>GSM is where we see services like voice mail, SMS, call waiting</li></ul><p><strong>EDGE</strong></p><p>So everyone who is old like me remembers EDGE on cellphones, including the original iPhone I waited in line for. EDGE was effectively a retrofit you could put on top of an existing GSM network, keeping the cost for adding it low. You got speeds on 9.6-200kbps. </p><h3 id="3g">3G</h3><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-17.png" class="kg-image" alt="How Mobile Networks Work" loading="lazy" width="259" height="194"></figure><ul><li>Welcome to the year 2000</li><li>Frequency spectrum of 3G transmissions is 1900-2025MHz and 2110-2200MHz.</li><li>UTMS takes over for GSM and CDMA2000 takes over from CDMA. </li><li>Maxes out around 8-10Mbps</li><li>IMT-2000 = 3G </li></ul><p>So let&apos;s just recap quickly how we got here. </p><ul><li>2G (GSM): Initially focused on voice communication and slow data services (up to 9.6 kbps using Circuit Switched Data).</li><li>2.5G (GPRS): Introduced packet-switched data with rates of 40-50 kbps. It allowed more efficient use of radio resources for data services.</li><li>2.75G (EDGE): Enhanced the data rate by improving modulation techniques (8PSK). This increased data rates to around 384 kbps, making it more suitable for early mobile internet usage.</li></ul><p>EDGE introduced 8-PSK (8-Phase Shift Keying) modulation, which allowed the encoding of 3 bits per symbol (as opposed to 1 bit per symbol with the original GSM&#x2019;s GMSK (Gaussian Minimum Shift Keying) modulation). This increased spectral efficiency and data throughput.</p><p>EDGE had really high latency so it wasn&apos;t really usable for things like video streaming or online gaming. </p><ul><li>3G (WCDMA): Max data rate: 2 Mbps (with improvements over EDGE in practice). Introduced spread-spectrum (CDMA) technology with QPSK modulation.</li><li>3.5G (HSDPA): Enhanced WCDMA by introducing adaptive modulation (AMC), HARQ, and NodeB-based scheduling. Max data rate: 14.4 Mbps (downlink).</li></ul><p>So when we say 3G we actually mean a pretty wide range of technologies all underneath the same umbrella. </p><h3 id="4g">4G</h3><ul><li>4G or as it is sometimes called LTE evolved from WCDMA. Instead of developing new radio interfaces and new technology existing and newly developed wireless system like GPRS, EDGE, Bluetooth, WLAN and Hiper-LAN were integrated together</li><li>4G has a download speed of 67.65Mbps and upload speed of 29.37Mbps</li><li>4G operates at frequency bands of 2500-2570MHz for uplink and 2620-2690MHz for downlink with channel bandwidth of 1.25-20MHz</li><li>4G has a few key technologies, mainly OFDM, SDR and Multiple-Input Multiple-Output (MIMO). <ul><li>OFDM (Orthogonal Frequency Division Multiplexing)<ul><li>Allows for more efficient use of the available bandwidth by breaking down data into smaller pieces and sending them simultaneously</li><li>Since each channel uses a different frequency, if one channel experiences interference or errors, the others remain unaffected.</li><li>OFDM can adapt to changing network conditions by dynamically adjusting the power levels and frequencies used for each channel.</li></ul></li><li>SDR (Software Defined Radio)<ul><li>Like it sounds, it is a technology that enables flexible and efficient implementation of wireless communication systems by using software algorithms to control and process radio signals in real-time. In cellular 4G, SDR is used to improve performance, reduce costs, and enable advanced features like multi-band support and spectrum flexibility.</li></ul></li><li>MIMO (multiple-input multiple-output)<ul><li>A technology used in cellular 4G to improve the performance and capacity of wireless networks. It allows for the simultaneous transmission and reception of multiple data streams over the same frequency band, using multiple antennas at both the base station and mobile device.</li><li>Works by having both the base station and the mobile device equipped with multiple antennas</li><li>Each antenna transmits and receives a separate data stream, allowing for multiple streams to be transmitted over the same frequency band</li><li>There is Spatial Multiplexing where multiple data streams are transmitted over the same frequency band using different antennas. Then Beamforming where advanced signal processing techniques to direct the transmitted beams towards specific users, improving signal quality and reducing interference. Finally Massive MIMO where you use a lot of antennas (64 or more) to improve capacity and performance. </li></ul></li></ul></li></ul><h3 id="5g">5G</h3><ul><li>The International Telecommunication Union (ITU) defines 5G as a wireless communication system that supports speeds of at least 20 Gbps (gigabits per second), with ultra-low latency of less than 1 ms (millisecond).</li><li>5G operates on a much broader range of frequency bands than 4G<ul><li><strong>Low-band frequencies:</strong> These frequencies are typically below 3 GHz and are used for coverage in rural areas or indoor environments. Examples include the 600 MHz, 700 MHz, and 850 MHz bands.</li><li><strong>Mid-band frequencies:</strong> These frequencies range from approximately 3-10 GHz and are used for both coverage and capacity in urban areas. Examples include the 4.5 GHz, 6 GHz, and 24 GHz bands.</li><li><strong>High-band frequencies:</strong> These frequencies range from approximately 10-90 GHz and are used primarily for high-speed data transfer in dense urban environments. Examples include the 28 GHz, 39 GHz, and 73 GHz bands.</li></ul></li><li>5g network designs are a step up in complexity from their 4g predecessors, with a control plane and a userplane with each plane using a separate network function. 4G networks have a single plane. </li><li>5G uses advanced modulation schemes such as 256-Quadrature Amplitude Modulation (QAM) to achieve higher data transfer rates than 4G, which typically uses 64-QAM or 16-QAM</li><li>All the MIMO stuff discussed above. </li></ul><p><strong>What the hell is Quadrature Amplitude Modulation?</strong></p><p>I know, it sounds like a Star Trek thing. It is a way to send digital information over a communication channel, like a wireless network or cable. It&apos;s a method of &quot;modulating&quot; the signal, which means changing its characteristics in a way that allows us to transmit data.</p><p>When we say 256-QAM, it refers to the specific type of modulation being used. Here&apos;s what it means:</p><ul><li><strong>Quadrature</strong>: This refers to the fact that the signal is being modulated using two different dimensions (or &quot;quadratures&quot;). Think of it like a coordinate system with x and y axes.</li><li><strong>Amplitude Modulation</strong> (AM): This is the way we change the signal&apos;s characteristics. In this case, we&apos;re changing the amplitude (magnitude) of the signal to represent digital information.</li><li><strong>256</strong>: This refers to the number of possible states or levels that the signal can take on. Think of it like a binary alphabet with 2^8 = 256 possible combinations.</li></ul><p>Why does 5G want this?</p><ul><li><strong>More information per symbol</strong>: With 256-QAM, each &quot;symbol&quot; (or signal change) can represent one of 256 different values. This means we can pack more data into the same amount of time.</li><li><strong>Faster transmission speeds</strong>: As a result, we can transmit data at higher speeds without compromising quality.</li></ul><h3 id="kubernetes-and-5g">Kubernetes and 5G</h3><p>Kubernetes is a popular technology in 5G and is used for a number of functions, including the following:</p><ul><li><strong>Virtual Network Functions (VNFs):</strong> VNFs are software-based implementations of traditional network functions, such as firewalls or packet filters. Kubernetes is used to deploy and manage these VNFs.</li><li><strong>Cloud-Native Network Functions (CNFs):</strong> CNFs are cloud-native applications that provide network function capabilities, such as traffic management or security filtering. Kubernetes is used to deploy and manage these CNFs.</li><li><strong>Network Function Virtualization (NFV) Infrastructure:</strong> NFV infrastructure provides the underlying hardware and software resources for running VNFs and CNFs. Kubernetes is used to orchestrate and manage this infrastructure.</li></ul><h3 id="conclusion">Conclusion</h3><p>So one of the common sources of frustration for developers I&apos;ve worked with when debugging cellular network problems is that often while there is plenty of bandwidth for what they are trying to do, the latency involved can be quite variable. If you look at all the complexity behind the scenes and then factor in that the network radio on the actual cellular device is constantly flipping between an Active and Idle state in an attempt to save battery life, this suddenly makes sense. </p><p>Because all of the complexity I&apos;m talking about ultimately gets you back to the same TCP stack we&apos;ve been using for years with all the overhead involved in that back and forth. We&apos;re still ending up with a SYN -&gt; SYN-ACK. There are tools you can use to shorten this process somewhat (TCP Fast Open) and changing the initial congestion window but still you are mostly dealing with the same level of overhead you always dealt with. </p><p>Ultimately there isn&apos;t much you can do with this information, as developers have almost no control over the elements present here. However I think it&apos;s useful as cellular networks continue to become the dominant default Internet for the Earth&apos;s population that more folks understand the pieces happening in the background of this stack.  </p>


guid: 66fa558f0c2ad20001fc208f
title: Post-Quantum Cryptography Basics
author: Mathew Duggan
url: https://matduggan.com/post-quantum-cryptography-basics/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<img src="https://images.unsplash.com/photo-1667581386428-ed64b9bee6a6?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=M3wxMTc3M3wwfDF8c2VhcmNofDE4fHxxdWFudHVtfGVufDB8fHx8MTcyNzY5NjAzMnww&amp;ixlib=rb-4.0.3&amp;q=80&amp;w=2000" alt="Post-Quantum Cryptography Basics"><p>Every few years I will be on a team and the topic of quantum computing will come up. Inevitably the question will get asked &quot;well is there something we are supposed to be doing about that or is it just a looming threat?&quot; We will all collectively stare at each other and shrug, then resume writing stuff exactly like we were writing it before. </p><p>In 2024 it would be hard to make a strong justification for worrying a lot about post-quantum cryptography in a world where your most likely attack vector is someone breaking into your company Slack and just asking for access to something. However it is a question developers like to worry about because it involves a lot of math and cool steampunk looking computers. It&apos;s definitely a more interesting problem than how to get everyone to stop blindly approving access to the company Confluence. </p><p><a href="https://9to5mac.com/2024/09/27/up-to-600-million-facebook-and-instagram-passwords-stored-in-plain-text/" rel="noreferrer">In a universe where Facebook decided it was a good idea to store hundreds of millions of passwords in plain text, it&apos;s clear nobody is even getting the basics right.</a> But it&apos;s more exciting to talk about post-quantum so let&apos;s do that. </p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/09/image-18.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="599" height="799"><figcaption><span style="white-space: pre-wrap;">Looks like something in Star Trek someone would trip and pull a bunch of wires out of. </span></figcaption></figure><p>Since I get asked the question every few year and I basically have no idea what I&apos;m talking about, I figured I&apos;ll do the research now and then refer back to this in the future when someone asks and I need to look clever in a hurry. </p><p><strong>TL/DR: </strong>The tooling to create post-quantum safe secrets exists and mostly works, but for normal developers dealing with data that is of little interest 12 months after it is created, I think this is more a &quot;nice to have&quot;. That said, these approaches are different enough from encryption now that developers operating with more important data would be well-served in investing the time in doing the research now on how to integrate some of these. Now that the standard is out I suspect there will be more professional interest in supporting these approaches and the tooling will get more open source developer contributions.</p><p>There is basically one implementation: <a href="https://github.com/PQClean/PQClean">https://github.com/PQClean/PQClean</a> and then that is being ported over to other languages: <a href="https://pypi.org/project/quantcrypt/">https://pypi.org/project/quantcrypt/</a>. If you are interested in exploring this technology I recommend starting with PQClean since it is easy to read. </p><p>Libraries I tested and seemed to work:</p><ul><li>quantcrypt - Python</li><li>CIRCL (Cloudflare Interoperable Reusable Cryptographic Library) - Golang</li><li>pqcrypto crate - Rust</li></ul><h3 id="quantum-computers-explained-through-pokemon-cards">Quantum Computers Explained Through Pokemon Cards</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/09/image-19.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="480" height="360"><figcaption><span style="white-space: pre-wrap;">About the level of math my brain can handle</span></figcaption></figure><p>Think of a conventional computer like a regular Pok&#xE9;mon player. This player makes decisions based on clear rules, one at a time, and can only do one move at a time.</p><p>In the Pok&#xE9;mon card game, you have:</p><ul><li>A limited number of cards (like a computer&#x2019;s memory)</li><li>You play one card at a time (like a computer performing one calculation at a time)</li><li>You follow a clear set of rules (like how classical computers follow step-by-step instructions)</li></ul><p>Every time you want to pick a card, attack, or use a move, you do it one by one in a specific order, just like a classical computer processes 0s and 1s in a step-by-step manner. If you want to calculate something or figure out the best strategy, you would test one option, then another, and so on, until you find the right solution. This makes conventional computers good at handling problems that can be broken down into simple steps.</p><p><strong>Quantum Computers</strong>:</p><p>Now, imagine a quantum computer is like a player who can somehow look at all the cards in their deck at the same time and choose the best one without flipping through each card individually.</p><p>In the quantum world:</p><ul><li>Instead of playing one card at a time, it&#x2019;s like you could play multiple cards at once, but in a way that combines all possibilities (like a super-powered move).</li><li>You don&#x2019;t just pick one strategy, you could explore all possible strategies at once. It&#x2019;s as if you&#x2019;re thinking of all possible moves simultaneously, which could lead to discovering new ways to win the game much faster than in a regular match.</li><li>Quantum computers rely on something called superposition, which is like having your Pok&#xE9;mon be both active and benched at the same time, until you need to make a decision. Then, they &#x201C;collapse&#x201D; into one state&#x2014;either active or benched.</li></ul><p>This gives quantum computers the ability to solve certain types of problems much faster because they are working with many possibilities at once, unlike classical computers that work on problems step-by-step.</p><h3 id="why-arent-quantum-computers-more-relevant-to-me">Why Aren&apos;t Quantum Computers More Relevant To Me?</h3><p>We&apos;ll explain this with Pokemon cards again. </p><p>The deck of cards (which represents the quantum system) in a quantum player&#x2019;s game is extremely fragile. The cards are like quantum bits (qubits), and they can be in many states at once (active, benched, etc.). However, if someone bumps the table or even just looks at the cards wrong, the whole system can collapse and go back to a simple state.</p><p>In the Pok&#xE9;mon analogy, this would be like having super rare and powerful cards, but they&#x2019;re so sensitive that if you shuffle too hard or drop the deck, the cards get damaged or lost. Because of this, it&#x2019;s hard to keep the quantum player&#x2019;s strategy intact without ruining their game.</p><p>In real life, quantum computers need extremely controlled environments to work&#x2014;like keeping them at near absolute zero temperatures. Otherwise, they make too many errors to be useful for most tasks.</p><p>The quantum player might be amazing at playing certain types of Pok&#xE9;mon battles, like tournaments that require deep strategy or involve many complex moves. However, if they try to play a quick, casual game with a simple strategy, their special abilities don&#x2019;t help much. They may even be worse at simple games than regular players.</p><h3 id="got-it-so-post-quantum-cryptography">Got it, so Post-Quantum Cryptography</h3><p>So conventional encryption algorithms often work with the following design. They select 2 very large prime numbers and then multiply them to obtain an even larger number. The act of multiplying the prime numbers is easy, but it&apos;s hard to figure out what you used to make the output. These two numbers are known as the prime factors and are what you are talking about obtaining when you are talking about breaking encryption. </p><p>Sometimes you hear this referred to as &quot;the RSA problem&quot;. How do you get the private key with only the public key. Since this not-yet-existing quantum computer would be good at finding these prime numbers, a lot of the assumptions we have about how encryption works would be broken. For years and years the idea that it is safe to share a public key has been an underpinning of much of the software that has been written. Cue much panic.  </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-20.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="480" height="360"></figure><p>But since it takes 20 years for us to do anything as an industry we have to start planning now even though it seems more likely in 20-30 years we&apos;ll be struggling to keep any component of the internet functional through massive heat waves and water wars. Anywho. </p><p>So the NIST starting in 2016 asked for help selecting some post-quantum standards and ended up settling on 3 of them. Let&apos;s talk about them and why they are (probably) better to solve this problem. </p><h2 id="fips-203-module-lattice-based-key-encapsulation-mechanism-standard">FIPS 203 (Module-Lattice-Based Key-Encapsulation Mechanism Standard)</h2><h3 id></h3><p>You can read the full standard here: <a href="https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.203.pdf">https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.203.pdf</a></p><p>Basically we have two different things happening here. We have a Key-Encapsulation Mechanism, which is a known thing you have probably used. Layered on top of that is a Module-Lattice-Based KEM. </p><p><strong>Key-Encapsulation Mechanism</strong></p><p>You and another entity need to establish a private key between the two of you, but only using non-confidential communication. Basically the receiver generates a key pair and transmits the public key to the sender. The sender needs to ensure they got the right public key. The sender, using that public key, generates another key and encrypted text and then sends that back to the receiver over a channel that could either be secure or insecure. You&apos;ve probably done this a lot in your career in some way or another. </p><h3 id="more-lattices">More Lattices</h3><p>There are two common algorithms that allow us to secure key-encapsulation mechanisms. </p><ul><li>Ring Learning with Errors</li><li>Learning with Errors</li></ul><p><strong>Ring Learning with Errors</strong></p><p>So we have three parts to this:</p><ul><li><strong>Ring</strong>: A set of polynomials where the variables are limited to a specific range. If you, like me, forgot what a polynomial is I&apos;ve got you.</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/09/image-21.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="787" height="322" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-21.png 600w, https://matduggan.com/content/images/2024/09/image-21.png 787w" sizes="(min-width: 720px) 720px"></figure><ul><li><strong>Modulus</strong>: The maximum value of the variable in the ring (e.g.,&#xA0;<code>q = 1024</code>).</li><li><strong>Error Term</strong>: Random values added during key generation, simulating noise.</li></ul><p><strong>How Does It Work?</strong></p><ol><li><strong>Key generation:</strong><ol><li>Choose a large prime number (p)</li><li>Generate two random polynomials within the ring:&#xA0;<code>a</code>&#xA0;and&#xA0;<code>s</code>. These will be used to create the public and private keys.</li></ol></li><li><strong>Public Key Creation</strong><ol><li>Compute the product of&#xA0;<code>a</code>&#xA0;and a fixed polynomial&#xA0;<code>x</code>, which is part of the key generation algorithm (<code>ax = s + e</code>). The error term&#xA0;<code>e</code>&#xA0;represents the &quot;noise&quot; added to simulate real-world conditions.</li></ol></li><li><strong>Private Key</strong>: Keep&#xA0;<code>s</code>&#xA0;secret. It&apos;s used for decryption.</li><li><strong>Public Key</strong>: Publish&#xA0;<code>a</code>. This is how others can send you encrypted messages.</li></ol><p>Assuming you have a public key, in order to encrypt stuff you need to do the following:</p><ol><li>Generate a random polynomial <code>r</code></li><li>Encrypt the message using&#xA0;<code>a</code>,&#xA0;<code>r</code>, and some additional computation (<code>c = ar + e&apos;</code>). The error term&#xA0;<code>e&apos;</code>&#xA0;represents the &quot;noise&quot; added during encryption.</li></ol><p>To decrypt:</p><ol><li>Computing the difference between the ciphertext and a multiple of the public key (<code>d = c - as</code>). This eliminates the noise introduced during encryption.</li><li>Solving for<strong>&#xA0;<code>r</code></strong>: Since we know that&#xA0;<code>c = ar + e&apos;</code>, subtracting&#xA0;<code>as</code>&#xA0;from both sides gives us an equation to solve for&#xA0;<code>r</code>.</li><li>Extracting the shared secret key: Once you have&#xA0;<code>r</code>, use it as a shared secret key.</li></ol><p>What do this look like in Python?</p><p><strong>Note: This is not a good example to use for real data. I&apos;m trying to show how it works at a basic level. Never use a randos Python script to encrypt actual real data.</strong></p><pre><code>import numpy as np

def rlsr_keygen(prime):
    # Generate large random numbers within the ring
    poly_degree = 4

    # Create a polynomial for key generation
    s = np.random.randint(0, 2**12, size=poly_degree)

    # Compute product of a and x, adding an error term (noise) during key generation
    A = np.random.randint(0, 2**12, size=(poly_degree, poly_degree))
    e = np.random.randint(-2**11, 2**11, size=poly_degree)

    return s, A

def rlsr_encapsulate(A, message):
    # Generate random polynomial to be used for encryption
    modulus = 2**12

    r = np.random.randint(0, 2**12, size=4)

    # Compute ciphertext with noise
    e_prime = np.random.randint(-modulus//2, modulus//2, size=4)
    c = np.dot(A, r) + e_prime

    return c

def rlsr_decapsulate(s, A, c):
    # Compute difference between ciphertext and a multiple of the public key
    d = np.subtract(c, np.dot(A, s))

    # Solve for r (short vector in the lattice)
    # In practice, this is done using various algorithms like LLL reduction

    return d

def generate_shared_secret_key():
    prime = 2**16 + 1  # Example value
    modulus = 2**12

    s, A = rlsr_keygen(prime)

    # Generate a random message (example value)
    message = np.random.randint(0, 256, size=4)

    c = rlsr_encapsulate(A, message)

    # Compute shared secret key
    d = rlsr_decapsulate(s, A, c)

    return d

shared_secret_key = generate_shared_secret_key()
print(shared_secret_key)</code></pre><p></p><h2 id="fips-204-module-lattice-based-digital-signature-standard">FIPS 204 (Module-Lattice-Based Digital<br>Signature Standard)</h2><p></p><p>You can see the link here: <a href="https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf">https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.204.pdf</a></p><p><strong>Digital Signatures</strong></p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/09/image-22.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="806" height="746" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-22.png 600w, https://matduggan.com/content/images/2024/09/image-22.png 806w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Link: </span><a href="https://www.geeksforgeeks.org/digital-signature-algorithm-dsa/"><span style="white-space: pre-wrap;">https://www.geeksforgeeks.org/digital-signature-algorithm-dsa/</span></a></figcaption></figure><p>A digital signature is a way to verify the authenticity and integrity of electronic documents, messages, or data. This is pretty important for software supply chains and packaging along with a million other things. </p><p><strong>How It Works</strong></p><ol><li><strong>Key Generation</strong>: Two large prime numbers <code>p</code> and <code>q</code> are chosen, along with their product <code>n</code>. A random matrix <code>A</code> is generated within this lattice. This process creates two keys:<ul><li><strong>Public Key (<code>A</code>)</strong>: Published for others to use when verifying a digital signature.</li><li><strong>Private Key (<code>s</code>)</strong>: Kept secret by the sender and used to create a digital signature.</li></ul></li><li><strong>Message Hashing</strong>: The sender takes their message or document, which is often large in size, and converts it into a fixed-size string of characters called a <strong>message digest</strong> or <strong>hash value</strong> using a hash function (e.g., SHA-256). This process ensures that any small change to the original message will result in a completely different hash value.</li><li><strong>Digital Signature Creation</strong>: The sender encrypts their private key (<code>s</code>) with the public key of the recipient (<code>A</code>) and then combines it with the message digest using a mathematical operation like exponentiation modulo <code>n</code>. This produces a unique digital signature for the original message.</li><li><strong>Message Transmission</strong>: The sender transmits the digitally signed message (message + digital signature) to the recipient.</li></ol><p><strong>Digital Signature Verification:</strong></p><p>When receiving the digitally signed message, the recipient can verify its authenticity using their public key (<code>A</code>). Here&apos;s how:</p><ol><li><strong>Recover Private Key (<code>s</code>)</strong>: The recipient uses their public key (<code>A</code>) and the received digital signature to recover the private key used by the sender.</li><li><strong>Message Hashing (Again)</strong>: The recipient recreates the message digest from the original message, which should match the one obtained during the digital signature creation process.</li><li><strong>Verification</strong>: If the two hash values match, it confirms that the original message hasn&apos;t been tampered with and was indeed signed by the sender.</li></ol><h3 id="module-lattice-based-digital-signature">Module-Lattice-Based Digital Signature</h3><p>So a lot of this is the same as the stuff in FIPS 203. I&apos;ll provide a Python example for you to see how similar it is. </p><pre><code>import numpy as np

def rlsr_keygen(prime):
    # Generate large random numbers within the ring
    modulus = 2**12
    poly_degree = 4
    
    s = np.random.randint(0, 2**12, size=poly_degree)
    
    A = np.random.randint(0, 2**12, size=(poly_degree, poly_degree))
    e = np.random.randint(-2**11, 2**11, size=poly_degree)

    return s, A

def rlsr_sign(A, message):
    # Generate random polynomial to be used for signing
    modulus = 2**12
    
    r = np.random.randint(0, 2**12, size=4)
    
    # Compute signature with noise
    e_prime = np.random.randint(-modulus//2, modulus//2, size=4)
    c = np.dot(A, r) + e_prime
    
    return c

def rlsr_verify(s, A, c):
    # Compute difference between ciphertext and a multiple of the public key
    d = np.subtract(c, np.dot(A, s))
    
    # Check if message can be recovered from signature (in practice, this involves solving for r using LLL reduction)
    return True

def generate_signature():
    prime = 2**16 + 1  # Example value
    modulus = 2**12
    
    s, A = rlsr_keygen(prime)
    
    message = np.random.randint(0, 256, size=4)
    c = rlsr_sign(A, message)
    
    signature_validity = rlsr_verify(s, A, c)
    if signature_validity:
        print(&quot;Signature is valid.&quot;)
        return True
    else:
        print(&quot;Signature is invalid.&quot;)
        return False

generate_signature()
</code></pre><p>Basically the same concept as before but for signatures. </p><h2 id="fips-205-stateless-hash-based-digital-signature-standard">FIPS 205 (Stateless Hash-Based Digital Signature Standard)</h2><p></p><p>Link: <a href="https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf">https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.205.pdf</a></p><p>The Stateful-Light Hash-based Digital Signature Scheme (SLH-DSA) is a family of digital signature schemes that use hash functions and do not require any intermediate computations or stored state. SLH-DSAs are designed to be highly efficient and secure, making them suitable for various applications.</p><p>Basically because they use hashes and are stateless they are more resistant to quantum computers. </p><h3 id="basic-parts">Basic Parts</h3><ol><li>Forest of Random Subsets (FORS): A collection of random subsets generated from a large set.</li><li>Hash Functions: Used to compute the hash values for the subsets.</li><li>Subset Selection: A mechanism for selecting a subset of subsets based on the message to be signed.</li></ol><h3 id="how-it-works">How It Works</h3><ol><li>Key Generation: Generate multiple random subsets from a large set using a hash function (e.g., SHA-256).</li><li>Message Hashing: Compute the hash value of the message to be signed.</li><li>Subset Selection: Select a subset of subsets based on the hash value of the message.</li><li>Signature Generation: Generate a signature by combining the selected subsets.</li></ol><h3 id="multi-time-signature-scheme-extended-merkle-signature-scheme-xmss">Multi-Time Signature Scheme: Extended Merkle Signature Scheme (XMSS)</h3><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/09/image-23.png" class="kg-image" alt="Post-Quantum Cryptography Basics" loading="lazy" width="1280" height="550" srcset="https://matduggan.com/content/images/size/w600/2024/09/image-23.png 600w, https://matduggan.com/content/images/size/w1000/2024/09/image-23.png 1000w, https://matduggan.com/content/images/2024/09/image-23.png 1280w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Mathy </span><a href="https://en.wikipedia.org/wiki/Merkle_signature_scheme#/media/File:MerkleTree1.1.svg" rel="noreferrer"><span style="white-space: pre-wrap;">source</span></a></figcaption></figure><p></p><p>The Extended Merkle Signature Scheme (XMSS) is a multi-time signature scheme that uses the Merkle Tree technique to generate digital signatures. It is basically the following 4 steps to use. </p><ol><li><strong>Key Generation</strong>: Generate a Merkle Tree using multiple levels of random hash values.</li><li><strong>Message Hashing</strong>: Compute the hash value of the message to be signed.</li><li><strong>Tree Traversal</strong>: Traverse the Merkle Tree to select nodes that correspond to the message&apos;s hash value.</li><li><strong>Signature Generation</strong>: Generate a signature by combining the selected nodes.</li></ol><p><strong>Can I have a Python example?</strong></p><p>Honestly I really tried on this. But there was not a lot on the internet on how to do this. I will give you what I wrote, but it doesn&apos;t work and I&apos;m not sure exactly how to fix it. </p><ol><li><strong>Python QRL library.</strong> This seems like it&apos;ll work but I couldn&apos;t get the package to install successfully with Python 3.10, 3.11 or 3.12. </li><li><strong>Quantcrypt: </strong>This worked but honestly the &quot;example&quot; doesn&apos;t really show you anything interesting except that it seems to output what you think it should output. </li><li><strong>Standard library: </strong>I messed part of it up but I&apos;m not sure exactly where I went wrong. </li></ol><pre><code>import hashlib
import os

# Utility function to generate a hash of data
def hash_data(data):
    return hashlib.sha256(data).digest()

# Generate a pair of keys (simplified as random bytes for the demo)
def generate_keypair():
    private_key = os.urandom(32)  # Private key (random 32 bytes)
    public_key = hash_data(private_key)  # Public key derived by hashing the private key
    return private_key, public_key

# Create a simplified Merkle tree with n leaves
def create_merkle_tree(leaf_keys):
    # Create parent nodes by hashing pairs of leaf nodes
    current_level = [hash_data(k) for k in leaf_keys]
    
    while len(current_level) &gt; 1:
        next_level = []
        # Pair nodes and hash them to create the next level
        for i in range(0, len(current_level), 2):
            left_node = current_level[i]
            right_node = current_level[i+1] if i + 1 &lt; len(current_level) else left_node  # Handle odd number of nodes
            parent_node = hash_data(left_node + right_node)
            next_level.append(parent_node)
        current_level = next_level
    
    return current_level[0]  # Root of the Merkle tree

# Sign a message using a given private key
def sign_message(message, private_key):
    # Hash the message and then &quot;sign&quot; it by using the private key
    message_hash = hash_data(message)
    signature = hash_data(private_key + message_hash)
    return signature

def verify_signature(message, signature, public_key):
    message_hash = hash_data(message)
    # Instead of using public_key, regenerate what the signature would be if valid
    expected_signature = hash_data(public_key + message_hash)  # Modify this logic
    return expected_signature == signature

# Example of using the above functions

# 1. Generate key pairs for leaf nodes in the Merkle tree
tree_height = 4  # This allows for 2^tree_height leaves
num_leaves = 2 ** tree_height
key_pairs = [generate_keypair() for _ in range(num_leaves)]
private_keys, public_keys = zip(*key_pairs)

# 2. Create the Merkle tree from the public keys (leaf nodes)
merkle_root = create_merkle_tree(public_keys)
print(f&quot;Merkle Tree Root: {merkle_root.hex()}&quot;)

# 3. Sign a message using one of the private keys (simplified signing)
message = b&quot;Hello, this is a test message for XMSS-like scheme&quot;
leaf_index = 0  # Choose which key to sign with (0 in this case)
private_key = private_keys[leaf_index]
public_key = public_keys[leaf_index]

signature = sign_message(message, private_key)
print(f&quot;Signature: {signature.hex()}&quot;)

# 4. Verify the signature
is_valid = verify_signature(message, signature, public_key)
print(&quot;Signature is valid!&quot; if is_valid else &quot;Signature is invalid!&quot;)</code></pre><p>It always says signature is invalid. If you spot what I did wrong let me know, but honestly I sort of lost enthusiasm for this as we went. Hopefully the code you shouldn&apos;t be using at least provides some context. </p><h2 id="is-this-something-i-should-worry-about-now">Is This Something I Should Worry About Now?</h2><p>That really depends on what data you are dealing with. If I was dealing with tons of super-sensitive data, I would probably start preparing the way now. This isn&apos;t a change you are going to want to make quickly, in no small part to account for the performance difference in using some of these approaches vs more standard encryption. Were I working on something like a medical device or secure communications it would definitely be something I&apos;d at least spike out and try to see what it looked like. </p><p>So basically if someone asks you about this, I hope now you can at least talk intelligently about it for 5 minutes until they wander away from boredom. If this is something you actually have to deal with, start with PQClean and work from there.  </p>

enclosure_url:
https://images.unsplash.com/photo-1667581386428-ed64b9bee6a6?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDE4fHxxdWFudHVtfGVufDB8fHx8MTcyNzY5NjAzMnww&ixlib=rb-4.0.3&q=80&w=2000


guid: 673de310c0267b0001c01786
title: Self-Hosting Isn't a Solution; It's A Patch
author: Mathew Duggan
url: https://matduggan.com/self-hosting-isnt-a-solution-its-a-patch/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>For as long as I&#x2019;ve been around tech enthusiasts, there has been a recurring &#x201C;decentralization dream.&#x201D; While the specifics evolve, the essence remains the same: everyone would own a domain name and host their digital identity. This vision promises that people, liberated from the chore of digital maintenance, would find freedom in owning their slice of the internet. The basic gist is at some point people would wake up to how important online services are to them and demand some ownership over how they work. </p><p>This idea, however, always fails. From hosting email and simple HTML websites in my youth to the current attempts at decentralized Twitter- or YouTube-like platforms, the tech community keeps waiting for everyday people to take the baton of self-hosting. They never will&#x2014;because the effort and cost of maintaining self-hosted services far exceeds the skill and interest of the audience. The primary &#x201C;feature&#x201D; of self-hosting is, for most, a fatal flaw: it&#x2019;s a chore. It&#x2019;s akin to being &#x201C;free&#x201D; to change the oil in your car&#x2014;it&#x2019;s an option, but not a welcome one for most.</p><p>Inadvertently, self-hosting advocates may undermine their own goal of better privacy and data ownership. By promoting open-source, self-hosted tools as the solution for those concerned about their privacy, they provide an escape valve for companies and regulators alike. Companies can claim, &#x201C;Well, if users care about privacy, they can use X tool.&#x201D; This reduces the pressure for legal regulation. Even Meta&#x2019;s Threads, with its integration of ActivityPub, can claim to be open and competitive, deflecting criticism and regulation&#x2014;despite this openness being largely flowing from Threads to ActivityPub and not the other way around. </p><p>What people actually need are laws. Regulations like the GDPR must become the international standard for platforms handling personal data. These laws ensure a basic level of privacy and data rights, independent of whether a judge forces a bored billionaire to buy your favorite social network. Suggesting self-hosting as a solution in the absence of such legal protections is as naive as believing encrypted messaging platforms alone can protect you from government or employer overreach.</p><h3 id="what-do-users-actually-deserve">What do users actually deserve?</h3><p>We don&apos;t need to treat this as a hypothetical. What citizens in the EU get is the logical &quot;floor&quot; of what citizens around the world should demand. </p><ul><li>Right to access<ul><li>What data do you have on me?</li><li>How long do you keep it?</li><li>Why do you have it? What purpose does it serve?</li></ul></li><li>Right to Rectification<ul><li>Fix errors in your personal data.  </li></ul></li><li>Right to be Forgotten <ul><li>There&apos;s no reason when you leave a platform that they should keep your contribution forever. </li></ul></li><li>Right to Data Portability<ul><li>Transfer your data to another platform in a standardized machine-readable format. </li></ul></li><li>Right to Withdraw Consent<ul><li>Opt out of data collection whenever you want, even if you originally agreed.</li></ul></li></ul><p>These are not all GDPR rights, but they form the backbone of what allows users to engage with platforms confidently, knowing they have levers to control their data. Regulations like these are binding and create accountability&#x2014;something neither self-hosting nor relying on tech billionaires can achieve.</p><p>Riding this roller coaster of &quot;I need digital platforms to provide me essential information and access&quot; and trying to balance it with &quot;whatever rich bored people are doing this week&quot; has been a disaster. It&apos;s time to stop pretending these companies are our friends and force them to do the things they say they&apos;ll do when they&apos;re attempting to attract new users. </p><h3 id="the-fallacies-of-decentralization-as-a-solution"><strong>The fallacies of decentralization as a solution</strong></h3><p>The decentralization argument often assumes that self-hosted platforms or volunteer-driven networks are inherently superior. But this isn&#x2019;t practical:</p><ol><li><strong>Self-hosting platforms are fragile.</strong><ol><li>Shutting down a small self-hosted platform running on a VPS provider is pretty trivial. These are basically paid for by one or two people and they would be insane to fight any challenge, even a bad one. How many self-hosted platforms would stand up to a threatening letter from a lawyer, much less an actual government putting pressure on their hosting provider? </li><li>Even without external pressure there isn&apos;t any practical way to fund these efforts. You can ask for donations, but that&apos;s not a reliable source of revenue for a cost that will only grow over time. At a certain size the maintainer will need to form a nonprofit in order to continue collecting the donations, a logistical and legal challenge well outside of the skillset of the people we&apos;re talking about. </li><li>It&apos;s effectively free labor. You are taking a job, running a platform, removing the pay for that job, adding in all the complexity of running a nonprofit and adding in the joys of being the CPA, the CEO, the sysadmin, etc. At some point people get sick, they lose interest, etc. </li></ol></li><li><strong>Decentralization doesn&#x2019;t replace regulation.</strong><ol><li>While decentralization aligns with the internet&#x2019;s original ethos, it doesn&#x2019;t negate the need for legal protections. Regulations like GDPR raise the minimum level of privacy and security, while decentralization remains an optional enhancement. You lose nothing by moving the floor up. </li></ol></li><li><strong>Regulation is not inherently bad.</strong><ol><li>A common refrain among technical enthusiasts is a libertarian belief that market pressures and having a superior technical product will &quot;win out&quot; and legislation is bad because it constrains future development. You saw this a lot in the US tech press over the EU move from proprietary chargers to USB-C, a sense of &quot;well when the next big thing comes we won&apos;t be able to use it because of silly government regulation&quot;. <ol><li>Global legislation forces all companies&#x2014;not just a niche few catering to privacy enthusiasts&#x2014;to respect users&#x2019; rights. Unlike market-driven solutions or self-hosting, laws are binding and provide universal protections.</li><li>It is impossible for an average user to keep track of who owns which platforms and what their terms of service are now. Since they can be changed with almost no notice, whatever &quot;protections&quot; they can provide are laughably weak. In resisting legislation you make the job of large corporations easier, not harder. </li></ol></li></ol></li></ol><h3 id="the-reality-of-privacy-as-a-privilege">The reality of privacy as a privilege </h3><h3 id> </h3><p>Right now, privacy often depends on technical skills, financial resources, or sheer luck:</p><p>&#x2022; <strong>I value privacy and have money:</strong> You can pay for premium platforms like Apple or Fastmail. These platforms <em>could</em> change the rules whenever they want to but <em>likely</em> won&apos;t because their entire brand is based on the promise of privacy.</p><p>&#x2022; <strong>I value privacy and have technical skills:</strong> You can self-host and manage your own services.</p><p>&#x2022; <strong>I value privacy but lack money and technical skills:</strong> You&#x2019;re left hoping that volunteers or nonprofits continue offering free tools&#x2014;and that they don&#x2019;t disappear overnight. Or you try and keep abreast of a constant churning ecosystem where companies change hands all the time and the rules change whenever they want. </p><p>This is a gatekeeping problem. Privacy should not be a luxury or dependent on arbitrary skill sets. Everyone deserves it.</p><h3 id="it-actually-makes-a-difference">It actually makes a difference</h3><p>As someone who has experienced the difference between the U.S. and the EU&#x2019;s approach to privacy, I can attest to how much better life is with stronger regulations. GDPR isn&#x2019;t perfect, but it provides a foundation that improves quality of life for everyone. Instead of treating regulation as burdensome or unrealistic, we should view it as essential.</p><p>The dream of a decentralized internet isn&#x2019;t inherently wrong, but waiting for it to materialize as a universal solution is a mistake. Laws&#x2014;not utopian ideals&#x2014;are the only way to ensure that users everywhere have the protections they deserve. It&#x2019;s time to stop pretending companies will prioritize ethics on their own and instead force them to. </p>


guid: 6749837ec0267b0001c01816
title: Making Fun Flask Apps for No Reason
author: Mathew Duggan
url: https://matduggan.com/making-fun-flask-apps-for-no-reason/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>Sometimes life gets you down. Maybe it&apos;s a crushing political situation in your home country, perhaps you read the latest scientific data about global warming or hey sometimes you just need to make something stupid to remind yourself why you ever enjoyed doing this. Whatever the reason, let&apos;s take a load off and make a pointless Flask app. You can do it too! </p><h3 id="pokemon-tcg-pocket-friend-website">Pokemon TCG Pocket Friend Website</h3><p>I want to find some friends for the mobile game Pokemon TCG Pocket, but I don&apos;t want to make a new Reddit account and I don&apos;t want to join a Discord. So let&apos;s make one. It&apos;s a pretty good, straightforward one-day kind of problem. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/12/image-3.png" class="kg-image" alt loading="lazy" width="1099" height="613" srcset="https://matduggan.com/content/images/size/w600/2024/12/image-3.png 600w, https://matduggan.com/content/images/size/w1000/2024/12/image-3.png 1000w, https://matduggan.com/content/images/2024/12/image-3.png 1099w" sizes="(min-width: 720px) 720px"></figure><h3 id="why-flask">Why Flask?</h3><p>Python Flask is the best web framework for dumb ideas that you want to see turned into websites with as little work as possible. Designed for people like me who can hold no more than 3 complex ideas in their heads at a time, it feels like working with Rails if Rails didn&apos;t try to constantly wrench the steering wheel away from you and drive the car. </p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://matduggan.com/content/images/2024/11/image-2.png" class="kg-image" alt loading="lazy" width="1500" height="1120" srcset="https://matduggan.com/content/images/size/w600/2024/11/image-2.png 600w, https://matduggan.com/content/images/size/w1000/2024/11/image-2.png 1000w, https://matduggan.com/content/images/2024/11/image-2.png 1500w" sizes="(min-width: 720px) 720px"><figcaption><span style="white-space: pre-wrap;">Rails wants to drive for awhile</span></figcaption></figure><p>It&apos;s easy to start using, pretty hard to break and extremely easy to troubleshoot. </p><p>We&apos;re gonna try to time limit this pretty aggressively. I don&apos;t want to put in a ton of time on this project, because I think a <em>critical</em> part of a fun project is to get something out onto the Internet as quickly as possible. The difference between fun projects and work projects is the time gap between &quot;idea&quot; and &quot;thing that exists for people to try&quot;. We&apos;re also not going to obsess about trying to get everything perfectly right. Instead we&apos;ll take some small steps to try and limit the damage if we do something wrong. </p><h3 id="let-me-just-see-what-you-made-and-skip-the-tutorial">Let me just see what you made and skip the tutorial</h3><figure class="kg-card kg-bookmark-card"><a class="kg-bookmark-container" href="https://poketcg.club/"><div class="kg-bookmark-content"><div class="kg-bookmark-title">Pok&#xE9;mon Pocket TCG Friend Code Exchange</div><div class="kg-bookmark-description">Find and exchange Pok&#xE9;mon Pocket TCG friend codes with players worldwide.</div><div class="kg-bookmark-metadata"><img class="kg-bookmark-icon" src="https://matduggan.com/content/images/icon/favicon-19.ico" alt><span class="kg-bookmark-author">Pok&#xE9;mon Friend Finder</span></div></div><div class="kg-bookmark-thumbnail"><img src="https://poketcg.club/images/pokemon_logo.png" alt onerror="this.style.display = &apos;none&apos;"></div></a></figure><p>Source code here: <a href="https://gitlab.com/matdevdug/pokemontcg-friend-finder">https://gitlab.com/matdevdug/pokemontcg-friend-finder</a></p><p>Feel FREE to use this as the beginning template for anything fun that you make and please let me know if you make something cool I can try. </p><h3 id="note">Note:</h3><p>This is not a &quot;how do I Flask&quot; tutorial. This is showing you how you can use Flask to do fun stuff quickly, not the basics of how the framework operates. There&apos;s a good Flask tutorial you&apos;ll have to do in order to do the stuff I&apos;m talking about: <a href="https://flask.palletsprojects.com/en/stable/tutorial/">https://flask.palletsprojects.com/en/stable/tutorial/</a></p><h3 id="getting-started"><strong>Getting Started</strong></h3><p>Alright let&apos;s set this bad-boy up. We&apos;ll kick it off with my friend <code>venv</code>. Assuming you got Python from The Internet somewhere, let&apos;s start writing some routes. </p><pre><code>python3.12 -m venv venv
source venv/bin/activate
pip install flask</code></pre><p>Make a <code>hello.py</code> with </p><pre><code>from flask import Flask

app = Flask(__name__)


@app.route(&apos;/&apos;)
def hello():
    return &apos;Hello, World!&apos;</code></pre><p>Run it with <code>python hello.py</code> and enjoy your hello world. </p><h3 id="lets-start-writing-stuff">Let&apos;s start writing stuff</h3><p>Basically Flask apps have a few parts. There&apos;s a <code>config</code>, the <code>app</code>, <code>templates</code> and <code>static</code>. But before we start all that, let&apos;s just quickly define what we actually need. </p><ul><li>We need an index.html as the /</li><li>We need a /sitemap.xml for search engines</li><li>Gonna need a /register for people to add their codes</li><li>Probably want some sort of /search</li><li>If we have user accounts you probably want a /profile</li><li>Finally gonna need a /login and /logout</li></ul><p>So to store all that junk we&apos;ll probably want a database but not something complicated because it&apos;s friend codes and we&apos;re not looking to make something serious here. SQLite it is! Also nice because we&apos;re trying to bang this out in one day so easier to test. </p><p>At a basic level Flask apps work like this. You define a <code>route</code> in your <code>app.py</code> (or whatever you want to call it. </p><pre><code>@app.route(&apos;/&apos;)
def main():
    return render_template(&quot;index.html&quot;)</code></pre><p>Then inside of your <code>templates</code> directory you have some Jinja2 templates that will get rendered back to the client. Here is my <code>index.html</code></p><pre><code>{% extends &quot;base.html&quot; %}

{% block content %}
&lt;div class=&quot;container mt-4&quot;&gt;
    &lt;h1 class=&quot;text-center text-danger&quot;&gt;Pok&#xE9;mon TCG Friend Finder&lt;/h1&gt;
    &lt;p&gt;Welcome to the Pok&#xE9;mon TCG Friend Finder, where you can connect with players from all over the world!&lt;/p&gt;

    &lt;div class=&quot;mt-4&quot;&gt;
        &lt;h4&gt;How to Find Friend Codes:&lt;/h4&gt;
        &lt;p&gt;To browse friend codes shared by other players, simply visit our &lt;a class=&quot;btn btn-primary btn-sm&quot; href=&quot;{{ url_for(&apos;find_friends&apos;) }}&quot;&gt;Find Friends&lt;/a&gt; page. No registration is required!&lt;/p&gt;
    &lt;/div&gt;

    &lt;div class=&quot;mt-4&quot;&gt;
        &lt;h4&gt;Want to Share Your Friend Code?&lt;/h4&gt;
        &lt;p&gt;If you&apos;d like to share your own friend code and country, you need to register for an account. It&apos;s quick and free!&lt;/p&gt;
        &lt;p&gt;
            {% if current_user.is_authenticated %}
                &lt;a class=&quot;btn btn-primary&quot; href=&quot;{{ url_for(&apos;find_friends&apos;) }}&quot;&gt;Visit Find Friends&lt;/a&gt;
            {% else %}
                &lt;a class=&quot;btn btn-success&quot; href=&quot;{{ url_for(&apos;register&apos;) }}&quot;&gt;Register&lt;/a&gt; or
                &lt;a class=&quot;btn btn-info&quot; href=&quot;{{ url_for(&apos;login&apos;) }}&quot;&gt;Log in&lt;/a&gt; to get started!
            {% endif %}
        &lt;/p&gt;
    &lt;/div&gt;

    &lt;div class=&quot;mt-4&quot;&gt;
        &lt;h4&gt;Spread the Word:&lt;/h4&gt;
        &lt;p&gt;Let others know about this platform and grow the Pok&#xE9;mon TCG community!&lt;/p&gt;
        &lt;div class=&quot;share-buttons&quot;&gt;
            &lt;a href=&quot;#&quot; onclick=&quot;shareOnFacebook()&quot; title=&quot;Share on Facebook&quot;&gt;
                &lt;img src=&quot;{{ url_for(&apos;static&apos;, filename=&apos;images/facebook.png&apos;) }}&quot; alt=&quot;Share on Facebook&quot; style=&quot;width: 64px;&quot;&gt;
            &lt;/a&gt;
            &lt;a href=&quot;#&quot; onclick=&quot;shareOnTwitter()&quot; title=&quot;Share on Twitter&quot;&gt;
                &lt;img src=&quot;{{ url_for(&apos;static&apos;, filename=&apos;images/twitter.png&apos;) }}&quot; alt=&quot;Share on Twitter&quot; style=&quot;width: 64px;&quot;&gt;
            &lt;/a&gt;
            &lt;a href=&quot;#&quot; onclick=&quot;shareOnReddit()&quot; title=&quot;Share on Reddit&quot;&gt;
                &lt;img src=&quot;{{ url_for(&apos;static&apos;, filename=&apos;images/reddit.png&apos;) }}&quot; alt=&quot;Share on Reddit&quot; style=&quot;width: 64px;&quot;&gt;
            &lt;/a&gt;
        &lt;/div&gt;
    &lt;/div&gt;
&lt;/div&gt;

&lt;!-- JavaScript for sharing --&gt;
&lt;script&gt;
    const url = encodeURIComponent(window.location.href);
    const title = encodeURIComponent(&quot;Check out Pok&#xE9;mon TCG Friend Finder!&quot;);

    function shareOnFacebook() {
        window.open(`https://www.facebook.com/sharer/sharer.php?u=${url}`, &apos;_blank&apos;);
    }

    function shareOnTwitter() {
        window.open(`https://twitter.com/intent/tweet?url=${url}&amp;text=${title}`, &apos;_blank&apos;);
    }

    function shareOnReddit() {
        window.open(`https://www.reddit.com/submit?url=${url}&amp;title=${title}`, &apos;_blank&apos;);
    }
&lt;/script&gt;
{% endblock %}</code></pre><p>Some quick notes:</p><ul><li>I am using Bootstrap because Bootstrap let&apos;s people who are not good at frontend do one of those really quickly: <a href="https://getbootstrap.com/">https://getbootstrap.com/</a></li><li>You&apos;ll notice the <code>base.html</code> which lets you import the base template across all the other templates so you don&apos;t need to redo the basic menu navigation. You can see that here: <a href="https://gitlab.com/matdevdug/pokemontcg-friend-finder/-/blob/main/templates/base.html?ref_type=heads">https://gitlab.com/matdevdug/pokemontcg-friend-finder/-/blob/main/templates/base.html?ref_type=heads</a></li></ul><p>Basically that&apos;s it. You make a route on Flask that points to a template, the template is populated from data from your database and you proudly display it for the world to see. </p><p>Instead let me run you through what I did that isn&apos;t &quot;in the box&quot; with Flask and why I think it helps.</p><h3 id="recommendations-to-do-this-real-fast">Recommendations to do this real fast</h3><ol><li><strong>Start with it inside of a container from the beginning. </strong></li></ol><pre><code>FROM python:3.12-slim

# Create a non-root user
RUN groupadd -r nonroot &amp;&amp; useradd -r -g nonroot nonroot
WORKDIR /app

COPY requirements.txt .

RUN pip3 install -r requirements.txt

COPY . .

RUN chown -R nonroot:nonroot /app

USER nonroot

ENTRYPOINT [&quot;./gunicorn.sh&quot;]</code></pre><p>You are going to have to use a different HTTP server for Flask anyway, gunicorn is.....one of those. So you might as well practice like you play. Here is the compose file</p><pre><code>version: &apos;3.8&apos;

services:
  app:
    build: .
    ports:
      - &quot;9000:9000&quot;
    volumes:
      - ~/Documents/data:/data
    environment:
      - FLASK_ENV=development
      - DATABASE_PATH=/data/users.db
      - SECRET_KEY=&quot;pleasechangeme&quot;
      - API_KEY=&quot;secrettttsss&quot;</code></pre><p>Change the volumes to be wherever you want the database mounted. This is for local development but switching it to &quot;prod&quot; should be pretty straight forward. </p><ol start="2"><li><strong>Move <code>config</code> and <code>models</code> to their own thing. </strong></li></ol><p>Inside of the app.py (available here: <a href="https://gitlab.com/matdevdug/pokemontcg-friend-finder/-/blob/main/app.py?ref_type=heads">https://gitlab.com/matdevdug/pokemontcg-friend-finder/-/blob/main/app.py?ref_type=heads</a>) you&apos;ll see <code>from config import Config</code> and <code>from models import db, bcrypt, User</code>. This isn&apos;t required but I find mentally it helps to have them in their own files. </p><p><code>config</code> is just &quot;the stuff you are using to configure your application&quot; </p><pre><code>import os

class Config:
    SECRET_KEY = os.environ.get(&quot;SECRET_KEY&quot;) or &quot;secretttssss&quot;
    SQLALCHEMY_DATABASE_URI = f&quot;sqlite:///{os.getenv(&apos;DATABASE_PATH&apos;, &apos;/data/users.db&apos;)}&quot;
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    WTF_CSRF_ENABLED = True
    if os.getenv(&apos;FLASK_ENV&apos;) == &apos;development&apos;:
        DEBUG = True
    else:
        DEBUG = False
        SERVER_NAME = &quot;poketcg.club&quot;</code></pre><p>Finally the <code>models</code> stuff is just the database things broken out to their own file. </p><pre><code>from flask_sqlalchemy import SQLAlchemy
from flask_bcrypt import Bcrypt
from flask_login import UserMixin, LoginManager

db = SQLAlchemy()
bcrypt = Bcrypt()
login_manager = LoginManager()

class User(db.Model, UserMixin):
    id = db.Column(db.Integer, primary_key=True)
    username = db.Column(db.String(150), unique=True, nullable=False)
    password = db.Column(db.String(150), nullable=False)
    friend_code = db.Column(db.String(50))
    country = db.Column(db.String(50))
    friend_requests = db.Column(db.Integer, default=0)</code></pre><p>You&apos;ll probably want to do a better job of defining the data you are inputting into the database than I did, but move fast break things etc. </p><ol start="3"><li><strong>Logs are pretty much all you are gonna have</strong></li></ol><p>Python logging library is unfortunately relatively basic, but important to note that this is going to be pretty much the only way you will know if something is working or not. </p><pre><code># Logging configuration
logging.basicConfig(level=logging.INFO,
                    format=&apos;%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d]&apos;,
                    handlers=[logging.FileHandler(&quot;app.log&quot;), logging.StreamHandler()])</code></pre><p>That&apos;s writing out to a log file and also stdout. You can choose either/or depending on what you want, with the understanding that it&apos;s more container-y to run them just as stdout. </p><ol start="4"><li><strong>Monitor Basic Response Times</strong></li></ol><p>So when I&apos;m just making the app and I want to see &quot;how long it takes to do x&quot; I&apos;ll add a very basic logging element to track &quot;how long did Flask thing the request took&quot;. </p><pre><code># Performance tracking
@app.before_request
def before_request():
    request.start_time = time.perf_counter()

@app.after_request
def after_request(response):
    if hasattr(request, &apos;start_time&apos;):
        duration = time.perf_counter() - request.start_time
        logging.info(
            f&quot;Request to {request.path} took {duration:.2f} seconds. &quot;
            f&quot;{response.status_code}.&quot;
        )
        return response</code></pre><p>This doesn&apos;t tell you a lot but it usually tells me &quot;whoa that took WAY too long something is wrong&quot;. It&apos;s pretty easy to put OpenTelemetry into Flask but that&apos;s sort of overkill for what we&apos;re talking about. </p><ol start="5"><li><strong>Skipping Emails and Password Resets</strong></li></ol><p>One thing that consumes a ton of time when working on something like this is coming up with the account recovery story. I&apos;ve written a ton on this before so I won&apos;t bore you with that again, but my recommendation for fun apps is just to skip it. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/12/image.png" class="kg-image" alt loading="lazy" width="1142" height="392" srcset="https://matduggan.com/content/images/size/w600/2024/12/image.png 600w, https://matduggan.com/content/images/size/w1000/2024/12/image.png 1000w, https://matduggan.com/content/images/2024/12/image.png 1142w" sizes="(min-width: 720px) 720px"></figure><p>In terms of account management make it <em>super easy</em> for the user to delete their account. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2024/12/image-1.png" class="kg-image" alt loading="lazy" width="1137" height="288" srcset="https://matduggan.com/content/images/size/w600/2024/12/image-1.png 600w, https://matduggan.com/content/images/size/w1000/2024/12/image-1.png 1000w, https://matduggan.com/content/images/2024/12/image-1.png 1137w" sizes="(min-width: 720px) 720px"></figure><h3 id="deploying-to-production">Deploying to Production</h3><p>The most straightforward way to do this is Docker Compose with a file that looks something like the following:</p><pre><code>services:
  app:
    build: .
    volumes:
      - /mnt/data:/data
    environment:
      - FLASK_ENV=production
      - DATABASE_PATH=/data/users.db
      - SECRET_KEY=&quot;make a good secret here please&quot;

  caddy:
    image: caddy:latest
    ports:
      - &quot;80:80&quot;
      - &quot;443:443&quot;
    volumes:
      - caddy_data:/data
      - caddy_config:/config
      - ./Caddyfile:/etc/caddy/Caddyfile
    environment:
      - CADDY_AUTOHTTPS=on

volumes:
  caddy_data:
  caddy_config:</code></pre><p>Then you need the <code>Caddyfile</code> which looks something like this:</p><pre><code>example.com {
    reverse_proxy app:9000
}</code></pre><p>There are even easier options I outline here: <a href="https://matduggan.com/easier-alternative-to-nginx-lets-encrypt-with-caddy/">https://matduggan.com/easier-alternative-to-nginx-lets-encrypt-with-caddy/</a></p><p><strong>Some quick checks</strong></p><ul><li>Set up seccomp profiles on your container: <a href="https://docs.docker.com/reference/cli/docker/container/run/#security-opt">https://docs.docker.com/reference/cli/docker/container/run/#security-opt</a> <code>--security-opt=&quot;no-new-privileges=true&quot;</code></li><li>Setting up the network correctly</li></ul><pre><code>networks:
  internal:
    driver: bridge

services:
  app:
    networks:
      - internal
  caddy:
    networks:
      - internal
      - default</code></pre><p>You can decide how complicated or simple you want to make this, but you should be able to (pretty easily) set this up on anything from a Pi to a $5 a month server. </p><h3 id="see-wasnt-that-hard">See? Wasn&apos;t that hard!</h3><p>So is my website a giant success? Absolutely not. I&apos;ve only gotten a handful of users on it and I&apos;m not optimistic anyone will ever use it. But I did have a ton of fun making it, so honestly mission success. </p><p>Questions/comments/concerns: <a href="https://c.im/@matdevdug">https://c.im/@matdevdug</a></p>


guid: 67516d5ac0267b0001c018d3
title: Career Advice for New Tech Workers in 2025
author: Mathew Duggan
url: https://matduggan.com/career-advice-for-new-tech-workers-in-2025-2/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>Giving advice is always tricky&#x2014;especially now, in a world where everything seems to change every week. Recently, I had the chance to chat with some folks who&#x2019;ve either just graduated or are about to graduate and are looking for jobs in tech. It was an informal, unstructured conversation, which, frankly, was refreshing.</p><p>A lot of the exchange surprised me in good ways. But one thing stood out: how little they actually knew about <em>working</em> for a tech company.</p><p>Most of what they knew came from one of three places:</p><p>1. Corporate blog posts gushing about how amazing life is for employees.</p><p>2. LinkedIn (because those who don&apos;t work post on LinkedIn about work).</p><p>3. Random anecdotes from people they&#x2019;d bumped into after graduating and starting their first job.</p><p>Spoiler alert: Much of this information is either lies, corporate fantasy, or both.</p><p>If history is written by the victors, then the narrative about life inside tech companies is written (or at least approved) by the marketing department. It&#x2019;s polished, rehearsed, and utterly useless for preparing you for the reality of it all. That&#x2019;s what I&#x2019;m here to do: share a bit of that reality.</p><p>I&#x2019;ve been fortunate enough to work at both Fortune 500 companies and scrappy startups with 50 people, across the US and Europe. I&#x2019;m not claiming to know <em>everything</em> (far from it). In fact, I&#x2019;d strongly encourage others to chime in and add their own insights. The more voices, the better.</p><p>That said, I think I&#x2019;ve been around long enough to offer a perspective worth sharing.</p><p>One last thing: this advice is engineering-specific. I haven&#x2019;t worked in other roles, so while some of this might apply to other fields, don&#x2019;t hold me accountable if it doesn&#x2019;t. Consider yourself warned.</p><h3 id="where-should-i-apply">Where Should I Apply?</h3><p>A lot of newcomers gravitate toward companies they&#x2019;ve heard of&#x2014;and honestly, that&#x2019;s not the approach I&#x2019;d recommend. The bigger and fancier a tech company is, the less their tech stack resembles anything you&#x2019;ll ever see anywhere else.</p><p>Take it from someone who&#x2019;s watched ex-Googlers and former Facebook folks struggle to understand GitHub: starting at the top isn&#x2019;t always the blessing it sounds like. (Seriously, just because you worked on a world-class distributed system doesn&#x2019;t mean you know how to use Postgres.)</p><p>Finding your &#x201C;sweet spot&#x201D; takes time. For me, it&#x2019;s companies with 200 to 1,000 employees. That size means they&#x2019;re big enough to have actual resources but not so bloated with internal politics that all your time is spent in meetings about meetings. Bonus: you might even get to ship something users will see!</p><p>Here&#x2019;s a pro tip: <strong>Don&#x2019;t assume name recognition = better place to work.</strong> Plenty of people at Apple hate their jobs. Sometimes, smaller companies will give you way better experience and way fewer existential crises.</p><p>And remember: your first job isn&#x2019;t your forever job. The goal here is just to get something solid on your resume and learn the ropes. That&#x2019;s it.</p><h3 id="interview">Interview</h3><p>Congratulations! You&#x2019;ve slogged through a million online applications and finally landed your first interview. Exciting, right? Before you dive in, here are some tips for navigating this bizarre ritual, especially if you&#x2019;ve never done it before.</p><ul><li><strong>Interviews &#x2260; The Job </strong><ul><li>Let&#x2019;s get this straight: interviews have almost <em>nothing</em> to do with the job you&#x2019;ll actually be doing. They&#x2019;re really just opportunities for engineers to ask whatever random questions they think are good &#x201C;signals.&#x201D;</li><li>Someone once said that technical interviews are like &#x201C;showing you can drive a race car so you can get a job driving a garbage truck.&#x201D; They weren&#x2019;t wrong.</li></ul></li><li><strong>Protect Your Time </strong><ul><li>Here&#x2019;s the deal: some companies will waste your time. A lot of it.</li><li>I&#x2019;ve been through this circus&#x2014;take-home tests, calls with unrelated questions, and in-person interviews where no one has even glanced at my earlier work. It&#x2019;s maddening.</li><li>&#x2022; <strong>Take-Home Assignments</strong>: Fine, but they should replace <em>some</em> in-person tests, not stack on top of them.</li><li>&#x2022; <strong>In-Person Interviews</strong>: If they insist on multiple rounds, the interviewers better be sharing notes. If not, walk away.</li><li>Remember: the average interview doesn&#x2019;t lead to a job. Don&#x2019;t ditch other opportunities because you&#x2019;re on round three with a &#x201C;Big Company.&#x201D; You&#x2019;re not as close to an offer as you might think.</li></ul></li><li><strong>Watch for the Sunk-Cost Fallacy</strong><ul><li>If you find yourself doing two take-homes, six interviews, and a whiteboard session, ask yourself: <em>Is this really worth it?</em></li><li>I&#x2019;ve ended interview processes midstream when I realized they were wasting my time&#x2014;and I&#x2019;ve <em>never</em> regretted it. On the flip side, every time I&#x2019;ve stuck it out for those marathon processes, the resulting job was&#x2026;meh. Not worth it.</li></ul></li><li><strong>Ask the Hard Questions</strong><ul><li>This is your chance to interview <em>them</em>. Don&#x2019;t hold back:<ul><li><strong>On-Call Work</strong>: Ask to see recent pages. Are they fixing real problems, or are you signing up for a never-ending nightmare?</li><li><strong>Testing</strong>: What&#x2019;s their test coverage like? Are the tests actually useful, or are they just there to pass &#x201C;sometimes&#x201D;?</li><li><strong>Turnover</strong>: What&#x2019;s the average tenure? If everyone&#x2019;s left within 18 months, that&#x2019;s a big, waving red flag.</li></ul></li></ul></li><li><strong>&#xA0;Embrace the Leetcode Circus</strong><ul><li>I know, I know&#x2014;Leetcode is frustrating and ridiculous, but there&#x2019;s no way around it. Companies <em>love</em> to make you grind through live coding problems. Just prepare for it, get through it, and move on.</li></ul></li><li><strong>Failure Is Normal</strong><ul><li>Failing an interview means absolutely nothing. It stings, sure, especially after investing so much time. But rejection is just part of the process. Don&#x2019;t dwell on it.</li></ul></li></ul><p><strong>Common Questions</strong></p><ul><li>Why are interviews so stupid if they&apos;re also really expensive to do? <ul><li>Because interview time is treated like &#x201C;free time.&#x201D; Your time, their time&#x2014;it&#x2019;s all apparently worthless. Why? Probably something they heard in a TED talk.</li></ul></li><li>Should I contribute to an open-source project to improve my resume?<ul><li>No. Companies that rely on open-source rarely contribute themselves, so why should you? Unless you want to be an unpaid idealist, skip it.</li></ul></li><li><strong>&#xA0;</strong>I&#x2019;ve always dreamed of working at <em>[Insert Company]</em>. Should I show my enthusiasm?<ul><li>Absolutely not. Recruiters can <em>smell</em> enthusiasm, and it works against you. For some reason, acting mildly disinterested is the best way to convince a company you&#x2019;re a great fit. Don&#x2019;t ask me why&#x2014;it just works.</li></ul></li></ul><h3 id="getting-started">Getting Started</h3><p>One of the biggest mistakes newcomers make in tech is believing that technical decisions are what truly matter. Spoiler: they&#x2019;re not.</p><p>Projects don&#x2019;t succeed or fail because of technical issues, and teams don&#x2019;t thrive or collapse based on coding prowess alone. The reality is this: <strong>human relationships</strong> are the most critical factor in determining your success. Everything else is just noise that can (usually) be worked around or fixed.</p><p><strong>Your Manager Is Key</strong></p><p>When starting out, the most important relationship you&#x2019;ll have is with your manager. Unfortunately, management in tech is&#x2026;well, often terrible.</p><p>Most managers fall into two categories:</p><p>1. <strong>The Engineer Turned Manager</strong>: An engineer who got told, &#x201C;Congrats, you&#x2019;re a manager now!&#x201D; without any training.</p><p>2. <strong>The People Manager&#x2122;</strong>: Someone who claims to be an &#x201C;expert&#x201D; at managing people but doesn&#x2019;t understand (or care about) how software development actually works.</p><p>Both types can be problematic, but the former engineer is usually the lesser evil. Great managers <em>do</em> exist, but they&#x2019;re rare. Teams with good managers tend to have low turnover, so odds are, as a junior, you&#x2019;re more likely to get stuck with a bad one.</p><p>The way you know you have a good manager is they understand the role is one of service. It&apos;s not about extracting from you, it&apos;s about enabling you to do the thing you know how to do. They do the following stuff:</p><ul><li>They don&apos;t shy away from hard tasks</li><li>They understand the tasks their team is working on and can effortlessly answer questions about them</li><li>Without thinking about it you would say they add value to most interactions they are in</li><li>They have a sense of humor. It&apos;s not all grim productivity. </li></ul><p>Let&#x2019;s meet the cast of bad managers you might encounter&#x2014;and how to survive them.</p><h3 id="1800s-factory-tycoon">1800s Factory Tycoon</h3><p>This manager runs their team like an industrial assembly line and sees engineers as interchangeable cogs. Their dream? Climbing the corporate ladder as quickly as possible.</p><p><strong>Signs You Work for a Tycoon</strong></p><ul><li><strong>Obsession with &#x201C;Keeping Things Moving&#x201D;</strong>: Complaints or objections? Clearly, someone&#x2019;s trying to sabotage the conveyor belt.</li><li><strong>&#x201C;Everyone Is Replaceable&#x201D;</strong>: They dismiss concepts like institutional knowledge or individual skill levels. An engineer is just someone who &#x201C;writes code at a computer.&#x201D;</li><li><strong>No Onboarding</strong>: &#x201C;Here&#x2019;s the codebase. It&#x2019;s self-documenting.&#x201D; Then they wander off.</li><li><strong>Fear of Change</strong>: Nothing new gets approved unless it meets impossible criteria:<ul><li>Zero disruption to current work.</li><li>Requires no training.</li><li>Produces a shiny metric showing immediate improvement.</li><li>No one else needs to do anything differently.</li></ul></li><li><strong>I Am The Star</strong>. They&apos;re the face of every success and the final word in every debate. You exist to implement <em>their dream</em> and when the dream doesn&apos;t match what they imagined <em>it&apos;s because you suck</em>. </li></ul><p><strong>Tips for Dealing With The Tycoon</strong></p><ul><li>Stay out of their way.</li><li>Expect zero career support&#x2014;build your own network and contacts.</li><li>Be patient. Tycoons often self-destruct when their lack of results catches up to them. A new manager will eventually step in.</li></ul><h3 id="sad-engineer">Sad Engineer </h3><p>This person was a good engineer who got promoted into management&#x2026;without any guidance. They mean well but often struggle with the non-technical aspects of leadership.</p><p>&#x2022; <strong>Can&#x2019;t Stay Out of the Code</strong>: They take on coding tasks, but no one on the team feels comfortable critiquing their PRs.</p><p>&#x2022; <strong>Poor Communication</strong>: They&#x2019;ll multitask during meetings, avoid eye contact, or act distracted&#x2014;leaving their team feeling undervalued.</p><p>&#x2022; <strong>Technical Debates &gt; Team Conflicts</strong>: Great at discussing system architecture, useless at resolving interpersonal issues.</p><p>&#x2022; <strong>No Work/Life Balance</strong>: They&#x2019;re often workaholics, unintentionally setting a toxic example for their team.</p><p><strong>Tips For Dealing with the SE</strong></p><ul><li>Be direct. Subtle hints about what you need or dislike will go over their head.</li><li>Use their technical expertise to your advantage&#x2014;they love helping with code.</li><li>Don&#x2019;t get attached. SEs rarely stick with management roles for long.</li></ul><h3 id="jira-as-a-religion-jaar"><strong>Jira as a Religion (JaaR)</strong></h3><p>The JaaR manager believes in the divine power of <em>process</em>. They dream of turning the chaos of software development into a predictable, assembly-line-like utopia. Spoiler: it never works.</p><p>&#x2022; <strong>Obsessed with Estimates</strong>: They treat software delivery like Amazon package tracking, demanding updates down to the minute.</p><p>&#x2022; <strong>Can&#x2019;t Say No</strong>: They agree to every request from other teams, leaving their own team drowning in extra work.</p><p>&#x2022; <strong>The Calendar Always Wins</strong>: Quality doesn&#x2019;t matter; deadlines are sacred. If you ship a mess, that&#x2019;s fine&#x2014;as long as it&#x2019;s on time.</p><p>&#x2022; <strong>Meetings. Endless Meetings.</strong>: Every decision requires a meeting, a slide deck, and 20 attendees. Progress crawls.</p><p><strong>Tips for Surviving the JaaR</strong></p><p>&#x2022; <strong>Find the Real Decision-Makers</strong>: JaaRs defer to others. Identify who actually calls the shots and work with them directly.</p><p>&#x2022; <strong>Play Their Game</strong>: Turn everything into a ticket. They love tickets. Detailed ones make them feel productive, even if nothing gets done.</p><h2 id="your-team">Your Team</h2><p><strong>The Core Reality of Technical Work</strong></p><p>Technical work is often driven by passion. Many engineers enjoy solving problems and building things, even in their free time. However, one universal truth often emerges: <strong>organizational busywork expands to fill the available working day.</strong> No matter how skilled or motivated your team is, you will face an inevitable struggle against the growing tide of meetings, updates, and bureaucracy.</p><p>Every action you and your teammates do is designed to try and <em>delay</em> this harsh reality for as long as possible. Nobody escapes it, eventually you will be crushed by endless amounts of status updates, tickets, syncs and Slack channels. You&apos;ll know it&apos;s happened when you are commuting home and can&apos;t remember anything you actually did that week. </p><p>Teams will often pitch one of the following in an attempt to escape the looming black hole of busywork. Don&apos;t let yourself get associated with one of these ideas that almost never works out. </p><p><strong>Common Losing Concepts in Team Discussions</strong></p><ol><li><strong>&#x201C;If we adopt X, all our problems go away.&#x201D;</strong><br>Tools and technologies can solve some problems but always introduce new challenges. Stay cautious when someone pitches a &#x201C;silver bullet&#x201D; solution.</li><li><strong>&#x201C;We&#x2019;re being left behind!&#x201D; </strong><br>Tech loves fads and you don&apos;t have to participate in every one. Don&apos;t stress about &quot;well if I don&apos;t work with Y technology then nobody will ever hire me&quot;. It&apos;s not true. Productivity in software development doesn&apos;t grow by leaps and bounds year over year regardless of how &quot;fast&quot; you adopt new junk</li><li><strong>&#x201C;We need to tackle the backlog.&#x201D; </strong>Backlogs often consist of: <ol><li><strong>Important but complex tasks</strong> requiring effort and time.</li><li><strong>Low-value tasks</strong> that remain because no one has questioned their necessity.</li></ol></li></ol><p>If a thing brings a lot of value to users and isn&apos;t terribly painful to write, developers will almost always snap it up for the dopamine rush of making it. This means addressing a backlog without critically evaluating its contents is a waste of resources.</p><p>4. <strong>&#x201C;We need detailed conventions for everything.&#x201D;</strong></p><p>While consistency is good, overengineering processes can be counterproductive. Practical examples and lightweight guidelines often work better than rigid rules.</p><p>5. <strong>&#x201C;Let&#x2019;s rewrite everything in [new language].&#x201D;</strong></p><p>Rewrites are rarely worth the cost. Evolutionary changes and refactoring are almost always more effective than starting over.</p><h3 id="team-dynamics">Team Dynamics</h3><p></p><p><strong>The Golden Rule</strong></p><p><strong>People can either be nice or good at their jobs, both are equally valuable. </strong></p><p>Teams need balance. While &#x201C;10x engineers&#x201D; may excel at writing code, they often disrupt team dynamics by overengineering or pursuing unnecessary rewrites. A harmonious team with diverse strengths typically achieves more than one overloaded with &#x201C;superstars.&#x201D;</p><p><strong>Tips for Thriving on a Team</strong></p><p><strong>1. Quietly Grind on the Problem</strong></p><p>As a junior developer, expect to spend a lot of time simply reading code and figuring things out. While asking for help is encouraged, much of your growth comes from independently wrestling with problems.</p><ul><li>Start with small, low-risk changes to learn the codebase and deployment process.</li><li>Accept that understanding a system takes time, and no explanation can replace hands-on exploration.</li></ul><p><strong>2. Understand How Your Team Communicates</strong></p><p>Some teams use Slack for everything; others rely on email or tools like GitHub. Learn the preferred methods of communication and actively participate.</p><p><strong>3. Create a &#x201C;How We Make Money&#x201D; Diagram</strong></p><p>Understanding your company&#x2019;s business model helps you prioritize your work effectively. Map out how the company generates revenue and identify critical systems.</p><p>&#x2022; Focus on less-sensitive parts of the system for experimentation or learning.</p><p>&#x2022; Warn teammates when working on the &#x201C;make money&#x201D; areas to avoid unnecessary risks.</p><p><strong>4. Plan Before Coding</strong></p><p>The more complex a problem, the more time you should spend planning. Collaboration and context are key to successful implementation.</p><p>&#x2022; Discuss proposed changes thoroughly.</p><p>&#x2022; Understand the historical context behind existing decisions.</p><p>Assume most of the decisions you are looking at were made by a person as smart or smarter than you and work from there has been a good mental framework when joining a team for me. </p><h3 id="take-care-of-yourself">Take Care Of Yourself</h3><p>Tech work can be unpredictable&#x2014;projects get canceled, layoffs happen, or sometimes a lucky break might land you a new opportunity. Regardless, <strong>this is your career</strong>, and it&#x2019;s essential to make sure you&#x2019;re learning and advancing at each job, because you never know when you might need to move on.</p><ul><li><strong>All Offices Suck</strong><ul><li>Open-office layouts are a disaster for anyone who values focus, especially for new employees. You&#x2019;re constantly bombarded by conversations and interruptions, often with no warning. The office environment may look sleek and modern, but it&#x2019;s rarely conducive to concentration or productivity.</li><li>If you need to drown out the sound of people talking or noise with headphones, your leadership cares more about how an office looks to a visitor than how it functions. Try to find a quiet spot where you can sit and work on your tasks. </li></ul></li><li><strong>Monitor Turnover</strong><br>High employee turnover can be one of the most damaging issues for a team. Not only does it drain time and resources through offboarding, interviewing, and training, but it also kills morale and focus.<ul><li>Turnover, through voluntary leavings or layoffs, is one of the most destructive things to your team dynamics and overall morale. Turnover by itself costs about 20% of all the hours invested by a team (offboarding, interviewing, bringing someone new on), but there&apos;s a more sinister angle. <ul><li>Teams with high turnover don&apos;t care about the future. You&apos;ll often get trapped in a death spiral of watching a looming problem grow and grow but unable to get anybody invested because they&apos;re already planning their exit. </li><li>Now high-turnover teams also promote really quickly, so this could be a good opportunity to get some more senior titles on your resume. But be warned that these teams are incredibly chaotic and often are shut down with little warning by upper management. </li><li>High turnover also says management is shitty at cost-benefit analysis. It takes <em>forever</em> to onboard engineers, especially in niche or high-complexity sectors. </li></ul></li><li>Try to find out <em>why</em> people are leaving. <ul><li>Is this &quot;just a job&quot; to them? Fine, let it be that way to you. </li><li>Do they feel disposable or ignored? That suggests more serious management issues that go up the chain. </li><li>&quot;Loyalty is for idiots&quot;. People naturally want to be loyal to organizations and teams, so if the sentiment is &quot;only a sucker would be loyal to this place&quot;, understand you don&apos;t likely have a long career here. </li></ul></li></ul></li><li><strong>Corporate Goals Don&apos;t Matter</strong><ul><li>A lot of companies assume that whatever their stated high-level goals are somehow matter to you at the bottom. They don&apos;t and it&apos;s a little crazy to think they would. </li><li>Managers align more strongly with these high-level goals because that is the goal of their team. They don&apos;t matter to you because that&apos;s not what drives your happiness. Doing work that feels fulfilling, interesting and positive matters. Seeing the profit numbers go up doesn&apos;t do anything for you. </li></ul></li><li><strong>Programming Is Lonely</strong><ul><li>It can be hard for people to transition from university to a silent tech office where people rarely speak outside of meetings.</li><li>Try to make friends across departments. I personally always loved hanging out during breaks with the sales folks, mostly because their entire lives are social contact (and they&apos;re often naturally funny people). It&apos;s important to escape your group from time to time. </li></ul></li></ul><h3 id="desired-end-state">Desired End State</h3><p>Our goal with all of this is to get ourselves onto a good team. You&apos;ll know you are on a good team when you spend most of your time just clearing the highway of problems, such is your pace of progress. People are legitimately happy, even if the subject matter is boring. Some of my favorite teams in my career worked on mind-numbingly mundane things but we just had a great time doing it. </p><p>Once you find one of these, my advice is to ride it out for as long as possible. They don&apos;t last forever, management will come in and destroy it if given enough time. But these are the groups where you will do the most learning and develop the most valuable connections. These are the people who you will hire (or will hire you) five or ten years after you stop working together. That&apos;s how incredible the bond of a high-functioning group is and that&apos;s what you need to find. </p><p>But it takes awhile. Until you do, you just need to keep experimenting. Don&apos;t stay at a job you don&apos;t like for more than 2-3 years. Instead take those learnings and move on to the next chance. </p><p>Ultimately, your career is about learning, growing, and being part of a team that values you. The road can be long, and it&#x2019;s okay to experiment with different roles and environments until you find the right fit. Stay focused on what matters to you, take care of yourself, and don&#x2019;t be afraid to move on if the job isn&#x2019;t fulfilling your needs.</p><p></p><p></p><p></p><h3 id></h3>


guid: 677af04672833c0001a9a822
title: Kubernetes as a Distro
author: Mathew Duggan
url: https://matduggan.com/kubernetes-as-a-distro/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>I recently read this great piece by Timur Tukaev discussing how to approach the growing complexity of Kubernetes. <a href="https://blog.aenix.io/the-inevitable-future-of-kubernetes-why-the-orchestrator-should-follow-the-path-of-the-linux-367f49916712" rel="noreferrer">You can read it here.</a> Basically as Kubernetes continues to expand to be the everything platform, the amount of functionality it contains is taking longer and longer to learn. </p><p>Right now every business that adopts Kubernetes is basically rolling their own bespoke infrastructure. Timur&apos;s idea is to try and solve this problem by following the Linux distro model. You&apos;d have groups of people with similar needs work together to make an out-of-the-box Kubernetes setup geared towards their specific needs. I wouldn&apos;t start from a blank cluster, but a cluster already configured for my specific usecase (ML, web applications, batch job processing). </p><p>I understand the idea, but I think the root cause of all of this is simply a lack of a meaningful package management system for Kubernetes. Helm has done the best it can, but practically speaking it&apos;s really far from where we would need to be in order to have something <em>even approaching</em> a Linux package manager. </p><p>More specifically we need something between the very easy to use but easy to mess up Helm and the highly bespoke and complex to write Operator concept. </p><ul><li><strong>Centralized State Management</strong><ul><li>Maintain a robust, centralized state store for all deployed resources, akin to a package database.</li><li>Provide consistency checks to detect and reconcile drifts between the desired and actual states.</li></ul></li><li><strong>Advanced Dependency Resolution</strong><ul><li>Implement dependency trees with conflict resolution.</li><li>Ensure dependencies are satisfied dynamically, including handling version constraints and providing alternatives where possible.</li></ul></li><li><strong>Granular Resource Lifecycle Control</strong><ul><li>Include better support for orchestrating changes across interdependent Kubernetes objects.</li></ul></li><li><strong>Secure Packaging Standards</strong><ul><li>Enforce package signing and verification mechanisms with a centralized trust system. </li><li>Include better support for orchestrating changes across interdependent Kubernetes objects.</li></ul></li><li><strong>Native Support for Multi-Cluster Management</strong><ul><li>Allow packages to target multiple clusters and namespaces with standardized overrides.</li><li>Provide tools to synchronize package versions across clusters efficiently.</li></ul></li><li><strong>Rollback Mechanisms</strong><ul><li>Improve rollback functionality by snapshotting cluster states (beyond Helm&#x2019;s existing rollback features) and ensuring consistent recovery even after partial failures.</li></ul></li><li><strong>Declarative and Immutable Design</strong><ul><li>Introduce a declarative approach where the desired state is managed directly (similar to GitOps) rather than relying on templates.</li></ul></li><li><strong>Integration with Kubernetes APIs</strong><ul><li>Directly leverage Kubernetes APIs like Custom Resource Definitions (CRDs) for managing installed packages and versions.</li><li>Provide better integration with Kubernetes-native tooling (e.g., kubectl, kustomize).</li></ul></li></ul><p>Again a lot of this is Operators, but Operators are proving too complicated for normal people to write. I think we could reuse a lot of that work, keep that functionality and create something similar to what the Operator allows you to do with less maintenance complexity. </p><p>Still I&apos;d love to see the Kubernetes folks do anything in this area. The current state of the world is so bespoke and frankly broken there is a ton of low hanging fruit in this space. </p>


guid: 677bd2fc72833c0001a9a88d
title: Review of Orbit by Mozilla
author: Mathew Duggan
url: https://matduggan.com/review-of-orbit-by-mozilla/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>Recently Mozilla announced the first of what will presumably be a number of LLM-powered tools designed to assist them with advancing a future with &quot;trustworthy AI&quot;. <a href="https://blog.mozilla.org/en/mozilla/introducing-mozilla-ai-investing-in-trustworthy-ai/" rel="noreferrer">You can read their whole thing here.</a> This first stab at the concept is a browser extension called &quot;Orbit&quot;. It&apos;s a smart, limited risk approach to AI, unlike what Apple did which was serve everyone raw cake and tell you it&apos;s cooked with Apple Intelligence. Or Google destroying their search results, previously the &quot;crown jewels&quot; of the company. </p><p>Personally I&apos;m not a huge fan of LLMs. I don&apos;t really think there is something like a &quot;trustworthy LLM&quot;. But I think this is an interesting approach by Mozilla setting up what seems like a very isolated from their core infrastructure LLM appliance running Mistral LLM (Mistral 7B). </p><h3 id="taking-orbit-apart">Taking Orbit Apart</h3><p>You can install the Add-On here: <a href="https://orbitbymozilla.com/">https://orbitbymozilla.com/</a>. I figured we should grab it, rip it apart and see what it does. First I grabbed the file with <code>wget </code><a href="https://addons.mozilla.org/firefox/downloads/file/4392515/orbit_summarizer-1.2.2.xpi"><code>https://addons.mozilla.org/firefox/downloads/file/4392515/orbit_summarizer-1.2.2.xpi</code></a>` , renamed to .zip and peeked around. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image.png" class="kg-image" alt loading="lazy" width="1122" height="564" srcset="https://matduggan.com/content/images/size/w600/2025/01/image.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/image.png 1000w, https://matduggan.com/content/images/2025/01/image.png 1122w" sizes="(min-width: 720px) 720px"></figure><p><strong>background.js</strong></p><p>Going through this file, it seems to be mostly doing what you would expect a &quot;background.js&quot; file does. I was originally thrown off by the <code>chrome.</code> thing until I saw that this is a convention for the WebExtensions API and many WebExtensions APIs in Firefox use the chrome.* namespace for compatibility with Chrome extensions, even though they also support the browser.* namespace as an alias.</p><ol><li><strong>Sentry: </strong>I&apos;m surprised a Mozilla add-on uses Sentry. However it does have a _sentryDebugIdIdentifier. </li><li><strong>Event listeners</strong>: The code sets up event listeners for various WebExtensions API events:<ul><li><code>chrome.runtime.onConnect</code>: Listens for incoming connections from the background script.</li><li><code>chrome.runtime.onMessage</code>: Listens for messages from the browser.</li><li><code>chrome.runtime.onInstalled</code>: Listens for installation and update events.</li><li><code>chrome.contextMenus</code>: Listens for context menu clicks.</li><li><code>chrome.tabs</code>: Listens for tab updates and removals.</li></ul></li><li><strong>Background script</strong>: The code creates a background script that listens for incoming messages from the browser and executes various tasks, including:<ul><li>Handling popup initialization.</li><li>Updating the &quot;isEnabled&quot; setting when the extension is installed or updated.</li><li>Creating new tabs with a welcome page URL (or an onboarding page).</li><li>Aborting video playback when the tab is removed.</li></ul></li></ol><p><strong>Context menu items</strong>: The code creates two context menu items: &quot;Summarize selection with Orbit&quot; and &quot;Summarize page with Orbit&quot;. These items trigger messages to be sent to the browser, which are then handled by the background script.</p><p><strong>Server Elements</strong></p><p>So it looks like Mozilla has a server set up to run the LLM powering Orbit. </p><pre><code>            De = function (t, e) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/prompt/stream&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify({ prompt: t, chat_token: e }) });
            },
            $e = function (t, e) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/chat_history/reinstate_session&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: e }, body: JSON.stringify({ token: t.sessionToken }) });
            },
            Me = function (t) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/chat_history/clear_session&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify({ token: t }) });
            },
            Re = function (t, e, n) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/chat_history/update_context_history&quot;, {
                    method: &quot;POST&quot;,
                    headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: e },
                    body: JSON.stringify({ prev_resp: t, token: n }),
                });
            },
            Ge = function (t, e) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/chat_history/index&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: e }, body: JSON.stringify({ page: t }) });
            },
            Ue = function (t, e, n, r, o, i, a, s) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/prompt/update&quot;, {
                    method: &quot;POST&quot;,
                    headers: { &quot;X-Orbit-Version&quot;: chrome.runtime.getManifest().version, &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: a },
                    body: JSON.stringify({ prompt: t, ai_context: o, context: e, title: n, chat_token: i, type: s, icon_url: r }),
                });
            },
            ze = function (t, e, n) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/prompt/store_result&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: n }, body: JSON.stringify({ ai_context: t, chat_token: e }) });
            },
            Fe = function (t) {
                return fetch(&quot;https://orbitbymozilla.com/v1/users/show&quot;, { method: &quot;GET&quot;, mode: &quot;cors&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: t } });
            },
            qe = function (t, e) {
                return fetch(&quot;https://orbitbymozilla.com/v1/users/sign_in&quot;, { method: &quot;POST&quot;, mode: &quot;cors&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify({ user: { email: t, password: e } }) });
            },
            He = function (t, e) {
                return fetch(&quot;https://orbitbymozilla.com/v1/users&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify({ user: { email: t, password: e } }) });
            },
            Ye = function (t) {
                return fetch(&quot;https://orbitbymozilla.com/v1/users/provider_auth&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify({ provider_name: &quot;google&quot;, token: t }) });
            },
            We = function (t) {
                return fetch(&quot;https://orbitbymozilla.com/v1/users/sign_out&quot;, { method: &quot;DELETE&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: t } });
            },
            Be = function (t) {
                return fetch(t, { method: &quot;GET&quot;, headers: { Host: &quot;docs.google.com&quot;, origin: &quot;https://docs.google.com&quot;, Accept: &quot;text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8&quot; } });
            },
            Je = function (t) {
                return fetch(&quot;https://orbitbymozilla.com/v1/orbit/feedback&quot;, { method: &quot;POST&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot; }, body: JSON.stringify(t) });
            },
            Ke = function (t, e) {
                return fetch(&quot;&quot;.concat(&quot;https://orbitbymozilla.com/&quot;, &quot;v1/orbit/chat_history/&quot;).concat(t), { method: &quot;DELETE&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: e } });
            },
            Ve = function (t) {
                return fetch(&quot;&quot;.concat(&quot;https://orbitbymozilla.com/&quot;, &quot;v1/orbit/chat_history/delete_all&quot;), { method: &quot;DELETE&quot;, headers: { &quot;Content-Type&quot;: &quot;application/json&quot;, Authorization: t } });
            };
        function Xe(t) {
            return (
                (Xe =
                    &quot;function&quot; == typeof Symbol &amp;&amp; &quot;symbol&quot; == typeof Symbol.iterator
                        ? function (t) {
                              return typeof t;
                          }
                        : function (t) {
                              return t &amp;&amp; &quot;function&quot; == typeof Symbol &amp;&amp; t.constructor === Symbol &amp;&amp; t !== Symbol.prototype ? &quot;symbol&quot; : typeof t;
                          }),
                Xe(t)
            );
        }</code></pre><p>Nothing too shocking here. I don&apos;t fully understand what the Google Docs is doing there. </p><p><strong>content.js</strong></p><p>This is where Orbit gets initialized. The code sets up several event listeners to handle incoming messages from the browser&apos;s runtime API</p><p>This code sets up event listeners for several message types, including <code>&quot;startup&quot;</code>, <code>&quot;updatedUrl&quot;</code>, and <code>&quot;responseStream&quot;</code>. The <code>chrome.runtime.onMessage</code> function is used to listen for incoming messages from the browser&apos;s runtime API.</p><h3 id="slightly-mysterious-stuff">Slightly Mysterious Stuff</h3><p>So multiple times in the code you see reference to <code>gmail</code>, <code>youtube</code>, <code>outlook</code> specifically. It seems that this plugin is geared towards providing summaries on those specific websites. Not to say that it won&apos;t attempt it on other sites, but those are the ones where it is hardcoded to work. The <code>google.json</code> is as follows:</p><pre><code>{
  &quot;main&quot;: &quot;div[jsaction*=\&quot;CyXzrf:.CLIENT\&quot;]&quot;,
  &quot;emailThread&quot;: &quot;*[data-thread-perm-id*=\&quot;thread-f\&quot;], *[data-thread-perm-id*=\&quot;thread-a\&quot;]&quot;,
  &quot;emailChains&quot;: &quot;.h7[role*=\&quot;listitem\&quot;], .kv[role*=\&quot;listitem\&quot;]&quot;
}
</code></pre><p>My guess about how this works is as follows:</p><p><strong><code>main</code>:</strong> <code>&quot;div[jsaction*=\&quot;CyXzrf:.CLIENT\&quot;]&quot;</code></p><p>This selector targets <code>div</code> elements that have an attribute called <code>jsaction</code> with a value that starts with <code>CyXzrf:</code> and ends with <code>.CLIENT</code>. This suggests that the plugin is trying to select elements that contain a specific JavaScript action.</p><p><strong><code>emailThread</code>:</strong> <code>&quot;*[data-thread-perm-id*=\&quot;thread-f\&quot;]&quot;, &quot;[data-thread-perm-id*=\&quot;thread-a\&quot;]&quot;</code></p><p>This selector targets two types of elements:<br>* Elements with an attribute called <code>data-thread-perm-id</code> that has a value starting with <code>thread-f</code>.<br>* Elements with an attribute called <code>data-thread-perm-id</code> that has a value starting with <code>thread-a</code>.</p><p>The double quotes around the selectors indicate that they are using the CSS <code>attribute selector</code> syntax, which allows selecting elements based on their attributes.</p><p><strong><code>emailChains</code>:</strong> <code>&quot; .h7[role*=\&quot;listitem\&quot;], .kv[role*=\&quot;listitem\&quot;]&quot;</code></p><p>This selector targets two types of elements:<br>* Elements with a class called <code>h7</code> that have an attribute called <code>role</code> with a value containing the substring &quot;listitem&quot;.<br>* Elements with a class called <code>kv</code> that have an attribute called <code>role</code> with a value containing the substring &quot;listitem&quot;.</p><p>Weirdly the <code>outlook.json</code> and the <code>youtube.json</code> are both empty. I&apos;m guessing for YouTube it grabs the text transcript and ships that off to the server. I don&apos;t see any reference to Outlook in the JS code, so I&apos;m not clear how that works. </p><h3 id="orbit-network-traffic">Orbit Network Traffic </h3><p>So going into <code>about:debugging#/runtime/this-firefox</code> I grabbed the Network Inspector for this plugin. </p><p>Running a summary on a random Ars Technica article looked like this:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-6.png" class="kg-image" alt loading="lazy" width="1491" height="103" srcset="https://matduggan.com/content/images/size/w600/2025/01/image-6.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/image-6.png 1000w, https://matduggan.com/content/images/2025/01/image-6.png 1491w" sizes="(min-width: 720px) 720px"></figure><p>Inspecting the requests, it looks like this is how it works. </p><p>POST to <a href="https://orbitbymozilla.com/v1/orbit/prompt/update">https://orbitbymozilla.com/v1/orbit/prompt/update</a></p><p>It has <code>null</code> ai_context and a context of: </p><blockquote>&apos;Here is a page: Skip to content Story text * Subscribers only Learn more As many of us celebrated the year-end holidays, a small group of researchers worked overtime tracking a startling discovery: At least 33 browser extensions hosted in Google&#x2019;s Chrome Web Store, some for as long as 18 months, were surreptitiously siphoning sensitive data from roughly 2.6 million devices.The compromises came to light with the discovery by data loss prevention service Cyberhaven that a Chrome extension used by 400,000 of &#x2026;Worlds 10 Most Breathtakingly Beautiful Women.OMGIFactsUndoShe Was Everyones Dream Girl In 90s, This Is Her Recently.InvestructorUndoShe Was Everyones Dream Girl In 90s, This Is Her RecentlyBoite A ScoopUndo20 Classic Cars That Remain In High Demand For a ReasonChillingHistory.comUndo2024 Latest Stair Lifts: Ideal for Elderly, No Installation NeededStair Lift | Search AdsUndoCopenhagen - StairLift Elevators Could Be A Dream Come True For SeniorsStair Lift | Search AdUndo Search dialog... Sign in dialog...&apos;</blockquote><p>The response is clearly a reference for the plugin to wait and grab the result later: <code>&quot;orbit_message_store:488b357b-20cc-408d-a55e-bfa5219b102f_e057453954543a1733eaaa2996904b81d0fa3cef6fe15efeccdfe644228173f0&quot;</code></p><p>Then we POST to <code>https://prod.orbit-ml-front-api.fakespot.prod.webservices.mozgcp.net/invoke</code></p><pre><code>{&quot;config&quot;:{&quot;configurable&quot;:{&quot;key&quot;:&quot;488b357b-20cc-408d-a55e-bfa5219b102f_e057453954543a1733eaaa2996904b81d0fa3cef6fe15efeccdfe644228173f0&quot;},&quot;metadata&quot;:{&quot;key&quot;:&quot;488b357b-20cc-408d-a55e-bfa5219b102f_e057453954543a1733eaaa2996904b81d0fa3cef6fe15efeccdfe644228173f0&quot;},&quot;recursionLimit&quot;:25,&quot;tags&quot;:[]},&quot;input&quot;:{&quot;human_input&quot;:&quot;generate_suggested_justquestions&quot;},&quot;kwargs&quot;:{}}</code></pre><p>This service clearly generates the &quot;suggested questions&quot; in the plugin as seen in the response. </p><blockquote>&quot; Question 1: Which Chrome extensions were found to have surreptitiously collected sensitive data from users&apos; devices, and how many users were affected?\n\nQuestion 2: How did the attackers exploit the Google OAuth permission system to upload malicious versions of the Chrome extensions to the Chrome Web Store?\n\nQuestion 3: What types of sensitive data were stolen by the malicious Chrome extensions, and which websites&apos; authentication credentials were targeted?\n\nQuestion 4: What steps should organizations take to manage and secure the use of browser extensions in their security programs?\n\nQuestion 5: What is the potential impact on users who ran one of the compromised Chrome extensions, and what actions should they take to protect themselves?&quot;</blockquote><p>Finally we run a GET to prod.orbit-ml-front-api.fakespot.prod.webservices.mozgcp.net which seems to get the actual summary. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-7.png" class="kg-image" alt loading="lazy" width="950" height="586" srcset="https://matduggan.com/content/images/size/w600/2025/01/image-7.png 600w, https://matduggan.com/content/images/2025/01/image-7.png 950w" sizes="(min-width: 720px) 720px"></figure><p>Finally we <code>store_result</code>. <a href="https://orbitbymozilla.com/v1/orbit/prompt/store_result">https://orbitbymozilla.com/v1/orbit/prompt/store_result</a> is what we POST to with the following. </p><blockquote>&quot; At least 33 malicious Chrome extensions, some of which had been available in Google&apos;s Chrome Web Store for up to 18 months, were discovered to have been stealing sensitive data from approximately 2.6 million devices. The compromises came to light when a Chrome extension used by 400,000 of Cyberhaven&apos;s customers was found to have been updated with malicious code. The malicious version, available for 31 hours between Christmas Day and Boxing Day, automatically downloaded and installed on Chrome browsers running Cyberhaven during that window. The attacker gained access to the extension by sending a spear phishing email to the developers, posing as Google and warning that the extension would be revoked unless immediate action was taken. The attacker then used the granted permission to upload the malicious version. Other extensions were also targeted using the same method, with at least 19 identified as of Thursday afternoon. The earliest known compromise occurred in May 2023.&quot;</blockquote><pre><code>orbit_message_store:bb1d4e09-0ff9-4ba1-93e5-172f77257022_cfe62c87f51e4a81c5f0bbb85b3938bb4a7513a3ac9f11b3e537d35918d97f5b</code></pre><p>I&apos;m a little confused by this. I thought the summary wasn&apos;t stored server-side as explained by this:</p><blockquote>Does Orbit save the content of the pages I visit or summaries generated?</blockquote><blockquote>No, it does not.</blockquote><blockquote>When you use Orbit, we receive a payload back that contains the contents of your query; information about the model queried (such as the name and version number); information about technical problems with processing the query, if any; the number of tokens required to process the query; and the model outputs in response to the query. We do not store this data beyond temporarily caching it to process your query and return the outputs to you.</blockquote><blockquote>Orbit summaries are only available on the page that you are actually on. As soon as you navigate away from that page, Orbit erases the session.</blockquote><p>I&apos;m not <em>exactly clear</em> how this can be true if it seems like I&apos;m sending the summary back to Mozilla to an endpoint called <code>store_result</code>. They do say they &quot;temporarily cache&quot; the results so I guess they are covered in terms of how it works. It doesn&apos;t appear to me like an endpoint called <code>store_result</code> is a temporary storage endpoint, but who knows. </p><h2 id="does-it-work-well">Does it work well?</h2><p>So now that I&apos;ve checked the extension and installed it, agreeing to this somewhat alarming permission screen:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/SCR-20250106-jjt.png" class="kg-image" alt loading="lazy" width="418" height="191"></figure><p>It gives me a strange....hovering black hole icon? </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-1.png" class="kg-image" alt loading="lazy" width="130" height="90"></figure><p>When you click on the hovering black hole you get the following options:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-2.png" class="kg-image" alt loading="lazy" width="173" height="146"></figure><p>You can do the following:</p><ul><li>Ask Orbit, which is asking the plugin about anything <em>on the specific page you are on</em>. </li><li>Settings.</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-3.png" class="kg-image" alt loading="lazy" width="322" height="566"></figure><p>I strongly recommend &quot;Minimal&quot; to get rid of the large Bonzi Buddy type floating icon on the screen. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-4.png" class="kg-image" alt loading="lazy" width="1200" height="900" srcset="https://matduggan.com/content/images/size/w600/2025/01/image-4.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/image-4.png 1000w, https://matduggan.com/content/images/2025/01/image-4.png 1200w" sizes="(min-width: 720px) 720px"></figure><ul><li>Summarize</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-5.png" class="kg-image" alt loading="lazy" width="342" height="1099"></figure><ul><li>Finally <code>Snooze Orbit</code>. </li></ul><h3 id="testing">Testing</h3><p>So let&apos;s try it out on some junk! We&apos;ll start with the Ars article from before. <a href="https://arstechnica.com/security/2025/01/dozens-of-backdoored-chrome-extensions-discovered-on-2-6-million-devices/">https://arstechnica.com/security/2025/01/dozens-of-backdoored-chrome-extensions-discovered-on-2-6-million-devices/</a></p><p>The summary is ok, if a little brief. </p><blockquote>At least 33 malicious Chrome extensions, some of which had been available in Google&apos;s Chrome Web Store for up to 18 months, were discovered to have been stealing sensitive data from approximately 2.6 million devices. The compromises came to light when a Chrome extension used by 400,000 of Cyberhaven&apos;s customers was found to have been updated with malicious code. The malicious version, available for 31 hours between Christmas Day and Boxing Day, automatically downloaded and installed on Chrome browsers running Cyberhaven during that window. The attacker gained access to the extension by sending a spear phishing email to the developers, posing as Google and warning that the extension would be revoked unless immediate action was taken. The attacker then used the granted permission to upload the malicious version. Other extensions were also targeted using the same method, with at least 19 identified as of Thursday afternoon. The earliest known compromise occurred in May 2023.</blockquote><p>Switching the settings to have it produce &quot;paragraphs&quot; resulted in a much more robust answer. </p><blockquote>At least 33 malicious Chrome extensions, some of which had been available in Google&apos;s Chrome Web Store for up to 18 months, were discovered to have been stealing sensitive data from approximately 2.6 million devices. The compromises came to light when data loss prevention service Cyberhaven discovered that their own extension, used by 400,000 of its customers, had been updated with malicious code that stole their sensitive data. The malicious version was available for just 31 hours, from December 25 at 1:32 AM UTC to Dec 26 at 2:50 AM UTC, and automatically downloaded and installed on Chrome browsers running Cyberhaven during that window.<br>The malicious code was configured to work with different payloads that were downloaded from cyberhavenext.pro, a malicious site registered to give the appearance it was affiliated with the company. One recovered payload scoured user devices for browser cookies and authentication credentials for the facebook.com domain, while another stole cookies and credentials for chatgpt.com.<br>The attacker gained access to the Cyberhaven extension by sending a spear phishing email to the developers on Christmas Eve, warning that the extension wasn&apos;t in compliance with Google terms and would be revoked unless immediate action was taken. The email contained a link to a Google consent screen requesting access permission for an OAuth application named Privacy Policy Extension. A Cyberhaven developer granted the permission, unknowingly giving the attacker the ability to upload new versions of Cyberhaven&apos;s Chrome extension to the Chrome Web Store.<br>Other extensions were also targeted using the same spear phishing campaign, with at least 19 other Chrome extensions compromised as of Thursday afternoon. In every case, the attacker used spear phishing to push a new malicious version and custom, look-alike domains to issue payloads and receive authentication credentials. Collectively, the 20 extensions had 1.46 million downloads.<br>The earliest compromise occurred in May 2023. The full list of affected extensions and their details is provided in the article.<br>Browser extensions have long been a weak link in the security chain, with malicious extensions stealing sensitive data from 4 million devices in 2019, including those of companies like Tesla, Blue Origin, FireEye, Symantec, TMobile, and Reddit.<br>To mitigate the risk of malicious extensions, organizations can compile a browser asset management list that allows only selected extensions to run and blocks all others. However, this would not have prevented the Cyberhaven customers from installing the malicious extension version unless a specific version was trusted and all others were blocked.<br>Anyone who ran one of these compromised extensions is advised to carefully consider changing their passwords and other authentication credentials. Additional indicators of compromise can be found in the article and other posts.</blockquote><p>The suggested questions are as follows:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-8.png" class="kg-image" alt loading="lazy" width="326" height="218"></figure><blockquote>According to the text, at least 33 Chrome extensions were found to have surreptitiously collected sensitive data from roughly 2.6 million devices. Some of the affected extensions include Cyberhaven, Uvoice, Internxt VPN, and Castorus. The malicious code was pushed out through a spear phishing campaign that targeted the developers of these extensions.</blockquote><p>That&apos;s the answer to the first one. Since the article has a list of impacted extensions I would have expected it to return the full list of extensions. </p><blockquote>The attackers exploited the Google OAuth permission system by tricking a Cyberhaven developer into granting them permission to upload new versions of the Chrome extension to the Chrome Web Store. They did this by sending a spear phishing email on Christmas Eve, posing as a Google representative and warning that the extension would be revoked unless immediate action was taken. The link in the email led to a Google consent screen, where the developer unknowingly granted the attacker the ability to upload new versions.</blockquote><p>Second answer is a little bit stronger. </p><p><strong>OpenAI Comparison</strong></p><blockquote>A recent investigation has uncovered that at least 33 malicious Chrome extensions, collectively installed on approximately 2.6 million devices, have been covertly extracting sensitive user data.</blockquote><blockquote>Key Findings:<br>&#x2022; Discovery and Impact: The issue came to light when Cyberhaven, a data loss prevention service, found that a Chrome extension used by 400,000 of its customers had been compromised. The malicious version was available for 31 hours between December 25 and 26, during which it harvested sensitive information from users&#x2019; devices.<br>&#x2022; Method of Compromise: Attackers employed spear-phishing emails targeting extension developers, tricking them into granting permissions that allowed the attackers to upload malicious versions of legitimate extensions to the Chrome Web Store. These versions were configured to download additional payloads from domains mimicking legitimate services, enabling the theft of browser cookies and authentication credentials.<br>&#x2022; Scope of Affected Extensions: Beyond Cyberhaven&#x2019;s extension, at least 19 other Chrome extensions were similarly compromised through this method, affecting a total of 1.46 million users. The attackers used custom, look-alike domains to distribute payloads and collect stolen credentials.</blockquote><blockquote>Recommendations for Users:<br>&#x2022; Review Installed Extensions: Users should audit their browser extensions and remove any that are unnecessary or unfamiliar.<br>&#x2022; Stay Informed: Regularly check for updates from trusted sources regarding compromised extensions to ensure your browser remains secure.<br>&#x2022; Be Cautious with Permissions: Before installing extensions, scrutinize the permissions they request and be wary of granting access that seems excessive for the extension&#x2019;s functionality.</blockquote><blockquote>This incident underscores the importance of vigilance when managing browser extensions, as they can become vectors for significant security breaches if compromised.</blockquote><p><strong>Review: </strong>Honestly they&apos;re both pretty bad. The first summary is missing a lot of relatively critical information. The second summary has most of that information, but is so long you might as well read the original source material. OpenAI decides it doesn&apos;t want to even read the article and sorta goes off on its own there. </p><p><strong>Another One: </strong><a href="https://lwn.net/Articles/1001783/">https://lwn.net/Articles/1001783/</a></p><p>I was hoping the plain web style would help the plugin, but it seemed to have trouble right away. I suspect the comments section threw it off. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-9.png" class="kg-image" alt loading="lazy" width="296" height="132"></figure><p>This one was a complete failure. </p><blockquote>provider of plugins for a CMS that has a 62% market share.<br>That&apos;s not quite accurate. Automattic is the provider of the official plugin repository for WordPress, which is the most popular CMS. However, there are other plugin repositories and marketplaces, such as GitHub, where plugins can be distributed and installed directly.<br>The issue here is that Automattic has taken actions to prevent WP Engine from distributing plugins through the official repository, which WP Engine argues is damaging their business. WP Engine is seeking an injunction to restore their access to the repository.<br>The court has found that WP Engine is likely to succeed on the merits of their claim for tortious interference with contractual relations, as they have demonstrated that Automattic intentionally interfered with their contractual relationship with their customers by blocking their access to the repository. The court has also found that WP Engine is likely to suffer irreparable harm if the injunction is not granted, as they will continue to lose customers and revenue.<br>However, it&apos;s important to note that this is just a preliminary injunction, and it does not necessarily mean that WP Engine will ultimately prevail in the case. The trial has not yet begun, and there will be further proceedings where both sides will present their evidence and arguments.<br>Legal jargon is significant.</blockquote><p><strong>Review: </strong>Disaster. </p><p></p><p><strong>Try again! </strong><a href="https://www.theverge.com/2025/1/5/24336154/samsung-the-frame-pro-features-specs">https://www.theverge.com/2025/1/5/24336154/samsung-the-frame-pro-features-specs</a></p><p>Ok this is at least a readable summary. </p><blockquote>Samsung has announced the release of The Frame Pro, an upgraded version of its popular TV that aims to deliver both aesthetics and improved performance. The Frame Pro features Mini LED technology, which promises enhanced contrast, brightness, and black levels, although it does not offer the same level of local dimming as traditional Mini LED TVs. Additionally, the maximum refresh rate has been boosted to 144Hz for smoother visuals for PC gamers. However, Dolby Vision support is still not available. The Frame Pro also eliminates the need for a wire between the TV and Samsung&apos;s breakout box, resulting in less cable clutter. The TV is expected to ship in the spring, with pricing details yet to be announced.</blockquote><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-10.png" class="kg-image" alt loading="lazy" width="327" height="201"></figure><blockquote>The Frame Pro TV by Samsung introduces two major improvements compared to the regular model. First, it uses Mini LED technology along the bottom of the screen, which Samsung claims produces some level of local dimming. Second, it offers a higher maximum refresh rate of 144Hz for smoother visuals for PC gamers.</blockquote><blockquote>The Frame Pro by Samsung utilizes Mini LEDs placed along the bottom of the screen, which Samsung claims still produces some level of local dimming. However, this approach seems more similar to edge-lit TVs as it doesn&apos;t involve the large number of small dimming zones typically found in traditional Mini LED TVs.</blockquote><p>This is fine. I think it&apos;s missing a few important things from the article. First it doesn&apos;t really convey any of the skepticism of the author.</p><blockquote>The Frame has always been something of a compromise; maybe you&#x2019;ve got a significant other who refuses to allow a dull black rectangle into the living room. So you, being the good and considerate person you are, ultimately agree to &#x201C;settle&#x201D; on The Frame. After first hearing about The Frame Pro, I was hopeful that it would be much less of a compromise.</blockquote><blockquote>But this asterisk around Mini LED has me a little less excited. Like recent models, The Frame Pro&#x2019;s display has a matte finish to give your preferred art a more authentic appearance and mask the reality that you&#x2019;re looking at a screen. But matte screens can sometimes lessen a display&#x2019;s punch, so genuine Mini LED backlighting could&#x2019;ve helped quite a bit in that regard.</blockquote><p>It&apos;s also a pretty bare-boned summary. Here is the OpenAI comparison:</p><blockquote>Samsung has announced The Frame Pro, an enhanced version of its popular TV model, The Frame, which combines home decor aesthetics with television functionality. &#xFFFC;</blockquote><blockquote>Key Features of The Frame Pro:<br>&#x2022; Mini LED Technology: The Frame Pro transitions to Mini LED technology, aiming to improve contrast, brightness, and black levels. Unlike traditional Mini LED setups with backlighting, Samsung places the Mini LEDs along the bottom of the screen, which may still result in an edge-lit TV feel. &#xFFFC;<br>&#x2022; Increased Refresh Rate: The maximum refresh rate is boosted from 120Hz to 144Hz, appealing to PC gamers seeking smoother visuals. &#xFFFC;<br>&#x2022; Wireless One Connect Box: The wired connection to the breakout box is eliminated, replaced by the Wireless One Connect Box supporting Wi-Fi 7. This box can function from up to 10 meters away, reducing cable clutter and enhancing the TV&#x2019;s seamless integration into home decor. &#xFFFC;<br>&#x2022; AI-Powered Features: The Frame Pro includes AI-driven functionalities such as &#x201C;Click to Search,&#x201D; which provides information about actors, scenes, or clothing displayed on-screen; &#x201C;Samsung Food,&#x201D; recognizing dishes onscreen and offering recipes; and &#x201C;Live Translate,&#x201D; instantly translating closed captions on live broadcasts in up to seven languages. &#xFFFC;</blockquote><blockquote>Design Considerations:<br>&#x2022; Matte Finish Display: The TV maintains a matte finish to give displayed art a more authentic appearance, aligning with its design ethos of blending seamlessly into home decor. &#xFFFC;</blockquote><blockquote>Availability and Pricing:<br>&#x2022; Samsung plans to announce pricing details in the spring, coinciding with the product&#x2019;s shipment schedule. The cost will be a significant factor in determining The Frame Pro&#x2019;s appeal compared to the regular model. &#xFFFC;</blockquote><blockquote>The Frame Pro aims to enhance the balance between aesthetic appeal and advanced television functionality, building upon the success of its predecessor by introducing technological upgrades and user-friendly features. &#xFFFC;</blockquote><p>This is still missing some of the subtext from the author about why the product is good or bad, but it does provide much more detail. </p><p><strong>Review: </strong>Meh, it&apos;s pretty ok? I guess if you are really pressed for time. It seems to be missing a lot of important context about what this product is and what people might have wanted from it. </p><p><strong>Last Test: </strong>email</p><p>I have my email on Fastmail, so we&apos;ll see how it does there. Here is the email. </p><blockquote>At the heart of conservation are people like you &#x2014; individuals who bring unique strengths to sustain and grow this vital work. This year, we&#x2019;ve shared stories of the incredible women and communities driving change on the ground, and your support has been an integral part of those stories and our mission. Thanks to your generosity, Akashinga achieved remarkable milestones in 2024:</blockquote><blockquote>&#x2022; Expanded our reach into key conservation areas, such as the Kunene Highlands in Namibia, safeguarding even more critical habitats.<br>&#x2022; Trained over 70 new female rangers, equipping them to protect wildlife and support their communities.<br>&#x2022; Protected 13 million acres of wilderness, ensuring a safe haven for countless species.<br>&#x2022; Grew our global workforce to over 900 individuals, with 98% of staff being local and/or indigenous to the countries they are working in.<br>&#x2022; Advanced community healthcare and infrastructure by implementing mobile health clinics and initiating additional solar-powered borehole projects within local communities.<br>Every dollar you gave directly strengthened conservation efforts and empowered communities across Southern and East Africa. Your commitment doesn&#x2019;t just fund our mission &#x2014; it makes you part of the Akashinga family, united in protecting our planet and fostering sustainability.</blockquote><blockquote>Thank you for being a vital voice in the Akashinga story.</blockquote><p>Here is the summary:</p><blockquote>Mathew received an email from Akashinga acknowledging his support in 2024, which significantly contributed to their conservation efforts. Akashinga expanded their reach into new areas, trained new female rangers, protected 13 million acres of wilderness, and grew their global workforce. Mathew&apos;s donation also funded community healthcare and infrastructure projects. Akashinga expressed gratitude for his commitment and considered him part of their family. The email was a thank you note and update on their achievements in 2024.</blockquote><p><strong>Review: </strong>I mean again like it&apos;s not wrong, it&apos;s just such a short summary. It seems to be missing a lot of important information. I&apos;m not clear why the tool doesn&apos;t give me a bit more info even when I have &quot;paragraphs&quot; selected. </p><h3 id="conclusion">Conclusion</h3><p>I get what Mozilla is going for here, but frankly this is a miss. I&apos;m not really clear what it is doing with my summaries of websites, but it feels like they&apos;re storing them in a cache so they don&apos;t need to redo the summary every time. Outside of privacy, the summary is just too short to provide almost any useful information. </p><p>If you are someone who is drowning in websites and emails and just needs a super fast <em>extremely</em> high level overview of their content, give it a shot. It works pretty well for that and you can&apos;t beat the price. But if you are looking for a more nuanced understanding of articles and emails where you can ask meaningful follow-up questions and get answers, keep looking. This isn&apos;t there yet, although since most of the magic seems to be happening in their server, I guess there&apos;s nothing stopping them from improving the results over time. </p>


guid: 677e420f72833c0001a9a945
title: Stop Trying To Schedule A Call With Me
author: Mathew Duggan
url: https://matduggan.com/stop-trying-to-schedule-a-call-with-me/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<img src="https://images.unsplash.com/photo-1508517911809-9fb266784e95?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=M3wxMTc3M3wwfDF8c2VhcmNofDZ8fGNhbGxpbmd8ZW58MHx8fHwxNzM2MzM1NzE5fDA&amp;ixlib=rb-4.0.3&amp;q=80&amp;w=2000" alt="Stop Trying To Schedule A Call With Me"><p>One of the biggest hurdles for me when trying out a new service or product is the inevitable harassment that follows. It always starts innocuously:</p><p><em>&#x201C;Hey, I saw you were checking out our service. Let me know if you have any questions!&#x201D;</em></p><p>Fine, whatever. You have documentation, so I&#x2019;m not going to email you, but I understand that we&#x2019;re all just doing our jobs.</p><p>Then, it escalates. </p><p><em>&#x201C;Hi, I&#x2019;m your customer success fun-gineer! Just checking in to make sure you&#x2019;re having the best possible experience with your trial!&#x201D;</em></p><p>Chances are, I signed up to see if your tool can do one specific thing. If it doesn&#x2019;t, I&#x2019;ve already mentally moved on and forgotten about it. So, when you email me, I&#x2019;m either actively evaluating whether to buy your product, or I have no idea why you&#x2019;re reaching out.</p><p>And now, I&#x2019;m stuck on your mailing list forever. I get notifications about all your new releases and launches, which forces me to make a choice every time:</p><p>&#x2022; <em>&#x201C;Obviously, I don&#x2019;t care about this anymore.&#x201D;</em></p><p>&#x2022; <em>&#x201C;But what if they&#x2019;ve finally added the feature I wanted?&#x201D;</em></p><p>Since your mailing list is apparently the only place on Earth to find out if Platform A has added Feature X (because putting release notes somewhere accessible is apparently too hard), I have to weigh unsubscribing every time I see one of your marketing emails.</p><p>And that&#x2019;s not even the worst-case scenario. The absolute worst case is when, <em>god forbid</em>, I can actually use your service, but now I&#x2019;m roped into setting up a <em>&#x201C;series of calls.&#x201D;</em></p><p>You can&apos;t just let me input a credit card number into a web site. Now I need to form a bunch of interpersonal relationships with strangers over Microsoft Teams. </p><h3 id="lets-jump-on-a-call">Let&apos;s Jump On A Call</h3><p>Every SaaS sales team has this classic duo.</p><p>First, there&#x2019;s the salesperson. They&#x2019;re friendly enough but only half paying attention. Their main focus is inputting data into the CRM. Whether they&#x2019;re selling plastic wrap or missiles, their approach wouldn&#x2019;t change much. Their job is to keep us moving steadily toward <em>The Sale</em>.</p><p>Then, there&#x2019;s their counterpart: the &#x201C;sales engineer,&#x201D; &#x201C;customer success engineer,&#x201D; or whatever bastardized title with the word <em>engineer</em> they&#x2019;ve decided on this week. This person is one of the few people at the company who has actually read all the documentation. They&#x2019;re brought in to explain&#x2014;always with an air of exhaustion&#x2014;how this is really my new <em>&#x201C;everything platform.&#x201D;</em></p><p><em>&#x201C;Our platform does everything you could possibly want. We are very secure&#x2014;maybe too secure. Our engineers are the best in the world. Every release is tested through a 300-point inspection process designed by our CTO, who interned at Google once, so we strongly imply they held a leadership position there.&#x201D;</em></p><p>I will then endure a series of demos showcasing functionality I&#x2019;ll never use because I&#x2019;m only here for one or two specific features. You know this, but the rigid demo template doesn&#x2019;t allow for flexibility, so we have to slog through the whole thing.</p><p>To placate me, the salesperson will inevitably say something like,</p><p><em>&#x201C;Mat is pretty technical&#x2014;he probably already knows this.&#x201D;</em></p><p>As if this mild flattery will somehow make me believe that a lowly nerd like me and a superstar salesperson like you could ever be friends. Instead, my empathy will shift to the sales engineer, whose demo will, without fail, break at the worst possible time. Their look of pure despair will resonate with me deeply.</p><p><em>&#x201C;Uh, I promise this normally works.&#x201D;</em></p><p>There, there. I know. It&#x2019;s all held together with tape and string.</p><p>At some point, I&#x2019;ll ask about compliance and security, prompting you to send over a pile of meaningless certifications. These documents don&#x2019;t actually prove you <em>did</em> the things outlined in them; they just demonstrate that you could <em>plausibly fake</em> having done them.</p><p>We both know this. If I got you drunk, you&#x2019;d probably tell me horror stories about engineers fixing databases by copying them to their laptops, or how user roles don&#x2019;t really work and everyone is secretly an admin.</p><p>But this is still the dating phase of our relationship, so we&#x2019;re pretending to be on our best behavior.</p><p><em>&#x201C;Very impressive SOC-2.&#x201D;</em></p>
<!--kg-card-begin: html-->
<iframe src="https://giphy.com/embed/j0AAjIX38E4nrj0AVe" width="480" height="480" style frameborder="0" class="giphy-embed" allowfullscreen></iframe><p><a href="https://giphy.com/gifs/oceansprayinc-ocean-spray-oceanspray-power-your-party-j0AAjIX38E4nrj0AVe">via GIPHY</a></p>
<!--kg-card-end: html-->
<h3 id="getting-someone-to-pay-you">Getting Someone To Pay You</h3><p>We&#x2019;ve gone through the demos. You&#x2019;ve tried to bond with me, forming a &#x201C;team&#x201D; that will supposedly work together against the people who actually matter and make decisions at my company. Now you want to bring my boss&#x2019;s boss into the call to pitch them directly.</p>
<!--kg-card-begin: html-->
<iframe src="https://giphy.com/embed/l4pTo1zFkq02vqvHW" width="480" height="295" style frameborder="0" class="giphy-embed" allowfullscreen></iframe><p><a href="https://giphy.com/gifs/cbc-funny-reaction-l4pTo1zFkq02vqvHW">via GIPHY</a></p>
<!--kg-card-end: html-->
<p>Here&#x2019;s the problem: that person would rather be set on fire than sit through 12 of these pitches a week from various companies. So, naturally, it becomes my job to &#x201C;put together the proposal.&#x201D;</p><p>This is where things start to fall apart. The salesperson grows increasingly irritated because they could close the deal if they didn&#x2019;t have to talk to me and could just pitch directly to leadership. Meanwhile, the sales engineer&#x2014;who, for some reason, is still forced to attend these calls&#x2014;stares into the middle distance like an orphan in a war zone.</p><p><em>&#x201C;Look, can we just loop in the leadership on your side and wrap this up?&#x201D;</em> the salesperson asks, visibly annoyed.</p><p><em>&#x201C;They pay me so they don&#x2019;t have to talk to you,&#x201D;</em> I&#x2019;ll respond, a line you first thought was a joke but have since realized was an honest admission you refused to hear early in our relationship.</p><p>If I really, really care about your product, I&#x2019;ll contact the 300 people I need on my side to get it approved. This process will take at least a month. Why? Who knows&#x2014;it just always does. If I work for a Fortune 500 company, it&#x2019;ll take a minimum of three months, assuming everything goes perfectly.</p><p>By this point, I hate myself for ever clicking that cursed link and discovering your product existed. What was supposed to save me time has now turned into a massive project. I start to wonder if I should&#x2019;ve just reverse-engineered your tool myself.</p><p>Eventually, it&#x2019;s approved. Money is exchanged, and the salesperson disappears forever. Now, I&#x2019;m handed off to Customer Service&#x2014;aka a large language model (LLM).</p><h3 id="the-honeymoon-is-over">The Honeymoon Is Over</h3><p>It doesn&#x2019;t take long to realize that your &#x201C;limitless, cloud-based platform designed by the best in the business&#x201D; is, in fact, quite limited. One day, everything works fine. The next, I unknowingly exceed some threshold, and the whole thing collapses in on itself.</p><p>I&#x2019;ll turn to your documentation, which has been meticulously curated to highlight your strengths&#x2014;because god forbid potential customers see any warnings. Finding no answers, I&#x2019;ll engage Customer Service. After wasting precious moments of my life with an LLM that links me to the same useless documentation, I&#x2019;ll finally be allowed to email a real person.</p><p>The SLA on that support email will be absurdly long&#x2014;72 business hours&#x2014;because I didn&#x2019;t opt for the <em>Super Enterprise Plan&#x2122;</em>. Eventually, I&#x2019;ll get a response explaining that I&#x2019;ve hit some invisible limit and need to restructure my workflows to avoid it.</p><p>As I continue using your product, I&#x2019;ll develop a growing list of undocumented failure modes:</p><p><em>&#x201C;If you click those two buttons too quickly, the iFrame throws an error.&#x201D;</em></p><p>I&#x2019;ll actually say this to another human being, as if we&#x2019;re in some cyberpunk dystopia where flying cars randomly explode in the background because they were built by idiots. Despite your stack presumably logging these errors, no one will ever reach out to explain them or help me fix anything.</p><h3 id="account-reps">Account Reps</h3><p>Then, out of the blue, I&#x2019;ll hear from my new account rep. They&#x2019;ll want a call to &#x201C;discuss how I&#x2019;m using the product&#x201D; and &#x201C;see how they can help.&#x201D; Don&#x2019;t be fooled&#x2014;this isn&#x2019;t an attempt to gather feedback or fix what&#x2019;s broken. It&#x2019;s just another sales pitch.</p><p>After listening to my litany of issues and promising to &#x201C;look into them,&#x201D; the real purpose of the call emerges: convincing me to buy more features. These &#x201C;new features&#x201D; are things that cost you almost nothing but make a huge difference to me&#x2014;like SSO or API access. Now I&#x2019;m forced to decide whether to double down on your product or rip it out entirely and move on with my life.</p><p>Since it&#x2019;s not my money, I&#x2019;ll probably agree to give you more just to get basic functionality that should&#x2019;ve been included in the first place.</p><h3 id="fond-farewell">Fond Farewell</h3><p>Eventually, one of those open-source programmers&#x2014;the kind who gleefully release free tools and then deal with endless complaints for life&#x2014;will create something that does what your product does. It&#x2019;ll have a ridiculous name like CodeSquish, Dojo, or GitCharm.</p><p>I&#x2019;ll hear about it from a peer. When I mention I use your product, they&#x2019;ll turn to me, eyes wide, and say, <em>&#x201C;Why don&#x2019;t you just use CodeSquish?&#x201D;</em></p><p>Not wanting to admit ignorance, I&#x2019;ll make up a reason on the spot. Later, in the bathroom, I&#x2019;ll Google CodeSquish and discover it does everything I need, costs nothing, and is 100x more performant&#x2014;even though it&#x2019;s maintained by a single recluse who only emerges from their Vermont farm to push code to their self-hosted git repo.</p><p>We&#x2019;ll try it out. Despite the fact that its only &#x201C;forum&#x201D; is a Discord server, it&#x2019;ll still be miles ahead of your commercial product.</p><p>Then comes the breakup. I&#x2019;ll put it off for as long as possible because we probably signed a contract. Eventually, I&#x2019;ll tell Finance not to renew it. Suddenly, I&#x2019;ll get a flurry of attention from your team. You&#x2019;ll pitch me on why the open-source tool is <em>actually inferior</em> (which we both know isn&#x2019;t true).</p><p>I&#x2019;ll tell you, <em>&#x201C;We&#x2019;ll discuss it on our side.&#x201D;</em> We won&#x2019;t. The only people who cared about your product were me and six others. Finally, like the coward I am, I&#x2019;ll break up with you over email&#x2014;and then block your domain.</p>

enclosure_url:
https://images.unsplash.com/photo-1508517911809-9fb266784e95?crop=entropy&cs=tinysrgb&fit=max&fm=jpg&ixid=M3wxMTc3M3wwfDF8c2VhcmNofDZ8fGNhbGxpbmd8ZW58MHx8fHwxNzM2MzM1NzE5fDA&ixlib=rb-4.0.3&q=80&w=2000


guid: 6788eff072833c0001a9aa34
title: The AI Bubble Is Bursting
author: Mathew Duggan
url: https://matduggan.com/the-ai-bubble-is-bursting/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>Last week I awoke to Google deciding I hadn&apos;t had enough AI shoved down my throat. With no warning they decided to take the previously $20/user/month Gemini add-on and make it &quot;free&quot; and on by default. If that wasn&apos;t bad enough, they also decided to remove my ability as an admin to turn it off. Despite me hitting all the Off buttons I could find:</p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-12.png" class="kg-image" alt loading="lazy" width="1206" height="325" srcset="https://matduggan.com/content/images/size/w600/2025/01/image-12.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/image-12.png 1000w, https://matduggan.com/content/images/2025/01/image-12.png 1206w" sizes="(min-width: 720px) 720px"></figure><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image--1-.png" class="kg-image" alt loading="lazy" width="610" height="223" srcset="https://matduggan.com/content/images/size/w600/2025/01/image--1-.png 600w, https://matduggan.com/content/images/2025/01/image--1-.png 610w"></figure><p>Users were still seeing giant Gemini chat windows, advertisements and harassment to try out Gemini. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/Screenshot-2025-01-16-at-14.27.56.png" class="kg-image" alt loading="lazy" width="1251" height="591" srcset="https://matduggan.com/content/images/size/w600/2025/01/Screenshot-2025-01-16-at-14.27.56.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/Screenshot-2025-01-16-at-14.27.56.png 1000w, https://matduggan.com/content/images/2025/01/Screenshot-2025-01-16-at-14.27.56.png 1251w" sizes="(min-width: 720px) 720px"></figure><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/Screenshot-2025-01-16-at-14.19.49.png" class="kg-image" alt loading="lazy" width="542" height="647"></figure><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/Screenshot-2025-01-16-at-14.32.30.png" class="kg-image" alt loading="lazy" width="998" height="659" srcset="https://matduggan.com/content/images/size/w600/2025/01/Screenshot-2025-01-16-at-14.32.30.png 600w, https://matduggan.com/content/images/2025/01/Screenshot-2025-01-16-at-14.32.30.png 998w" sizes="(min-width: 720px) 720px"></figure><p>This situation is especially frustrating because we had already evaluated Gemini. I sat through sales calls, read the documentation, and tested it extensively. Our conclusion? It&#x2019;s a bad service, inferior to everything on the market including free LLMs.  Yet, now, to disable Gemini, I&#x2019;m left with two unappealing options:</p><ul><li><em>upgrade</em> my account to the next tier up and pay Google more money for less AI garbage. </li><li>Beg customer service to turn it off, after enduring misleading responses from several Google sources who claimed it wasn&#x2019;t possible.</li></ul><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image--2--1.png" class="kg-image" alt loading="lazy" width="986" height="1320" srcset="https://matduggan.com/content/images/size/w600/2025/01/image--2--1.png 600w, https://matduggan.com/content/images/2025/01/image--2--1.png 986w" sizes="(min-width: 720px) 720px"></figure><p></p><p>Taking how I feel about AI out of the equation, this is one of the most desperate and pathetic things I&apos;ve ever seen a company do. Nobody was willing to pay you for your AI so, after wasting billions making it, you decide to force enable it and raise the overall price of Google Workspaces, a platform companies cannot easily migrate off of. Suddenly we&apos;ve repriced AI from $20/user/month to &quot;we hope this will help smooth over us raising the price by $2/user/month&quot;. </p><p>Plus, we know that Google knew I wouldn&apos;t like this because they <em>didn&apos;t tell me they were going to do it</em>. They didn&apos;t update their docs when they did it, meaning all my help documentation searching was pointless because they kept pointing me to when Gemini was a per-user subscription, not a UI nightmare that they decided to force everyone to use. Like a bad cook at a dinner party trying to sneak their burned appetizers onto my place, Google clearly understood I didn&apos;t want their garbage and decided what I wanted didn&apos;t matter. </p><p>If it were just Google, I might dismiss this as the result of having a particularly lackluster AI product. But it&#x2019;s not just Google. Microsoft and Apple seem equally desperate to find anyone who wants these AI features.</p><blockquote>Google&#x2019;s not the only company walking back its AI up-charge: Microsoft announced in November that its own Copilot Pro AI features, which had also previously been a $20 monthly upgrade, would become <a href="https://www.theverge.com/2024/11/7/24290268/microsoft-copilot-office-features-microsoft-365">part of the standard Microsoft 365 subscription</a>. So far, that&#x2019;s only for the Personal and Family subscriptions, and only in a few places. But these companies all understand that this is their moment to teach people new ways to use their products and win new customers in the process. They&#x2019;re betting that the cost of rolling out all these AI features to everyone will be worth it in the long run. <a href="https://www.theverge.com/2025/1/15/24343794/google-workspace-ai-features-free" rel="noreferrer">Source</a></blockquote><p>Despite billions in funding, stealing the output of all humans from all time and being free for consumers to try, not enough users are sufficiently impressed that they are going to work and asking for the premium package. If that isn&apos;t bad enough, it also seems that these services are <em>extremely</em> expensive to offer, with even OpenAI&apos;s $200 a month Pro subscription losing money. </p><figure class="kg-card kg-image-card"><img src="https://matduggan.com/content/images/2025/01/image-14.png" class="kg-image" alt loading="lazy" width="1200" height="396" srcset="https://matduggan.com/content/images/size/w600/2025/01/image-14.png 600w, https://matduggan.com/content/images/size/w1000/2025/01/image-14.png 1000w, https://matduggan.com/content/images/2025/01/image-14.png 1200w" sizes="(min-width: 720px) 720px"></figure><h3 id="watch-the-bubble-burst">Watch The Bubble Burst</h3><p>None of this should be taken to mean &quot;LLMs serve no purpose&quot;. LLMs are real tools and they can serve a useful function, in very specific applications. It just doesn&apos;t seem like those applications matter enough to normal people to actually pay anyone for them. </p><p>Given the enormous cost of building and maintaining these systems, companies were faced with a choice. Apple took its foot off the LLM gas pedal with the following changes in the iOS beta. </p><ul><li>When you enable notification summaries, iOS 18.3 will make it clearer that the feature &#x2013; like all Apple Intelligence features &#x2013; is a beta.</li><li>You can now disable notification summaries for an app directly from the Lock Screen or Notification Center by swiping, tapping &#x201C;Options,&#x201D; then choosing the &#x201C;Turn Off Summaries&#x201D; option.</li><li>On the Lock Screen, notification summaries now use italicized text to better distinguish them from normal notifications.</li><li>In the Settings app, Apple now warns users that notification summaries &#x201C;may contain errors.&#x201D;</li><li>Additionally, notification summaries have been temporarily disabled entirely for the News &amp; Entertainment category of apps. Notification summaries will be re-enabled for this category with a future software update as Apple continues to refine the experience.</li></ul><p>This is smart, it wasn&apos;t working that well and the very public failures are a bad look for any tech company. Microsoft has decided to go pretty much the exact opposite direction <a href="https://arstechnica.com/gadgets/2025/01/amid-a-flurry-of-hype-microsoft-reorganizes-entire-dev-team-around-ai/" rel="noreferrer">and reorganize their entire developer-centric division around AI. </a> Ironically the Amazon Echo teams seems more interested in accuracy than Apple and have committed to getting hallucinations as close to zero as possible. <a href="https://arstechnica.com/ai/2025/01/amazon-must-solve-hallucination-problem-before-launching-ai-enabled-alexa/" rel="noreferrer">Source</a></p><p>High level though, AI is starting to look a lot like executive vanity. A desperate desire to show investors that your company isn&apos;t behind the curve of innovation and, once you have committed financially, doing real reputational harm to some core products in order to be convincing. I never imagined a world where Google would act so irresponsibly with some of the crown jewels of their portfolio of products, but as we saw with Search, they&apos;re no longer interested in what users want, even paying users. </p>


guid: 679a00f89b9c350001c9c6bb
title: TIL How To Make Brown Noise in Python
author: Mathew Duggan
url: https://matduggan.com/til-how-to-make-brown-noise-in-python/
feedurl: https://matduggan.com/rss/
content_mime_type: text/html
content:
<p>My daughter has been a terrible sleeper since we brought her home from the hospital and the only thing that makes a difference is white noise. We learned this while I was riding the Copenhagen Metro late at night with her so that my wife could get some sleep. I realized she was almost immediately falling asleep when we got on the subway.</p><p>After that we experimented with a lot of &quot;white noise&quot; machines, which worked but ultimately all died. The machines themselves are expensive and only last about 6-8 months of daily use. I decided to rig up a simple Raspberry Pi MP3 player with a speaker and a battery which worked great. Once it&apos;s not a rats nest of cables I&apos;ll post the instructions on how I did that, but honestly there isn&apos;t much to it. </p><p>It took some experimentation to get the &quot;layered brown noise&quot; effect I wanted. There are obviously simpler ways to do it that are less computationally expensive but I like how this sounds. </p><pre><code class="language-Python">import numpy as np
from scipy.io.wavfile import write
from scipy import signal

# Parameters for the brown noise generation
sample_rate = 44100  # Sample rate in Hz
duration_hours = 1  # Duration of the audio in hours
noise_length = duration_hours * sample_rate * 3600  # Total number of samples

# Generate white noise
white_noise = np.random.randn(noise_length)

# Define frequency bands and corresponding low-pass filter parameters
freq_bands = [5, 10, 20, 40, 80, 160, 320]  # Frequency bands in Hz
filter_order = 4
low_pass_filters = []

for freq in freq_bands:
    b, a = signal.butter(filter_order, freq / (sample_rate / 2), btype=&apos;low&apos;)
    low_pass_filters.append((b, a))

# Generate multiple layers of brown noise with different frequencies
brown_noise_layers = []
for b, a in low_pass_filters:
    filtered_noise = np.convolve(white_noise, np.ones(filter_order)/filter_order, mode=&apos;same&apos;)
    filtered_noise = signal.lfilter(b, a, filtered_noise)
    brown_noise_layers.append(filtered_noise)

# Mix all layers together
brown_noise_mixed = np.sum(np.vstack(brown_noise_layers), axis=0)

# Normalize the noise to be within the range [-1, 1]
brown_noise_mixed /= max(abs(brown_noise_mixed))

# Convert to int16 as required by .wav file format
audio_data = (brown_noise_mixed * 32768).astype(np.int16)

# Write the audio data to a .wav file
write(&apos;brown_noise.wav&apos;, sample_rate, audio_data)</code></pre><p>Then to convert it from .wav to mp3 I just ran this: <code>ffmpeg -i brown_noise.wav -ab 320k brown_noise.mp3</code></p><p>So in case you love brown noise and wanted to make a 12 hour or whatever long mp3, this should get you a nice premium multilayer sounding version. </p>


guid: https://mitjafelicijan.com/most-likely-to-succeed-in-year-of-2011.html
title: Most likely to succeed in the year of 2011
url: https://mitjafelicijan.com/most-likely-to-succeed-in-year-of-2011.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>The year of 2010 was definitely the year of Geo-location. The market responded
beautifully and lots of very cool services were launched. We all have to thank
the mobile market for such extensive adoption. With new generations of mobile
phones that are not only buffed with high-tech hardware but are also affordable.
We can now manage tasks that were not so long time ago, almost Star Trek’ish.
And all this had and has great influence on the destination to which we are
going now.</p>
<p>Reading all this articles about new innovation about new thriving technologies
makes me wonder what’s the next step. The future is the mesh, like Lisa Gansky
said in her book The Mesh.</p>
<p>Many still have conservative views on distributed systems. The problems with
security of information. Fear of not controlling every aspect of information
flow. I am very opened to distributed systems and heterogeneous applications,
and I think this is the correct and best way to proceed.</p>
<p>This year will definitely be about communication platforms. Mobile to mobile.
Machine to mobile and vice versa. All the tech is available and ready to put
into action. Wireless is today’s new mantra. And the concept of semantic web is
now ready for industry.</p>
<p>Applications and developers now can gain access to new layers of systems and can
prepare and build solutions to meet the high quality needs of market. The speed
is everything now.</p>
<p>My vote goes to “Machine to Machine” and “Embedded Systems”!</p>
<ul>
<li><a href="http://en.wikipedia.org/wiki/Machine-to-Machine">Machine-to-Machine</a></li>
<li><a href="http://www.bitxml.org/">The ultimate M2M communication protocol</a></li>
<li><a href="http://www.coosproject.org/maven-site/1.0.0/project-info.html">COOS Project (connectivity initiative)</a></li>
<li><a href="http://m2m.com/index.jspa">Community for machine-to-machine</a></li>
<li><a href="http://en.wikipedia.org/wiki/Embedded_system">Embedded system</a></li>
</ul>


guid: https://mitjafelicijan.com/led-technology-not-so-eco.html
title: LED technology might not be as eco-friendly as you think
url: https://mitjafelicijan.com/led-technology-not-so-eco.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>There is a lot of talk about LED technology. It is beginning to infiltrate
industry at a fast rate, and it’s a challenge for designers and also engineers.
I wondered when a weakness will be revealed. Then I stomped on an article
talking about harm in using LED technology. It looks like this magical
technology is not so magical and eco-friendly.</p>
<p>A new study from the University of California indicates that LED lights contain
toxic metals, and should be produced, used and disposed of carefully.  Besides
the lead and nickel, the bulbs and their associated parts were also found to
contain arsenic, copper, and other metals that have been linked to different
cancers, neurological damage, kidney disease, hypertension, skin rashes and
other illnesses in humans, and to ecological damage in waterways.</p>
<p>Since then, I haven’t yet found any regulation for disposal of LED lights or any
other regulation or standard. This might be a problem in the future. And it is a
massive drawback. This might have quite an impact on consumer market.</p>
<p>Nevertheless, there is a potential, and I am sure the market will adapt. I also
hope I will be reading documents regarding solution for this concern soon.</p>
<p><strong>Additional resources:</strong></p>
<ul>
<li><a href="http://ezinearticles.com/?Recycling-and-Disposal-of-Light-Bulbs&amp;id=1091304">Recycling and Disposal of Light Bulbs</a></li>
<li><a href="http://www.ehow.com/how_7483442_dispose-lowenergy-light-bulb.html">How to Dispose of a Low-Energy Light Bulb</a></li>
</ul>


guid: https://mitjafelicijan.com/wireless-sensor-networks.html
title: Wireless sensor networks
url: https://mitjafelicijan.com/wireless-sensor-networks.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>Zigbee networks have this wonderful capability to self-heal, which means they
can reorder connections between them if one of them is inoperable. This works
our of the box when you deploy them. But you have to have in mind that achieving
this is not as easy as you would think. None of it is plug&amp;play.  So to make
your life a bit easier, here are some pointers which, I hope, will help you.</p>
<ul>
<li>Be careful when you are ordering your equipment abroad. There are many rules
and regulations you need to comply before you get your Xbee radios. What they
do is they wait until you prove that you won’t use the technology for some
kind of evil take over control of the world project :). For this, they have
EAR (Export Administration Regulations) which basically means “This product
may require a license to export from the United States.”.</li>
<li>I don’t know if this applies for every country, but when we purchased our Xbee
radios from Mouser, this was mandatory! What we needed to do was to print out
a form and write information about our company and send them a copy via
email. With this document, we proved that we are a legitimate company.</li>
<li>When you complete your purchase and send all the documentation, you are not
clear yet. Then customs will take it from there :). There will be some
additional costs. Before purchasing, make sure you have as much information
about costs as possible. Because it can get costly in the end.</li>
<li>I suggest you use companies from your country. You can seriously cut your
costs. Here in Slovenia, the best option so far as I know is Farnell. And
based on my personal experience, they rock! All I need to say!</li>
<li>Make plans when ordering larger quantities. Do not, I say, do not make your
orders in December! :) Believe me! You will have problems with stock they can
provide for you. So, we were forced to buy some things from Mouser, which was
extremely painful because of all the regulations you need to obey when
importing goods from the USA.</li>
<li>Make sure that firmware version on your Xbee radios is exactly the same! Do
not get creative!!! I propose using templates. You can get template by
exporting settings/profile in X-CTU application. Make sure you have enabled
“Upgrade firmware” so you can be sure each radio has the same firmware.</li>
<li>And again: make plans! Plan everything! In months advanced! You will thank me
later :)</li>
<li>Test, test, test. Wireless networks can be tricky.</li>
</ul>
<p>If you are serious, I suggest you buy this book, Building Wireless Sensor
Networks. You will get a glimpse of how networks work in lumens terms. It is a
good starting point for everybody who wants to build wireless networks.</p>
<p><strong>Additional resources:</strong></p>
<ul>
<li><a href="http://www.digi.com/aboutus/export/generalexportinfo">http://www.digi.com/aboutus/export/generalexportinfo</a></li>
<li><a href="http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons">http://doresearch.stanford.edu/research-scholarship/export-controls/export-controlled-or-embargoed-countries-entities-and-persons</a></li>
<li><a href="http://www.bis.doc.gov/licensing/exportingbasics.htm">http://www.bis.doc.gov/licensing/exportingbasics.htm</a></li>
</ul>


guid: https://mitjafelicijan.com/software-development-pitfalls.html
title: Software development and my favorite pitfalls
url: https://mitjafelicijan.com/software-development-pitfalls.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>Over the years I had the privilege to work on some very excited projects both in
software development field and also in electronics field and every experience
taught me some invaluable lessons about how NOT TO approach development. And
through this post I will try to point out some absurd, outdated techniques I
find the most annoying and damaging during a development cycle. There will be
swearing because this topic really gets on my nerves and I never coherently
tried to explain them in writing. So if I get heated up, please bear with me.</p>
<p>As new methods of project management are emerging, underlying processes still
stay old and outdated. This is mainly because we as people are unable to
completely shift away from these approaches.</p>
<p>I was always struggling with communication, and many times that cost me a
relationship or two because I was not on the ball all the time. Through every
experience, I became more convinced that I am the problem and never ever doubted
that the problem may be that communication never evolved a single step from
emails. And if you think for a second, not many things have changed around this
topic. We just have different representations of email (message boards, chats,
project management tools). And I believe this is the real issue we are facing
now.</p>
<p>There are many articles written about hyper connectivity and the effects that
are a direct result of it. But mainstream does nothing towards it. We are just
putting out fires, and we do nothing to prevent it. I am certain this will be a
major source of grief in coming years. And what we all can do to avoid this is
to change our mindset and experiment on our communication skills, development
approaches. We need to maximize possible output that a person can give. And to
achieve this we need to listen to them, encourage them. I know that not
everybody is a naturally born leader, but with enough practice and encouragement
they also can become active participants in leadership.</p>
<p>There are many talks now about methodologies such as Scrum, Kanban, Cleanroom
and they all fucking piss me of :). These are all boxes that imprison people and
take away their freedom of thought. This is a straightforward mindfuck /
amputation of creativity.</p>
<p>Let me list a couple of things that I find really destructive and bad for a
project and in a long run company.</p>
<h2 id="ping-emails">Ping emails</h2>
<p>Ping emails are emails you have to write as soon as you receive an email. Its
sole purpose is to inform the sender that you received their email, and you are
working on it. Its result is only to calm down the sender that their task is
being dealt with. It’s intent basically is, I did my job by sending you this
email, so I am on clear grounds. I categorize this email as fuck you email.
This is one of the most irritating types of emails I need to write. This is the
ultimate control freak show you can experience, and it gives the sender a false
feeling of control. Newsflash: We do not live in 1982 where there was a
possibility that email never reached the destination. I really hate this from
the bottom of my heart.</p>
<p>They should be like: “Yes, I am fucking alive, and I am at your service my
leash!”. I guess if I would reply like this, I wouldn’t have to write any more
of this kind of messages.</p>
<h2 id="everybody-is-a-project-manager">Everybody is a project manager</h2>
<p>Well, this is a tough one. I noticed that as soon as you let people to give
their suggestions, you are basically screwed. There is a truth in the saying:
“Give low expectations and deliver little more than you promised.”.</p>
<p>People tend to take a role of a manager as soon as they are presented with an
opportunity. And by getting angry at them, you only provoke yourself. They are
not at fault. You just need to tell them they are only giving suggestions and
not tasks at the beginning and everything will be alright. But if you give them
a feeling that they are in control, you will have immense problems explaining
why their features are not in current release.</p>
<p>Project mission must be always leading project requirements and any deviation
from it will result in major project butchering. And by this, I mean that the
project will get its own path, and you will be left with half done software that
helps nobody. Clear mission goals and clean execution will allow you to develop
software will clear intent.</p>
<h2 id="we-are-never-wrong">We are never wrong</h2>
<p>I find this type of arrogance the worst. We must always conduct ourselves that
we are infallible and cannot make mistakes. As soon as a procedure or process is
established, there is no room for changes or improvements. This is the most
idiotic thing someone can say of think. I think that processes need to involve
and change over time. This is imperative and need to have in your organization
if you want to improve and develop company. We all need to grow balls and change
everything in order to adapt to current situations. Being a prisoner of
predefined processes kills creativity.</p>
<p>I am constantly trying new software for project managing and communication.  I
believe every team has its own dynamic, and it needs to be discovered
organically and naturally through many experiments. By putting the team in a
box, you are amputating their creativity and therefore minimizing their
potential. But if you talk to an executive, you will mainly find archetypical
thinking and a strong need to compartmentalize everything from business
processes to resource management. And this type of management that often
displays micromanagement techniques only works for short periods (couple of
years) and then employees either leave the company or become basically retarded
drones on autopilot.</p>
<h2 id="micromanaging">Micromanaging</h2>
<p>This basically implies that everybody on the team is an idiot who needs to have
a to-do list that they cannot write themselves. How about spoon-feeding the team
at launch because besides the team leader, everybody must be a retarded idiot at
best?</p>
<p>I prefer milestones as they give developers much more freedom and creativity in
developing and not waste their time checking some bizarre to-do list that was
not even thought through. Projects constantly change throughout the development
cycle, and all you are left at the end is a list of unchecked tasks and the
wrath of management why they are not completed. Best WTF moment!</p>
<h2 id="human-contact--no-need-for-it">Human contact — no need for it!</h2>
<p>We are vigorously trying to eliminate physical contact by replacing short
meetings with software, with no regards that we are not machines. Many times a
simple 5-min meeting at morning can solve most of the problems. In rapid
development, short bursts of man to man communication is possibly the best way
to go.</p>
<p>We now have all this software available, and all what we get out of it is a
giant clusterfuck. An obstacle and not a solution. So, why we still use them?</p>
<h2 id="mvp-is-killing-innovation">MVP is killing innovation</h2>
<p>Many will disagree with me on this one, but I stand strong by this statement.
What I noticed in my experience that all this buzz words around us only mislead
and capture us in a circle of solving issues that already have a solution, but
we are unable to see it without using some fancy word for it.</p>
<p>The toughest thing to do for a developer is to minimize requirements. Well, this
is though only for bad developers. Yes, I said it. There are many types of
developers out there. And those unable to minimize feature scope are the ones
you don’t need on your team. Their only goal is to solve problems that exist
only in their heads. And then you have to argue with them, and waste energy on
them, instead of developing your awesome product. They are a cancer and I
suggest you cut them off.</p>
<p>MVP as an idea is great, but sadly people don’t understand underlying
philosophy, and they spent too much time focusing and fixating on something that
every sane person with normal IQ will understand without some made up
acronym. And the result is a lot of talking and barely no execution.</p>
<p>Well, MVP is not directly killing innovation, but stupid people do when they try
to understand it.</p>
<h2 id="pressure-wasteland">Pressure wasteland</h2>
<p>You must never allow to be pressured into confirming a deadline if you are not
confident. We often feel a need that we are in service of others, which is true
to some extent. But it is also true that others are in service to us to some
extent. And we forget this all the time. We are all pressured all the time to
make decisions just to calm other people down. And when they leave your office
you experience WTF moment :) How the hell did they manage to fuck me up again?</p>
<p>People need to realize that the more pressure you put on somebody, the less they
will be able to do. So 5-min update email requests will only resolve in mental
breakdown and inability to work that day. Constant poking is probably the only
thing I lose my mind instantly. For all you that are doing this: “Stop bothering
us with your insecurities and let us do our job. We will do it quicker and
better without you breathing down our necks.”</p>
<p>If this happens to me, I end up with no energy at the end. Don’t you get it?
You will get much more from and out of me if you ask me like a human person and
not your personal butler. On a long run, you are destroying your relationships
and nobody would want to work with you. Your schizophrenic approach will damage
only you in a long run. Nobody is anybody’s property.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I am guilty of many things described in this post. And I find it hard sometimes
to acknowledge this. And I lie to myself and try vigorously to find some
explanation why I do these things. There is always space for growth. And maybe
you will also find some of yourself in this post and realize what needs to
change for you to evolve.</p>


guid: https://mitjafelicijan.com/golang-profiling-simplified.html
title: Golang profiling simplified
url: https://mitjafelicijan.com/golang-profiling-simplified.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>Many posts have been written regarding profiling in Golang and I haven’t found
proper tutorial regarding this. Almost all of them are missing some part of
important information and it gets pretty frustrating when you have a deadline
and are not finding simple distilled solution.</p>
<p>Nevertheless, after searching and experimenting I have found a solution that
works for me and probably should also for you.</p>
<h2 id="where-are-my-pprof-files">Where are my pprof files?</h2>
<p>By default pprof files are generated in /tmp/ folder. You can override folder
where this files are generated programmatically in your golang code as we will
see below in example.</p>
<h2 id="why-is-my-cpu-profile-empty">Why is my CPU profile empty?</h2>
<p>I have found out that sometimes CPU profile is empty because program was not
executing long enough. Programs, that execute too quickly don’t produce pprof
file in my cases. Well, file is generated but only contains 4KB of information.</p>
<h2 id="profiling">Profiling</h2>
<p>As you can see from examples we are executing dummy_benchmark functions to
ensure some sort of execution. Memory profiling can be done without such a
“complex” function. But CPU profiling needs it.</p>
<p>Both memory and CPU profiling examples are almost the same. Only parameters in
main function when calling profile.Start are different. When we set
profile.ProfilePath(“.”) we tell profiler to store pprof files in the same
folder as our program.</p>
<h3 id="memory-profiling">Memory profiling</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">package</span> main
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> (
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;fmt&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;time&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;github.com/pkg/profile&#34;</span>
</span></span><span style="display:flex;"><span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">func</span> dummy_benchmark() {
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fmt.Println(<span style="color:#a31515">&#34;first set ...&#34;</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">for</span> i := 0; i &lt; 918231333; i++ {
</span></span><span style="display:flex;"><span>    i *= 2
</span></span><span style="display:flex;"><span>    i /= 2
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;-time.After(time.Second*3)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fmt.Println(<span style="color:#a31515">&#34;sencond set ...&#34;</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">for</span> i := 0; i &lt; 9182312232; i++ {
</span></span><span style="display:flex;"><span>    i *= 2
</span></span><span style="display:flex;"><span>    i /= 2
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">func</span> main() {
</span></span><span style="display:flex;"><span>  <span style="color:#00f">defer</span> profile.Start(profile.MemProfile, profile.ProfilePath(<span style="color:#a31515">&#34;.&#34;</span>), profile.NoShutdownHook).Stop()
</span></span><span style="display:flex;"><span>  dummy_benchmark()
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><h3 id="cpu-profiling">CPU profiling</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">package</span> main
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> (
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;fmt&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;time&#34;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#a31515">&#34;github.com/pkg/profile&#34;</span>
</span></span><span style="display:flex;"><span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">func</span> dummy_benchmark() {
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fmt.Println(<span style="color:#a31515">&#34;first set ...&#34;</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">for</span> i := 0; i &lt; 918231333; i++ {
</span></span><span style="display:flex;"><span>    i *= 2
</span></span><span style="display:flex;"><span>    i /= 2
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;-time.After(time.Second*3)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  fmt.Println(<span style="color:#a31515">&#34;sencond set ...&#34;</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">for</span> i := 0; i &lt; 9182312232; i++ {
</span></span><span style="display:flex;"><span>    i *= 2
</span></span><span style="display:flex;"><span>    i /= 2
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">func</span> main() {
</span></span><span style="display:flex;"><span>  <span style="color:#00f">defer</span> profile.Start(profile.CPUProfile, profile.ProfilePath(<span style="color:#a31515">&#34;.&#34;</span>), profile.NoShutdownHook).Stop()
</span></span><span style="display:flex;"><span>  dummy_benchmark()
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><h3 id="generating-profiling-reports">Generating profiling reports</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># memory profiling</span>
</span></span><span style="display:flex;"><span>go build mem.go
</span></span><span style="display:flex;"><span>./mem
</span></span><span style="display:flex;"><span>go tool pprof -pdf ./mem mem.pprof &gt; mem.pdf
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># cpu profiling</span>
</span></span><span style="display:flex;"><span>go build cpu.go
</span></span><span style="display:flex;"><span>./cpu
</span></span><span style="display:flex;"><span>go tool pprof -pdf ./cpu cpu.pprof &gt; cpu.pdf
</span></span></code></pre><p>This will generate PDF document with visualized profile.</p>
<ul>
<li><a href="/assets/posts/go-profiling/golang-profiling-mem.pdf">Memory PDF profile example</a></li>
<li><a href="/assets/posts/go-profiling/golang-profiling-cpu.pdf">CPU PDF profile example</a></li>
</ul>


guid: https://mitjafelicijan.com/what-i-ve-learned-developing-ad-server.html
title: What I've learned developing ad server
url: https://mitjafelicijan.com/what-i-ve-learned-developing-ad-server.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>For the past year and half I have been developing native advertising server that
contextually matches ads and displays them in different template forms on
variety of websites. This project grew from serving thousands of ads per day to
millions.</p>
<p>The system is made from couple of core components:</p>
<ul>
<li>API for serving ads,</li>
<li>Utils - cronjobs and queue management tools,</li>
<li>Dashboard UI.</li>
</ul>
<p>Initial release was using <a href="https://www.mongodb.com/">MongoDB</a> for full-text
search but was later replaced by <a href="https://www.elastic.co/">Elasticsearch</a> for
better CPU utilization and better search performance. This provided us with many
amazing functionalities of <a href="https://www.elastic.co/">Elasticsearch</a>.  You should
check it out if you do any search related operations.</p>
<p>Because the premise of the server is to provide native ad experience, they are
rendered on the client side via simple templating engine. This ensures that ads
can be displayed number of different ways based on the visual style of the
page. And this makes JavaScript client library quite complex.</p>
<p>So now that you know basic information about the product lets get into the
lessons we learned.</p>
<h2 id="aggregate-everything">Aggregate everything</h2>
<p>After beta version was released everything (impressions, clicks, etc) was
written in nanosecond resolution in the database. At that time we were using
<a href="https://www.postgresql.org/">PostgreSQL</a> and database quickly grew way above
200GB in disk space. And that was problematic. Statistics took disturbingly long
time to aggregate. Also using indexes on stats table in database was no help
after we reached 500 million datapoints.</p>
<blockquote>
<p>There is a marketing product information and there is real life experience.
And the tend to be quite the opposite.</p>
</blockquote>
<p>This was the reason that now everything is aggregated on daily basis and this
data is then fed to Elastic in form of daily summary. With this we achieved we
can now track many more dimensions such as zone, channel and platform
information.  And with this information we can now adapt occurrences of ads on
specific places more precisely.</p>
<p>We have also adapted <a href="https://redis.io/">Redis</a> as a full-time citizen in our
stack. Because Redis also stores information on a local disk we have some sort
of backup if server would accidentally suffer some failure.</p>
<p>All the real-time statistics for ad serving and redirecting is presented as
counters in Redis instance and daily extracted and pushed to Elastic.</p>
<h2 id="measure-everything">Measure everything</h2>
<p>The thing about software is that we really don't know how well it is performing
under load until such load is presented. When testing locally everything is fine
but when on production things tend to fall apart.</p>
<p>As a solution for this we are measuring everything we can. Function execution
time (by encapsulating functions with timers), server performance (cpu, memory,
disk, etc), Nginx and <a href="https://uwsgi-docs.readthedocs.io/">uWSGI</a> performance.
We sacrifice a bit of performance for the sake of this information. And we store
all this information for later analysis.</p>
<p><strong>Example of function execution time</strong></p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  &#34;get_final_filtered_ads&#34;: {
</span></span><span style="display:flex;"><span>    &#34;counter&#34;: 1931250,
</span></span><span style="display:flex;"><span>    &#34;avg&#34;: 0.0066143431,
</span></span><span style="display:flex;"><span>    &#34;elapsed&#34;: 12773.9500310003
</span></span><span style="display:flex;"><span>  },
</span></span><span style="display:flex;"><span>  &#34;store_keywords_statistics&#34;: {
</span></span><span style="display:flex;"><span>    &#34;counter&#34;: 1931011,
</span></span><span style="display:flex;"><span>    &#34;avg&#34;: 0.0004605267,
</span></span><span style="display:flex;"><span>    &#34;elapsed&#34;: 889.2821669996
</span></span><span style="display:flex;"><span>  },
</span></span><span style="display:flex;"><span>  &#34;match_by_context&#34;: {
</span></span><span style="display:flex;"><span>    &#34;counter&#34;: 1931011,
</span></span><span style="display:flex;"><span>    &#34;avg&#34;: 0.0055960716,
</span></span><span style="display:flex;"><span>    &#34;elapsed&#34;: 10806.0758889999
</span></span><span style="display:flex;"><span>  },
</span></span><span style="display:flex;"><span>  &#34;match_by_high_performance&#34;: {
</span></span><span style="display:flex;"><span>    &#34;counter&#34;: 262,
</span></span><span style="display:flex;"><span>    &#34;avg&#34;: 0.0152770229,
</span></span><span style="display:flex;"><span>    &#34;elapsed&#34;: 4.00258
</span></span><span style="display:flex;"><span>  },
</span></span><span style="display:flex;"><span>  &#34;store_impression_stats&#34;: {
</span></span><span style="display:flex;"><span>    &#34;counter&#34;: 1931250,
</span></span><span style="display:flex;"><span>    &#34;avg&#34;: 0.0006189991,
</span></span><span style="display:flex;"><span>    &#34;elapsed&#34;: 1195.4419869999
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><p>We have also started profiling with <a href="https://pymotw.com/2/profile/">cProfile</a>
and then visualizing with <a href="http://kcachegrind.sourceforge.net/">KCachegrind</a>.
This provides much more detailed look into code execution.</p>
<h2 id="cache-control-is-your-friend">Cache control is your friend</h2>
<p>Because we use Javascript library for rendering ads we rely on this script
extensively and when in need we need to be able to change behavior of the script
quickly.</p>
<p>In our case we can not simply replace javascript url in html code. It usually
takes a day or two for the guys who maintain sites to change code or add
?ver=xxx attribute. And this makes rapid deployment and testing very difficult
and time consuming. There is a limitation of how much you can test locally.</p>
<p>We are now in the process of integrating <a href="https://www.google.com/analytics/tag-manager/">Google Tag
Manager</a> but couple of websites
are developed on ASP.net platform that have some problems with tag manager. With
a solution below we are certain that we are serving latest version of the
script.</p>
<p>And it only takes one mistake and users have the script cached and in case of
caching it for 1 year you probably know where the problem is.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># nginx ➜ /etc/nginx/sites-available/default
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#00f">location</span> <span style="color:#a31515">/static/</span> {
</span></span><span style="display:flex;"><span>  <span style="color:#00f">alias</span> <span style="color:#a31515">/path-to-static-content/</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">autoindex</span> off;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">charset</span> <span style="color:#a31515">utf-8</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">gzip</span> on;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">gzip_types</span> <span style="color:#a31515">text/plain</span> <span style="color:#a31515">application/javascript</span> <span style="color:#a31515">application/x-javascript</span> <span style="color:#a31515">text/javascript</span> <span style="color:#a31515">text/xml</span> <span style="color:#a31515">text/css</span>;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">location</span> ~<span style="color:#a31515">*</span> <span style="color:#a31515">\.(ico|gif|jpeg|jpg|png|woff|ttf|otf|svg|woff2|eot)</span>$ {
</span></span><span style="display:flex;"><span>    <span style="color:#00f">expires</span> <span style="color:#a31515">1y</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">add_header</span> <span style="color:#a31515">Pragma</span> <span style="color:#a31515">public</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">add_header</span> <span style="color:#a31515">Cache-Control</span> <span style="color:#a31515">&#34;public&#34;</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>  <span style="color:#00f">location</span> ~<span style="color:#a31515">*</span> <span style="color:#a31515">\.(css|js|txt)</span>$ {
</span></span><span style="display:flex;"><span>    <span style="color:#00f">expires</span> <span style="color:#a31515">3600s</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">add_header</span> <span style="color:#a31515">Pragma</span> <span style="color:#a31515">public</span>;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">add_header</span> <span style="color:#a31515">Cache-Control</span> <span style="color:#a31515">&#34;public,</span> <span style="color:#a31515">must-revalidate&#34;</span>;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><p>Also be careful when redirecting to url in your python code. We noticed that if
we didn't precisely setup cache control and expire headers in response we didn't
get the request on the server and therefore couldn't measure clicks.  So when
redirecting do as follows and there will be no problems.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># python ➜ bottlepy web micro-framework</span>
</span></span><span style="display:flex;"><span>response = bottle.HTTPResponse(status=302)
</span></span><span style="display:flex;"><span>response.set_header(<span style="color:#a31515">&#34;Cache-Control&#34;</span>, <span style="color:#a31515">&#34;no-store, no-cache, must-revalidate&#34;</span>)
</span></span><span style="display:flex;"><span>response.set_header(<span style="color:#a31515">&#34;Expires&#34;</span>, <span style="color:#a31515">&#34;Thu, 01 Jan 1970 00:00:00 GMT&#34;</span>)
</span></span><span style="display:flex;"><span>response.set_header(<span style="color:#a31515">&#34;Location&#34;</span>, url)
</span></span><span style="display:flex;"><span><span style="color:#00f">return</span> response
</span></span></code></pre><blockquote>
<p>Cache control in browsers is quite aggressive and you need to be precise to
avoid future problems. We learned that lesson the hard way.</p>
</blockquote>
<h2 id="learn-nginx">Learn NGINX</h2>
<p>When deciding on a web server we went with Nginx as a reverse proxy for our
applications. We adapted micro-service oriented architecture early in the
project to ensure when we scale we can easily add additional servers to our
cluster. And Nginx was crucial to perform load balancing and static content
delivery.</p>
<p>At first our config file was quite simple and later grew larger. After patching
and adding new settings I sat down and learned more about the guts of Nginx.
This proved to be very useful and we were able to squeeze much more out of our
setup. So I advise you to take your time and read through the
<a href="https://nginx.org/en/docs/">documentation</a>. This saved us a lot of headache.
Googling for solutions only goes so far.</p>
<h2 id="use-redismemcached">Use Redis/Memcached</h2>
<p>As explained above we are using caching basically for everything. It is the
corner stone of our services. At first we were very careful about the quantity
of things we stored in <a href="https://redis.io/">Redis</a>. But we later found out that
the memory footprint is very low even when storing large amount of data in it.</p>
<p>So we gradually increased our usage to caching whole HTML outputs of dashboard.
This improved our performance in order of magnitude. And by using native TTL
support this goes hand in hand with our needs.</p>
<p>The reason why we choose <a href="https://redis.io/">Redis</a> over
<a href="https://memcached.org/">Memcached</a> was the nature of scalability of Redis out
of the box. But all this can be achieved with Memcached.</p>
<h2 id="conclusion">Conclusion</h2>
<p>There are a lot more details that could have been written and every single topic
in here deserves it's own post but you probably got the idea about the problems
we faced.</p>


guid: https://mitjafelicijan.com/profiling-python-web-applications-with-visual-tools.html
title: Profiling Python web applications with visual tools
url: https://mitjafelicijan.com/profiling-python-web-applications-with-visual-tools.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>I have been profiling my software with KCachegrind for a long time now and I was
missing this option when I am developing API's or other web services. I always
knew that this is possible but never really took the time and dive into it.</p>
<p>Before we begin there are some requirements. We will need to:</p>
<ul>
<li>implement <a href="https://docs.python.org/2/library/profile.html#module-cProfile">cProfile</a> into our web app,</li>
<li>convert output to <a href="http://valgrind.org/docs/manual/cl-manual.html">callgrind</a> format with <a href="https://pypi.python.org/pypi/pyprof2calltree/">pyprof2calltree</a>,</li>
<li>visualize data with <a href="http://kcachegrind.sourceforge.net/html/Home.html">KCachegrind</a> or <a href="http://www.profilingviewer.com/">Profiling Viewer</a>.</li>
</ul>
<p>If you are using MacOS you should check out <a href="http://www.profilingviewer.com/">Profiling
Viewer</a> or
<a href="http://www.maccallgrind.com/">MacCallGrind</a>.</p>
<figure>
<img src="/assets/posts/python-profiling/kcachegrind.png" alt="KCachegrind" />
</figure>
<p>We will be dividing this post into two main categories:</p>
<ul>
<li>writing simple web-service,</li>
<li>visualize profile of this web-service.</li>
</ul>
<h2 id="simple-web-service">Simple web-service</h2>
<p>Let's use virtualenv so we won't pollute our base system. If you don't have
virtualenv installed on your system you can install it with pip command.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># let&#39;s install virtualenv globally</span>
</span></span><span style="display:flex;"><span>$ sudo pip install virtualenv
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># let&#39;s also install pyprof2calltree globally</span>
</span></span><span style="display:flex;"><span>$ sudo pip install pyprof2calltree
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we create project</span>
</span></span><span style="display:flex;"><span>$ mkdir demo-project
</span></span><span style="display:flex;"><span>$ cd demo-project/
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now let&#39;s create folder where we will store profiles</span>
</span></span><span style="display:flex;"><span>$ mkdir prof
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we create empty virtualenv in venv/ folder</span>
</span></span><span style="display:flex;"><span>$ virtualenv --no-site-packages venv
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we now need to activate virtualenv</span>
</span></span><span style="display:flex;"><span>$ source venv/bin/activate
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># you can check if virtualenv was correctly initialized by</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># checking where your python interpreter is located</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># if command bellow points to your created directory and not some</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># system dir like /usr/bin/python then everything is fine</span>
</span></span><span style="display:flex;"><span>$ which python
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we can check now if all is good ➜ if ok couple of</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># lines will be displayed</span>
</span></span><span style="display:flex;"><span>$ pip freeze
</span></span><span style="display:flex;"><span><span style="color:#008000"># appdirs==1.4.3</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># packaging==16.8</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># pyparsing==2.2.0</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># six==1.10.0</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we are ready to install bottlepy ➜ web micro-framework</span>
</span></span><span style="display:flex;"><span>$ pip install bottle
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># you can deactivate virtualenv but you will then go</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># under system domain ➜ for now don&#39;t deactivate</span>
</span></span><span style="display:flex;"><span>$ deactivate
</span></span></code></pre><p>We are now ready to write simple web service. Let's create file app.py and paste
code bellow in this newly created file.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># -*- coding: utf-8 -*-</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> bottle
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> random
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> cProfile
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>app = bottle.Bottle()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># this function is a decorator and encapsulates function</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># and performs profiling and then saves it to subfolder</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># prof/function-name.prof</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># in our example only awesome_random_number function will</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># be profiled because it has do_cprofile defined</span>
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> do_cprofile(func):
</span></span><span style="display:flex;"><span>  <span style="color:#00f">def</span> profiled_func(*args, **kwargs):
</span></span><span style="display:flex;"><span>    profile = cProfile.Profile()
</span></span><span style="display:flex;"><span>    <span style="color:#00f">try</span>:
</span></span><span style="display:flex;"><span>      profile.enable()
</span></span><span style="display:flex;"><span>      result = func(*args, **kwargs)
</span></span><span style="display:flex;"><span>      profile.disable()
</span></span><span style="display:flex;"><span>      <span style="color:#00f">return</span> result
</span></span><span style="display:flex;"><span>    <span style="color:#00f">finally</span>:
</span></span><span style="display:flex;"><span>      profile.dump_stats(<span style="color:#a31515">&#34;prof/&#34;</span> + str(func.__name__) + <span style="color:#a31515">&#34;.prof&#34;</span>)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> profiled_func
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we use profiling over specific function with including</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># @do_cprofile above function declaration</span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/&#34;</span>)
</span></span><span style="display:flex;"><span>@do_cprofile
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> awesome_random_number():
</span></span><span style="display:flex;"><span>  awesome_random_number = random.randint(0, 100)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> <span style="color:#a31515">&#34;awesome random number is &#34;</span> + str(awesome_random_number)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/test&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> test():
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> <span style="color:#a31515">&#34;dummy test&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">if</span> __name__ == <span style="color:#a31515">&#39;__main__&#39;</span>:
</span></span><span style="display:flex;"><span>  bottle.run(
</span></span><span style="display:flex;"><span>    app = app,
</span></span><span style="display:flex;"><span>    host = <span style="color:#a31515">&#34;0.0.0.0&#34;</span>,
</span></span><span style="display:flex;"><span>    port = 4000
</span></span><span style="display:flex;"><span>  )
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># run with &#39;python app.py&#39;</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># open browser &#39;http://0.0.0.0:4000&#39;</span>
</span></span></code></pre><p>When browser hits awesome_random_number() function profile is created in prof/
subfolder.</p>
<h2 id="visualize-profile">Visualize profile</h2>
<p>Now let's create callgrind format from this cProfile output.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>$ cd prof/
</span></span><span style="display:flex;"><span>$ pyprof2calltree -i awesome_random_number.prof
</span></span><span style="display:flex;"><span><span style="color:#008000"># this creates &#39;awesome_random_number.prof.log&#39; file in the same folder</span>
</span></span></code></pre><p>This file can be opened with visualizing tools listed above. In this case we
will be using Profilling Viewer under MacOS. You can open image in new tab.  As
you can see from this example there is hierarchy of execution order of your
code.</p>
<figure>
<img src="/assets/posts/python-profiling/profiling-viewer.png" alt="Profilling Viewer" />
</figure>
<blockquote>
<p>Make sure you convert output of the cProfile output every time you want to
refresh and take a look at your possible optimizations because cProfile updates
.prof file every time browser hits the function.</p>
</blockquote>
<p>This is just a simple example but when you are developing real-life applications
this can be very illuminating, especially to see which parts of your code are
bottlenecks and need to be optimized.</p>
<h2 id="update-2017-04-22">Update 2017-04-22</h2>
<p>Reddit user <a href="https://www.reddit.com/user/mvt">mvt</a> also recommended this awesome
web based profile visualizer <a href="https://jiffyclub.github.io/snakeviz/">SnakeViz</a>
that directly takes output from
<a href="https://docs.python.org/2/library/profile.html#module-cProfile">cProfile</a>
module.</p>
<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="583880c1-002e-41ed-a373-020a0ef2cff9" data-embed-created="2017-04-22T19:46:54.810Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dgljhsb/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># let&#39;s install it globally as well</span>
</span></span><span style="display:flex;"><span>$ sudo pip install snakeviz
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now let&#39;s visualize</span>
</span></span><span style="display:flex;"><span>$ cd prof/
</span></span><span style="display:flex;"><span>$ snakeviz awesome_random_number.prof
</span></span><span style="display:flex;"><span><span style="color:#008000"># this automatically opens browser window and</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># shows visualized profile</span>
</span></span></code></pre><figure>
<img src="/assets/posts/python-profiling/snakeviz.png" alt="SnakeViz" />
</figure>
<p>Reddit user <a href="https://www.reddit.com/user/ccharles">ccharles</a> suggested a better
way for installing pip software by targeting user level instead of using sudo.</p>
<div class="reddit-embed" data-embed-media="www.redditmedia.com" data-embed-parent="false" data-embed-live="false" data-embed-uuid="f4f0459e-684d-441e-bebe-eb49b2f0a31d" data-embed-created="2017-04-22T19:46:10.874Z"><a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/dglpzkx/">Comment</a> from discussion <a href="https://www.reddit.com/r/Python/comments/66v373/profiling_python_web_applications_with_visual/">Profiling Python web applications with visual tools</a>.</div><script async src="https://www.redditstatic.com/comment-embed.js"></script>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># now we need to add this path to our $PATH variable</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we do this my adding this line at the end of your</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># ~/.bashrc file</span>
</span></span><span style="display:flex;"><span>PATH=$PATH:$HOME/.local/bin/
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># in order to use this new configuration you can close</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># and reopen terminal or reload .bashrc file</span>
</span></span><span style="display:flex;"><span>$ source ~/.bashrc
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now let&#39;s test if new directory is present in $PATH</span>
</span></span><span style="display:flex;"><span>$ echo $PATH
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we can install on user level by adding --user</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># without use of sudo</span>
</span></span><span style="display:flex;"><span>$ pip install snakeviz --user
</span></span></code></pre><p>Or as suggested by <a href="https://www.reddit.com/user/mvt">mvt</a> you can
use <a href="https://github.com/mitsuhiko/pipsi">pipsi</a>.</p>


guid: https://mitjafelicijan.com/simple-iot-application.html
title: Simple IOT application supported by real-time monitoring and data history
url: https://mitjafelicijan.com/simple-iot-application.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<h2 id="initial-thoughts">Initial thoughts</h2>
<p>I have been developing these kind of application for the better part of my last
5 years and people keep asking me how to approach developing such application
and I will give a try explaining it here.</p>
<p>IOT applications are really no different than any other kind of applications.
We have data that needs to be collected and visualized in some form of tables or
charts. The main difference here is that most of the times these data is
collected by some kind of device foreign to developer that mainly operates in
web domain. But fear not, it's not that different than writing some JavaScript.</p>
<p>There are many devices able to transmit data via wireless or wired network by
default but for the sake of example we will be using commonly known Arduino with
wireless module already on the board → <a href="https://store.arduino.cc/arduino-mkr1000">Arduino
MKR1000</a>.</p>
<p>In order to make this little project as accessible to others as possible I will
try to make it as inexpensive as possible. And by this I mean that I will avoid
using hosted virtual servers and will be using my own laptop as a server. But
you must buy Arduino MKR1000 to follow steps below. But if you would want to
deploy this software I would suggest using
<a href="https://www.digitalocean.com">DigitalOcean</a> → smallest VPS is only per month
making this one of the most affordable option out there. Please notice that this
software will not run on stock web hosting that only supports LAMP (Linux,
Apache, MySQL, and PHP).</p>
<p>But before we begin please take notice that this is strictly experimental code
and not well optimized and there are much better ways in handling some aspects
of the application but that requires much deeper knowledge of technology that is
not needed for an example like this.</p>
<p><strong>Development steps</strong></p>
<ol>
<li>Simple Python API that will receive and store incoming data.</li>
<li>Prototype C++ code that will read &quot;sensor data&quot; and transmit it to API.</li>
<li>Data visualization with charts → extends Python web application.</li>
</ol>
<p>Step 1. and 3. will share the same web application. One route will be dedicated
to API and another to serving HTML with chart.</p>
<p>Schema below represents what we will try to achieve and how different parts
correlates to each other.</p>
<figure>
<img src="/assets/posts/iot-application/simple-iot-application-overview.svg" alt="Overview" />
</figure>
<h2 id="simple-python-api">Simple Python API</h2>
<p>I have always been a fan of simplicity so we will be using <a href="https://bottlepy.org/docs/dev/">Bottle: Python Web
Framework</a>.  It is a single file web framework
that seriously simplifies working with routes, templating and has built-in web
server that satisfies our need in this case.</p>
<p>First we need to install bottle package. This can be done by downloading
<code>bottle.py</code> and placing it in the root of your application or by using pip
software <code>pip install bottle --user</code>.</p>
<p>If you are using Linux or MacOS then Python is already installed. If you will
try to test this on Windows please install <a href="https://www.python.org/downloads/windows/">Python for
Windows</a>.  There may be some problems
with path when you will try to launch <code>python webapp.py</code> so please take care
of this before you continue.</p>
<h3 id="basic-web-application">Basic web application</h3>
<p>Most basic bottle application is quite simple. Paste code below in
<code>webapp.py</code> file and save.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># -*- coding: utf-8 -*-</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> bottle
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># initializing bottle app</span>
</span></span><span style="display:flex;"><span>app = bottle.Bottle()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># triggered when / is accessed from browser</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># only accepts GET → no POST allowed</span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/&#34;</span>, method=[<span style="color:#a31515">&#34;GET&#34;</span>])
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> route_default():
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> <span style="color:#a31515">&#34;howdy from python&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># starting server on http://0.0.0.0:5000</span>
</span></span><span style="display:flex;"><span><span style="color:#00f">if</span> __name__ == <span style="color:#a31515">&#34;__main__&#34;</span>:
</span></span><span style="display:flex;"><span>  bottle.run(
</span></span><span style="display:flex;"><span>    app = app,
</span></span><span style="display:flex;"><span>    host = <span style="color:#a31515">&#34;0.0.0.0&#34;</span>,
</span></span><span style="display:flex;"><span>    port = 5000,
</span></span><span style="display:flex;"><span>    debug = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    reloader = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    catchall = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>  )
</span></span></code></pre><p>To run this simple application you should open command prompt or terminal on
your machine and go to the folder containing your file and type <code>python webapp.py</code>. If everything goes ok then open your web browser and point it to
<code>http://0.0.0.0:5000</code>.</p>
<p>If you would like change the port of your application (like port 80) and not use
root to run your app this will present a problem. The TCP/IP port numbers below
1024 are privileged ports → this is a security feature. So in order of
simplicity and security use a port number above 1024 like I have used port 5000.</p>
<p>If this fails at any time please fix it before you continue, because nothing
below will work otherwise.</p>
<p>We use 0.0.0.0 as default host so that this app is available over your local
network. If you find your local ip <code>ifconfig</code> and try accessing this site
with your phone (if on same network/router as your machine) this should work as
well (example of such ip <code>http://192.168.1.15:5000</code>). This is a must have
because Arduino will be accessing this application to send it's data.</p>
<h3 id="web-application-security">Web application security</h3>
<p>There is a lot to be said about security and is a topic of many books. Of course
all this can not be written here but to just establish some basic security → you
should always use SSL with your application. Some fantastic free certificates
are available by <a href="https://letsencrypt.org">Let's Encrypt - Free SSL/TLS
Certificates</a>.  With SSL certificate installed you
should then make use of HTTP headers and send your &quot;API key&quot; via a header. If
your key is send via header then this key is encrypted by SSL and send encrypted
over the network. Never send your api keys by GET parameter like
<code>http://example.com/?api_key=somekeyvalue</code>.  The problem that this kind of
sending presents is that this key is visible in logs and by network sniffers.</p>
<p>There is a fantastic article describing some aspects about security: <a href="https://www.keycdn.com/blog/web-application-security-best-practices/">11 Web
Application Security Best
Practices</a>. Please
check it out.</p>
<h3 id="simple-api-for-writing-data-points">Simple API for writing data-points</h3>
<p>We will now be using boilerplate code from example above and extend it to be
SQLite3 because it plays well with Python and can store quite large amount of
able to write data received by API to local storage. For example use I will use
data. I have been using it to collect gigabytes of data in a single database
without any corruption or problems → your experience may vary.</p>
<p>To avoid learning SQLite I will be using <a href="https://dataset.readthedocs.io/en/latest/index.html">Dataset: databases for lazy
people</a>.  This package
abstracts SQL and simplifies writing and reading data from database. You should
install this package with pip software <code>pip install dataset --user</code>.</p>
<p>Because API will use POST method I will be testing if code works correctly by
using <a href="https://chrome.google.com/webstore/detail/restlet-client-rest-api-t/aejoelaoggembcahagimdiliamlcdmfm">Restlet Client for Google
Chrome</a>.
This software also allows you to set headers → for basic security with API_KEY.</p>
<p>To quickly generate passwords or API keys I usually use this nifty website
<a href="https://randomkeygen.com/">RandomKeygen</a>.</p>
<p>Copy and paste code below over your previous code in file <code>webapp.py</code>.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># -*- coding: utf-8 -*-</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> time
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> bottle
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> random
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> dataset
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># initializing bottle app</span>
</span></span><span style="display:flex;"><span>app = bottle.Bottle()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># connects to sqlite database</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># check_same_thread=False allows using it in multi-threaded mode</span>
</span></span><span style="display:flex;"><span>app.config[<span style="color:#a31515">&#34;dsn&#34;</span>] = dataset.connect(<span style="color:#a31515">&#34;sqlite:///data.db?check_same_thread=False&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># api key that will be used in Arduino code</span>
</span></span><span style="display:flex;"><span>app.config[<span style="color:#a31515">&#34;api_key&#34;</span>] = <span style="color:#a31515">&#34;JtF2aUE5SGHfVJBCG5SH&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># triggered when /api is accessed from browser</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># only accepts POST → no GET allowed</span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/api&#34;</span>, method=[<span style="color:#a31515">&#34;POST&#34;</span>])
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> route_default():
</span></span><span style="display:flex;"><span>  status = 400
</span></span><span style="display:flex;"><span>  ts = int(time.time()) <span style="color:#008000"># current timestamp</span>
</span></span><span style="display:flex;"><span>  value = bottle.request.body.read() <span style="color:#008000"># data from device</span>
</span></span><span style="display:flex;"><span>  api_key = bottle.request.get_header(<span style="color:#a31515">&#34;Api_Key&#34;</span>) <span style="color:#008000"># api key from header</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># outputs to console received data for debug reason</span>
</span></span><span style="display:flex;"><span>  print <span style="color:#a31515">&#34;&gt;&gt;&gt; </span><span style="color:#a31515">{}</span><span style="color:#a31515"> :: </span><span style="color:#a31515">{}</span><span style="color:#a31515">&#34;</span>.format(value, api_key)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># if api_key is correct and value is present</span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># then writes attribute to point table</span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">if</span> api_key == app.config[<span style="color:#a31515">&#34;api_key&#34;</span>] <span style="color:#00f">and</span> value:
</span></span><span style="display:flex;"><span>    app.config[<span style="color:#a31515">&#34;dsn&#34;</span>][<span style="color:#a31515">&#34;point&#34;</span>].insert(dict(ts=ts, value=value))
</span></span><span style="display:flex;"><span>    status = 200
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># we only need to return status</span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> bottle.HTTPResponse(status=status, body=<span style="color:#a31515">&#34;&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># starting server on http://0.0.0.0:5000</span>
</span></span><span style="display:flex;"><span><span style="color:#00f">if</span> __name__ == <span style="color:#a31515">&#34;__main__&#34;</span>:
</span></span><span style="display:flex;"><span>  bottle.run(
</span></span><span style="display:flex;"><span>    app = app,
</span></span><span style="display:flex;"><span>    host = <span style="color:#a31515">&#34;0.0.0.0&#34;</span>,
</span></span><span style="display:flex;"><span>    port = 5000,
</span></span><span style="display:flex;"><span>    debug = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    reloader = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    catchall = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>  )
</span></span></code></pre><p>To run this simply go to folder containing python file and run <code>python webapp.py</code> from terminal. If everything goes ok you should have simple API
available via POST method on /api route.</p>
<p>After testing the service with Restlet Client you should be able to view your
data in a database file <code>data.db</code>.</p>
<figure>
<img src="/assets/posts/iot-application/iot-rest-example.png" alt="REST settings example" />
</figure>
<p>You can also check the contents of new database file by using desktop client
for SQLite → <a href="http://sqlitebrowser.org/">DB Browser for SQLite</a>.</p>
<figure>
<img src="/assets/posts/iot-application/iot-sqlite-db.png" alt="SQLite database example" />
</figure>
<p>Table structure is as simple as it can be. We have ts (timestamp) and value
(value from Arduino). As you can see timestamp is generated on API side. If you
would happen to have atomic clock on Arduino it would be then better to generate
and send timestamp with the value. This would be particularity useful if we
would be collecting sensor data at a higher frequency and then sending this data
in bulk to API.</p>
<p>If you will deploy this app with uWSGI and multi-threaded, use DSN (Data Source
Name) url with <code>?check_same_thread=False</code>.</p>
<p>Ok, now that we have some sort of a working API with some basic security so
unwanted people can not post data to your database can we proceed further and
try to program Arduino to send data to API.</p>
<h2 id="sending-data-to-api-with-arduino-mkr1000">Sending data to API with Arduino MKR1000</h2>
<p>First of all you should have MKR1000 module and microUSB cable to proceed.  If
you have ever done any work with Arduino you should know that you also need
<a href="https://www.arduino.cc/en/Main/Software">Arduino IDE</a>. On provided link you
should be able to download and install IDE. Once that task is completed and you
have successfully run blink example you should proceed to the next step.</p>
<p>In order to use wireless capabilities of MKR1000 you need to first install
<a href="https://www.arduino.cc/en/Reference/WiFi101">WiFi101 library</a> in Arduino IDE.
Please check before you install, you may already have it installed.</p>
<p>Code below is a working example that sends data to API. Before you try to test
your code make sure you have run Python web application. Then change settings
for wifi, api endpoint and api_key. If by some reason code bellow doesn't work
for you please leave a comment and I'll try to help.</p>
<p>Once you have opened IDE and copied this code try to compile and upload it.
Then open &quot;Serial monitor&quot; to see if any output is presented by Arduino.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">#include</span> <span style="color:#00f">&lt;WiFi101.h&gt;</span><span style="color:#00f">
</span></span></span><span style="display:flex;"><span><span style="color:#00f"></span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// wifi settings
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#2b91af">char</span> ssid[] = <span style="color:#a31515">&#34;ssid-name&#34;</span>;
</span></span><span style="display:flex;"><span><span style="color:#2b91af">char</span> pass[] = <span style="color:#a31515">&#34;ssid-password&#34;</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// api server enpoint
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#2b91af">char</span> server[] = <span style="color:#a31515">&#34;192.168.6.22&#34;</span>;
</span></span><span style="display:flex;"><span><span style="color:#2b91af">int</span> port = 5000;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// api key that must be the same as the one in Python code
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>String api_key = <span style="color:#a31515">&#34;JtF2aUE5SGHfVJBCG5SH&#34;</span>;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// frequency data is sent in ms - every 5 seconds
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#2b91af">int</span> timeout = 1000 * 5;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#2b91af">int</span> status = WL_IDLE_STATUS;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#2b91af">void</span> setup() {
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// initialize serial and wait for port to open:
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  Serial.begin(9600);
</span></span><span style="display:flex;"><span>  delay(1000);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// check for the presence of the shield
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  <span style="color:#00f">if</span> (WiFi.status() == WL_NO_SHIELD) {
</span></span><span style="display:flex;"><span>    Serial.println(<span style="color:#a31515">&#34;WiFi shield not present&#34;</span>);
</span></span><span style="display:flex;"><span>    <span style="color:#00f">while</span> (true);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// attempt to connect to wifi network
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  <span style="color:#00f">while</span> (status != WL_CONNECTED) {
</span></span><span style="display:flex;"><span>    Serial.print(<span style="color:#a31515">&#34;Attempting to connect to SSID: &#34;</span>);
</span></span><span style="display:flex;"><span>    Serial.println(ssid);
</span></span><span style="display:flex;"><span>    status = WiFi.begin(ssid, pass);
</span></span><span style="display:flex;"><span>    <span style="color:#008000">// wait 10 seconds for connection
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>    delay(10000);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// output wifi status to serial monitor
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  Serial.print(<span style="color:#a31515">&#34;SSID: &#34;</span>);
</span></span><span style="display:flex;"><span>  Serial.println(WiFi.SSID());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  IPAddress ip = WiFi.localIP();
</span></span><span style="display:flex;"><span>  Serial.print(<span style="color:#a31515">&#34;IP Address: &#34;</span>);
</span></span><span style="display:flex;"><span>  Serial.println(ip);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#2b91af">long</span> rssi = WiFi.RSSI();
</span></span><span style="display:flex;"><span>  Serial.print(<span style="color:#a31515">&#34;signal strength (RSSI):&#34;</span>);
</span></span><span style="display:flex;"><span>  Serial.print(rssi);
</span></span><span style="display:flex;"><span>  Serial.println(<span style="color:#a31515">&#34; dBm&#34;</span>);
</span></span><span style="display:flex;"><span>}
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#2b91af">void</span> loop() {
</span></span><span style="display:flex;"><span>  WiFiClient client;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">if</span> (client.connect(server, port)) {
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000">// I use random number generator for this example
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>    <span style="color:#008000">// but you can use analog or digital inputs from arduino
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>    String content = String(random(1000));
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    client.println(<span style="color:#a31515">&#34;POST /api HTTP/1.1&#34;</span>);
</span></span><span style="display:flex;"><span>    client.println(<span style="color:#a31515">&#34;Connection: close&#34;</span>);
</span></span><span style="display:flex;"><span>    client.println(<span style="color:#a31515">&#34;Api-Key: &#34;</span> + api_key);
</span></span><span style="display:flex;"><span>    client.println(<span style="color:#a31515">&#34;Content-Length: &#34;</span> + String(content.length()));
</span></span><span style="display:flex;"><span>    client.println();
</span></span><span style="display:flex;"><span>    client.println(content);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    delay(100);
</span></span><span style="display:flex;"><span>    client.stop();
</span></span><span style="display:flex;"><span>    Serial.println(<span style="color:#a31515">&#34;Data sent successfully ...&#34;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  } <span style="color:#00f">else</span> {
</span></span><span style="display:flex;"><span>    Serial.println(<span style="color:#a31515">&#34;Problem sending data ...&#34;</span>);
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// waits for x seconds and continue looping
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  delay(timeout);
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><p>As seen from example you can notice that Arduino is generating random integer
between [ 0 .. 1000 ]. You can easily replace this with a temperature sensor or
any other kind of sensor.</p>
<p>Now that we have API under the hood and Arduino is sending demo data we can now
focus on data visualization.</p>
<h2 id="data-visualization">Data visualization</h2>
<p>Before we continue we should examine our project folder structure. Currently we
only have two files in our project:</p>
<p><em>simple-iot-app/</em></p>
<ul>
<li><em>webapp.py</em></li>
<li><em>data.db</em></li>
</ul>
<p>We will now add HTML template that will contain CSS and JavaScript code inline
for the simplicity reason. And for the bottle framework to be able to scan root
application folder for templates we will add <code>bottle.TEMPLATE_PATH.insert(0, &quot;./&quot;)</code> in <code>webapp.py</code>. By default bottle framework uses <code>views/</code>
subfolder to store templates. This is not the ideal situation and if you will
use bottle to develop web applications you should use native behavior and store
templates in it's predefined folder. But for the sake of example we will
over-ride this. Be careful to fully replace your code with new code that is
provided below. Avoid partially replacing code in file :) Also new code for
reading data-points is provided in Python example below.</p>
<p>First we add new route to our web application. It should be trigger when browser
hits root of application <code>http://0.0.0.0:5000/</code>. This route will do nothing
more than render <code>frontend.html</code> template. This is done by <code>return bottle.template(&quot;frontend.html&quot;)</code>. Check code below to further examine how
exactly this is done.</p>
<p>Now we will expand <code>/api</code> route and use different methods to write or read
data-points. For writing data-point we will use POST method and for reading
points we will use GET method. GET method will return JSON object with latest
readings and historical data.</p>
<p>There is a fantastic JavaScript library for plotting time-series charts called
<a href="https://www.metricsgraphicsjs.org">MetricsGraphics.js</a> that is based on
<a href="https://d3js.org/">D3.js</a> library for visualizing data.</p>
<p>Data schema required by MetricsGraphics.js → to achieve this we need to
transform data from database into this format:</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>[
</span></span><span style="display:flex;"><span>  {
</span></span><span style="display:flex;"><span>    &#34;date&#34;: <span style="color:#a31515">&#34;2017-08-11 01:07:20&#34;</span>,
</span></span><span style="display:flex;"><span>    &#34;value&#34;: 933
</span></span><span style="display:flex;"><span>  },
</span></span><span style="display:flex;"><span>  {
</span></span><span style="display:flex;"><span>    &#34;date&#34;: <span style="color:#a31515">&#34;2017-08-11 01:07:30&#34;</span>,
</span></span><span style="display:flex;"><span>    &#34;value&#34;: 743
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>]
</span></span></code></pre><p>Web application is now complete and we only need <code>frontend.html</code> that we
will develop now. If you would try to start web app now and go to root app this
will return error because we don't have frontend.html yet.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># -*- coding: utf-8 -*-</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> time
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> bottle
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> json
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> datetime
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> random
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> dataset
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># initializing bottle app</span>
</span></span><span style="display:flex;"><span>app = bottle.Bottle()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># adds root directory as template folder</span>
</span></span><span style="display:flex;"><span>bottle.TEMPLATE_PATH.insert(0, <span style="color:#a31515">&#34;./&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># connects to sqlite database</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># check_same_thread=False allows using it in multi-threaded mode</span>
</span></span><span style="display:flex;"><span>app.config[<span style="color:#a31515">&#34;db&#34;</span>] = dataset.connect(<span style="color:#a31515">&#34;sqlite:///data.db?check_same_thread=False&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># api key that will be used in Arduino code</span>
</span></span><span style="display:flex;"><span>app.config[<span style="color:#a31515">&#34;api_key&#34;</span>] = <span style="color:#a31515">&#34;JtF2aUE5SGHfVJBCG5SH&#34;</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># triggered when / is accessed from browser</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># only accepts GET → no POST allowed</span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/&#34;</span>, method=[<span style="color:#a31515">&#34;GET&#34;</span>])
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> route_default():
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> bottle.template(<span style="color:#a31515">&#34;frontend.html&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># triggered when /api is accessed from browser</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># accepts POST and GET</span>
</span></span><span style="display:flex;"><span>@app.route(<span style="color:#a31515">&#34;/api&#34;</span>, method=[<span style="color:#a31515">&#34;GET&#34;</span>, <span style="color:#a31515">&#34;POST&#34;</span>])
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> route_default():
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># if method is POST then we write datapoint</span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">if</span> bottle.request.method == <span style="color:#a31515">&#34;POST&#34;</span>:
</span></span><span style="display:flex;"><span>    status = 400
</span></span><span style="display:flex;"><span>    ts = int(time.time()) <span style="color:#008000"># current timestamp</span>
</span></span><span style="display:flex;"><span>    value = bottle.request.body.read() <span style="color:#008000"># data from device</span>
</span></span><span style="display:flex;"><span>    api_key = bottle.request.get_header(<span style="color:#a31515">&#34;Api-Key&#34;</span>) <span style="color:#008000"># api key from header</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000"># outputs to console recieved data for debug reason</span>
</span></span><span style="display:flex;"><span>    print <span style="color:#a31515">&#34;&gt;&gt;&gt; </span><span style="color:#a31515">{}</span><span style="color:#a31515"> :: </span><span style="color:#a31515">{}</span><span style="color:#a31515">&#34;</span>.format(value, api_key)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000"># if api_key is correct and value is present</span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000"># then writes attribute to point table</span>
</span></span><span style="display:flex;"><span>    <span style="color:#00f">if</span> api_key == app.config[<span style="color:#a31515">&#34;api_key&#34;</span>] <span style="color:#00f">and</span> value:
</span></span><span style="display:flex;"><span>      app.config[<span style="color:#a31515">&#34;db&#34;</span>][<span style="color:#a31515">&#34;point&#34;</span>].insert(dict(ts=ts, value=value))
</span></span><span style="display:flex;"><span>      status = 200
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      <span style="color:#008000"># we only need to return status</span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">return</span> bottle.HTTPResponse(status=status, body=<span style="color:#a31515">&#34;&#34;</span>)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000"># if method is GET then we read datapoint</span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">else</span>:
</span></span><span style="display:flex;"><span>    response = []
</span></span><span style="display:flex;"><span>    datapoints = app.config[<span style="color:#a31515">&#34;db&#34;</span>][<span style="color:#a31515">&#34;point&#34;</span>].all()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#00f">for</span> point <span style="color:#00f">in</span> datapoints:
</span></span><span style="display:flex;"><span>      response.append({
</span></span><span style="display:flex;"><span>        <span style="color:#a31515">&#34;date&#34;</span>: datetime.datetime.fromtimestamp(int(point[<span style="color:#a31515">&#34;ts&#34;</span>])).strftime(<span style="color:#a31515">&#34;%Y-%m-</span><span style="color:#a31515">%d</span><span style="color:#a31515"> %H:%M:%S&#34;</span>),
</span></span><span style="display:flex;"><span>        <span style="color:#a31515">&#34;value&#34;</span>: point[<span style="color:#a31515">&#34;value&#34;</span>]
</span></span><span style="display:flex;"><span>      })
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    bottle.response.content_type = <span style="color:#a31515">&#34;application/json&#34;</span>
</span></span><span style="display:flex;"><span>    <span style="color:#00f">return</span> json.dumps(response)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># starting server on http://0.0.0.0:5000</span>
</span></span><span style="display:flex;"><span><span style="color:#00f">if</span> __name__ == <span style="color:#a31515">&#34;__main__&#34;</span>:
</span></span><span style="display:flex;"><span>  bottle.run(
</span></span><span style="display:flex;"><span>    app = app,
</span></span><span style="display:flex;"><span>    host = <span style="color:#a31515">&#34;0.0.0.0&#34;</span>,
</span></span><span style="display:flex;"><span>    port = 5000,
</span></span><span style="display:flex;"><span>    debug = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    reloader = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>    catchall = <span style="color:#00f">True</span>,
</span></span><span style="display:flex;"><span>  )
</span></span></code></pre><p>And now finally we can implement <code>frontend.html</code>. Create file with this name
and copy code below. When you are done you can start web application.  Steps for
this part are listed below the code.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">&lt;!DOCTYPE html&gt;</span>
</span></span><span style="display:flex;"><span>&lt;html&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;head&gt;
</span></span><span style="display:flex;"><span>    &lt;meta charset=<span style="color:#a31515">&#34;utf-8&#34;</span>&gt;
</span></span><span style="display:flex;"><span>    &lt;title&gt;Simple IOT application&lt;/title&gt;
</span></span><span style="display:flex;"><span>  &lt;/head&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;h1&gt;Simple IOT application&lt;/h1&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;div class=<span style="color:#a31515">&#34;chart-placeholder&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;div id=<span style="color:#a31515">&#34;chart&#34;</span>&gt;&lt;/div&gt;
</span></span><span style="display:flex;"><span>    &lt;/div&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000">&lt;!-- application main script --&gt;</span>
</span></span><span style="display:flex;"><span>    &lt;script src=<span style="color:#a31515">&#34;https://cdnjs.cloudflare.com/ajax/libs/jquery/3.2.1/jquery.min.js&#34;</span>&gt;&lt;/script&gt;
</span></span><span style="display:flex;"><span>    &lt;script src=<span style="color:#a31515">&#34;https://cdnjs.cloudflare.com/ajax/libs/d3/4.10.0/d3.min.js&#34;</span>&gt;&lt;/script&gt;
</span></span><span style="display:flex;"><span>    &lt;script src=<span style="color:#a31515">&#34;https://cdnjs.cloudflare.com/ajax/libs/metrics-graphics/2.11.0/metricsgraphics.min.js&#34;</span>&gt;&lt;/script&gt;
</span></span><span style="display:flex;"><span>    &lt;script&gt;
</span></span><span style="display:flex;"><span>      <span style="color:#00f">function</span> fetch_and_render() {
</span></span><span style="display:flex;"><span>        d3.json(<span style="color:#a31515">&#34;/api&#34;</span>, <span style="color:#00f">function</span>(data) {
</span></span><span style="display:flex;"><span>          data = MG.convert.date(data, <span style="color:#a31515">&#34;date&#34;</span>, <span style="color:#a31515">&#34;%Y-%m-%d %H:%M:%S&#34;</span>);
</span></span><span style="display:flex;"><span>          MG.data_graphic({
</span></span><span style="display:flex;"><span>            data: data,
</span></span><span style="display:flex;"><span>            chart_type: <span style="color:#a31515">&#34;line&#34;</span>,
</span></span><span style="display:flex;"><span>            full_width: <span style="color:#00f">true</span>,
</span></span><span style="display:flex;"><span>            height: 270,
</span></span><span style="display:flex;"><span>            target: document.getElementById(<span style="color:#a31515">&#34;chart&#34;</span>),
</span></span><span style="display:flex;"><span>            x_accessor: <span style="color:#a31515">&#34;date&#34;</span>,
</span></span><span style="display:flex;"><span>            y_accessor: <span style="color:#a31515">&#34;value&#34;</span>
</span></span><span style="display:flex;"><span>          });
</span></span><span style="display:flex;"><span>        });
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>      window.onload = <span style="color:#00f">function</span>() {
</span></span><span style="display:flex;"><span>        <span style="color:#008000">// initial call for rendering
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>        fetch_and_render();
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#008000">// updates chart every 5 seconds
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>        setInterval(<span style="color:#00f">function</span>() {
</span></span><span style="display:flex;"><span>          fetch_and_render();
</span></span><span style="display:flex;"><span>        }, 5000);
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>    &lt;/script&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#008000">&lt;!-- application styles --&gt;</span>
</span></span><span style="display:flex;"><span>    &lt;style&gt;
</span></span><span style="display:flex;"><span>      body {
</span></span><span style="display:flex;"><span>        <span style="color:#00f">font</span>: 13<span style="color:#2b91af">px</span> <span style="color:#00f">sans-serif</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#00f">padding</span>: 20<span style="color:#2b91af">px</span> 50<span style="color:#2b91af">px</span>;
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>      .<span style="color:#2b91af">chart-placeholder</span> {
</span></span><span style="display:flex;"><span>        <span style="color:#00f">border</span>: 2<span style="color:#2b91af">px</span> <span style="color:#00f">solid</span> #ccc;
</span></span><span style="display:flex;"><span>        <span style="color:#00f">width</span>: 100<span style="color:#2b91af">%</span>;
</span></span><span style="display:flex;"><span>        <span style="color:#00f">user-select</span>: <span style="color:#00f">none</span>;
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>      <span style="color:#008000">/* chart styles */</span>
</span></span><span style="display:flex;"><span>      .<span style="color:#2b91af">mg-line1-color</span> {
</span></span><span style="display:flex;"><span>        stroke: <span style="color:#00f">red</span>;
</span></span><span style="display:flex;"><span>        stroke-width: 2;
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>      .<span style="color:#2b91af">mg-main-area</span>, .<span style="color:#2b91af">mg-main-line</span> {
</span></span><span style="display:flex;"><span>        fill: #fff;
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>      .<span style="color:#2b91af">mg-x-axis</span> line, .<span style="color:#2b91af">mg-y-axis</span> line {
</span></span><span style="display:flex;"><span>        stroke: #b3b2b2;
</span></span><span style="display:flex;"><span>        stroke-width: 1<span style="color:#2b91af">px</span>;
</span></span><span style="display:flex;"><span>      }
</span></span><span style="display:flex;"><span>    &lt;/style&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;/body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>&lt;/html&gt;
</span></span></code></pre><p>Now the folder structure should look like:</p>
<p><em>simple-iot-app/</em></p>
<ul>
<li><em>webapp.py</em></li>
<li><em>data.db</em></li>
<li><em>frontend.html</em></li>
</ul>
<p>Ok, lets now start application and start feeding it data.</p>
<ol>
<li><code>python webapp.py</code></li>
<li>connect Arduino MKR1000 to power source</li>
<li>open browser and go to <code>http://0.0.0.0:5000</code></li>
</ol>
<p>If everything goes well you should be seeing new data-points rendered on chart
every 5 seconds.</p>
<p>If you navigate to <code>http://0.0.0.0:5000</code> you should see rendered chart as
shown on picture below.</p>
<figure>
<img src="/assets/posts/iot-application/iot-app-output.png" alt="Application output" />
</figure>
<p>Complete application with all the code is available for
<a href="/assets/posts/iot-application/simple-iot-application.zip">download</a>.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I hope this clarifies some aspects of IOT application development. Of course
this is a minimal example and is far from what can be done in real life with
some further dive into other technologies.</p>
<p>If you would like to continue exploring IOT world here are some interesting
resources for you to examine:</p>
<ul>
<li><a href="https://www.allaboutcircuits.com/projects/reading-sensors-with-an-arduino/">Reading Sensors with an Arduino</a></li>
<li><a href="http://www.hivemq.com/blog/how-to-get-started-with-mqtt">MQTT 101 – How to Get Started with the lightweight IoT Protocol</a></li>
<li><a href="https://www.html5rocks.com/en/tutorials/eventsource/basics/">Stream Updates with Server-Sent Events</a></li>
<li><a href="http://www.tutorialspoint.com/internet_of_things/">Internet of Things (IoT) Tutorials</a></li>
</ul>
<p>Any comment or additional ideas are welcomed in comments below.</p>


guid: https://mitjafelicijan.com/using-digitalocean-spaces-object-storage-with-fuse.html
title: Using DigitalOcean Spaces Object Storage with FUSE
url: https://mitjafelicijan.com/using-digitalocean-spaces-object-storage-with-fuse.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>Couple of months ago <a href="https://www.digitalocean.com">DigitalOcean</a> introduced new
product called
<a href="https://blog.digitalocean.com/introducing-spaces-object-storage/">Spaces</a> which
is Object Storage very similar to Amazon's S3. This really peaked my interest,
because this was something I was missing and even the thought of going over the
internet for such functionality was in no interest to me. Also in fashion with
their previous pricing this also is very cheap and pricing page is a no-brainer
compared to AWS or GCE. <a href="https://www.digitalocean.com/pricing/">Prices are clearly and precisely defined and
outlined</a>.  You must love them for that
:)</p>
<h2 id="initial-requirements">Initial requirements</h2>
<ul>
<li>Is it possible to use them as a mounted drive with FUSE? (tl;dr YES)</li>
<li>Will the performance degrade over time and over different sizes of objects?
(tl;dr NO&amp;YES)</li>
<li>Can storage be mounted on multiple machines at the same time and be writable?
(tl;dr YES)</li>
</ul>
<blockquote>
<p>Let me be clear. This scripts I use are made just for benchmarking and are not
intended to be used in real-life situations. Besides that, I am looking into
using this approaches but adding caching service in front of it and then
dumping everything as an object to storage. This could potentially be some
interesting post of itself. But in case you would need real-time data without
eventual consistency please take this scripts as they are: not usable in such
situations.</p>
</blockquote>
<h2 id="is-it-possible-to-use-them-as-a-mounted-drive-with-fuse">Is it possible to use them as a mounted drive with FUSE?</h2>
<p>Well, actually they can be used in such manor. Because they are similar to <a href="https://aws.amazon.com/s3/">AWS
S3</a> many tools are available and you can find many
articles and <a href="https://stackoverflow.com/search?q=s3+fuse">Stackoverflow items</a>.</p>
<p>To make this work you will need DigitalOcean account. If you don't have one you
will not be able to test this code. But if you have an account then you go and
<a href="https://cloud.digitalocean.com/droplets/new?size=s-1vcpu-1gb&amp;region=ams3&amp;distro=debian&amp;distroImage=debian-9-x64&amp;options=private_networking,install_agent">create new
Droplet</a>.
If you click on this link you will already have preselected Debian 9 with
smallest VM option.</p>
<ul>
<li>Please be sure to add you SSH key, because we will login to this machine
remotely.</li>
<li>If you change your region please remember which one you choose because we will
need this information when we try to mount space to our machine.</li>
</ul>
<p>Instuctions on how to use SSH keys and how to setup them are available in
article <a href="https://www.digitalocean.com/community/tutorials/how-to-use-ssh-keys-with-digitalocean-droplets">How To Use SSH Keys with DigitalOcean
Droplets</a>.</p>
<figure>
<img src="/assets/posts/do-fuse/fuse-droplets.png" alt="DigitalOcean Droplets" />
</figure>
<p>After we created Droplet it's time to create new Space. This is done by clicking
on a button <a href="https://cloud.digitalocean.com/spaces/new">Create</a> (right top
corner) and selecting Spaces. Choose pronounceable <code>Unique name</code> because we
will use it in examples below. You can either choose Private or Public, it
doesn't matter in our case. And you can always change that in the future.</p>
<p>When you have created new Space we should <a href="https://cloud.digitalocean.com/settings/api/tokens">generate Access
key</a>.  This link will guide
to the page when you can generate this key. After you create new one, please
save provided Key and Secret because Secret will not be shown again.</p>
<figure>
<img src="/assets/posts/do-fuse/fuse-spaces.png" alt="DigitalOcean Spaces" />
</figure>
<p>Now that we have new Space and Access key we should SSH into our machine.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># replace IP with the ip of your newly created droplet</span>
</span></span><span style="display:flex;"><span>ssh root@IP
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># this will install utilities for mounting storage objects as FUSE</span>
</span></span><span style="display:flex;"><span>apt install s3fs
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we now need to provide credentials (access key we created earlier)</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># replace KEY and SECRET with your own credentials but leave the colon between them</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># we also need to set proper permissions</span>
</span></span><span style="display:flex;"><span>echo <span style="color:#a31515">&#34;KEY:SECRET&#34;</span> &gt; .passwd-s3fs
</span></span><span style="display:flex;"><span>chmod 600 .passwd-s3fs
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we mount space to our machine</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># replace UNIQUE-NAME with the name you choose earlier</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># if you choose different region for your space be careful about -ourl option (ams3)</span>
</span></span><span style="display:flex;"><span>s3fs UNIQUE-NAME /mnt/ -ourl=https://ams3.digitaloceanspaces.com -ouse_cache=/tmp
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we try to create a file</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># once you mount it may take a couple of seconds to retrieve data</span>
</span></span><span style="display:flex;"><span>echo <span style="color:#a31515">&#34;Hello cruel world&#34;</span> &gt; /mnt/hello.txt
</span></span></code></pre><p>After all this you can return to your browser and go to <a href="https://cloud.digitalocean.com/spaces">DigitalOcean
Spaces</a> and click on your created
space. If file hello.txt is present you have successfully mounted space to your
machine and wrote data to it.</p>
<p>I choose the same region for my Droplet and my Space but you don't have to.  You
can have different regions. What this actually does to performance I don't know.</p>
<p>Additional information on FUSE:</p>
<ul>
<li><a href="https://github.com/s3fs-fuse/s3fs-fuse">Github project page for s3fs</a></li>
<li><a href="https://en.wikipedia.org/wiki/Filesystem_in_Userspace">FUSE - Filesystem in Userspace</a></li>
</ul>
<h2 id="will-the-performance-degrade-over-time-and-over-different-sizes-of-objects">Will the performance degrade over time and over different sizes of objects?</h2>
<p>For this task I didn't want to just read and write text files or uploading
images. I actually wanted to figure out if using something like SQlite is viable
in this case.</p>
<h3 id="measurement-experiment-1-file-copy">Measurement experiment 1: File copy</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000"># first we create some dummy files at different sizes</span>
</span></span><span style="display:flex;"><span>dd <span style="color:#00f">if</span>=/dev/zero of=10KB.dat bs=1024 count=10 <span style="color:#008000">#10KB</span>
</span></span><span style="display:flex;"><span>dd <span style="color:#00f">if</span>=/dev/zero of=100KB.dat bs=1024 count=100 <span style="color:#008000">#100KB</span>
</span></span><span style="display:flex;"><span>dd <span style="color:#00f">if</span>=/dev/zero of=1MB.dat bs=1024 count=1024 <span style="color:#008000">#1MB</span>
</span></span><span style="display:flex;"><span>dd <span style="color:#00f">if</span>=/dev/zero of=10MB.dat bs=1024 count=10240 <span style="color:#008000">#10MB</span>
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now we set time command to only return real</span>
</span></span><span style="display:flex;"><span>TIMEFORMAT=%R
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># now lets test it</span>
</span></span><span style="display:flex;"><span>(time cp 10KB.dat /mnt/) |&amp; tee -a 10KB.results.txt
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># and now we automate</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># this will perform the same operation 100 times</span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># this will output results into separated files based on objecty size</span>
</span></span><span style="display:flex;"><span>n=0; <span style="color:#00f">while</span> (( n++ &lt; 100 )); <span style="color:#00f">do</span> (time cp 10KB.dat /mnt/10KB.$n.dat) |&amp; tee -a 10KB.results.txt; <span style="color:#00f">done</span>
</span></span><span style="display:flex;"><span>n=0; <span style="color:#00f">while</span> (( n++ &lt; 100 )); <span style="color:#00f">do</span> (time cp 100KB.dat /mnt/100KB.$n.dat) |&amp; tee -a 100KB.results.txt; <span style="color:#00f">done</span>
</span></span><span style="display:flex;"><span>n=0; <span style="color:#00f">while</span> (( n++ &lt; 100 )); <span style="color:#00f">do</span> (time cp 1MB.dat /mnt/1MB.$n.dat) |&amp; tee -a 1MB.results.txt; <span style="color:#00f">done</span>
</span></span><span style="display:flex;"><span>n=0; <span style="color:#00f">while</span> (( n++ &lt; 100 )); <span style="color:#00f">do</span> (time cp 10MB.dat /mnt/10MB.$n.dat) |&amp; tee -a 10MB.results.txt; <span style="color:#00f">done</span>
</span></span></code></pre><p>Files of size 100MB were not successfully transferred and ended up displaying
error (cp: failed to close '/mnt/100MB.1.dat': Operation not permitted).</p>
<p>As I suspected, object size is not really that important. Sadly I don't have the
time to test performance over periods of time. But if some of you would do it
please send me your data. I would be interested in seeing results.</p>
<p><strong>Here are plotted results</strong></p>
<p>You can download <a href="/assets/posts/do-fuse/copy-benchmarks.tsv">raw result here</a>.
Measurements are in seconds.</p>
<script src="//cdn.plot.ly/plotly-latest.min.js"></script>
<div id="copy-benchmarks"></div>
<script>
(function(){
	var request = new XMLHttpRequest();
	request.open("GET", "/assets/posts/do-fuse/copy-benchmarks.tsv", true);
	request.onload = function() {
		if (request.status >= 200 && request.status < 400) {
			var payload = request.responseText.trim();
			var tsv = payload.split("\n");
			for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
			var traces = [];
			var headers = tsv[0];
			tsv.shift();
			Array.prototype.forEach.call(headers, function(el, idx) {
				var x = [];
				var y = [];
				for (var j=0; j<tsv.length; j++) {
					x.push(j);
					y.push(parseFloat(tsv[j][idx].replace(",", ".")));
				}
				traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
			});
			var copy = Plotly.newPlot("copy-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 40, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } }, xaxis: { title: "fn(i)", titlefont: { size: 12 } } });
		} else { }
	};
	request.onerror = function() { };
	request.send(null);
})();
</script>
<p>As far as these tests show, performance is quite stable and can be predicted
which is fantastic. But this is a small test and spans only over couple of
hours. So you should not completely trust them.</p>
<h3 id="measurement-experiment-2-sqlite-performanse">Measurement experiment 2: SQLite performanse</h3>
<p>I was unable to use database file directly from mounted drive so this is a no-go
as I suspected. So I executed code below on a local disk just to get some
benchmarks. I inserted 1000 records with DROPTABLE, CREATETABLE, INSERTMANY,
FETCHALL, COMMIT for 1000 times to generate statistics. As you can see
performance of SQLite is quite amazing. You could then potentially just copy
file to mounted drive and be done with it.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">import</span> time
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> sqlite3
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> sys
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">if</span> len(sys.argv) &lt; 3:
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;usage: python sqlite-benchmark.py DB_PATH NUM_RECORDS REPEAT&#34;</span>)
</span></span><span style="display:flex;"><span>  exit()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">def</span> data_iter(x):
</span></span><span style="display:flex;"><span>  <span style="color:#00f">for</span> i <span style="color:#00f">in</span> range(x):
</span></span><span style="display:flex;"><span>    <span style="color:#00f">yield</span> <span style="color:#a31515">&#34;m&#34;</span> + str(i), <span style="color:#a31515">&#34;f&#34;</span> + str(i*i)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>header_line = <span style="color:#a31515">&#34;</span><span style="color:#a31515">%s</span><span style="color:#a31515">\t</span><span style="color:#a31515">%s</span><span style="color:#a31515">\t</span><span style="color:#a31515">%s</span><span style="color:#a31515">\t</span><span style="color:#a31515">%s</span><span style="color:#a31515">\t</span><span style="color:#a31515">%s</span><span style="color:#a31515">\n</span><span style="color:#a31515">&#34;</span> % (<span style="color:#a31515">&#34;DROPTABLE&#34;</span>, <span style="color:#a31515">&#34;CREATETABLE&#34;</span>, <span style="color:#a31515">&#34;INSERTMANY&#34;</span>, <span style="color:#a31515">&#34;FETCHALL&#34;</span>, <span style="color:#a31515">&#34;COMMIT&#34;</span>)
</span></span><span style="display:flex;"><span><span style="color:#00f">with</span> open(<span style="color:#a31515">&#34;sqlite-benchmarks.tsv&#34;</span>, <span style="color:#a31515">&#34;w&#34;</span>) <span style="color:#00f">as</span> fp:
</span></span><span style="display:flex;"><span>  fp.write(header_line)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>start_time = time.time()
</span></span><span style="display:flex;"><span>conn = sqlite3.connect(sys.argv[1])
</span></span><span style="display:flex;"><span>c = conn.cursor()
</span></span><span style="display:flex;"><span>end_time = time.time()
</span></span><span style="display:flex;"><span>result_time = CONNECT = end_time - start_time
</span></span><span style="display:flex;"><span>print(<span style="color:#a31515">&#34;CONNECT: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>start_time = time.time()
</span></span><span style="display:flex;"><span>c.execute(<span style="color:#a31515">&#34;PRAGMA journal_mode=WAL&#34;</span>)
</span></span><span style="display:flex;"><span>c.execute(<span style="color:#a31515">&#34;PRAGMA temp_store=MEMORY&#34;</span>)
</span></span><span style="display:flex;"><span>c.execute(<span style="color:#a31515">&#34;PRAGMA synchronous=OFF&#34;</span>)
</span></span><span style="display:flex;"><span>result_time = PRAGMA = end_time - start_time
</span></span><span style="display:flex;"><span>print(<span style="color:#a31515">&#34;PRAGMA: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">for</span> i <span style="color:#00f">in</span> range(int(sys.argv[3])):
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;#</span><span style="color:#a31515">%i</span><span style="color:#a31515">&#34;</span> % (i))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  start_time = time.time()
</span></span><span style="display:flex;"><span>  c.execute(<span style="color:#a31515">&#34;drop table if exists test&#34;</span>)
</span></span><span style="display:flex;"><span>  end_time = time.time()
</span></span><span style="display:flex;"><span>  result_time = DROPTABLE = end_time - start_time
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;DROPTABLE: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  start_time = time.time()
</span></span><span style="display:flex;"><span>  c.execute(<span style="color:#a31515">&#34;create table if not exists test(a,b)&#34;</span>)
</span></span><span style="display:flex;"><span>  end_time = time.time()
</span></span><span style="display:flex;"><span>  result_time = CREATETABLE = end_time - start_time
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;CREATETABLE: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  start_time = time.time()
</span></span><span style="display:flex;"><span>  c.executemany(<span style="color:#a31515">&#34;INSERT INTO test VALUES (?, ?)&#34;</span>, data_iter(int(sys.argv[2])))
</span></span><span style="display:flex;"><span>  end_time = time.time()
</span></span><span style="display:flex;"><span>  result_time = INSERTMANY = end_time - start_time
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;INSERTMANY: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  start_time = time.time()
</span></span><span style="display:flex;"><span>  c.execute(<span style="color:#a31515">&#34;select count(*) from test&#34;</span>)
</span></span><span style="display:flex;"><span>  res = c.fetchall()
</span></span><span style="display:flex;"><span>  end_time = time.time()
</span></span><span style="display:flex;"><span>  result_time = FETCHALL = end_time - start_time
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;FETCHALL: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  start_time = time.time()
</span></span><span style="display:flex;"><span>  conn.commit()
</span></span><span style="display:flex;"><span>  end_time = time.time()
</span></span><span style="display:flex;"><span>  result_time = COMMIT = end_time - start_time
</span></span><span style="display:flex;"><span>  print(<span style="color:#a31515">&#34;COMMIT: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  print
</span></span><span style="display:flex;"><span>  log_line = <span style="color:#a31515">&#34;</span><span style="color:#a31515">%f</span><span style="color:#a31515">\t</span><span style="color:#a31515">%f</span><span style="color:#a31515">\t</span><span style="color:#a31515">%f</span><span style="color:#a31515">\t</span><span style="color:#a31515">%f</span><span style="color:#a31515">\t</span><span style="color:#a31515">%f</span><span style="color:#a31515">\n</span><span style="color:#a31515">&#34;</span> % (DROPTABLE, CREATETABLE, INSERTMANY, FETCHALL, COMMIT)
</span></span><span style="display:flex;"><span>  <span style="color:#00f">with</span> open(<span style="color:#a31515">&#34;sqlite-benchmarks.tsv&#34;</span>, <span style="color:#a31515">&#34;a&#34;</span>) <span style="color:#00f">as</span> fp:
</span></span><span style="display:flex;"><span>    fp.write(log_line)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>start_time = time.time()
</span></span><span style="display:flex;"><span>conn.close()
</span></span><span style="display:flex;"><span>end_time = time.time()
</span></span><span style="display:flex;"><span>result_time = CLOSE = end_time - start_time
</span></span><span style="display:flex;"><span>print(<span style="color:#a31515">&#34;CLOSE: </span><span style="color:#a31515">%g</span><span style="color:#a31515"> seconds&#34;</span> % (result_time))
</span></span></code></pre><p>You can download <a href="/assets/posts/do-fuse/sqlite-benchmarks.tsv">raw result here</a>. And
again, these results are done on a local block storage and do not represent
capabilities of object storage. With my current approach and state of the test
code these can not be done. I would need to make Python code much more robust
and check locking etc.</p>
<div id="sqlite-benchmarks"></div>
<script>
(function(){
	var request = new XMLHttpRequest();
	request.open("GET", "/assets/posts/do-fuse/sqlite-benchmarks.tsv", true);
	request.onload = function() {
		if (request.status >= 200 && request.status < 400) {
			var payload = request.responseText.trim();
			var tsv = payload.split("\n");
			for (var i=0; i<tsv.length; i++) { tsv[i] = tsv[i].split("\t"); }
			var traces = [];
			var headers = tsv[0];
			tsv.shift();
			Array.prototype.forEach.call(headers, function(el, idx) {
				var x = [];
				var y = [];
				for (var j=0; j<tsv.length; j++) {
					x.push(j);
					y.push(parseFloat(tsv[j][idx].replace(",", ".")));
				}
				traces.push({ x: x, y: y, type: "scatter", name: el, line: { width: 1, shape: "spline" } });
			});
			var sqlite = Plotly.newPlot("sqlite-benchmarks", traces, { legend: {"orientation": "h"}, height: 400, margin: { l: 50, r: 0, b: 20, t: 30, pad: 0 }, yaxis: { title: "execution time in seconds", titlefont: { size: 12 } } });
		} else { }
	};
	request.onerror = function() { };
	request.send(null);
})();
</script>
<h2 id="can-storage-be-mounted-on-multiple-machines-at-the-same-time-and-be-writable">Can storage be mounted on multiple machines at the same time and be writable?</h2>
<p>Well, this one didn't take long to test. And the answer is <strong>YES</strong>. I mounted
space on both machines and measured same performance on both machines. But
because file is downloaded before write and then uploaded on complete there
could potentially be problems is another process is trying to access the same
file.</p>
<h2 id="observations-and-conslusion">Observations and conslusion</h2>
<p>Using Spaces in this way makes it easier to access and manage files. But besides
that you would need to write additional code to make this one play nice with you
applications.</p>
<p>Nevertheless, this was extremely simple to setup and use and this is just
another excellent product in DigitalOcean product line. I found this exercise
very valuable and am thinking about implementing some sort of mechanism for
SQLite, so data can be stored on Spaces and accessed by many VM's. For a project
where data doesn't need to be accessible in real-time and can have couple of
minutes old data this would be very interesting. If any of you find this
proposal interesting please write in a comment box below or shoot me an email
and I will keep you posted.</p>


guid: https://mitjafelicijan.com/encoding-binary-data-into-dna-sequence.html
title: Encoding binary data into DNA sequence
url: https://mitjafelicijan.com/encoding-binary-data-into-dna-sequence.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<h2 id="initial-thoughts">Initial thoughts</h2>
<p>Imagine a world where you could go outside and take a leaf from a tree and put
it through your personal DNA sequencer and get data like music, videos or
computer programs from it. Well, this is all possible now. It was not done on a
large scale because it is quite expensive to create DNA strands but it's
possible.</p>
<p>Encoding data into DNA sequence is relatively simple process once you understand
the relationship between binary data and nucleotides and scientists have been
making large leaps in this field in order to provide viable long-term storage
solution for our data that would potentially survive our specie if case of
global disaster. We could imprint all the world's knowledge into plants and
ensure the survival of our knowledge.</p>
<p>More optimistic usage for this technology would be easier storage of ever
growing data we produce every day. Once machines for sequencing DNA become fast
enough and cheaper this could mean the next evolution of storing data and
abandoning classical hard and solid state drives in data warehouses.</p>
<p>As we currently stand this is still not viable but it is quite an amazing and
cool technology.</p>
<p>My interests in this field are purely in encoding processes and experimental
testing mainly because I don't have the access to this expensive machines. My
initial goal was to create a toolkit that can be used by everybody to encode
their data into a proper DNA sequence.</p>
<h2 id="glossary">Glossary</h2>
<p><strong>deoxyribose</strong> A five-carbon sugar molecule with a hydrogen atom rather than a
hydroxyl group in the 2′ position; the sugar component of DNA nucleotides.</p>
<p><strong>double helix</strong> The molecular shape of DNA in which two strands of nucleotides
wind around each other in a spiral shape.</p>
<p><strong>nitrogenous base</strong> A nitrogen-containing molecule that acts as a base; often
referring to one of the purine or pyrimidine components of nucleic acids.</p>
<p><strong>phosphate group</strong> A molecular group consisting of a central phosphorus atom
bound to four oxygen atoms.</p>
<p><strong>RGB</strong> The RGB color model is an additive color model in which red, green and
blue light are added together in various ways to reproduce a broad array of
colors.</p>
<p><strong>GCC</strong> The GNU Compiler Collection is a compiler system produced by the GNU
Project supporting various programming languages.</p>
<h2 id="data-encoding">Data encoding</h2>
<p><strong>TL;DR:</strong> Encoding involves the use of a code to change original data into a
form that can be used by an external process.</p>
<p>Encoding is the process of converting data into a format required for a number
of information processing needs, including:</p>
<ul>
<li>Program compiling and execution</li>
<li>Data transmission, storage and compression/decompression</li>
<li>Application data processing, such as file conversion</li>
</ul>
<p>Encoding can have two meanings:</p>
<ul>
<li>In computer technology, encoding is the process of applying a specific code,
such as letters, symbols and numbers, to data for conversion into an
equivalent cipher.</li>
<li>In electronics, encoding refers to analog to digital conversion.</li>
</ul>
<h2 id="quick-history-of-dna">Quick history of DNA</h2>
<ul>
<li><strong>1869</strong> - Friedrich Miescher identifies &quot;nuclein&quot;.</li>
<li><strong>1900s</strong> - The Eugenics Movement.</li>
<li><strong>1900</strong> – Mendel's theories are rediscovered by researchers.</li>
<li><strong>1944</strong> - Oswald Avery identifies DNA as the 'transforming principle'.</li>
<li><strong>1952</strong> - Rosalind Franklin photographs crystallized DNA fibres.</li>
<li><strong>1953</strong> - James Watson and Francis Crick discover the double helix structure of DNA.</li>
<li><strong>1965</strong> - Marshall Nirenberg is the first person to sequence the bases in each codon.</li>
<li><strong>1983</strong> - Huntington's disease is the first mapped genetic disease.</li>
<li><strong>1990</strong> - The Human Genome Project begins.</li>
<li><strong>1995</strong> - Haemophilus Influenzae is the first bacterium genome sequenced.</li>
<li><strong>1996</strong> - Dolly the sheep is cloned.</li>
<li><strong>1999</strong> - First human chromosome is decoded.</li>
<li><strong>2000</strong> – Genetic code of the fruit fly is decoded.</li>
<li><strong>2002</strong> – Mouse is the first mammal to have its genome decoded.</li>
<li><strong>2003</strong> – The Human Genome Project is completed.</li>
<li><strong>2013</strong> – DNA Worldwide and Eurofins Forensic discover identical twins have differences in their genetic makeup.</li>
</ul>
<h2 id="what-is-dna">What is DNA?</h2>
<p>Deoxyribonucleic acid, a self-replicating material which is <strong>present in nearly
all living organisms</strong> as the main constituent of chromosomes. It is the
<strong>carrier of genetic information</strong>.</p>
<blockquote>
<p>The nitrogen in our DNA, the calcium in our teeth, the iron in our blood,
the carbon in our apple pies were made in the interiors of collapsing stars.
We are made of starstuff.
<strong>-- Carl Sagan, Cosmos</strong></p>
</blockquote>
<p>The nucleotide in DNA consists of a sugar (deoxyribose), one of four bases
(cytosine (C), thymine (T), adenine (A), guanine (G)), and a phosphate.
Cytosine and thymine are pyrimidine bases, while adenine and guanine are purine
bases. The sugar and the base together are called a nucleoside.</p>
<figure>
<img src="/assets/posts/dna-sequence/dna-basics.jpg" alt="DNA" />
</figure>
<p><em>DNA (a) forms a double stranded helix, and (b) adenine pairs with thymine and
cytosine pairs with guanine. (credit a: modification of work by Jerome Walker,
Dennis Myts)</em></p>
<h2 id="encode-binary-data-into-dna-sequence">Encode binary data into DNA sequence</h2>
<p>As an input file you can use any file you want:</p>
<ul>
<li>ASCII files,</li>
<li>Compiled programs,</li>
<li>Multimedia files (MP3, MP4, MVK, etc),</li>
<li>Images,</li>
<li>Database files,</li>
<li>etc.</li>
</ul>
<p>Note: If you would copy all the bytes from RAM to file or pipe data to file you
could encode also this data as long as you provide file pointer to the encoder.</p>
<h3 id="basic-encoding">Basic Encoding</h3>
<p>As already mentioned, the Basic Encoding is based on a simple mapping. Since DNA
is composed of 4 nucleotides (Adenine, Cytosine, Guanine, Thymine; usually
referred using the first letter). Using this technique we can encode</p>
<p><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 -907.9672135000189 11313.37788460873 1185.0382429179317" style="width: 26.259ex; height: 2.721ex; vertical-align: -0.68ex; margin: 1px 0px;"><g stroke="black" fill="black" stroke-width="0" transform="matrix(1 0 0 -1 0 0)"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-6C"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-6F" x="303" y="0"/><g transform="translate(793,0)"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-67"/><use transform="scale(0.7071067811865476)" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-32" x="681" y="-213"/></g><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-28" x="1732" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-34" x="2126" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-29" x="2631" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-3D" x="3302" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-6C" x="4363" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-6F" x="4666" y="0"/><g transform="translate(5156,0)"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-67"/><use transform="scale(0.7071067811865476)" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-32" x="681" y="-213"/></g><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-28" x="6095" y="0"/><g transform="translate(6489,0)"><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-32"/><use transform="scale(0.7071067811865476)" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-32" x="714" y="583"/></g><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-29" x="7451" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-3D" x="8123" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMAIN-32" x="9184" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-62" x="9689" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-69" x="10123" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-74" x="10473" y="0"/><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#MJMATHI-73" x="10839" y="0"/></g><defs id="MathJax_SVG_glyphs"><path id="MJSZ2-2211" stroke-width="10" d="M60 948Q63 950 665 950H1267L1325 815Q1384 677 1388 669H1348L1341 683Q1320 724 1285 761Q1235 809 1174 838T1033 881T882 898T699 902H574H543H251L259 891Q722 258 724 252Q725 250 724 246Q721 243 460 -56L196 -356Q196 -357 407 -357Q459 -357 548 -357T676 -358Q812 -358 896 -353T1063 -332T1204 -283T1307 -196Q1328 -170 1348 -124H1388Q1388 -125 1381 -145T1356 -210T1325 -294L1267 -449L666 -450Q64 -450 61 -448Q55 -446 55 -439Q55 -437 57 -433L590 177Q590 178 557 222T452 366T322 544L56 909L55 924Q55 945 60 948Z"/><path id="MJMATHI-69" stroke-width="10" d="M184 600Q184 624 203 642T247 661Q265 661 277 649T290 619Q290 596 270 577T226 557Q211 557 198 567T184 600ZM21 287Q21 295 30 318T54 369T98 420T158 442Q197 442 223 419T250 357Q250 340 236 301T196 196T154 83Q149 61 149 51Q149 26 166 26Q175 26 185 29T208 43T235 78T260 137Q263 149 265 151T282 153Q302 153 302 143Q302 135 293 112T268 61T223 11T161 -11Q129 -11 102 10T74 74Q74 91 79 106T122 220Q160 321 166 341T173 380Q173 404 156 404H154Q124 404 99 371T61 287Q60 286 59 284T58 281T56 279T53 278T49 278T41 278H27Q21 284 21 287Z"/><path id="MJMAIN-3D" stroke-width="10" d="M56 347Q56 360 70 367H707Q722 359 722 347Q722 336 708 328L390 327H72Q56 332 56 347ZM56 153Q56 168 72 173H708Q722 163 722 153Q722 140 707 133H70Q56 140 56 153Z"/><path id="MJMAIN-30" stroke-width="10" d="M96 585Q152 666 249 666Q297 666 345 640T423 548Q460 465 460 320Q460 165 417 83Q397 41 362 16T301 -15T250 -22Q224 -22 198 -16T137 16T82 83Q39 165 39 320Q39 494 96 585ZM321 597Q291 629 250 629Q208 629 178 597Q153 571 145 525T137 333Q137 175 145 125T181 46Q209 16 250 16Q290 16 318 46Q347 76 354 130T362 333Q362 478 354 524T321 597Z"/><path id="MJMATHI-6E" stroke-width="10" d="M21 287Q22 293 24 303T36 341T56 388T89 425T135 442Q171 442 195 424T225 390T231 369Q231 367 232 367L243 378Q304 442 382 442Q436 442 469 415T503 336T465 179T427 52Q427 26 444 26Q450 26 453 27Q482 32 505 65T540 145Q542 153 560 153Q580 153 580 145Q580 144 576 130Q568 101 554 73T508 17T439 -10Q392 -10 371 17T350 73Q350 92 386 193T423 345Q423 404 379 404H374Q288 404 229 303L222 291L189 157Q156 26 151 16Q138 -11 108 -11Q95 -11 87 -5T76 7T74 17Q74 30 112 180T152 343Q153 348 153 366Q153 405 129 405Q91 405 66 305Q60 285 60 284Q58 278 41 278H27Q21 284 21 287Z"/><path id="MJMAIN-28" stroke-width="10" d="M94 250Q94 319 104 381T127 488T164 576T202 643T244 695T277 729T302 750H315H319Q333 750 333 741Q333 738 316 720T275 667T226 581T184 443T167 250T184 58T225 -81T274 -167T316 -220T333 -241Q333 -250 318 -250H315H302L274 -226Q180 -141 137 -14T94 250Z"/><path id="MJMAIN-2B" stroke-width="10" d="M56 237T56 250T70 270H369V420L370 570Q380 583 389 583Q402 583 409 568V270H707Q722 262 722 250T707 230H409V-68Q401 -82 391 -82H389H387Q375 -82 369 -68V230H70Q56 237 56 250Z"/><path id="MJMAIN-31" stroke-width="10" d="M213 578L200 573Q186 568 160 563T102 556H83V602H102Q149 604 189 617T245 641T273 663Q275 666 285 666Q294 666 302 660V361L303 61Q310 54 315 52T339 48T401 46H427V0H416Q395 3 257 3Q121 3 100 0H88V46H114Q136 46 152 46T177 47T193 50T201 52T207 57T213 61V578Z"/><path id="MJMAIN-29" stroke-width="10" d="M60 749L64 750Q69 750 74 750H86L114 726Q208 641 251 514T294 250Q294 182 284 119T261 12T224 -76T186 -143T145 -194T113 -227T90 -246Q87 -249 86 -250H74Q66 -250 63 -250T58 -247T55 -238Q56 -237 66 -225Q221 -64 221 250T66 725Q56 737 55 738Q55 746 60 749Z"/><path id="MJMAIN-32" stroke-width="10" d="M109 429Q82 429 66 447T50 491Q50 562 103 614T235 666Q326 666 387 610T449 465Q449 422 429 383T381 315T301 241Q265 210 201 149L142 93L218 92Q375 92 385 97Q392 99 409 186V189H449V186Q448 183 436 95T421 3V0H50V19V31Q50 38 56 46T86 81Q115 113 136 137Q145 147 170 174T204 211T233 244T261 278T284 308T305 340T320 369T333 401T340 431T343 464Q343 527 309 573T212 619Q179 619 154 602T119 569T109 550Q109 549 114 549Q132 549 151 535T170 489Q170 464 154 447T109 429Z"/><path id="MJMATHI-6C" stroke-width="10" d="M117 59Q117 26 142 26Q179 26 205 131Q211 151 215 152Q217 153 225 153H229Q238 153 241 153T246 151T248 144Q247 138 245 128T234 90T214 43T183 6T137 -11Q101 -11 70 11T38 85Q38 97 39 102L104 360Q167 615 167 623Q167 626 166 628T162 632T157 634T149 635T141 636T132 637T122 637Q112 637 109 637T101 638T95 641T94 647Q94 649 96 661Q101 680 107 682T179 688Q194 689 213 690T243 693T254 694Q266 694 266 686Q266 675 193 386T118 83Q118 81 118 75T117 65V59Z"/><path id="MJMATHI-6F" stroke-width="10" d="M201 -11Q126 -11 80 38T34 156Q34 221 64 279T146 380Q222 441 301 441Q333 441 341 440Q354 437 367 433T402 417T438 387T464 338T476 268Q476 161 390 75T201 -11ZM121 120Q121 70 147 48T206 26Q250 26 289 58T351 142Q360 163 374 216T388 308Q388 352 370 375Q346 405 306 405Q243 405 195 347Q158 303 140 230T121 120Z"/><path id="MJMATHI-67" stroke-width="10" d="M311 43Q296 30 267 15T206 0Q143 0 105 45T66 160Q66 265 143 353T314 442Q361 442 401 394L404 398Q406 401 409 404T418 412T431 419T447 422Q461 422 470 413T480 394Q480 379 423 152T363 -80Q345 -134 286 -169T151 -205Q10 -205 10 -137Q10 -111 28 -91T74 -71Q89 -71 102 -80T116 -111Q116 -121 114 -130T107 -144T99 -154T92 -162L90 -164H91Q101 -167 151 -167Q189 -167 211 -155Q234 -144 254 -122T282 -75Q288 -56 298 -13Q311 35 311 43ZM384 328L380 339Q377 350 375 354T369 368T359 382T346 393T328 402T306 405Q262 405 221 352Q191 313 171 233T151 117Q151 38 213 38Q269 38 323 108L331 118L384 328Z"/><path id="MJMAIN-34" stroke-width="10" d="M462 0Q444 3 333 3Q217 3 199 0H190V46H221Q241 46 248 46T265 48T279 53T286 61Q287 63 287 115V165H28V211L179 442Q332 674 334 675Q336 677 355 677H373L379 671V211H471V165H379V114Q379 73 379 66T385 54Q393 47 442 46H471V0H462ZM293 211V545L74 212L183 211H293Z"/><path id="MJMATHI-62" stroke-width="10" d="M73 647Q73 657 77 670T89 683Q90 683 161 688T234 694Q246 694 246 685T212 542Q204 508 195 472T180 418L176 399Q176 396 182 402Q231 442 283 442Q345 442 383 396T422 280Q422 169 343 79T173 -11Q123 -11 82 27T40 150V159Q40 180 48 217T97 414Q147 611 147 623T109 637Q104 637 101 637H96Q86 637 83 637T76 640T73 647ZM336 325V331Q336 405 275 405Q258 405 240 397T207 376T181 352T163 330L157 322L136 236Q114 150 114 114Q114 66 138 42Q154 26 178 26Q211 26 245 58Q270 81 285 114T318 219Q336 291 336 325Z"/><path id="MJMATHI-74" stroke-width="10" d="M26 385Q19 392 19 395Q19 399 22 411T27 425Q29 430 36 430T87 431H140L159 511Q162 522 166 540T173 566T179 586T187 603T197 615T211 624T229 626Q247 625 254 615T261 596Q261 589 252 549T232 470L222 433Q222 431 272 431H323Q330 424 330 420Q330 398 317 385H210L174 240Q135 80 135 68Q135 26 162 26Q197 26 230 60T283 144Q285 150 288 151T303 153H307Q322 153 322 145Q322 142 319 133Q314 117 301 95T267 48T216 6T155 -11Q125 -11 98 4T59 56Q57 64 57 83V101L92 241Q127 382 128 383Q128 385 77 385H26Z"/><path id="MJMATHI-73" stroke-width="10" d="M131 289Q131 321 147 354T203 415T300 442Q362 442 390 415T419 355Q419 323 402 308T364 292Q351 292 340 300T328 326Q328 342 337 354T354 372T367 378Q368 378 368 379Q368 382 361 388T336 399T297 405Q249 405 227 379T204 326Q204 301 223 291T278 274T330 259Q396 230 396 163Q396 135 385 107T352 51T289 7T195 -10Q118 -10 86 19T53 87Q53 126 74 143T118 160Q133 160 146 151T160 120Q160 94 142 76T111 58Q109 57 108 57T107 55Q108 52 115 47T146 34T201 27Q237 27 263 38T301 66T318 97T323 122Q323 150 302 164T254 181T195 196T148 231Q131 256 131 289Z"/></defs></svg></p>
<p>using a single nucleotide. In this way, we are able to use the 4 bases that
compose the DNA strand to encode each byte of data.</p>
<table>
<thead>
<tr>
<th>Two bits</th>
<th>Nucleotides</th>
</tr>
</thead>
<tbody>
<tr>
<td>00</td>
<td><strong>A</strong> (Adenine)</td>
</tr>
<tr>
<td>10</td>
<td><strong>G</strong> (Guanine)</td>
</tr>
<tr>
<td>01</td>
<td><strong>C</strong> (Cytosine)</td>
</tr>
<tr>
<td>11</td>
<td><strong>T</strong> (Thymine)</td>
</tr>
</tbody>
</table>
<p>With this in mind we can simply encode any data by using two-bit to Nucleotides
conversion.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>{ Algorithm 1: Naive byte array to DNA encode }
</span></span><span style="display:flex;"><span>procedure EncodeToDNASequence(f) string
</span></span><span style="display:flex;"><span>begin
</span></span><span style="display:flex;"><span>  enc string
</span></span><span style="display:flex;"><span>  <span style="color:#00f">while</span> <span style="color:#00f">not</span> eof(f) do
</span></span><span style="display:flex;"><span>    c byte := buffer[0]                             { Read 1 byte <span style="color:#00f">from</span> buffer }
</span></span><span style="display:flex;"><span>    bin integer := sprintf(<span style="color:#a31515">&#39;08b&#39;</span>, c)                { Convert to string binary }
</span></span><span style="display:flex;"><span>    <span style="color:#00f">for</span> e <span style="color:#00f">in</span> range[0, 2, 4, 6] do
</span></span><span style="display:flex;"><span>      <span style="color:#00f">if</span> e[0] == 48 <span style="color:#00f">and</span> e[1] == 48 then             { 0x00 - A (Adenine) }
</span></span><span style="display:flex;"><span>        enc += <span style="color:#a31515">&#39;A&#39;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">else</span> <span style="color:#00f">if</span> e[0] == 48 <span style="color:#00f">and</span> e[1] == 49 then        { 0x01 - G (Guanine) }
</span></span><span style="display:flex;"><span>        enc += <span style="color:#a31515">&#39;G&#39;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">else</span> <span style="color:#00f">if</span> e[0] == 49 <span style="color:#00f">and</span> e[1] == 48 then        { 0x10 - C (Cytosine) }
</span></span><span style="display:flex;"><span>        enc += <span style="color:#a31515">&#39;C&#39;</span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">else</span> <span style="color:#00f">if</span> e[0] == 49 <span style="color:#00f">and</span> e[1] == 49 then        { 0x11 - T (Thymine) }
</span></span><span style="display:flex;"><span>        enc += <span style="color:#a31515">&#39;T&#39;</span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> enc                                        { Return DNA sequence }
</span></span><span style="display:flex;"><span>end
</span></span></code></pre><p>Another encoding would be <strong>Goldman encoding</strong>. Using this encoding helps with
Nonsense mutation (amino acids replaced by a stop codon) that occurs and is the
most problematic during translation because it leads to truncated amino acid
sequences, which in turn results in truncated proteins.</p>
<p><a href="https://www.youtube.com/watch?v=a4PiGWNsIEU">Where to store big data? In DNA: Nick Goldman at TEDxPrague</a></p>
<h3 id="fasta-file-format">FASTA file format</h3>
<p>In bioinformatics, FASTA format is a text-based format for representing either
nucleotide sequences or peptide sequences, in which nucleotides or amino acids
are represented using single-letter codes. The format also allows for sequence
names and comments to precede the sequences. The format originates from the
FASTA software package, but has now become a standard in the field of
bioinformatics.</p>
<p>The first line in a FASTA file started either with a &quot;&gt;&quot; (greater-than) symbol
or, less frequently, a &quot;;&quot; (semicolon) was taken as a comment. Subsequent lines
starting with a semicolon would be ignored by software. Since the only comment
used was the first, it quickly became used to hold a summary description of the
sequence, often starting with a unique library accession number, and with time
it has become commonplace to always use &quot;&gt;&quot; for the first line and to not use
&quot;;&quot; comments (which would otherwise be ignored).</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>;LCBO - Prolactin precursor - Bovine
</span></span><span style="display:flex;"><span>; a sample sequence in FASTA format
</span></span><span style="display:flex;"><span>MDSKGSSQKGSRLLLLLVVSNLLLCQGVVSTPVCPNGPGNCQVSLRDLFDRAVMVSHYIHDLSS
</span></span><span style="display:flex;"><span>EMFNEFDKRYAQGKGFITMALNSCHTSSLPTPEDKEQAQQTHHEVLMSLILGLLRSWNDPLYHL
</span></span><span style="display:flex;"><span>VTEVRGMKGAPDAILSRAIEIEEENKRLLEGMEMIFGQVIPGAKETEPYPVWSGLPSLQTKDED
</span></span><span style="display:flex;"><span>ARYSAFYNLLHCLRRDSSKIDTYLKLLNCRIIYNNNC*
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>&gt;MCHU - Calmodulin - Human, rabbit, bovine, rat, and chicken
</span></span><span style="display:flex;"><span>ADQLTEEQIAEFKEAFSLFDKDGDGTITTKELGTVMRSLGQNPTEAELQDMINEVDADGNGTID
</span></span><span style="display:flex;"><span>FPEFLTMMARKMKDTDSEEEIREAFRVFDKDGNGYISAAELRHVMTNLGEKLTDEEVDEMIREA
</span></span><span style="display:flex;"><span>DIDGDGQVNYEEFVQMMTAK*
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>&gt;gi|5524211|gb|AAD44166.1| cytochrome b [Elephas maximus maximus]
</span></span><span style="display:flex;"><span>LCLYTHIGRNIYYGSYLYSETWNTGIMLLLITMATAFMGYVLPWGQMSFWGATVITNLFSAIPYIGTNLV
</span></span><span style="display:flex;"><span>EWIWGGFSVDKATLNRFFAFHFILPFTMVALAGVHLTFLHETGSNNPLGLTSDSDKIPFHPYYTIKDFLG
</span></span><span style="display:flex;"><span>LLILILLLLLLALLSPDMLGDPDNHMPADPLNTPLHIKPEWYFLFAYAILRSVPNKLGGVLALFLSIVIL
</span></span><span style="display:flex;"><span>GLMPFLHTSKHRSMMLRPLSQALFWTLTMDLLTLTWIGSQPVEYPYTIIGQMASILYFSIILAFLPIAGX
</span></span><span style="display:flex;"><span>IENY
</span></span></code></pre><p>FASTA format was extended by <a href="https://en.wikipedia.org/wiki/FASTQ_format">FASTQ</a>
format from the <a href="https://www.sanger.ac.uk/">Sanger Centre</a> in Cambridge.</p>
<h3 id="png-encoded-dna-sequence">PNG encoded DNA sequence</h3>
<table>
<thead>
<tr>
<th>Nucleotides</th>
<th>RGB</th>
<th>Color name</th>
</tr>
</thead>
<tbody>
<tr>
<td>A ➞ Adenine</td>
<td>(0,0,255)</td>
<td>Blue</td>
</tr>
<tr>
<td>G ➞ Guanine</td>
<td>(0,100,0)</td>
<td>Green</td>
</tr>
<tr>
<td>C ➞ Cytosine</td>
<td>(255,0,0)</td>
<td>Red</td>
</tr>
<tr>
<td>T ➞ Thymine</td>
<td>(255,255,0)</td>
<td>Yellow</td>
</tr>
</tbody>
</table>
<p>With this in mind we can create a simple algorithm to create PNG representation
of a DNA sequence.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>{ Algorithm 2: Naive DNA to PNG encode <span style="color:#00f">from</span> FASTA file }
</span></span><span style="display:flex;"><span>procedure EncodeDNASequenceToPNG(f)
</span></span><span style="display:flex;"><span>begin
</span></span><span style="display:flex;"><span>  i image
</span></span><span style="display:flex;"><span>  <span style="color:#00f">while</span> <span style="color:#00f">not</span> eof(f) do
</span></span><span style="display:flex;"><span>    c char := buffer[0]                             { Read 1 char <span style="color:#00f">from</span> buffer }
</span></span><span style="display:flex;"><span>    case c of
</span></span><span style="display:flex;"><span>      <span style="color:#a31515">&#39;A&#39;</span>: color := RGB(0, 0, 255)                  { Blue }
</span></span><span style="display:flex;"><span>      <span style="color:#a31515">&#39;G&#39;</span>: color := RGB(0, 100, 0)                  { Green }
</span></span><span style="display:flex;"><span>      <span style="color:#a31515">&#39;C&#39;</span>: color := RGB(255, 0, 0)                  { Red }
</span></span><span style="display:flex;"><span>      <span style="color:#a31515">&#39;T&#39;</span>: color := RGB(255, 255, 0)                { Yellow }
</span></span><span style="display:flex;"><span>    drawRect(i, [x, y], color)
</span></span><span style="display:flex;"><span>  save(i)                                           { Save PNG image }
</span></span><span style="display:flex;"><span>end
</span></span></code></pre><h2 id="encoding-text-file-in-practice">Encoding text file in practice</h2>
<p>In this example we will take a simple text file as our input stream for
encoding. This file will have a quote from Niels Bohr and saved as txt file.</p>
<blockquote>
<p>How wonderful that we have met with a paradox. Now we have some hope of
making progress.
― Niels Bohr</p>
</blockquote>
<p>First we encode text file into FASTA file.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>./dnae-encode -i quote.txt -o quote.fa
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Gathering input file stats
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Starting encoding ...
</span></span><span style="display:flex;"><span> 106 B / 106 B [==================================] 100.00% 0s
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Saving to FASTA file ...
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Output FASTA file length is 438 B
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Process took 987.263µs
</span></span><span style="display:flex;"><span>2019/01/10 00:38:29 Done ...
</span></span></code></pre><p>Output of <code>quote.fa</code> file contains the encoded DNA sequence in ASCII format.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>&gt;SEQ1
</span></span><span style="display:flex;"><span>GACAGCTTGTGTACAAGTGTGCTTGCTCGCGAGCGGGTACGCGCGTGGGCTAACAAGTGA
</span></span><span style="display:flex;"><span>GCCAGCAGGTGAACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGCTGGCGGGTGA
</span></span><span style="display:flex;"><span>ACAAGTGTGCCGGTGAGCCAACAAGCAGACAAGTAAGCAGGTACGCAGGCGAGCTTGTCA
</span></span><span style="display:flex;"><span>ACTCACAAGATCGCTTGTGTACAAGTGTGCGGACAAGCCAGCAGGTGCGCGGACAAGTAT
</span></span><span style="display:flex;"><span>GCTTGCTGGCGGACAAGCCAGCTTGTAAGCGGACAAGCTTGCGCACAAGCTGGCAGGCCT
</span></span><span style="display:flex;"><span>GCCGGCTCGCGTACAAATTCACAAGTAAGTACGCTTGCGTGTACGCGGGTATGTATACTC
</span></span><span style="display:flex;"><span>AACCTCACCAAACGGGACAAGATCGCCGGCGGGCTAGTATACAAGAACGCTTGCCAGTAC
</span></span><span style="display:flex;"><span>AACC
</span></span></code></pre><p>Then we encode FASTA file from previous operation to encode this data into PNG.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>./dnae-png -i quote.fa -o quote.png
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Gathering input file stats ...
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Deconstructing FASTA file ...
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Compositing image file ...
</span></span><span style="display:flex;"><span> 424 / 424 [==================================] 100.00% 0s
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Saving output file ...
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Output image file length is 1.1 kB
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Process took 19.036117ms
</span></span><span style="display:flex;"><span>2019/01/10 00:40:09 Done ...
</span></span></code></pre><p>After encoding into PNG format this file looks like this.</p>
<figure>
<img src="/assets/posts/dna-sequence/quote.png" alt="Encoded Quote in PNG format" />
</figure>
<p>The larger the input stream is the larger the PNG file would be.</p>
<p>Compiled basic Hello World C program with
<a href="https://www.gnu.org/software/gcc/">GCC</a> would <a href="/assets/posts/dna-sequence/sample.png">look
like</a>.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#008000">// gcc -O3 -o sample sample.c
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#00f">#include</span> <span style="color:#00f">&lt;stdio.h&gt;</span><span style="color:#00f">
</span></span></span><span style="display:flex;"><span><span style="color:#00f"></span>
</span></span><span style="display:flex;"><span>main() {
</span></span><span style="display:flex;"><span>  printf(<span style="color:#a31515">&#34;Hello, world!</span><span style="color:#a31515">\n</span><span style="color:#a31515">&#34;</span>);
</span></span><span style="display:flex;"><span>  <span style="color:#00f">return</span> 0;
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><h2 id="toolkit-for-encoding-data">Toolkit for encoding data</h2>
<p>I have created a toolkit with two main programs:</p>
<ul>
<li>dnae-encode (encodes file into FASTA file)</li>
<li>dnae-png (encodes FASTA file into PNG)</li>
</ul>
<p>Toolkit with full source code is available on
<a href="https://github.com/mitjafelicijan/dna-encoding">github.com/mitjafelicijan/dna-encoding</a>.</p>
<h3 id="dnae-encode">dnae-encode</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>&gt; ./dnae-encode --help
</span></span><span style="display:flex;"><span>usage: dnae-encode --input=INPUT [&lt;flags&gt;]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>A command-line application that encodes file into DNA sequence.
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>Flags:
</span></span><span style="display:flex;"><span>      --help             Show context-sensitive help (also try --help-long and --help-man).
</span></span><span style="display:flex;"><span>  -i, --input=INPUT      Input file (ASCII or binary) which will be encoded into DNA sequence.
</span></span><span style="display:flex;"><span>  -o, --output=<span style="color:#a31515">&#34;out.fa&#34;</span>  Output file which stores DNA sequence in FASTA format.
</span></span><span style="display:flex;"><span>  -s, --sequence=SEQ1    The description line (defline) or header/identifier line, gives a name and/or a unique identifier <span style="color:#00f">for</span> the sequence.
</span></span><span style="display:flex;"><span>  -c, --columns=60       Row characters length (no more than 120 characters). Devices preallocate fixed line sizes in software.
</span></span><span style="display:flex;"><span>      --version          Show application version.
</span></span></code></pre><h3 id="dnae-png">dnae-png</h3>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>&gt; ./dnae-png --help
</span></span><span style="display:flex;"><span>usage: dnae-png --input=INPUT [&lt;flags&gt;]
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>A command-line application that encodes FASTA file into PNG image.
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>Flags:
</span></span><span style="display:flex;"><span>      --help              Show context-sensitive help (also try --help-long and --help-man).
</span></span><span style="display:flex;"><span>  -i, --input=INPUT       Input FASTA file which will be encoded into PNG image.
</span></span><span style="display:flex;"><span>  -o, --output=<span style="color:#a31515">&#34;out.png&#34;</span>  Output file in PNG format that represents DNA sequence in graphical way.
</span></span><span style="display:flex;"><span>  -s, --size=10           Size of pairings of DNA bases on image in pixels (lower resolution lower file size).
</span></span><span style="display:flex;"><span>      --version           Show application version.
</span></span></code></pre><h2 id="benchmarks">Benchmarks</h2>
<p>First we generate some binary sample data with dd.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>dd <span style="color:#00f">if</span>=&lt;(openssl enc -aes-256-ctr  -pass pass:<span style="color:#a31515">&#34;</span><span style="color:#00f">$(</span>dd <span style="color:#00f">if</span>=/dev/urandom bs=128 count=1 2&gt;/dev/null | base64<span style="color:#00f">)</span><span style="color:#a31515">&#34;</span> -nosalt &lt; /dev/zero) of=1KB.bin bs=1KB count=1 iflag=fullblock
</span></span></code></pre><figure>
<img src="/assets/posts/dna-sequence/sample-binary-file.png" alt="Sample binary file 1KB" />
</figure>
<p>Our freshly generated 1KB file looks something like this (its full of
garbage data as intended).</p>
<p>We create following binary files:</p>
<ul>
<li>1KB.bin</li>
<li>10KB.bin</li>
<li>100KB.bin</li>
<li>1MB.bin</li>
<li>10MB.bin</li>
<li>100MB.bin</li>
</ul>
<p>After this we create FASTA files for all the binary files by encoding them
into DNA sequence.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>./dnae-encode -i 100MB.bin -o 100MB.fa
</span></span></code></pre><p>Then we GZIP all the FASTA files to see how much the can be compressed.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>gzip -9 &lt; 10MB.fa &gt; 10MB.fa.gz
</span></span></code></pre><figure>
<img src="/assets/posts/dna-sequence/chart-speed.svg" alt="Encode to FASTA" />
</figure>
<p>The speed increase that occurs when encoding to FASTA format.</p>
<figure>
<img src="/assets/posts/dna-sequence/chart-size.svg" alt="File sizes" />
</figure>
<p>Size of the out file after encoding.</p>
<p><a href="/assets/posts/dna-sequence/benchmarks.csv">Download CSV file with benchmarks</a>.</p>
<h2 id="references">References</h2>
<ul>
<li><a href="https://www.techopedia.com/definition/948/encoding">https://www.techopedia.com/definition/948/encoding</a></li>
<li><a href="https://www.dna-worldwide.com/resource/160/history-dna-timeline">https://www.dna-worldwide.com/resource/160/history-dna-timeline</a></li>
<li><a href="https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/">https://opentextbc.ca/biology/chapter/9-1-the-structure-of-dna/</a></li>
<li><a href="https://arxiv.org/abs/1801.04774">https://arxiv.org/abs/1801.04774</a></li>
<li><a href="https://en.wikipedia.org/wiki/FASTA_format">https://en.wikipedia.org/wiki/FASTA_format</a></li>
</ul>


guid: https://mitjafelicijan.com/simplifying-and-reducing-clutter.html
title: Simplifying and reducing clutter in my life and work
url: https://mitjafelicijan.com/simplifying-and-reducing-clutter.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<p>I recently moved my main working machine back from Hachintosh to Linux. Well the
experiment was interesting and I have done some great work on macOS but it was
time to move back.</p>
<p>I actually really missed Linux. The simplicity of <code>apt-get</code> or just the amount
of software that exists for Linux should be a no-brainer. I spent most of my
time on macOS finding solutions to make things work. Using
<a href="https://brew.sh/">Brew</a> was just a horrible experience and far from package
managers of Linux. At least they managed to get that <code>sudo</code> debacle sorted.</p>
<p>Not all was bad. macOS in general was a perfectly good environment. Things like
Docker and tooling like this worked without any hiccups. My normal tools like
coding IDE worked flawlessly and the whole look and feel is just superb. I have
been using MacBook Air for couple of years so I was used to the system but never
as a daily driver.</p>
<p>One of the things I did after I installed Linux back on my machine was cleaning
up my Dropbox folder. I have everything on Dropbox. Even projects folder. I
write code for living so my whole life revolves around couple of megs of code
(with assets). So it's not like I have huge files on my machine. I don't have
movies or music or pictures on my PC. All of that stuff is in cloud. I use
Google music and I have Netflix account which is more than enough for me.</p>
<p>I also went and deleted some of the repositories on my Github account. I have
deleted more code than deployed. People find this strange but for me deleting
something feels so cathartic and also forces me to write better code next time
around when I am faced with similar problem. That was a huge relief if I am
being totally honest.</p>
<p>Next step was to do something with my webpage. I have been using some scripts I
wrote a while ago to generate static pages from markdown source posts. I kept on
adding and adding stuff on top of it and it became a source of a
frustration. And this is just a simple blog and I was using gulp and npm.
Anyways after couple of hours of searching and testing static generators I found
an interesting one
<a href="https://github.com/piranha/gostatic">https://github.com/piranha/gostatic</a> and I
just decided to use this one. It was the only one that had a simple templating
engine, not that I really need one. But others had this convoluted way of trying
to solve everything and at the end just required quite bigger learning curve I
was ready to go with. So I deleted couple of old posts, simplified HTML, trashed
most of the CSS and went with
<a href="https://motherfuckingwebsite.com/">https://motherfuckingwebsite.com/</a>
aesthetics.  Yeah, the previous site was more visually stimulating but all I
really care is the content at this point. And Times New Roman font is kind of
awesome.</p>
<p>I stopped working on most of the projects in the past couple of months because
the overhead was just too insane. There comes a point when you stretch yourself
too much and then you stop progressing and with that comes dissatisfaction.</p>
<p>So that's about it. Moving forward minimal style.</p>


guid: https://mitjafelicijan.com/using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html
title: Using sentiment analysis for clickbait detection in RSS feeds
url: https://mitjafelicijan.com/using-sentiment-analysis-for-clickbait-detection-in-rss-feeds.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<h2 id="initial-thoughts">Initial thoughts</h2>
<p>One of the things that interested me for a while now is if major well
established news sites use click bait titles to drive additional traffic to
their sites and generate additional impressions.</p>
<p>Goal is to see how article titles and actual content of article differ from each
other and see if titles are clickbaited.</p>
<h2 id="preparing-and-cleaning-data">Preparing and cleaning data</h2>
<p>For this example I opted to just use RSS feed from a new website and decided to
go with <a href="https://www.theguardian.com">The Guardian</a> World news. While this gets
us limited data (~40) articles and also description (actual content) is trimmed
this really doesn't reflect the actual article contents.</p>
<p>To get better content I could use web scraping and use RSS as link list and
fetch contents directly from website, but for this simple example this will
suffice.</p>
<p>There are couple of requirements we need to install before we continue:</p>
<ul>
<li><code>pip3 install feedparser</code> (parses RSS feed from url)</li>
<li><code>pip3 install vaderSentiment</code> (does sentiment polarity analysis)</li>
<li><code>pip3 install matplotlib</code> (plots chart of results)</li>
</ul>
<p>So first we need to fetch RSS data and sanitize HTML content from description.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">import</span> re
</span></span><span style="display:flex;"><span><span style="color:#00f">import</span> feedparser
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>feed_url = <span style="color:#a31515">&#34;https://www.theguardian.com/world/rss&#34;</span>
</span></span><span style="display:flex;"><span>feed = feedparser.parse(feed_url)
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000"># sanitize html</span>
</span></span><span style="display:flex;"><span><span style="color:#00f">for</span> item <span style="color:#00f">in</span> feed.entries:
</span></span><span style="display:flex;"><span>    item.description = re.sub(<span style="color:#a31515">&#39;&lt;[^&lt;]+?&gt;&#39;</span>, <span style="color:#a31515">&#39;&#39;</span>, item.description)
</span></span></code></pre><h2 id="perform-sentiment-analysis">Perform sentiment analysis</h2>
<p>Since we now have cleaned up data in our <code>feed.entries</code> object we can start with
performing sentiment analysis.</p>
<p>There are many sentiment analysis libraries available that range from rule-based
sentiment analysis up to machine learning supported analysis. To keep things
simple I decided to use rule-based analysis library
<a href="https://github.com/cjhutto/vaderSentiment">vaderSentiment</a> from
<a href="https://github.com/cjhutto">C.J. Hutto</a>. Really nice library and quite easy to
use.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">from</span> vaderSentiment.vaderSentiment <span style="color:#00f">import</span> SentimentIntensityAnalyzer
</span></span><span style="display:flex;"><span>analyser = SentimentIntensityAnalyzer()
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>sentiment_results = []
</span></span><span style="display:flex;"><span><span style="color:#00f">for</span> item <span style="color:#00f">in</span> feed.entries:
</span></span><span style="display:flex;"><span>    sentiment_title = analyser.polarity_scores(item.title)
</span></span><span style="display:flex;"><span>    sentiment_description = analyser.polarity_scores(item.description)
</span></span><span style="display:flex;"><span>    sentiment_results.append([sentiment_title[<span style="color:#a31515">&#39;compound&#39;</span>], sentiment_description[<span style="color:#a31515">&#39;compound&#39;</span>]])
</span></span></code></pre><p>Now that we have this data in a shape that is compatible with matplotlib we can
plot results to see the difference between title and description sentiment of an
article.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">import</span> matplotlib.pyplot <span style="color:#00f">as</span> plt
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>plt.rcParams[<span style="color:#a31515">&#39;figure.figsize&#39;</span>] = (15, 3)
</span></span><span style="display:flex;"><span>plt.plot(sentiment_results, drawstyle=<span style="color:#a31515">&#39;steps&#39;</span>)
</span></span><span style="display:flex;"><span>plt.title(<span style="color:#a31515">&#39;Sentiment analysis relationship between title and description (Guardian World News)&#39;</span>)
</span></span><span style="display:flex;"><span>plt.legend([<span style="color:#a31515">&#39;title&#39;</span>, <span style="color:#a31515">&#39;description&#39;</span>])
</span></span><span style="display:flex;"><span>plt.show()
</span></span></code></pre><h2 id="results-and-assets">Results and assets</h2>
<ol>
<li>Because of the small sample size further conclusions are impossible to make.</li>
<li>Rule-based approach may not be the best way of doing this. By using deep
learning we would be able to get better insights.</li>
<li><strong>Next step would be to</strong> periodically fetch RSS items and store them over a
longer period of time and then perform analysis again and use either machine
learning or deep learning on top of it.</li>
</ol>
<figure>
<img src="/assets/posts/sentiment-analysis/guardian-sa-title-desc-relationship.png" alt="Relationship between title and description" />
</figure>
<p>Figure above displays difference between title and description sentiment for
specific RSS feed item. 1 means positive and -1 means negative sentiment.</p>
<p><a href="/assets/posts/sentiment-analysis/sentiment-analysis.ipynb">» Download Jupyter Notebook</a></p>
<h2 id="going-further">Going further</h2>
<ul>
<li><a href="https://github.com/bswiss/news_mood">Twitter Sentiment Analysis by Bryan Schwierzke</a></li>
<li><a href="https://github.com/thisandagain/sentiment">AFINN-based sentiment analysis for Node.js by Andrew Sliwinski</a></li>
<li><a href="https://github.com/adeshpande3/LSTM-Sentiment-Analysis">Sentiment Analysis with LSTMs in Tensorflow by Adit Deshpande</a></li>
<li><a href="https://github.com/abdulfatir/twitter-sentiment-analysis">Sentiment analysis on tweets using Naive Bayes, SVM, CNN, LSTM, etc. by Abdul Fatir</a></li>
</ul>


guid: https://mitjafelicijan.com/simple-server-sent-events-based-pubsub-server.html
title: Simple Server-Sent Events based PubSub Server
url: https://mitjafelicijan.com/simple-server-sent-events-based-pubsub-server.html
feedurl: https://mitjafelicijan.com/feed.xml
content_mime_type: text/html
content:
<h2 id="before-we-continue-">Before we continue ...</h2>
<p>Publisher Subscriber model is nothing new and there are many amazing solutions
out there, so writing a new one would be a waste of time if other solutions
wouldn't have quite complex install procedures and weren't so hard to maintain.
But to be fair, comparing this simple server with something like
<a href="https://kafka.apache.org/">Kafka</a> or <a href="https://www.rabbitmq.com/">RabbitMQ</a> is
laughable at the least. Those solutions are enterprise grade and have many
mechanisms there to ensure messages aren't lost and much more. Regardless of
these drawbacks, this method has been tested on a large website and worked until
now without any problems. So now, that we got that cleared up, let's continue.</p>
<p><em><strong>Wiki definition:</strong> Publish/subscribe messaging, or pub/sub messaging, is a
form of asynchronous service-to-service communication used in serverless and
microservices architectures. In a pub/sub model, any message published to a
topic is immediately received by all the subscribers to the topic.</em></p>
<h2 id="general-goals">General goals</h2>
<ul>
<li>provide a simple server that relays messages to all the connected clients,</li>
<li>messages can be posted on specific topics,</li>
<li>messages get sent via <a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">Server-Sent
Events</a>
to all the subscribers.</li>
</ul>
<h2 id="how-exactly-does-the-pubsub-model-work">How exactly does the pub/sub model work?</h2>
<p>The easiest way to explain this is with diagram bellow. Basic function is
simple. We have subscribers that receive messages, and we have publishers that
create and post messages. Similar model is also well know pattern that works on
a premise of consumers and producers, and they take similar roles.</p>
<figure>
<img src="/assets/posts/simple-pubsub-server/pubsub-overview.png" alt="How PubSub works" />
</figure>
<p><strong>These are some naive characteristics we want to achieve:</strong></p>
<ul>
<li>producer is publishing messages to subscribe topic,</li>
<li>consumer is receiving messages from subscribed topic,</li>
<li>servers is also known as Broker,</li>
<li>broker does not store messages or tracks success,</li>
<li>broker uses
<a href="https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)">FIFO</a> method
for delivering messages,</li>
<li>if consumer wants to receive messages from a topic, producer and consumer
topics must match,</li>
<li>consumer can subscribe to multiple topics,</li>
<li>producer can publish to multiple topics,</li>
<li>each message has a messageId.</li>
</ul>
<p><strong>Known drawbacks:</strong></p>
<ul>
<li>messages will not be stored in a persistent queue or unreceived messages like
<a href="https://en.wikipedia.org/wiki/Dead_letter_queue">DeadLetterQueue</a> so old
messages could be lost on server restart,</li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">Server-Sent
Events</a>
opens a long-running connection between the client and the server so make sure
if your setup is load balanced that the load balancer in this case can have
long opened connection,</li>
<li>no system moderation due to the dynamic nature of creating queues.</li>
</ul>
<h2 id="server-sent-events">Server-Sent Events</h2>
<p>Read more about it on <a href="https://html.spec.whatwg.org/multipage/server-sent-events.html">official specification
page</a>.</p>
<h3 id="current-browser-support">Current browser support</h3>
<figure>
<img src="/assets/posts/simple-pubsub-server/caniuse.png" alt="Browser support" />
</figure>
<p>Check
<a href="https://caniuse.com/#feat=eventsource">https://caniuse.com/#feat=eventsource</a>
for latest information about browser support.</p>
<h3 id="known-issues">Known issues</h3>
<ul>
<li>Firefox 52 and below do not support EventSource in web/shared workers</li>
<li>In Firefox prior to version 36 server-sent events do not reconnect
automatically in case of a connection interrupt (bug)</li>
<li>Reportedly, CORS in EventSource is currently supported in Firefox 10+, Opera
12+, Chrome 26+, Safari 7.0+.</li>
<li>Antivirus software may block the event streaming data chunks.</li>
</ul>
<p>Source: <a href="https://caniuse.com/#feat=eventsource">https://caniuse.com/#feat=eventsource</a></p>
<h3 id="message-format">Message format</h3>
<p>The simplest message that can be sent is only with data attribute:</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>data: this is a simple message
</span></span><span style="display:flex;"><span>&lt;blank line&gt;
</span></span></code></pre><p>You can send message IDs to be used if the connection is dropped:</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>id: 33
</span></span><span style="display:flex;"><span>data: this is line one
</span></span><span style="display:flex;"><span>data: this is line two
</span></span><span style="display:flex;"><span>&lt;blank line&gt;
</span></span></code></pre><p>And you can specify your own event types (the above messages will all trigger
the message event):</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>id: 36
</span></span><span style="display:flex;"><span>event: price
</span></span><span style="display:flex;"><span>data: 103.34
</span></span><span style="display:flex;"><span>&lt;blank line&gt;
</span></span></code></pre><h3 id="server-requirements">Server requirements</h3>
<p>The important thing is how you send headers and which headers are sent by the
server that triggers browser to threat response as a EventStream.</p>
<p>Headers responsible for this are:</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>Content-Type: text/event-stream
</span></span><span style="display:flex;"><span>Cache-Control: no-cache
</span></span><span style="display:flex;"><span>Connection: keep-alive
</span></span></code></pre><h3 id="debugging-with-google-chrome">Debugging with Google Chrome</h3>
<p>Google Chrome provides build-in debugging and exploration tool for <a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">Server-Sent
Events</a>
which is quite nice and available from Developer Tools under Network tab.</p>
<blockquote>
<p>You can debug only client side events that get received and not the server
ones. For debugging server events add <code>console.log</code> to <code>server.js</code> code and
print out events.</p>
</blockquote>
<figure>
<img src="/assets/posts/simple-pubsub-server/chrome-debugging.png" alt="Google Chrome Developer Tools EventStream" />
</figure>
<h2 id="server-implementation">Server implementation</h2>
<p>For the sake of this example we will use <a href="https://nodejs.org/en/">Node.js</a> with
<a href="https://expressjs.com">Express</a> as our router since this is the easiest way to
get started and we will use already written SSE library for node
<a href="https://www.npmjs.com/package/sse-pubsub">sse-pubsub</a> so we don't reinvent the
wheel.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>npm init --yes
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>npm install express
</span></span><span style="display:flex;"><span>npm install body-parser
</span></span><span style="display:flex;"><span>npm install sse-pubsub
</span></span></code></pre><p>Basic implementation of a server (<code>server.js</code>):</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">const</span> express = require(<span style="color:#a31515">&#39;express&#39;</span>);
</span></span><span style="display:flex;"><span><span style="color:#00f">const</span> bodyParser = require(<span style="color:#a31515">&#39;body-parser&#39;</span>);
</span></span><span style="display:flex;"><span><span style="color:#00f">const</span> SSETopic = require(<span style="color:#a31515">&#39;sse-pubsub&#39;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#00f">const</span> app = express();
</span></span><span style="display:flex;"><span><span style="color:#00f">const</span> port = process.env.PORT || 4000;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// topics container
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span><span style="color:#00f">const</span> sseTopics = {};
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>app.use(bodyParser.json());
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// open for all cors
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.all(<span style="color:#a31515">&#39;*&#39;</span>, (req, res, next) =&gt; {
</span></span><span style="display:flex;"><span>  res.header(<span style="color:#a31515">&#39;Access-Control-Allow-Origin&#39;</span>, <span style="color:#a31515">&#39;*&#39;</span>);
</span></span><span style="display:flex;"><span>  res.header(<span style="color:#a31515">&#39;Access-Control-Allow-Headers&#39;</span>, <span style="color:#a31515">&#39;X-Requested-With, Content-Type&#39;</span>);
</span></span><span style="display:flex;"><span>  next();
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// preflight request error fix
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.options(<span style="color:#a31515">&#39;*&#39;</span>, <span style="color:#00f">async</span> (req, res) =&gt; {
</span></span><span style="display:flex;"><span>  res.header(<span style="color:#a31515">&#39;Access-Control-Allow-Origin&#39;</span>, <span style="color:#a31515">&#39;*&#39;</span>);
</span></span><span style="display:flex;"><span>  res.header(<span style="color:#a31515">&#39;Access-Control-Allow-Headers&#39;</span>, <span style="color:#a31515">&#39;X-Requested-With, Content-Type&#39;</span>);
</span></span><span style="display:flex;"><span>  res.send(<span style="color:#a31515">&#39;OK&#39;</span>);
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// serve the event streams
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.get(<span style="color:#a31515">&#39;/stream/:topic&#39;</span>, <span style="color:#00f">async</span> (req, res, next) =&gt; {
</span></span><span style="display:flex;"><span>  <span style="color:#00f">const</span> topic = req.params.topic;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">if</span> (!(topic <span style="color:#00f">in</span> sseTopics)) {
</span></span><span style="display:flex;"><span>    sseTopics[topic] = <span style="color:#00f">new</span> SSETopic({
</span></span><span style="display:flex;"><span>      pingInterval: 0,
</span></span><span style="display:flex;"><span>      maxStreamDuration: 15000,
</span></span><span style="display:flex;"><span>    });
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#008000">// subscribing client to topic
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>  sseTopics[topic].subscribe(req, res);
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// accepts new messages into topic
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.post(<span style="color:#a31515">&#39;/publish&#39;</span>, <span style="color:#00f">async</span> (req, res) =&gt; {
</span></span><span style="display:flex;"><span>  <span style="color:#00f">let</span> body = req.body;
</span></span><span style="display:flex;"><span>  <span style="color:#00f">let</span> status = 200;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  console.log(<span style="color:#a31515">&#39;Incoming message:&#39;</span>, req.body);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  <span style="color:#00f">if</span> (
</span></span><span style="display:flex;"><span>    body.hasOwnProperty(<span style="color:#a31515">&#39;topic&#39;</span>) &amp;&amp;
</span></span><span style="display:flex;"><span>    body.hasOwnProperty(<span style="color:#a31515">&#39;event&#39;</span>) &amp;&amp;
</span></span><span style="display:flex;"><span>    body.hasOwnProperty(<span style="color:#a31515">&#39;message&#39;</span>)
</span></span><span style="display:flex;"><span>  ) {
</span></span><span style="display:flex;"><span>    <span style="color:#00f">const</span> topic = req.body.topic;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">const</span> event = req.body.event;
</span></span><span style="display:flex;"><span>    <span style="color:#00f">const</span> message = req.body.message;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    <span style="color:#00f">if</span> (topic <span style="color:#00f">in</span> sseTopics) {
</span></span><span style="display:flex;"><span>      <span style="color:#008000">// sends message to all the subscribers
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>      sseTopics[topic].publish(message, event);
</span></span><span style="display:flex;"><span>    }
</span></span><span style="display:flex;"><span>  } <span style="color:#00f">else</span> {
</span></span><span style="display:flex;"><span>    status = 400;
</span></span><span style="display:flex;"><span>  }
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  res.status(status).send({
</span></span><span style="display:flex;"><span>    status,
</span></span><span style="display:flex;"><span>  });
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// returns JSON object of all opened topics
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.get(<span style="color:#a31515">&#39;/status&#39;</span>, <span style="color:#00f">async</span> (req, res) =&gt; {
</span></span><span style="display:flex;"><span>  res.send(sseTopics);
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// health-check endpoint
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.get(<span style="color:#a31515">&#39;/&#39;</span>, <span style="color:#00f">async</span> (req, res) =&gt; {
</span></span><span style="display:flex;"><span>  res.send(<span style="color:#a31515">&#39;OK&#39;</span>);
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// return a 404 if no routes match
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.use((req, res, next) =&gt; {
</span></span><span style="display:flex;"><span>  res.set(<span style="color:#a31515">&#39;Cache-Control&#39;</span>, <span style="color:#a31515">&#39;private, no-store&#39;</span>);
</span></span><span style="display:flex;"><span>  res.status(404).end(<span style="color:#a31515">&#39;Not found&#39;</span>);
</span></span><span style="display:flex;"><span>});
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span><span style="color:#008000">// starts the server
</span></span></span><span style="display:flex;"><span><span style="color:#008000"></span>app.listen(port, () =&gt; {
</span></span><span style="display:flex;"><span>  console.log(<span style="color:#a31515">`PubSub server running on http://localhost:</span><span style="color:#a31515">${</span>port<span style="color:#a31515">}</span><span style="color:#a31515">`</span>);
</span></span><span style="display:flex;"><span>});
</span></span></code></pre><h3 id="our-custom-message-format">Our custom message format</h3>
<p>Each message posted on a server must be in a specific format that out server
accepts. Having structure like this allows us to have multiple separated type of
events on each topic.</p>
<p>With this we can separate streams and only receive events that belong to the
topic.</p>
<p>One example would be, that we have index page and we want to receive messages
about new upvotes or new subscribers but we don't want to follow events for
other pages. This reduces clutter and overall network. And structure is much
nicer and maintanable.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span>{
</span></span><span style="display:flex;"><span>  &#34;topic&#34;: <span style="color:#a31515">&#34;sample-topic&#34;</span>,
</span></span><span style="display:flex;"><span>  &#34;event&#34;: <span style="color:#a31515">&#34;sample-event&#34;</span>,
</span></span><span style="display:flex;"><span>  &#34;message&#34;: { &#34;name&#34;: <span style="color:#a31515">&#34;John&#34;</span> }
</span></span><span style="display:flex;"><span>}
</span></span></code></pre><h2 id="publisher-and-subscriber-clients">Publisher and subscriber clients</h2>
<h3 id="publisher-and-subscriber-in-action">Publisher and subscriber in action</h3>
<p><video src="/assets/posts/simple-pubsub-server/clients.m4v" controls></video></p>
<p>You can download <a href="../simple-pubsub-server/sse-pubsub-server.zip">the code</a> and
follow along.</p>
<h3 id="publisher">Publisher</h3>
<p>As talked about above publisher is the one that send messages to the
broker/server. Message inside the payload can be whatever you want (string,
object, array). I would however personally avoid send large chunks of data like
blobs and such.</p>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">&lt;!DOCTYPE html&gt;</span>
</span></span><span style="display:flex;"><span>&lt;html lang=<span style="color:#a31515">&#34;en&#34;</span>&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;head&gt;
</span></span><span style="display:flex;"><span>    &lt;meta charset=<span style="color:#a31515">&#34;UTF-8&#34;</span>&gt;
</span></span><span style="display:flex;"><span>    &lt;meta name=<span style="color:#a31515">&#34;viewport&#34;</span> content=<span style="color:#a31515">&#34;width=device-width, initial-scale=1.0&#34;</span>&gt;
</span></span><span style="display:flex;"><span>    &lt;title&gt;Publisher&lt;/title&gt;
</span></span><span style="display:flex;"><span>  &lt;/head&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;h1&gt;Publisher&lt;/h1&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;fieldset&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Server:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;server&#34;</span> value=<span style="color:#a31515">&#34;http://localhost:4000&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Topic:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;topic&#34;</span> value=<span style="color:#a31515">&#34;sample-topic&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Event:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;event&#34;</span> value=<span style="color:#a31515">&#34;sample-event&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Message:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;message&#34;</span> value=<span style="color:#a31515">&#39;{&#34;name&#34;: &#34;John&#34;}&#39;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;button type=<span style="color:#a31515">&#34;button&#34;</span> id=<span style="color:#a31515">&#34;button&#34;</span>&gt;Publish message to topic&lt;/button&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>    &lt;/fieldset&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;script&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> button = document.querySelector(<span style="color:#a31515">&#39;#button&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> server = document.querySelector(<span style="color:#a31515">&#39;#server&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> topic = document.querySelector(<span style="color:#a31515">&#39;#topic&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> event = document.querySelector(<span style="color:#a31515">&#39;#event&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> message = document.querySelector(<span style="color:#a31515">&#39;#message&#39;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      button.addEventListener(<span style="color:#a31515">&#39;click&#39;</span>, <span style="color:#00f">async</span> (evt) =&gt; {
</span></span><span style="display:flex;"><span>        <span style="color:#00f">const</span> req = <span style="color:#00f">await</span> fetch(<span style="color:#a31515">`</span><span style="color:#a31515">${</span>server.value<span style="color:#a31515">}</span><span style="color:#a31515">/publish`</span>, {
</span></span><span style="display:flex;"><span>          method: <span style="color:#a31515">&#39;post&#39;</span>,
</span></span><span style="display:flex;"><span>          headers: {
</span></span><span style="display:flex;"><span>            <span style="color:#a31515">&#39;Accept&#39;</span>: <span style="color:#a31515">&#39;application/json&#39;</span>,
</span></span><span style="display:flex;"><span>            <span style="color:#a31515">&#39;Content-Type&#39;</span>: <span style="color:#a31515">&#39;application/json&#39;</span>,
</span></span><span style="display:flex;"><span>          },
</span></span><span style="display:flex;"><span>          body: JSON.stringify({
</span></span><span style="display:flex;"><span>            topic: topic.value,
</span></span><span style="display:flex;"><span>            event: event.value,
</span></span><span style="display:flex;"><span>            message: JSON.parse(message.value),
</span></span><span style="display:flex;"><span>          }),
</span></span><span style="display:flex;"><span>        });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#00f">const</span> res = <span style="color:#00f">await</span> req.json();
</span></span><span style="display:flex;"><span>        console.log(res);
</span></span><span style="display:flex;"><span>      });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;/script&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;/body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>&lt;/html&gt;
</span></span></code></pre><h3 id="subscriber">Subscriber</h3>
<p>Subscriber is responsible for receiving new messages that come from server via
publisher. The code bellow is very rudimentary but works and follows the
implementation guidelines for EventSource.</p>
<p>You can use either Developer Tools Console to see incoming messages or you can
defer to Debugging with Google Chrome section above to see all EventStream
messages.</p>
<blockquote>
<p>Don't be alarmed if the subscriber gets disconnected from the server every so
often. The code we have here resets connection every 15s but it automatically
get reconnected and fetches all messages up to last received message id. This
setting can be adjusted in <code>server.js</code> file; search for the
<code>maxStreamDuration</code> variable.</p>
</blockquote>
<pre tabindex="0" style="background-color:#fff;"><code><span style="display:flex;"><span><span style="color:#00f">&lt;!DOCTYPE html&gt;</span>
</span></span><span style="display:flex;"><span>&lt;html lang=<span style="color:#a31515">&#34;en&#34;</span>&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;head&gt;
</span></span><span style="display:flex;"><span>    &lt;meta charset=<span style="color:#a31515">&#34;UTF-8&#34;</span>&gt;
</span></span><span style="display:flex;"><span>    &lt;meta name=<span style="color:#a31515">&#34;viewport&#34;</span> content=<span style="color:#a31515">&#34;width=device-width, initial-scale=1.0&#34;</span>&gt;
</span></span><span style="display:flex;"><span>    &lt;title&gt;Subscriber&lt;/title&gt;
</span></span><span style="display:flex;"><span>    &lt;link rel=<span style="color:#a31515">&#34;stylesheet&#34;</span> href=<span style="color:#a31515">&#34;style.css&#34;</span>&gt;
</span></span><span style="display:flex;"><span>  &lt;/head&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;h1&gt;Subscriber&lt;/h1&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;fieldset&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Server:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;server&#34;</span> value=<span style="color:#a31515">&#34;http://localhost:4000&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Topic:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;topic&#34;</span> value=<span style="color:#a31515">&#34;sample-topic&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;label&gt;Event:&lt;/label&gt;
</span></span><span style="display:flex;"><span>        &lt;input type=<span style="color:#a31515">&#34;text&#34;</span> id=<span style="color:#a31515">&#34;event&#34;</span> value=<span style="color:#a31515">&#34;sample-event&#34;</span>&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>      &lt;p&gt;
</span></span><span style="display:flex;"><span>        &lt;button type=<span style="color:#a31515">&#34;button&#34;</span> id=<span style="color:#a31515">&#34;button&#34;</span>&gt;Subscribe to topic&lt;/button&gt;
</span></span><span style="display:flex;"><span>      &lt;/p&gt;
</span></span><span style="display:flex;"><span>    &lt;/fieldset&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;script&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> button = document.querySelector(<span style="color:#a31515">&#39;#button&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> server = document.querySelector(<span style="color:#a31515">&#39;#server&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> topic = document.querySelector(<span style="color:#a31515">&#39;#topic&#39;</span>);
</span></span><span style="display:flex;"><span>      <span style="color:#00f">const</span> event = document.querySelector(<span style="color:#a31515">&#39;#event&#39;</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      button.addEventListener(<span style="color:#a31515">&#39;click&#39;</span>, <span style="color:#00f">async</span> (evt) =&gt; {
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        <span style="color:#00f">let</span> es = <span style="color:#00f">new</span> EventSource(<span style="color:#a31515">`</span><span style="color:#a31515">${</span>server.value<span style="color:#a31515">}</span><span style="color:#a31515">/stream/</span><span style="color:#a31515">${</span>topic.value<span style="color:#a31515">}</span><span style="color:#a31515">`</span>);
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        es.addEventListener(event.value, <span style="color:#00f">function</span> (evt) {
</span></span><span style="display:flex;"><span>          console.log(<span style="color:#a31515">`incoming message`</span>, JSON.parse(evt.data));
</span></span><span style="display:flex;"><span>        });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        es.addEventListener(<span style="color:#a31515">&#39;open&#39;</span>, <span style="color:#00f">function</span> (evt) {
</span></span><span style="display:flex;"><span>          console.log(<span style="color:#a31515">&#39;connected&#39;</span>, evt);
</span></span><span style="display:flex;"><span>        });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>        es.addEventListener(<span style="color:#a31515">&#39;error&#39;</span>, <span style="color:#00f">function</span> (evt) {
</span></span><span style="display:flex;"><span>          console.log(<span style="color:#a31515">&#39;error&#39;</span>, evt);
</span></span><span style="display:flex;"><span>        });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>      });
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>    &lt;/script&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>  &lt;/body&gt;
</span></span><span style="display:flex;"><span>
</span></span><span style="display:flex;"><span>&lt;/html&gt;
</span></span></code></pre><h2 id="reading-further">Reading further</h2>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events">Using server-sent events</a></li>
<li><a href="https://www.smashingmagazine.com/2018/02/sse-websockets-data-flow-http2/">Using SSE Instead Of WebSockets For Unidirectional Data Flow Over HTTP/2</a></li>
<li><a href="https://apifriends.com/api-streaming/server-sent-events/">What is Server-Sent Events?</a></li>
<li><a href="https://tools.ietf.org/id/draft-xie-bidirectional-messaging-01.html">An HTTP/2 extension for bidirectional messaging communication</a></li>
<li><a href="https://developers.google.com/web/fundamentals/performance/http2">Introduction to HTTP/2</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API">The WebSocket API (WebSockets)</a></li>
</ul>


